System

Abstract

To provide a system for securing the safety of a user at an intersection where there is no signal, especially under a situation where visibility is poor or a situation where many entities approach the intersection at the same time.SOLUTION: A system comprising: means for a terminal to obtain location information and speed information and send the location information and the speed information to a server; means for the server to calculate a predicted trajectory of each entity by using the received data and detect a collision risk; means for the server to send a warning message to a corresponding terminal when the collision risk is detected; and means for the terminal to issue a voice or visual signal to alert a user when the warning message is received.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Intersections without traffic lights present a problem in that the movements of vehicles, bicycles, and pedestrians are difficult to predict, making collisions more likely to occur. Currently, visual confirmation and warnings alone are insufficient, and the risk of accidents increases, especially in poor visibility conditions or when many entities approach the intersection at the same time. To ensure user safety under these conditions, a more effective accident prevention system is needed. [Means for solving the problem]

[0005] In order to solve the above-mentioned problems, the present invention provides the following means: A means for a terminal to acquire position information and speed information and send it to a server; A means for the server to calculate a predicted trajectory of each entity using the received data and detect a collision risk; A means for the server to send a warning message to the relevant terminal when a collision risk is detected; Further, a means for issuing an audio or visual signal to alert the user when the terminal receives the warning message. This system can reduce the risk of collision accidents at intersections and increase user safety.

[0006] A "terminal" is a device that acquires location information and speed information and communicates with a server.

[0007] "Server" refers to a central management system that receives data sent from the terminal, analyzes it, and generates and sends warning messages.

[0008] "Location information" is data that indicates the current location of an entity, and is data consisting of latitude and longitude.

[0009] "Speed ​​information" is data that indicates the moving speed of an entity.

[0010] A "predicted trajectory" is a calculation result that predicts the future position of an entity based on its current position information and speed information.

[0011] The "collision risk" is an evaluation result that indicates the possibility of entities colliding at an intersection based on the predicted trajectories.

[0012] A "warning message" is a notification message sent to a terminal to alert the user, and is displayed to the user audibly or visually.

[0013] An "audio or visual signal" is an audio message or a visual indicator, such as a flashing light, that the terminal uses to alert the user.

[0014] "Entity" refers to a moving object such as a vehicle, bicycle, or pedestrian that is monitored within the system. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention provides a system for monitoring the movements of vehicles, pedestrians, and bicycles at intersections without traffic lights in real time to reduce the risk of collisions. The system includes a means for a terminal to acquire position information and speed information and transmit it to a server, a means for the server to calculate a predicted trajectory based on the received data and detect a collision risk, a means for the server to send a warning message to the terminal when a collision risk is detected, and a means for the terminal to issue an audio or visual signal to a user when it receives the warning message.

[0037] System operation example

[0038] The device acquires location and speed information using the built-in GPS sensor and speed sensor. For example, when pedestrian A is walking towards an intersection, pedestrian A's device acquires location information (latitude X1, longitude Y1) and speed information (speed V1), which are periodically sent to the server.

[0039] Sending data:

[0040] The device sends location and speed information to the server, which receives it. The data sent is in the following format:

[0041] {

[0042] "id": "entity_1",

[0043] "position": {"lat": X1, "lon": Y1},

[0044] "velocity": V1,

[0045] "timestamp": T1

[0046] }

[0047] The server stores the received data and uses only the most recent information for analysis based on the timestamp.

[0048] Collision risk analysis:

[0049] The server calculates the predicted trajectory of each entity. For example, if Pedestrian A and Bicycle B are heading towards the same intersection, the server calculates their predicted positions in the next 2 seconds and evaluates the likelihood that the entities will arrive at the same location.

[0050] Calculating the predicted trajectory:

[0051] The server calculates the predicted location based on the current location and velocity information as follows:

[0052] next_position = current_position + velocity time_interval

[0053] Based on this predicted position, the entity's future positional relationship is analyzed.

[0054] Sending a warning message:

[0055] If the server detects a collision risk, it generates and sends a warning message to the device of the relevant entity. For example, if there is a high possibility that pedestrian A and cyclist B will collide, the server will generate the following warning message:

[0056] {

[0057] "id": "entity_1",

[0058] "warning": "collision_risk",

[0059] "message": "Attention! Possible collision ahead."

[0060] }

[0061] This is sent to the device.

[0062] Receive alerts and notifications:

[0063] When the device receives a warning message from the server, it notifies the user of the warning. Notification methods include voice messages and flashing lights:

[0064] Display: "Attention! Possible collision ahead."

[0065] Sound: Play warning sound

[0066] Light: Start blinking

[0067] The user can then review this notification and take appropriate action.

[0068] Specific example processing:

[0069] For example, consider a situation where pedestrian A is walking towards an intersection and cyclist B is simultaneously heading towards the intersection. The system operates in the following steps:

[0070] 1. The device acquires location and speed information.

[0071] Pedestrian A's device acquires location information (X1, Y1) and speed (V1), and cyclist B's device also acquires location information (X2, Y2) and speed (V2).

[0072] 2. The device sends the data to the server.

[0073] Both terminals send data to the server, which receives it.

[0074] 3. The server calculates the predicted trajectory.

[0075] The server calculates the predicted trajectories of both vehicles and assesses the risk of collision at the intersection.

[0076] 4. The server generates and sends a warning message.

[0077] If a collision risk is detected, the server sends a warning message to the devices of pedestrian A and cyclist B.

[0078] 5. The device receives the warning message and notifies you.

[0079] The device displays a notification to the user and alerts them with an audio or visual signal.

[0080] In this way, the present invention reduces the risk of collisions between entities within an intersection, ensuring safety.

[0081] The processing flow will be explained below.

[0082] Step 1: The device acquires location and speed information

[0083] The device uses the built-in GPS sensor to obtain the current location (latitude, longitude) and the speed sensor to obtain the moving speed. For example, the device of pedestrian A obtains the location information (X1, Y1) and speed (V1).

[0084] Step 2: The device sends the data to the server

[0085] The device periodically sends the acquired location and speed information to the server. The data sent includes the device ID, location information, speed information, and timestamp. For example,

[0086] {

[0087] "id": "entity_1",

[0088] "position": {"lat": X1, "lon": Y1},

[0089] "velocity": V1,

[0090] "timestamp": T1

[0091] }

[0092] It is sent in the format:

[0093] Step 3: The server receives the data

[0094] The server receives location and speed information from the device via API and stores it in memory or in a database. The server overwrites older data based on the timestamp to keep only the most recent data.

[0095] Step 4: The server calculates the predicted trajectory

[0096] The server calculates the predicted location of each entity at regular intervals based on its current location and velocity. For example, the predicted location for the next 2 seconds is calculated using the following formula:

[0097] next_position = current_position + velocity time_interval

[0098] Based on this predicted position, a future trajectory for each entity is generated.

[0099] Step 5: Server detects collision risk

[0100] The server analyzes the trajectories of all generated entities and evaluates whether they will intersect at an intersection. For example, if pedestrian A and cyclist B are likely to arrive at the same location, it identifies the time and location and determines the collision risk.

[0101] Step 6: The server generates and sends a warning message

[0102] If the server detects a collision risk, it generates and sends a warning message to the device of the relevant entity. For example, it generates a warning message like this:

[0103] {

[0104] "id": "entity_1",

[0105] "warning": "collision_risk",

[0106] "message": "Attention! Possible collision ahead."

[0107] }

[0108] Send this message to the terminal.

[0109] Step 7: The device receives the warning message and notifies the user

[0110] When the terminal receives the warning message sent from the server, it issues a warning to the user with an audio or visual signal, such as displaying the warning message on the terminal screen, sounding an alarm, or flashing a light to alert the user.

[0111] Step 8: The user acts on the device warning

[0112] The user checks the warning from the device and takes safe action. For example, pedestrian A slows down as he approaches the intersection, or the driver of bicycle B slows down to delay entering the intersection.

[0113] Example 1

[0114] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0115] At intersections without traffic lights, the risk of collisions increases because many moving objects, including vehicles, pedestrians, and bicycles, cross the intersection. In such situations, a system is needed that can monitor the position and speed of each entity in real time, detect danger in advance, and issue a warning. However, conventional systems have difficulty accurately calculating the predicted trajectories of multiple moving objects and issuing a warning quickly.

[0116] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0117] In this invention, the server includes a means for the terminal to acquire position information and speed information and transmit it to the server, a means for the server to calculate a predicted trajectory of each entity using the data received and detect a collision risk, and a means for transmitting a warning message to the corresponding terminal when the server detects a collision risk. This makes it possible to simultaneously calculate the predicted trajectories of multiple entities and evaluate the collision risk at an intersection.

[0118] A "terminal" is a device for obtaining location and velocity information of an entity and transmitting it to a server.

[0119] "Location information" is latitude and longitude data that indicates the current location of an entity.

[0120] "Speed ​​information" is data that indicates the moving speed of an entity.

[0121] A "server" is a device or system that receives data transmitted from a terminal and calculates a predicted trajectory and evaluates a collision risk.

[0122] A "predicted trajectory" is a future movement path calculated based on the entity's current position and speed information.

[0123] "Collision risk" is an assessment that indicates the likelihood that multiple entities will arrive at the same location in the future, resulting in a collision accident.

[0124] A "warning message" is a warning communication that is sent to the relevant terminal when the server detects a collision risk.

[0125] The "audio signal" is an audio notification that alerts the user when a warning message is received by the terminal.

[0126] A "visual signal" is a light or screen display notification that alerts the user when a warning message is received by the terminal.

[0127] A "timestamp" is information that records the time when data is acquired or transmitted.

[0128] A "location sensor" is a device that can determine the current location of an entity and obtain that location information.

[0129] The present invention provides a system for monitoring the movements of vehicles, pedestrians, and bicycles at intersections without traffic lights in real time to reduce the risk of collision accidents. The system includes a means for a terminal to acquire position information and speed information and transmit it to a server, a means for the server to calculate a predicted trajectory based on the received data and detect a collision risk, a means for the server to transmit a warning message to the terminal when a collision risk is detected, and a means for the terminal to issue an audio or visual signal to a user when it receives the warning message.

[0130] The device acquires current location information (latitude and longitude) and speed information using the built-in location sensor and speed sensor. For example, if a pedestrian is walking towards an intersection, the device acquires location information (X1, Y1) and speed information (V1) and periodically sends this to the server. The data sent by the device includes the ID, location information (latitude and longitude), speed information, and the timestamp when the data was acquired.

[0131] The server receives data sent from the device and stores it in a database. Data sent from each entity is recorded with a timestamp, and the server uses only the most recent data for analysis. Based on the received data, the server calculates the predicted trajectory of each entity. The current position and speed information are used to calculate the predicted trajectory, using the formula "next position = current position + speed x time interval."

[0132] The server simultaneously calculates the predicted positions of multiple entities and analyzes their future positional relationships. For example, if pedestrian A and cyclist B are heading towards the same intersection, the server calculates their predicted positions and evaluates the probability that they will arrive at the intersection at the same time. Based on this evaluation, the server determines the collision risk.

[0133] If a collision risk is detected, the server generates and sends a warning message to the relevant entity. For example, if there is a high risk of a collision between pedestrian A and cyclist B, the server generates a warning message and sends it to both devices. The warning message includes a warning message and details of the warning.

[0134] When the device receives a warning message, it notifies the user with an audio or visual signal, such as by displaying the message "Attention! Possible collision ahead" on the display, emitting a warning sound, or flashing a light to alert the user.

[0135] For example, if pedestrian A is walking towards an intersection and cyclist B is heading towards the same intersection, the system will operate as follows:

[0136] 1. Pedestrian A's device acquires location information (X1, Y1) and speed (V1), and cyclist B's device also acquires location information (X2, Y2) and speed (V2).

[0137] 2. Both devices send data to the server.

[0138] 3. The server stores the received data and calculates the predicted trajectories of both vehicles.

[0139] 4. The server evaluates the collision risk based on the predicted trajectory.

[0140] 5. If a collision risk is detected, the server generates a warning message and sends it to the devices of pedestrian A and cyclist B.

[0141] 6. The terminal receives the warning message and notifies the user of the warning.

[0142] Such a system can reduce the risk of collisions between entities and ensure safety even at intersections without traffic lights. An example of a prompt sentence could be, "Watch out for the next intersection. Stop and check your surroundings."

[0143] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0144] System program processing flow

[0145] Step 1: The device obtains location and speed information

[0146] The device uses the built-in location and speed sensors to obtain current location information (latitude and longitude) and speed information. The input is the entity's sensor information, and the output is location information (X, Y) and speed information (V). This specific operation involves obtaining GPS data and calculating speed using the acceleration sensor.

[0147] Step 2: The device sends data to the server

[0148] The device sends the acquired location and speed information to the server. The input is the location and speed information acquired in step 1, and the output is the transmission of a data packet to the server. The transmitted data has the following format:

[0149] {

[0150] "id": "entity_1",

[0151] "position": {"lat": X, "lon": Y},

[0152] "velocity": V,

[0153] "timestamp": T

[0154] }

[0155] Specifically, the data packet is sent to the server using an HTTP request or the like.

[0156] Step 3: The server receives and stores the data

[0157] The server receives the data sent from the device and stores it in a database. The input is the data packet sent in step 2, and the output is the stored location and speed information. The specific operation is a write operation to the database.

[0158] Step 4: The server calculates the predicted trajectory

[0159] The server calculates the predicted trajectory of each entity based on the received position and velocity information. The input is the saved current position and velocity information, and the output is the predicted position (next_position) after the next time interval. The predicted position is calculated using the following formula:

[0160] next_position = current_position + velocity time_interval

[0161] The specific operation is to use this formula to predict the future position of each entity.

[0162] Step 5: Server evaluates collision risk

[0163] The server evaluates the collision risk of multiple entities based on the predicted trajectories. The input is the predicted position calculated in step 4, and the output is the collision risk assessment result. The specific operation is to compare the predicted positions and calculate the probability that the entities will arrive at the same point.

[0164] Step 6: The server generates a warning message

[0165] If a collision risk is detected, the server generates a warning message to be sent to the relevant entity. The input is the collision risk assessment result from step 5, and the output is a warning message. The specific format of the warning message is as follows:

[0166] {

[0167] "id": "entity_1",

[0168] "warning": "collision_risk",

[0169] "message": "Attention! Possible collision ahead."

[0170] }

[0171] Specifically, the system dynamically generates warning content based on the evaluation results.

[0172] Step 7: The server sends a warning message

[0173] The server sends the generated warning message to the corresponding terminal. The input is the warning message generated in step 6, and the output is the transmission of warning data to the terminal. Specifically, this message is sent to the corresponding terminal using an HTTP request, etc.

[0174] Step 8: Your device receives a warning message

[0175] The terminal receives the warning message sent from the server. The input is the warning message from the server, and the output is the internal processing of this message. The specific operation is to store the received message in memory and add it to the processing queue.

[0176] Step 9: The device notifies the user of the warning

[0177] When the device receives the warning message, it notifies the user of the warning with an audio or visual signal. The input is the warning message received in step 8, and the output is the notification to the user. Specifically, the device displays "Attention! Possible collision ahead." and plays a warning sound or flashes a light.

[0178] In this way, the risk of collisions between entities can be reduced and safety ensured even at intersections without traffic lights.

[0179] (Application example 1)

[0180] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0181] At unsignalized intersections and in congested urban environments, the risk of collisions between autonomous vehicles and entities such as other vehicles, pedestrians, and cyclists is increasing. In such situations, real-time dynamic prediction and immediate warnings are necessary for autonomous vehicles to operate safely and prevent accidents. However, current technology is unable to meet these needs, and there is a lack of systems that can adequately reduce the risk of collisions between entities. Accidents resulting from this pose a major problem that threatens the safety of passengers and those around them.

[0182] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0183] In this invention, the server includes: means for the terminal to acquire position information and speed information and transmit it to the server; means for the server to calculate a predicted trajectory of each entity using the data received and detect a collision risk; means for transmitting a warning message to the corresponding terminal when the server detects a collision risk; means for the terminal to issue an audio or visual signal to alert the user when it receives the warning message; and means for the terminal to be mounted in an autonomous vehicle and notify the driver of the warning message via an in-vehicle display and audio guidance system. This enables autonomous vehicles to detect collision risks with other entities in real time, even at intersections or in crowded environments, and ensure safe operation.

[0184] A "terminal" is a device that acquires location information and speed information and transmits it to a server.

[0185] "Predicted trajectory" is a calculation method that predicts the future position of each entity based on its current position and speed information.

[0186] "Collision risk" is an assessment that indicates the likelihood of contact or collision between entities using the predicted trajectory.

[0187] A "warning message" is a notification message that is sent to the relevant terminal when a collision risk is detected.

[0188] An "audio or visual signal" is a signal, such as a voice message or a flashing light, that is emitted to notify the user of a warning.

[0189] An "autonomous vehicle" is a vehicle that is equipped with GPS and speed sensors and operates autonomously.

[0190] An "in-vehicle display" is a screen installed inside an autonomous vehicle to visually display information.

[0191] A "voice guidance system" is a system installed inside an autonomous vehicle that provides information through voice messages.

[0192] The present invention is a system for reducing the risk of collisions between autonomous vehicles and entities such as other vehicles, pedestrians, and cyclists at unsignalized intersections and in congested urban environments. The system includes a terminal, a server, an in-vehicle display, and a voice guidance system.

[0193] System hardware configuration

[0194] Device: Installed in an autonomous vehicle, it continuously acquires location and speed information using built-in positioning sensors (e.g., GPS sensors) and speed sensors.

[0195] Server: Located on the cloud, it receives data sent from the device, calculates the predicted trajectory, and evaluates the collision risk.

[0196] In-vehicle displays: installed inside autonomous vehicles, they visually display warning messages and other necessary information.

[0197] Voice guidance system: A system that works in conjunction with the in-car display to provide warnings to the driver through voice messages.

[0198] Software configuration and processing flow

[0199] 1. Data Acquisition

[0200] The device uses built-in positioning and speed measurement sensors to obtain real-time position and speed information of the autonomous vehicle.

[0201] 2. Data Transmission

[0202] The terminal periodically transmits the acquired location and speed information to the server. The transmitted data includes location information (latitude and longitude), speed information, and a timestamp.

[0203] 3. Collision risk assessment

[0204] The server uses the received data to calculate the predicted trajectory of the autonomous vehicle and other entities, including calculating future positions based on current position and speed information.

[0205] 4. Generating Warning Messages

[0206] The server evaluates the collision risk based on the predicted trajectory, and if a collision risk is detected, it generates and sends a warning message to the relevant terminal.

[0207] 5. Warning Notification

[0208] When the terminal receives a warning message from the server, it issues a visual and audio warning to the driver through the in-car display and voice guidance system.

[0209] Prompt Sentence Examples

[0210] Below are some example prompts to input to a generative AI model:

[0211] "Design a system that acquires real-time location and speed information of an autonomous vehicle, sends it to a server, and evaluates the risk of collision. The server calculates a predicted trajectory based on data from other entities, and if there is a risk of collision, warns the driver on the vehicle's display and voice guidance system. Please provide Python code that runs in real time."

[0212] As a specific example, consider the following scenario: As autonomous vehicle A approaches an intersection, it detects the presence of pedestrian B walking toward the same intersection. The server calculates the predicted trajectories of both vehicles, and if it detects a high risk of collision, it sends a warning message to the device of autonomous vehicle A. The in-vehicle display will show a message saying, "Possible collision with pedestrian at intersection ahead. Please be careful," and the voice guidance system will give instructions such as, "Pay attention to what is ahead." This notification will alert the driver to be careful at the intersection, ensuring safe driving.

[0213] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0214] Step 1:

[0215] The device obtains location and speed information. The device uses a location sensor (e.g., GPS sensor) and a speed sensor to obtain the current latitude, longitude, and speed of the autonomous vehicle. In this case, the input is the raw data provided by the vehicle's sensors, and the output is the latitude, longitude, and speed values.

[0216] Step 2:

[0217] The device sends the acquired location and speed information to the server. The device sends data to the server, including the acquired location information (latitude, longitude), speed information, and timestamp. This data is encoded in JSON or other appropriate data format. The location and speed information obtained in step 1 is used as input, and the data sent to the server is generated as output.

[0218] Step 3:

[0219] The server stores the received data and uses it for analysis. The server stores the location and speed information received from the device in a database and uses only the most recent data for analysis based on the timestamp. This eliminates duplicate and old data. The input is the data sent from the device, and the output is the latest data set to use for analysis and calculations.

[0220] Step 4:

[0221] The server calculates the predicted trajectory of each entity. Based on the received position and velocity information, the server predicts the future position of each entity. This calculation is done by multiplying the current position by the velocity, for example. The input is the latest position and velocity information, and the output is the predicted position at a specific point in the future.

[0222] Step 5:

[0223] The server evaluates the collision risk. The server compares the predicted trajectories of each entity and determines whether there is a possibility of an intersection. If the intersection points are very close, it determines that there is a collision risk. The input is the predicted trajectory data, and the output is the collision risk assessment result (risk / no risk).

[0224] Step 6:

[0225] If the server detects a collision risk, it sends a warning message to the relevant terminal. The server generates and sends a warning message to the terminal of the entity where the collision risk was detected. This message indicates that a collision risk exists and calls for attention. The input is the collision risk evaluation result, and the output is the warning message to be sent to each terminal.

[0226] Step 7:

[0227] The terminal receives the warning message and alerts the user. The terminal notifies the driver of the received warning message via the in-car display or voice guidance system. For example, the display may show a message saying "There is a pedestrian ahead. Please be careful," and the voice guidance system may play a voice message saying "Pay attention to what is ahead." The input is the warning message received from the server, and the output is the warning notification to the user.

[0228] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0229] The present invention combines an emotion engine with a system for monitoring the movements of entities such as vehicles, pedestrians, and bicycles at intersections without traffic lights in real time and reducing the risk of collision accidents. This system includes means for a terminal to acquire position information and speed information and send it to a server, means for the server to calculate a predicted trajectory based on the received data and detect a collision risk, means for the server to send a warning message to the relevant terminal when a collision risk is detected, means for the terminal to issue an audio or visual signal to alert the user when it receives the warning message, and an emotion engine that recognizes the user's emotions.

[0230] System Overview

[0231] The device acquires location and speed information using the built-in GPS sensor and speed sensor. For example, when pedestrian A is walking towards an intersection, pedestrian A's device acquires location information (latitude X1, longitude Y1) and speed information (speed V1), which are periodically sent to the server.

[0232] Detailed System Operation

[0233] Sending data:

[0234] The device periodically sends the location and speed information it has acquired to the server. The data sent is in the following format:

[0235] {

[0236] "id": "entity_1",

[0237] "position": {"lat": X1, "lon": Y1},

[0238] "velocity": V1,

[0239] "timestamp": T1

[0240] }

[0241] The server stores the received data and uses only the most recent information for analysis based on the timestamp.

[0242] Collision risk analysis:

[0243] The server calculates the predicted location of each entity at regular intervals based on its current location and velocity. For example, the predicted location for the next 2 seconds is calculated using the following formula:

[0244] next_position = current_position + velocity time_interval

[0245] Based on this predicted position, a future trajectory for each entity is generated.

[0246] Calculating the predicted trajectory:

[0247] The server analyzes the trajectories of all entities and evaluates whether they intersect at an intersection. For example, if pedestrian A and cyclist B are likely to arrive at the same location, it identifies the time and location and determines the collision risk.

[0248] Generate and send a warning message:

[0249] If the server detects a collision risk, it generates and sends a warning message to the device of the relevant entity. For example, it generates a warning message like this:

[0250] {

[0251] "id": "entity_1",

[0252] "warning": "collision_risk",

[0253] "message": "Attention! Possible collision ahead."

[0254] }

[0255] This is sent to the device.

[0256] Using the Emotion Engine:

[0257] The emotion engine analyzes the user's emotions in real time and recognizes their emotional state. For example, it uses the device's camera and microphone to analyze emotions from changes in the user's facial expressions and voice.

[0258] Dynamically adjusting warnings based on emotions:

[0259] If the emotion engine determines that the user is in a negative emotional state, such as tension or anxiety, the server will issue a more emphatic warning message, for example by making the warning sound louder or flashing visual signals more frequently.

[0260] Receiving and notifying warning messages:

[0261] When the device receives a warning message from the server, it notifies the user of the warning. Notification methods include voice messages and flashing lights:

[0262] Display: "Attention! Possible collision ahead."

[0263] Sound: Play warning sound

[0264] Light: Start blinking

[0265] The user can then review this notification and take appropriate action.

[0266] Processing of specific examples

[0267] For example, consider a situation where pedestrian A is walking towards an intersection and cyclist B is simultaneously heading towards the intersection. The system operates in the following steps:

[0268] 1. The device acquires location and speed information.

[0269] Pedestrian A's device acquires location information (X1, Y1) and speed (V1), and cyclist B's device also acquires location information (X2, Y2) and speed (V2).

[0270] 2. The device sends the data to the server.

[0271] Both terminals send data to the server, which receives it.

[0272] 3. The server calculates the predicted trajectory.

[0273] The server calculates the predicted trajectories of both vehicles and assesses the risk of collision at the intersection.

[0274] 4. The server generates and sends a warning message.

[0275] If a collision risk is detected, the server sends a warning message to the devices of pedestrian A and cyclist B.

[0276] 5. The emotion engine analyzes the user's emotions.

[0277] The emotion engine analyzes the emotional state of pedestrian A and cyclist B and dynamically adjusts the content and format of the warning as needed.

[0278] 6. The device receives the warning message and notifies you.

[0279] The device displays a notification to the user and alerts them with an audio or visual signal.

[0280] 7. The user acts in accordance with the warning on the device.

[0281] The user acknowledges the warning and takes safe action.

[0282] In this way, the present invention reduces the risk of collisions between entities at intersections and ensures safety. By combining emotion engines, the effectiveness of warnings is maximized and users' safety awareness is enhanced.

[0283] The processing flow will be explained below.

[0284] Step 1: The device acquires location and speed information

[0285] The device uses the built-in GPS sensor to obtain the current location (latitude, longitude) and the speed sensor to obtain the moving speed. For example, the device of pedestrian A obtains the location information (X1, Y1) and speed (V1).

[0286] Step 2: The device sends the data to the server

[0287] The device periodically sends the acquired location and speed information to the server. The data sent includes the device ID, location information, speed information, and timestamp. For example,

[0288] {

[0289] "id": "entity_1",

[0290] "position": {"lat": X1, "lon": Y1},

[0291] "velocity": V1,

[0292] "timestamp": T1

[0293] }

[0294] It is sent in the format:

[0295] Step 3: The server receives the data

[0296] The server receives location and speed information from the device via API and stores it in memory or in a database. The server overwrites older data based on the timestamp to keep only the most recent data.

[0297] Step 4: The server calculates the predicted trajectory

[0298] The server calculates the predicted location of each entity at regular intervals based on its current location and velocity. For example, the predicted location for the next 2 seconds is calculated using the following formula:

[0299] next_position = current_position + velocity time_interval

[0300] Based on this predicted position, a future trajectory for each entity is generated.

[0301] Step 5: Server detects collision risk

[0302] The server analyzes the trajectories of all generated entities and evaluates whether they will intersect at an intersection. For example, if pedestrian A and cyclist B are likely to arrive at the same location, it identifies the time and location and determines the collision risk.

[0303] Step 6: The emotion engine analyzes the user's emotions

[0304] Using the device's built-in camera and microphone, the emotion engine analyzes the user's emotional state from their facial expressions and voice. For example, if the user is nervous, the engine will analyze their emotional state in real time.

[0305] Step 7: The server generates and sends a warning message

[0306] When the server detects a collision risk, it dynamically adjusts the warning message based on the analysis results of the emotion engine. For example, if the user is nervous, it generates a more emphasized warning message. Examples of generated warning messages are:

[0307] {

[0308] "id": "entity_1",

[0309] "warning": "collision_risk",

[0310] "message": "Attention! Possible collision ahead. Please be careful!"

[0311] }

[0312] This is sent to the device.

[0313] Step 8: The device receives the warning message and notifies the user.

[0314] When the device receives the warning message sent from the server, it will warn the user with audio or visual signals, for example, by displaying a warning message on the device screen, sounding an alarm, or flashing a light to alert the user:

[0315] Display: "Attention! Possible collision ahead. Please be careful!"

[0316] Sound: Play warning sound

[0317] Light: Start blinking

[0318] Step 9: User acts on the device warning

[0319] The user checks the warning from the device and takes safe action. For example, pedestrian A slows down as he approaches the intersection, or the driver of bicycle B slows down to delay entering the intersection.

[0320] This series of processes reduces the risk of collisions between entities at intersections without traffic lights, ensuring safety. In addition, the use of an emotion engine maximizes the effectiveness of warnings and increases user safety awareness.

[0321] Example 2

[0322] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0323] Traffic accidents frequently occur at intersections without traffic lights, and the risk increases significantly when different types of entities, such as pedestrians, vehicles, and bicycles, enter the intersection at the same time. Conventional systems have difficulty monitoring the movements of these entities in real time and providing users with appropriate warnings to reduce collision risks. Furthermore, they lack the means to appropriately adjust the content and format of warnings based on the user's emotional state. Therefore, a system that can more efficiently and effectively reduce collision risks in real-world usage environments is needed.

[0324] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0325] In this invention, the server includes a means for the terminal to acquire position information and speed information and transmit it to the server, a means for the server to calculate a predicted trajectory of each entity using the received data and detect a collision risk, a means for the server to transmit a warning message to the corresponding terminal when the server detects a collision risk, and a means for the terminal to analyze the emotional state of the user and dynamically adjust the content and format of the warning. This makes it possible to monitor the movements of entities at intersections without signals in real time, detect collision risks in a timely manner, and further improve the effectiveness of the warning according to the emotional state of the user, thereby promoting safe behavior.

[0326] A "terminal" is an electronic device that acquires location information and speed information and transmits it to a server.

[0327] The "server" is a device or system that receives data transmitted from the terminal, calculates the predicted trajectory of the entity based on the data, and detects the risk of collision.

[0328] "Location information" is the entity's current latitude and longitude data.

[0329] "Speed ​​information" is data about the speed at which an entity is moving.

[0330] An "entity" is a moving object such as a pedestrian, vehicle, or bicycle that moves around within an intersection.

[0331] A "predicted trajectory" is the path or position of an entity after a certain period of time, calculated by the server based on the entity's current location and speed information.

[0332] "Collision risk" refers to a situation where the predicted trajectories of different entities intersect, potentially resulting in a physical collision.

[0333] A "warning message" is a notification that is generated by the server and sent to the relevant terminal when a collision risk is detected.

[0334] An "audio or visual signal" is a sound or a visual alarm such as an indicator light that is emitted to alert the user when the terminal receives a warning message.

[0335] "Emotional state" refers to the user's psychological state, such as tension, anxiety, excitement, etc., and is analyzed using sensors such as a camera and microphone.

[0336] "Dynamic adjustment" refers to changing the content and notification method of the warning message in real time based on the user's emotional state.

[0337] This invention is a system for monitoring the movements of entities such as vehicles, pedestrians, and bicycles in real time at intersections without traffic lights to reduce the risk of collision accidents. The system includes means for acquiring position information and speed information of entities and transmitting it to a server, means for calculating a predicted trajectory based on the data received by the server and detecting a collision risk, means for transmitting a warning message to an appropriate terminal when a collision risk is detected, means for issuing an audio or visual signal to alert the user, and means for recognizing the user's emotions and dynamically adjusting the content and format of the warning.

[0338] Hardware and software used

[0339] Hardware:

[0340] Device: Smartphones and wearable devices are used. These devices have built-in GPS and speed sensors. Typical examples of smartphones include the Qualcomm Snapdragon GPS and Ublox M8.

[0341] Server: The server is built on a cloud service, for example, a web server using Flask on an AWS EC2 instance.

[0342] software:

[0343] GPS and speed data analysis: Location information is analyzed using the Python "Geopy" library, and speed information is obtained directly from the device's built-in sensors.

[0344] Data processing: The received data is processed and saved using the Python "Pandas" library, and the necessary analysis is performed.

[0345] Machine learning model: Sentiment analysis is performed using "Scikit-learn".

[0346] Example of a system

[0347] First, the device acquires location information (latitude X1, longitude Y1) and speed information (speed V1) using the built-in GPS sensor and speed sensor. For example, when pedestrian A is walking towards an intersection, pedestrian A's device acquires location information and speed information and sends it to the server in JSON format.

[0348] json

[0349] Topic: GPS data formatting

[0350] {

[0351] "id": "walk_A",

[0352] "position": {"lat": 35.6895, "lon": 139.6917},

[0353] "velocity": 1.2,

[0354] "timestamp": "2023-10-10T12:00:00Z"

[0355] }

[0356] The server then calculates the entity's predicted trajectory based on the received data to detect collision risk. This calculation uses the current position and speed to derive a predicted position for a certain time period. For example, the predicted position for the next 2 seconds is calculated using the following formula:

[0357] python

[0358] next_position = current_position + velocity time_interval

[0359] The server analyzes the trajectories of all entities and evaluates whether they intersect at an intersection. For example, if pedestrian A and cyclist B are likely to arrive at the same location, it identifies the time and location and determines the collision risk.

[0360] If a collision risk is detected, the server generates and sends a warning message to the device of the relevant entity. For example, the following warning message is sent to pedestrian A:

[0361] json

[0362] Note: Example of a warning message

[0363] {

[0364] "id": "walk_A",

[0365] "warning": "collision_risk",

[0366] "message": "Attention! Possible collision ahead."

[0367] }

[0368] When the device receives a warning message, it notifies the user through audio, visual signals (flashing lights), or vibration. Examples of notifications include:

[0369] json

[0370] Alert methods: sound notification, flashing light, vibration

[0371] Display: "Attention! Possible collision ahead."

[0372] Sound: Play warning sound

[0373] Light: Start blinking

[0374] An emotion engine is also used, which uses the device's camera and microphone to analyze the user's emotions and, if the user is in a negative emotional state such as tension or anxiety, will issue a more emphatic warning message, for example by increasing the volume of the warning sound or speeding up the blinking of the light signal.

[0375] Prompt Sentence Examples

[0376] "A pedestrian and a cyclist are approaching an intersection. Write Python code to calculate the collision risk and generate and send a warning message if necessary."

[0377] This allows the system to monitor entity movements at unsignalized intersections in real time, and improve safety by detecting and notifying collision risks early. Dynamic warning adjustment based on the user's emotional state maximizes the effectiveness of warning messages and increases safety awareness.

[0378] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0379] Step 1:

[0380] The device acquires location and speed information.

[0381] Input: Built-in GPS and speed sensors

[0382] Specific behavior:

[0383] The device uses the built-in GPS sensor to obtain latitude (X) and longitude (Y), and the speed sensor to measure speed (V).

[0384] Output: Position information (latitude X, longitude Y) and speed information (V)

[0385] Step 2:

[0386] The terminal transmits location information and speed information to the server.

[0387] Input: Obtained location information (latitude X, longitude Y) and speed information (V)

[0388] Specific behavior:

[0389] The device converts the location and speed information into JSON format and sends it to the server via an HTTP request.

[0390] Output: JSON data sent to the server

[0391] json

[0392] {

[0393] "id": "entity_1",

[0394] "position": {"lat": X, "lon": Y},

[0395] "velocity": V,

[0396] "timestamp": T

[0397] }

[0398] Step 3:

[0399] The server stores the received data.

[0400] Input: JSON data sent from the terminal

[0401] Specific behavior:

[0402] The server receives the HTTP request, parses the received JSON data, and saves it in the database. At this time, it saves only the latest data based on the timestamp (T).

[0403] Output: Location, speed, and timestamp stored in a database

[0404] Step 4:

[0405] The server calculates the predicted trajectories of all entities.

[0406] Input: Position and velocity information of all entities stored in the database

[0407] Specific behavior:

[0408] The server uses the position (X, Y) and velocity (V) information of each entity to calculate the predicted position at the next time step using the following formula:

[0409] python

[0410] next_position = current_position + velocity time_interval

[0411] For example, calculate the predicted location for the next 2 seconds and generate future trajectories for all entities.

[0412] Output: predicted trajectory (future_position)

[0413] Step 5:

[0414] The server determines the collision risk and generates a warning message.

[0415] Input: predicted trajectory (future_position)

[0416] Specific behavior:

[0417] The server analyzes the predicted trajectories of all entities and evaluates the likelihood of them intersecting with each other. If there is a risk of collision, it generates a warning message for the relevant entity.

[0418] Output: Warning message

[0419] json

[0420] {

[0421] "id": "entity_1",

[0422] "warning": "collision_risk",

[0423] "message": "Attention! Possible collision ahead."

[0424] }

[0425] Step 6:

[0426] The server sends a warning message to the relevant terminal.

[0427] Input: The generated warning message

[0428] Specific behavior:

[0429] The server sends the generated warning message to the relevant terminal via an HTTP request.

[0430] Output: Warning message sent to terminal

[0431] Step 7:

[0432] The terminal receives the warning message and notifies the user.

[0433] Input: The warning message sent by the server

[0434] Specific behavior:

[0435] The device receives the warning message and notifies the user of the warning through a voice message, flashing light, or vibration.

[0436] Output: Warnings notified to the user

[0437] json

[0438] Display: "Attention! Possible collision ahead."

[0439] Sound: Play warning sound

[0440] Light: Start blinking

[0441] Step 8:

[0442] The emotion engine analyzes the user's emotions and dynamically adjusts the notification method.

[0443] Input: User facial and voice data

[0444] Specific behavior:

[0445] The device's built-in camera and microphone are used to analyze the user's facial expressions and voice changes in real time, which allows the system to determine the user's psychological state and dynamically adjust the volume of the warning sound and the flashing speed according to the user's emotional state.

[0446] Output: Adjusted notification method

[0447] Through the above steps, the system can monitor the movements of entities at unsignalized intersections in real time and ensure safety by providing appropriate collision warnings.

[0448] (Application example 2)

[0449] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0450] At intersections without traffic lights, there is a high risk of collisions between entities such as vehicles, pedestrians, and cyclists, making safety particularly important for autonomous vehicles. However, conventional systems lack the means to monitor entities' movements in real time and provide appropriate warnings. Furthermore, they do not dynamically adjust warnings according to the user's emotional state, resulting in insufficient improvements in safety.

[0451] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0452] In this invention, the server includes means for the terminal to acquire position information and speed information and transmit it to the server, means for the server to calculate a predicted trajectory of each entity using the received data and detect a collision risk, means for the server to transmit a warning message to the corresponding terminal when the server detects a collision risk, and means for the terminal to analyze the emotional state of the user using a built-in emotion analysis engine and dynamically adjust the content and format of the warning message based on the analysis result. This makes it possible to reduce the collision risk of entities in real time and provide effective warnings according to the emotional state of the user.

[0453] A "terminal" is a device that acquires location and speed information and is equipped with an emotion analysis engine.

[0454] A "server" is a central processing unit that receives, analyzes, and processes data sent from a terminal.

[0455] "Location information" is data that indicates the current geographical location of a terminal.

[0456] "Speed ​​information" is data indicating the speed at which the terminal is moving.

[0457] A "predicted trajectory" is a calculated prediction of the trajectory that an entity will move in the future.

[0458] "Collision risk" refers to the likelihood that two or more entities will collide when reaching an intersection.

[0459] A "warning message" is a warning notification sent to a terminal when a collision risk is detected.

[0460] An "audio signal" is an audio notification that the terminal issues to alert the user.

[0461] A "visual signal" is a notification using light or display that a device emits to alert the user.

[0462] An "emotion analysis engine" is a system component that analyzes the user's emotional state in real time and utilizes the analysis results.

[0463] "Dynamic adjustment" refers to flexibly changing the content and format of the warning message depending on the user's emotional state.

[0464] MODE FOR CARRYING OUT THE INVENTION

[0465] The present invention relates to a system installed in an autonomous vehicle, which includes a terminal, a server, and an emotion analysis engine to reduce the risk of collision and ensure safety.

[0466] Hardware and Software Configuration

[0467] Terminal

[0468] It will be installed in self-driving vehicles.

[0469] Location and speed information is obtained using the built-in GPS sensor and speed sensor.

[0470] It is equipped with an emotion analysis engine and uses the camera and microphone to analyze the user's emotional state.

[0471] server

[0472] Receives and analyzes location and speed information sent from the device.

[0473] The latest information is stored based on timestamps and collision risk is predicted.

[0474] If a collision risk is detected, a warning message is sent to the device.

[0475] Sentiment Analysis Engine

[0476] Analyze the user's emotional state in real time.

[0477] Data from the camera and microphone is processed using image analysis libraries such as TENSORFLOW (registered trademark) and OpenCV.

[0478] Data transmission and analysis

[0479] Terminal

[0480] The device periodically sends its location and speed information to the server. The data sent consists of the entity ID, location (latitude, longitude), speed, and timestamp. The format sent to the server is as follows:

[0481] {

[0482] "id": "vehicle_1",

[0483] "position": {"lat": X1, "lon": Y1},

[0484] "velocity": V1,

[0485] "timestamp": T1

[0486] }

[0487] server

[0488] The server calculates the predicted trajectory of each entity based on the received data, calculates the entity's position when it arrives at the intersection, and evaluates the collision risk with other entities. If a collision risk is detected, it sends a warning message to the relevant device.

[0489] Sentiment analysis and dynamic alert adjustment

[0490] Emotion analysis

[0491] The device's built-in emotion analysis engine analyzes data collected from the camera and microphone to recognize the user's emotional state. A machine learning model using TensorFlow analyzes facial expressions and voice data to detect negative emotional states such as tension or anxiety.

[0492] Adjustment of warning content

[0493] An emotion analysis engine recognizes the user's emotional state and dynamically adjusts the content and format of the alert message as needed - for example, if tension or anxiety is detected, the alert may sound louder or flash a visual signal more frequently.

[0494] Specific examples

[0495] For example, if autonomous vehicle "A" is located at latitude 35.6895 and longitude 139.6917 and is moving at a speed of 10 m / s, the following prompt sentence can be used to predict its location in the next 5 seconds and avoid the risk of collision:

[0496] If the current location of the vehicle is at latitude 35.6895, longitude 139.6917, and speed 10 m / s, predict the location of the vehicle in the next 5 seconds and generate a warning message to avoid a possible collision.

[0497] In this way, the present invention can reduce the risk of collisions between entities at intersections in autonomous vehicles and ensure user safety. The introduction of a sentiment analysis engine allows the content and format of warnings to be appropriately adjusted, enabling more effective warnings.

[0498] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0499] Step 1:

[0500] The device acquires location and speed information. The device uses the built-in GPS sensor and speed sensor to measure the current location (latitude, longitude) and speed of the autonomous vehicle. The input is real-time data from the GPS sensor and speed sensor, and the output is location and speed information.

[0501] Step 2:

[0502] The location and speed information acquired by the device is sent to the server. The acquired location information (latitude, longitude) and speed information are sent to the server along with a timestamp. The input is the location and speed information acquired in step 1, and the output is the data sent to the server.

[0503] Step 3:

[0504] The server uses the received data to calculate the predicted trajectory. The server receives the vehicle's current position and speed information as input, and uses this to calculate the predicted position for the next few seconds. The output is the calculated predicted position. The server predicts the position as follows:

[0505] Next position = Current position + Speed ​​Time interval

[0506] Step 4:

[0507] The server detects collision risk. The server analyzes the predicted trajectories of all entities and evaluates the collision risk at intersections. The input is the predicted trajectory of each entity, and the output is the collision risk judgment result. Specifically, the server lists overlapping predicted positions and checks whether they are at the same time.

[0508] Step 5:

[0509] The server generates a warning message and sends it to the device. If a collision risk is detected, the server generates a warning message and sends it to the device. The input is the collision risk judgment result obtained in step 4, and the output is a warning message to the device. The warning message contains the following content:

[0510] {

[0511] "id": "entity_1",

[0512] "warning": "collision_risk",

[0513] "message": "Attention! Possible collision ahead."

[0514] }

[0515] Step 6:

[0516] The device uses an emotion analysis engine to analyze the user's emotional state. The emotion analysis engine uses data acquired from the device's built-in camera and microphone as input and analyzes the user's emotional state. The output is the user's emotional state. Specifically, the data from the camera and microphone is analyzed using TensorFlow and OpenCV to evaluate the user's facial expressions and tone of voice.

[0517] Step 7:

[0518] The device dynamically adjusts the warning message based on the emotional state. Based on the results of the emotion analysis engine, the strength of the warning sound and visual signal is adjusted. The input is the emotional state obtained in step 6, and the output is the adjusted warning message. Specific actions include increasing the volume of the warning sound or the frequency of the flashing light to emphasize the notification of the audio or visual signal.

[0519] Step 8:

[0520] The user receives a warning message from the device and takes appropriate action. The user acknowledges the warning via audio or visual signals from the device and takes safe action. The input is the adjusted warning message and the output is the user's action.

[0521] This is the flow of processing in the program for the system that realizes this application example. Through this series of processing, it is possible to reduce the risk of collisions at intersections for autonomous vehicles and ensure the safety of users.

[0522] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0523] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0524] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0525] [Second embodiment]

[0526] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0527] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0528] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0529] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0530] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0531] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0532] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0533] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0534] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0535] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0536] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0537] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0538] The present invention provides a system for monitoring the movements of vehicles, pedestrians, and bicycles at intersections without traffic lights in real time to reduce the risk of collisions. The system includes a means for a terminal to acquire position information and speed information and transmit it to a server, a means for the server to calculate a predicted trajectory based on the received data and detect a collision risk, a means for the server to send a warning message to the terminal when a collision risk is detected, and a means for the terminal to issue an audio or visual signal to a user when it receives the warning message.

[0539] System operation example

[0540] The device acquires location and speed information using the built-in GPS sensor and speed sensor. For example, when pedestrian A is walking towards an intersection, pedestrian A's device acquires location information (latitude X1, longitude Y1) and speed information (speed V1), which are periodically sent to the server.

[0541] Sending data:

[0542] The device sends location and speed information to the server, which receives it. The data sent is in the following format:

[0543] {

[0544] "id": "entity_1",

[0545] "position": {"lat": X1, "lon": Y1},

[0546] "velocity": V1,

[0547] "timestamp": T1

[0548] }

[0549] The server stores the received data and uses only the most recent information for analysis based on the timestamp.

[0550] Collision risk analysis:

[0551] The server calculates the predicted trajectory of each entity. For example, if Pedestrian A and Bicycle B are heading towards the same intersection, the server calculates their predicted positions in the next 2 seconds and evaluates the likelihood that the entities will arrive at the same location.

[0552] Calculating the predicted trajectory:

[0553] The server calculates the predicted location based on the current location and velocity information as follows:

[0554] next_position = current_position + velocity time_interval

[0555] Based on this predicted position, the entity's future positional relationship is analyzed.

[0556] Sending a warning message:

[0557] If the server detects a collision risk, it generates and sends a warning message to the device of the relevant entity. For example, if there is a high possibility that pedestrian A and cyclist B will collide, the server will generate the following warning message:

[0558] {

[0559] "id": "entity_1",

[0560] "warning": "collision_risk",

[0561] "message": "Attention! Possible collision ahead."

[0562] }

[0563] This is sent to the device.

[0564] Receive alerts and notifications:

[0565] When the device receives a warning message from the server, it notifies the user of the warning. Notification methods include voice messages and flashing lights:

[0566] Display: "Attention! Possible collision ahead."

[0567] Sound: Play warning sound

[0568] Light: Start blinking

[0569] The user can then review this notification and take appropriate action.

[0570] Specific example processing:

[0571] For example, consider a situation where pedestrian A is walking towards an intersection and cyclist B is simultaneously heading towards the intersection. The system operates in the following steps:

[0572] 1. The device acquires location and speed information.

[0573] Pedestrian A's device acquires location information (X1, Y1) and speed (V1), and cyclist B's device also acquires location information (X2, Y2) and speed (V2).

[0574] 2. The device sends the data to the server.

[0575] Both terminals send data to the server, which receives it.

[0576] 3. The server calculates the predicted trajectory.

[0577] The server calculates the predicted trajectories of both vehicles and assesses the risk of collision at the intersection.

[0578] 4. The server generates and sends a warning message.

[0579] If a collision risk is detected, the server sends a warning message to the devices of pedestrian A and cyclist B.

[0580] 5. The device receives the warning message and notifies you.

[0581] The device displays a notification to the user and alerts them with an audio or visual signal.

[0582] In this way, the present invention reduces the risk of collisions between entities within an intersection, ensuring safety.

[0583] The processing flow will be explained below.

[0584] Step 1: The device acquires location and speed information

[0585] The device uses the built-in GPS sensor to obtain the current location (latitude, longitude) and the speed sensor to obtain the moving speed. For example, the device of pedestrian A obtains the location information (X1, Y1) and speed (V1).

[0586] Step 2: The device sends the data to the server

[0587] The device periodically sends the acquired location and speed information to the server. The data sent includes the device ID, location information, speed information, and timestamp. For example,

[0588] {

[0589] "id": "entity_1",

[0590] "position": {"lat": X1, "lon": Y1},

[0591] "velocity": V1,

[0592] "timestamp": T1

[0593] }

[0594] It is sent in the format:

[0595] Step 3: The server receives the data

[0596] The server receives location and speed information from the device via API and stores it in memory or in a database. The server overwrites older data based on the timestamp to keep only the most recent data.

[0597] Step 4: The server calculates the predicted trajectory

[0598] The server calculates the predicted location of each entity at regular intervals based on its current location and velocity. For example, the predicted location for the next 2 seconds is calculated using the following formula:

[0599] next_position = current_position + velocity time_interval

[0600] Based on this predicted position, a future trajectory for each entity is generated.

[0601] Step 5: Server detects collision risk

[0602] The server analyzes the trajectories of all generated entities and evaluates whether they will intersect at an intersection. For example, if pedestrian A and cyclist B are likely to arrive at the same location, it identifies the time and location and determines the collision risk.

[0603] Step 6: The server generates and sends a warning message

[0604] If the server detects a collision risk, it generates and sends a warning message to the device of the relevant entity. For example, it generates a warning message like this:

[0605] {

[0606] "id": "entity_1",

[0607] "warning": "collision_risk",

[0608] "message": "Attention! Possible collision ahead."

[0609] }

[0610] Send this message to the terminal.

[0611] Step 7: The device receives the warning message and notifies the user

[0612] When the terminal receives the warning message sent from the server, it issues a warning to the user with an audio or visual signal, such as displaying the warning message on the terminal screen, sounding an alarm, or flashing a light to alert the user.

[0613] Step 8: The user acts on the device warning

[0614] The user checks the warning from the device and takes safe action. For example, pedestrian A slows down as he approaches the intersection, or the driver of bicycle B slows down to delay entering the intersection.

[0615] Example 1

[0616] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0617] At intersections without traffic lights, the risk of collisions increases because many moving objects, including vehicles, pedestrians, and bicycles, cross the intersection. In such situations, a system is needed that can monitor the position and speed of each entity in real time, detect danger in advance, and issue a warning. However, conventional systems have difficulty accurately calculating the predicted trajectories of multiple moving objects and issuing a warning quickly.

[0618] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0619] In this invention, the server includes a means for the terminal to acquire position information and speed information and transmit it to the server, a means for the server to calculate a predicted trajectory of each entity using the data received and detect a collision risk, and a means for transmitting a warning message to the corresponding terminal when the server detects a collision risk. This makes it possible to simultaneously calculate the predicted trajectories of multiple entities and evaluate the collision risk at an intersection.

[0620] A "terminal" is a device for obtaining location and velocity information of an entity and transmitting it to a server.

[0621] "Location information" is latitude and longitude data that indicates the current location of an entity.

[0622] "Speed ​​information" is data that indicates the moving speed of an entity.

[0623] A "server" is a device or system that receives data transmitted from a terminal and calculates a predicted trajectory and evaluates a collision risk.

[0624] A "predicted trajectory" is a future movement path calculated based on the entity's current position and speed information.

[0625] "Collision risk" is an assessment that indicates the likelihood that multiple entities will arrive at the same location in the future, resulting in a collision accident.

[0626] A "warning message" is a warning communication that is sent to the relevant terminal when the server detects a collision risk.

[0627] The "audio signal" is an audio notification that alerts the user when a warning message is received by the terminal.

[0628] A "visual signal" is a light or screen display notification that alerts the user when a warning message is received by the terminal.

[0629] A "timestamp" is information that records the time when data is acquired or transmitted.

[0630] A "location sensor" is a device that can determine the current location of an entity and obtain that location information.

[0631] The present invention provides a system for monitoring the movements of vehicles, pedestrians, and bicycles at intersections without traffic lights in real time to reduce the risk of collision accidents. The system includes a means for a terminal to acquire position information and speed information and transmit it to a server, a means for the server to calculate a predicted trajectory based on the received data and detect a collision risk, a means for the server to transmit a warning message to the terminal when a collision risk is detected, and a means for the terminal to issue an audio or visual signal to a user when it receives the warning message.

[0632] The device acquires current location information (latitude and longitude) and speed information using the built-in location sensor and speed sensor. For example, if a pedestrian is walking towards an intersection, the device acquires location information (X1, Y1) and speed information (V1) and periodically sends this to the server. The data sent by the device includes the ID, location information (latitude and longitude), speed information, and the timestamp when the data was acquired.

[0633] The server receives data sent from the device and stores it in a database. Data sent from each entity is recorded with a timestamp, and the server uses only the most recent data for analysis. Based on the received data, the server calculates the predicted trajectory of each entity. The current position and speed information are used to calculate the predicted trajectory, using the formula "next position = current position + speed x time interval."

[0634] The server simultaneously calculates the predicted positions of multiple entities and analyzes their future positional relationships. For example, if pedestrian A and cyclist B are heading towards the same intersection, the server calculates their predicted positions and evaluates the probability that they will arrive at the intersection at the same time. Based on this evaluation, the server determines the collision risk.

[0635] If a collision risk is detected, the server generates and sends a warning message to the relevant entity. For example, if there is a high risk of a collision between pedestrian A and cyclist B, the server generates a warning message and sends it to both devices. The warning message includes a warning message and details of the warning.

[0636] When the device receives a warning message, it notifies the user with an audio or visual signal, such as by displaying the message "Attention! Possible collision ahead" on the display, emitting a warning sound, or flashing a light to alert the user.

[0637] For example, if pedestrian A is walking towards an intersection and cyclist B is heading towards the same intersection, the system will operate as follows:

[0638] 1. Pedestrian A's device acquires location information (X1, Y1) and speed (V1), and cyclist B's device also acquires location information (X2, Y2) and speed (V2).

[0639] 2. Both devices send data to the server.

[0640] 3. The server stores the received data and calculates the predicted trajectories of both vehicles.

[0641] 4. The server evaluates the collision risk based on the predicted trajectory.

[0642] 5. If a collision risk is detected, the server generates a warning message and sends it to the devices of pedestrian A and cyclist B.

[0643] 6. The terminal receives the warning message and notifies the user of the warning.

[0644] Such a system can reduce the risk of collisions between entities and ensure safety even at intersections without traffic lights. An example of a prompt sentence could be, "Watch out for the next intersection. Stop and check your surroundings."

[0645] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0646] System program processing flow

[0647] Step 1: The device obtains location and speed information

[0648] The device uses the built-in location and speed sensors to obtain current location information (latitude and longitude) and speed information. The input is the entity's sensor information, and the output is location information (X, Y) and speed information (V). This specific operation involves obtaining GPS data and calculating speed using the acceleration sensor.

[0649] Step 2: The device sends data to the server

[0650] The device sends the acquired location and speed information to the server. The input is the location and speed information acquired in step 1, and the output is the transmission of a data packet to the server. The transmitted data has the following format:

[0651] {

[0652] "id": "entity_1",

[0653] "position": {"lat": X, "lon": Y},

[0654] "velocity": V,

[0655] "timestamp": T

[0656] }

[0657] Specifically, the data packet is sent to the server using an HTTP request or the like.

[0658] Step 3: The server receives and stores the data

[0659] The server receives the data sent from the device and stores it in a database. The input is the data packet sent in step 2, and the output is the stored location and speed information. The specific operation is a write operation to the database.

[0660] Step 4: The server calculates the predicted trajectory

[0661] The server calculates the predicted trajectory of each entity based on the received position and velocity information. The input is the saved current position and velocity information, and the output is the predicted position (next_position) after the next time interval. The predicted position is calculated using the following formula:

[0662] next_position = current_position + velocity time_interval

[0663] The specific operation is to use this formula to predict the future position of each entity.

[0664] Step 5: Server evaluates collision risk

[0665] The server evaluates the collision risk of multiple entities based on the predicted trajectories. The input is the predicted position calculated in step 4, and the output is the collision risk assessment result. The specific operation is to compare the predicted positions and calculate the probability that the entities will arrive at the same point.

[0666] Step 6: The server generates a warning message

[0667] If a collision risk is detected, the server generates a warning message to be sent to the relevant entity. The input is the collision risk assessment result from step 5, and the output is a warning message. The specific format of the warning message is as follows:

[0668] {

[0669] "id": "entity_1",

[0670] "warning": "collision_risk",

[0671] "message": "Attention! Possible collision ahead."

[0672] }

[0673] Specifically, the system dynamically generates warning content based on the evaluation results.

[0674] Step 7: The server sends a warning message

[0675] The server sends the generated warning message to the corresponding terminal. The input is the warning message generated in step 6, and the output is the transmission of warning data to the terminal. Specifically, this message is sent to the corresponding terminal using an HTTP request, etc.

[0676] Step 8: Your device receives a warning message

[0677] The terminal receives the warning message sent from the server. The input is the warning message from the server, and the output is the internal processing of this message. The specific operation is to store the received message in memory and add it to the processing queue.

[0678] Step 9: The device notifies the user of the warning

[0679] When the device receives the warning message, it notifies the user of the warning with an audio or visual signal. The input is the warning message received in step 8, and the output is the notification to the user. Specifically, the device displays "Attention! Possible collision ahead." and plays a warning sound or flashes a light.

[0680] In this way, the risk of collisions between entities can be reduced and safety ensured even at intersections without traffic lights.

[0681] (Application example 1)

[0682] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0683] At unsignalized intersections and in congested urban environments, the risk of collisions between autonomous vehicles and entities such as other vehicles, pedestrians, and cyclists is increasing. In such situations, real-time dynamic prediction and immediate warnings are necessary for autonomous vehicles to operate safely and prevent accidents. However, current technology is unable to meet these needs, and there is a lack of systems that can adequately reduce the risk of collisions between entities. Accidents resulting from this pose a major problem that threatens the safety of passengers and those around them.

[0684] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0685] In this invention, the server includes: means for the terminal to acquire position information and speed information and transmit it to the server; means for the server to calculate a predicted trajectory of each entity using the data received and detect a collision risk; means for transmitting a warning message to the corresponding terminal when the server detects a collision risk; means for the terminal to issue an audio or visual signal to alert the user when it receives the warning message; and means for the terminal to be mounted in an autonomous vehicle and notify the driver of the warning message via an in-vehicle display and audio guidance system. This enables autonomous vehicles to detect collision risks with other entities in real time, even at intersections or in crowded environments, and ensure safe operation.

[0686] A "terminal" is a device that acquires location information and speed information and transmits it to a server.

[0687] "Predicted trajectory" is a calculation method that predicts the future position of each entity based on its current position and speed information.

[0688] "Collision risk" is an assessment that indicates the likelihood of contact or collision between entities using the predicted trajectory.

[0689] A "warning message" is a notification message that is sent to the relevant terminal when a collision risk is detected.

[0690] An "audio or visual signal" is a signal, such as a voice message or a flashing light, that is emitted to notify the user of a warning.

[0691] An "autonomous vehicle" is a vehicle that is equipped with GPS and speed sensors and operates autonomously.

[0692] An "in-vehicle display" is a screen installed inside an autonomous vehicle to visually display information.

[0693] A "voice guidance system" is a system installed inside an autonomous vehicle that provides information through voice messages.

[0694] The present invention is a system for reducing the risk of collisions between autonomous vehicles and entities such as other vehicles, pedestrians, and cyclists at unsignalized intersections and in congested urban environments. The system includes a terminal, a server, an in-vehicle display, and a voice guidance system.

[0695] System hardware configuration

[0696] Device: Installed in an autonomous vehicle, it continuously acquires location and speed information using built-in positioning sensors (e.g., GPS sensors) and speed sensors.

[0697] Server: Located on the cloud, it receives data sent from the device, calculates the predicted trajectory, and evaluates the collision risk.

[0698] In-vehicle displays: installed inside autonomous vehicles, they visually display warning messages and other necessary information.

[0699] Voice guidance system: A system that works in conjunction with the in-car display to provide warnings to the driver through voice messages.

[0700] Software configuration and processing flow

[0701] 1. Data Acquisition

[0702] The device uses built-in positioning and speed measurement sensors to obtain real-time position and speed information of the autonomous vehicle.

[0703] 2. Data Transmission

[0704] The terminal periodically transmits the acquired location and speed information to the server. The transmitted data includes location information (latitude and longitude), speed information, and a timestamp.

[0705] 3. Collision risk assessment

[0706] The server uses the received data to calculate the predicted trajectory of the autonomous vehicle and other entities, including calculating future positions based on current position and speed information.

[0707] 4. Generating Warning Messages

[0708] The server evaluates the collision risk based on the predicted trajectory, and if a collision risk is detected, it generates and sends a warning message to the relevant terminal.

[0709] 5. Warning Notification

[0710] When the terminal receives a warning message from the server, it issues a visual and audio warning to the driver through the in-car display and voice guidance system.

[0711] Prompt Sentence Examples

[0712] Below are some example prompts to input to a generative AI model:

[0713] "Design a system that acquires real-time location and speed information of an autonomous vehicle, sends it to a server, and evaluates the risk of collision. The server calculates a predicted trajectory based on data from other entities, and if there is a risk of collision, warns the driver on the vehicle's display and voice guidance system. Please provide Python code that runs in real time."

[0714] As a specific example, consider the following scenario: As autonomous vehicle A approaches an intersection, it detects the presence of pedestrian B walking toward the same intersection. The server calculates the predicted trajectories of both vehicles, and if it detects a high risk of collision, it sends a warning message to the device of autonomous vehicle A. The in-vehicle display will show a message saying, "Possible collision with pedestrian at intersection ahead. Please be careful," and the voice guidance system will give instructions such as, "Pay attention to what is ahead." This notification will alert the driver to be careful at the intersection, ensuring safe driving.

[0715] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0716] Step 1:

[0717] The device obtains location and speed information. The device uses a location sensor (e.g., GPS sensor) and a speed sensor to obtain the current latitude, longitude, and speed of the autonomous vehicle. In this case, the input is the raw data provided by the vehicle's sensors, and the output is the latitude, longitude, and speed values.

[0718] Step 2:

[0719] The device sends the acquired location and speed information to the server. The device sends data to the server, including the acquired location information (latitude, longitude), speed information, and timestamp. This data is encoded in JSON or other appropriate data format. The location and speed information obtained in step 1 is used as input, and the data sent to the server is generated as output.

[0720] Step 3:

[0721] The server stores the received data and uses it for analysis. The server stores the location and speed information received from the device in a database and uses only the most recent data for analysis based on the timestamp. This eliminates duplicate and old data. The input is the data sent from the device, and the output is the latest data set to use for analysis and calculations.

[0722] Step 4:

[0723] The server calculates the predicted trajectory of each entity. Based on the received position and velocity information, the server predicts the future position of each entity. This calculation is done by multiplying the current position by the velocity, for example. The input is the latest position and velocity information, and the output is the predicted position at a specific point in the future.

[0724] Step 5:

[0725] The server evaluates the collision risk. The server compares the predicted trajectories of each entity and determines whether there is a possibility of an intersection. If the intersection points are very close, it determines that there is a collision risk. The input is the predicted trajectory data, and the output is the collision risk assessment result (risk / no risk).

[0726] Step 6:

[0727] If the server detects a collision risk, it sends a warning message to the relevant terminal. The server generates and sends a warning message to the terminal of the entity where the collision risk was detected. This message indicates that a collision risk exists and calls for attention. The input is the collision risk evaluation result, and the output is the warning message to be sent to each terminal.

[0728] Step 7:

[0729] The terminal receives the warning message and alerts the user. The terminal notifies the driver of the received warning message via the in-car display or voice guidance system. For example, the display may show a message saying "There is a pedestrian ahead. Please be careful," and the voice guidance system may play a voice message saying "Pay attention to what is ahead." The input is the warning message received from the server, and the output is the warning notification to the user.

[0730] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0731] The present invention combines an emotion engine with a system for monitoring the movements of entities such as vehicles, pedestrians, and bicycles at intersections without traffic lights in real time and reducing the risk of collision accidents. This system includes means for a terminal to acquire position information and speed information and send it to a server, means for the server to calculate a predicted trajectory based on the received data and detect a collision risk, means for the server to send a warning message to the relevant terminal when a collision risk is detected, means for the terminal to issue an audio or visual signal to alert the user when it receives the warning message, and an emotion engine that recognizes the user's emotions.

[0732] System Overview

[0733] The device acquires location and speed information using the built-in GPS sensor and speed sensor. For example, when pedestrian A is walking towards an intersection, pedestrian A's device acquires location information (latitude X1, longitude Y1) and speed information (speed V1), which are periodically sent to the server.

[0734] Detailed System Operation

[0735] Sending data:

[0736] The device periodically sends the location and speed information it has acquired to the server. The data sent is in the following format:

[0737] {

[0738] "id": "entity_1",

[0739] "position": {"lat": X1, "lon": Y1},

[0740] "velocity": V1,

[0741] "timestamp": T1

[0742] }

[0743] The server stores the received data and uses only the most recent information for analysis based on the timestamp.

[0744] Collision risk analysis:

[0745] The server calculates the predicted location of each entity at regular intervals based on its current location and velocity. For example, the predicted location for the next 2 seconds is calculated using the following formula:

[0746] next_position = current_position + velocity time_interval

[0747] Based on this predicted position, a future trajectory for each entity is generated.

[0748] Calculating the predicted trajectory:

[0749] The server analyzes the trajectories of all entities and evaluates whether they intersect at an intersection. For example, if pedestrian A and cyclist B are likely to arrive at the same location, it identifies the time and location and determines the collision risk.

[0750] Generate and send a warning message:

[0751] If the server detects a collision risk, it generates and sends a warning message to the device of the relevant entity. For example, it generates a warning message like this:

[0752] {

[0753] "id": "entity_1",

[0754] "warning": "collision_risk",

[0755] "message": "Attention! Possible collision ahead."

[0756] }

[0757] This is sent to the device.

[0758] Using the Emotion Engine:

[0759] The emotion engine analyzes the user's emotions in real time and recognizes their emotional state. For example, it uses the device's camera and microphone to analyze emotions from changes in the user's facial expressions and voice.

[0760] Dynamically adjusting warnings based on emotions:

[0761] If the emotion engine determines that the user is in a negative emotional state, such as tension or anxiety, the server will issue a more emphatic warning message, for example by making the warning sound louder or flashing visual signals more frequently.

[0762] Receiving and notifying warning messages:

[0763] When the device receives a warning message from the server, it notifies the user of the warning. Notification methods include voice messages and flashing lights:

[0764] Display: "Attention! Possible collision ahead."

[0765] Sound: Play warning sound

[0766] Light: Start blinking

[0767] The user can then review this notification and take appropriate action.

[0768] Processing of specific examples

[0769] For example, consider a situation where pedestrian A is walking towards an intersection and cyclist B is simultaneously heading towards the intersection. The system operates in the following steps:

[0770] 1. The device acquires location and speed information.

[0771] Pedestrian A's device acquires location information (X1, Y1) and speed (V1), and cyclist B's device also acquires location information (X2, Y2) and speed (V2).

[0772] 2. The device sends the data to the server.

[0773] Both terminals send data to the server, which receives it.

[0774] 3. The server calculates the predicted trajectory.

[0775] The server calculates the predicted trajectories of both vehicles and assesses the risk of collision at the intersection.

[0776] 4. The server generates and sends a warning message.

[0777] If a collision risk is detected, the server sends a warning message to the devices of pedestrian A and cyclist B.

[0778] 5. The emotion engine analyzes the user's emotions.

[0779] The emotion engine analyzes the emotional state of pedestrian A and cyclist B and dynamically adjusts the content and format of the warning as needed.

[0780] 6. The device receives the warning message and notifies you.

[0781] The device displays a notification to the user and alerts them with an audio or visual signal.

[0782] 7. The user acts in accordance with the warning on the device.

[0783] The user acknowledges the warning and takes safe action.

[0784] In this way, the present invention reduces the risk of collisions between entities at intersections and ensures safety. By combining emotion engines, the effectiveness of warnings is maximized and users' safety awareness is enhanced.

[0785] The processing flow will be explained below.

[0786] Step 1: The device acquires location and speed information

[0787] The device uses the built-in GPS sensor to obtain the current location (latitude, longitude) and the speed sensor to obtain the moving speed. For example, the device of pedestrian A obtains the location information (X1, Y1) and speed (V1).

[0788] Step 2: The device sends the data to the server

[0789] The device periodically sends the acquired location and speed information to the server. The data sent includes the device ID, location information, speed information, and timestamp. For example,

[0790] {

[0791] "id": "entity_1",

[0792] "position": {"lat": X1, "lon": Y1},

[0793] "velocity": V1,

[0794] "timestamp": T1

[0795] }

[0796] It is sent in the format:

[0797] Step 3: The server receives the data

[0798] The server receives location and speed information from the device via API and stores it in memory or in a database. The server overwrites older data based on the timestamp to keep only the most recent data.

[0799] Step 4: The server calculates the predicted trajectory

[0800] The server calculates the predicted location of each entity at regular intervals based on its current location and velocity. For example, the predicted location for the next 2 seconds is calculated using the following formula:

[0801] next_position = current_position + velocity time_interval

[0802] Based on this predicted position, a future trajectory for each entity is generated.

[0803] Step 5: Server detects collision risk

[0804] The server analyzes the trajectories of all generated entities and evaluates whether they will intersect at an intersection. For example, if pedestrian A and cyclist B are likely to arrive at the same location, it identifies the time and location and determines the collision risk.

[0805] Step 6: The emotion engine analyzes the user's emotions

[0806] Using the device's built-in camera and microphone, the emotion engine analyzes the user's emotional state from their facial expressions and voice. For example, if the user is nervous, the engine will analyze their emotional state in real time.

[0807] Step 7: The server generates and sends a warning message

[0808] When the server detects a collision risk, it dynamically adjusts the warning message based on the analysis results of the emotion engine. For example, if the user is nervous, it generates a more emphasized warning message. Examples of generated warning messages are:

[0809] {

[0810] "id": "entity_1",

[0811] "warning": "collision_risk",

[0812] "message": "Attention! Possible collision ahead. Please be careful!"

[0813] }

[0814] This is sent to the device.

[0815] Step 8: The device receives the warning message and notifies the user.

[0816] When the device receives the warning message sent from the server, it will warn the user with audio or visual signals, for example, by displaying a warning message on the device screen, sounding an alarm, or flashing a light to alert the user:

[0817] Display: "Attention! Possible collision ahead. Please be careful!"

[0818] Sound: Play warning sound

[0819] Light: Start blinking

[0820] Step 9: User acts on the device warning

[0821] The user checks the warning from the device and takes safe action. For example, pedestrian A slows down as he approaches the intersection, or the driver of bicycle B slows down to delay entering the intersection.

[0822] This series of processes reduces the risk of collisions between entities at intersections without traffic lights, ensuring safety. In addition, the use of an emotion engine maximizes the effectiveness of warnings and increases user safety awareness.

[0823] Example 2

[0824] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0825] Traffic accidents frequently occur at intersections without traffic lights, and the risk increases significantly when different types of entities, such as pedestrians, vehicles, and bicycles, enter the intersection at the same time. Conventional systems have difficulty monitoring the movements of these entities in real time and providing users with appropriate warnings to reduce collision risks. Furthermore, they lack the means to appropriately adjust the content and format of warnings based on the user's emotional state. Therefore, a system that can more efficiently and effectively reduce collision risks in real-world usage environments is needed.

[0826] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0827] In this invention, the server includes a means for the terminal to acquire position information and speed information and transmit it to the server, a means for the server to calculate a predicted trajectory of each entity using the received data and detect a collision risk, a means for the server to transmit a warning message to the corresponding terminal when the server detects a collision risk, and a means for the terminal to analyze the emotional state of the user and dynamically adjust the content and format of the warning. This makes it possible to monitor the movements of entities at intersections without signals in real time, detect collision risks in a timely manner, and further improve the effectiveness of the warning according to the emotional state of the user, thereby promoting safe behavior.

[0828] A "terminal" is an electronic device that acquires location information and speed information and transmits it to a server.

[0829] The "server" is a device or system that receives data transmitted from the terminal, calculates the predicted trajectory of the entity based on the data, and detects the risk of collision.

[0830] "Location information" is the entity's current latitude and longitude data.

[0831] "Speed ​​information" is data about the speed at which an entity is moving.

[0832] An "entity" is a moving object such as a pedestrian, vehicle, or bicycle that moves around within an intersection.

[0833] A "predicted trajectory" is the path or position of an entity after a certain period of time, calculated by the server based on the entity's current location and speed information.

[0834] "Collision risk" refers to a situation where the predicted trajectories of different entities intersect, potentially resulting in a physical collision.

[0835] A "warning message" is a notification that is generated by the server and sent to the relevant terminal when a collision risk is detected.

[0836] An "audio or visual signal" is a sound or a visual alarm such as an indicator light that is emitted to alert the user when the terminal receives a warning message.

[0837] "Emotional state" refers to the user's psychological state, such as tension, anxiety, excitement, etc., and is analyzed using sensors such as a camera and microphone.

[0838] "Dynamic adjustment" refers to changing the content and notification method of the warning message in real time based on the user's emotional state.

[0839] This invention is a system for monitoring the movements of entities such as vehicles, pedestrians, and bicycles in real time at intersections without traffic lights to reduce the risk of collision accidents. The system includes means for acquiring position information and speed information of entities and transmitting it to a server, means for calculating a predicted trajectory based on the data received by the server and detecting a collision risk, means for transmitting a warning message to an appropriate terminal when a collision risk is detected, means for issuing an audio or visual signal to alert the user, and means for recognizing the user's emotions and dynamically adjusting the content and format of the warning.

[0840] Hardware and software used

[0841] Hardware:

[0842] Device: Smartphones and wearable devices are used. These devices have built-in GPS and speed sensors. Typical examples of smartphones include the Qualcomm Snapdragon GPS and Ublox M8.

[0843] Server: The server is built on a cloud service, for example, a web server using Flask on an AWS EC2 instance.

[0844] software:

[0845] GPS and speed data analysis: Location information is analyzed using the Python "Geopy" library, and speed information is obtained directly from the device's built-in sensors.

[0846] Data processing: The received data is processed and saved using the Python "Pandas" library, and the necessary analysis is performed.

[0847] Machine learning model: Sentiment analysis is performed using "Scikit-learn".

[0848] Example of a system

[0849] First, the device acquires location information (latitude X1, longitude Y1) and speed information (speed V1) using the built-in GPS sensor and speed sensor. For example, when pedestrian A is walking towards an intersection, pedestrian A's device acquires location information and speed information and sends it to the server in JSON format.

[0850] json

[0851] Topic: GPS data formatting

[0852] {

[0853] "id": "walk_A",

[0854] "position": {"lat": 35.6895, "lon": 139.6917},

[0855] "velocity": 1.2,

[0856] "timestamp": "2023-10-10T12:00:00Z"

[0857] }

[0858] The server then calculates the entity's predicted trajectory based on the received data to detect collision risk. This calculation uses the current position and speed to derive a predicted position for a certain time period. For example, the predicted position for the next 2 seconds is calculated using the following formula:

[0859] python

[0860] next_position = current_position + velocity time_interval

[0861] The server analyzes the trajectories of all entities and evaluates whether they intersect at an intersection. For example, if pedestrian A and cyclist B are likely to arrive at the same location, it identifies the time and location and determines the collision risk.

[0862] If a collision risk is detected, the server generates and sends a warning message to the device of the relevant entity. For example, the following warning message is sent to pedestrian A:

[0863] json

[0864] Note: Example of a warning message

[0865] {

[0866] "id": "walk_A",

[0867] "warning": "collision_risk",

[0868] "message": "Attention! Possible collision ahead."

[0869] }

[0870] When the device receives a warning message, it notifies the user through audio, visual signals (flashing lights), or vibration. Examples of notifications include:

[0871] json

[0872] Alert methods: sound notification, flashing light, vibration

[0873] Display: "Attention! Possible collision ahead."

[0874] Sound: Play warning sound

[0875] Light: Start blinking

[0876] An emotion engine is also used, which uses the device's camera and microphone to analyze the user's emotions and, if the user is in a negative emotional state such as tension or anxiety, will issue a more emphatic warning message, for example by increasing the volume of the warning sound or speeding up the blinking of the light signal.

[0877] Prompt Sentence Examples

[0878] "A pedestrian and a cyclist are approaching an intersection. Write Python code to calculate the collision risk and generate and send a warning message if necessary."

[0879] This allows the system to monitor entity movements at unsignalized intersections in real time, and improve safety by detecting and notifying collision risks early. Dynamic warning adjustment based on the user's emotional state maximizes the effectiveness of warning messages and increases safety awareness.

[0880] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0881] Step 1:

[0882] The device acquires location and speed information.

[0883] Input: Built-in GPS and speed sensors

[0884] Specific behavior:

[0885] The device uses the built-in GPS sensor to obtain latitude (X) and longitude (Y), and the speed sensor to measure speed (V).

[0886] Output: Position information (latitude X, longitude Y) and speed information (V)

[0887] Step 2:

[0888] The terminal transmits location information and speed information to the server.

[0889] Input: Obtained location information (latitude X, longitude Y) and speed information (V)

[0890] Specific behavior:

[0891] The device converts the location and speed information into JSON format and sends it to the server via an HTTP request.

[0892] Output: JSON data sent to the server

[0893] json

[0894] {

[0895] "id": "entity_1",

[0896] "position": {"lat": X, "lon": Y},

[0897] "velocity": V,

[0898] "timestamp": T

[0899] }

[0900] Step 3:

[0901] The server stores the received data.

[0902] Input: JSON data sent from the terminal

[0903] Specific behavior:

[0904] The server receives the HTTP request, parses the received JSON data, and saves it in the database. At this time, it saves only the latest data based on the timestamp (T).

[0905] Output: Location, speed, and timestamp stored in a database

[0906] Step 4:

[0907] The server calculates the predicted trajectories of all entities.

[0908] Input: Position and velocity information of all entities stored in the database

[0909] Specific behavior:

[0910] The server uses the position (X, Y) and velocity (V) information of each entity to calculate the predicted position at the next time step using the following formula:

[0911] python

[0912] next_position = current_position + velocity time_interval

[0913] For example, calculate the predicted location for the next 2 seconds and generate future trajectories for all entities.

[0914] Output: predicted trajectory (future_position)

[0915] Step 5:

[0916] The server determines the collision risk and generates a warning message.

[0917] Input: predicted trajectory (future_position)

[0918] Specific behavior:

[0919] The server analyzes the predicted trajectories of all entities and evaluates the likelihood of them intersecting with each other. If there is a risk of collision, it generates a warning message for the relevant entity.

[0920] Output: Warning message

[0921] json

[0922] {

[0923] "id": "entity_1",

[0924] "warning": "collision_risk",

[0925] "message": "Attention! Possible collision ahead."

[0926] }

[0927] Step 6:

[0928] The server sends a warning message to the relevant terminal.

[0929] Input: The generated warning message

[0930] Specific behavior:

[0931] The server sends the generated warning message to the relevant terminal via an HTTP request.

[0932] Output: Warning message sent to terminal

[0933] Step 7:

[0934] The terminal receives the warning message and notifies the user.

[0935] Input: The warning message sent by the server

[0936] Specific behavior:

[0937] The device receives the warning message and notifies the user of the warning through a voice message, flashing light, or vibration.

[0938] Output: Warnings notified to the user

[0939] json

[0940] Display: "Attention! Possible collision ahead."

[0941] Sound: Play warning sound

[0942] Light: Start blinking

[0943] Step 8:

[0944] The emotion engine analyzes the user's emotions and dynamically adjusts the notification method.

[0945] Input: User facial and voice data

[0946] Specific behavior:

[0947] The device's built-in camera and microphone are used to analyze the user's facial expressions and voice changes in real time, which allows the system to determine the user's psychological state and dynamically adjust the volume of the warning sound and the flashing speed according to the user's emotional state.

[0948] Output: Adjusted notification method

[0949] Through the above steps, the system can monitor the movements of entities at unsignalized intersections in real time and ensure safety by providing appropriate collision warnings.

[0950] (Application example 2)

[0951] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0952] At intersections without traffic lights, there is a high risk of collisions between entities such as vehicles, pedestrians, and cyclists, making safety particularly important for autonomous vehicles. However, conventional systems lack the means to monitor entities' movements in real time and provide appropriate warnings. Furthermore, they do not dynamically adjust warnings according to the user's emotional state, resulting in insufficient improvements in safety.

[0953] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0954] In this invention, the server includes means for the terminal to acquire position information and speed information and transmit it to the server, means for the server to calculate a predicted trajectory of each entity using the received data and detect a collision risk, means for the server to transmit a warning message to the corresponding terminal when the server detects a collision risk, and means for the terminal to analyze the emotional state of the user using a built-in emotion analysis engine and dynamically adjust the content and format of the warning message based on the analysis result. This makes it possible to reduce the collision risk of entities in real time and provide effective warnings according to the emotional state of the user.

[0955] A "terminal" is a device that acquires location and speed information and is equipped with an emotion analysis engine.

[0956] A "server" is a central processing unit that receives, analyzes, and processes data sent from a terminal.

[0957] "Location information" is data that indicates the current geographical location of a terminal.

[0958] "Speed ​​information" is data indicating the speed at which the terminal is moving.

[0959] A "predicted trajectory" is a calculated prediction of the trajectory that an entity will move in the future.

[0960] "Collision risk" refers to the likelihood that two or more entities will collide when reaching an intersection.

[0961] A "warning message" is a warning notification sent to a terminal when a collision risk is detected.

[0962] An "audio signal" is an audio notification that the terminal issues to alert the user.

[0963] A "visual signal" is a notification using light or display that a device emits to alert the user.

[0964] An "emotion analysis engine" is a system component that analyzes the user's emotional state in real time and utilizes the analysis results.

[0965] "Dynamic adjustment" refers to flexibly changing the content and format of the warning message depending on the user's emotional state.

[0966] MODE FOR CARRYING OUT THE INVENTION

[0967] The present invention relates to a system installed in an autonomous vehicle, which includes a terminal, a server, and an emotion analysis engine to reduce the risk of collision and ensure safety.

[0968] Hardware and Software Configuration

[0969] Terminal

[0970] It will be installed in self-driving vehicles.

[0971] Location and speed information is obtained using the built-in GPS sensor and speed sensor.

[0972] It is equipped with an emotion analysis engine and uses the camera and microphone to analyze the user's emotional state.

[0973] server

[0974] Receives and analyzes location and speed information sent from the device.

[0975] The latest information is stored based on timestamps and collision risk is predicted.

[0976] If a collision risk is detected, a warning message is sent to the device.

[0977] Sentiment Analysis Engine

[0978] Analyze the user's emotional state in real time.

[0979] Image analysis libraries such as TensorFlow and OpenCV are used to process data from the camera and microphone.

[0980] Data transmission and analysis

[0981] Terminal

[0982] The device periodically sends its location and speed information to the server. The data sent consists of the entity ID, location (latitude, longitude), speed, and timestamp. The format sent to the server is as follows:

[0983] {

[0984] "id": "vehicle_1",

[0985] "position": {"lat": X1, "lon": Y1},

[0986] "velocity": V1,

[0987] "timestamp": T1

[0988] }

[0989] server

[0990] The server calculates the predicted trajectory of each entity based on the received data, calculates the entity's position when it arrives at the intersection, and evaluates the collision risk with other entities. If a collision risk is detected, it sends a warning message to the relevant device.

[0991] Sentiment analysis and dynamic alert adjustment

[0992] Emotion analysis

[0993] The device's built-in emotion analysis engine analyzes data collected from the camera and microphone to recognize the user's emotional state. A machine learning model using TensorFlow analyzes facial expressions and voice data to detect negative emotional states such as tension or anxiety.

[0994] Adjustment of warning content

[0995] An emotion analysis engine recognizes the user's emotional state and dynamically adjusts the content and format of the alert message as needed - for example, if tension or anxiety is detected, the alert may sound louder or flash a visual signal more frequently.

[0996] Specific examples

[0997] For example, if autonomous vehicle "A" is located at latitude 35.6895 and longitude 139.6917 and is moving at a speed of 10 m / s, the following prompt sentence can be used to predict its location in the next 5 seconds and avoid the risk of collision:

[0998] If the current location of the vehicle is at latitude 35.6895, longitude 139.6917, and speed 10 m / s, predict the location of the vehicle in the next 5 seconds and generate a warning message to avoid a possible collision.

[0999] In this way, the present invention can reduce the risk of collisions between entities at intersections in autonomous vehicles and ensure user safety. The introduction of a sentiment analysis engine allows the content and format of warnings to be appropriately adjusted, enabling more effective warnings.

[1000] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1001] Step 1:

[1002] The device acquires location and speed information. The device uses the built-in GPS sensor and speed sensor to measure the current location (latitude, longitude) and speed of the autonomous vehicle. The input is real-time data from the GPS sensor and speed sensor, and the output is location and speed information.

[1003] Step 2:

[1004] The location and speed information acquired by the device is sent to the server. The acquired location information (latitude, longitude) and speed information are sent to the server along with a timestamp. The input is the location and speed information acquired in step 1, and the output is the data sent to the server.

[1005] Step 3:

[1006] The server uses the received data to calculate the predicted trajectory. The server receives the vehicle's current position and speed information as input, and uses this to calculate the predicted position for the next few seconds. The output is the calculated predicted position. The server predicts the position as follows:

[1007] Next position = Current position + Speed ​​Time interval

[1008] Step 4:

[1009] The server detects collision risk. The server analyzes the predicted trajectories of all entities and evaluates the collision risk at intersections. The input is the predicted trajectory of each entity, and the output is the collision risk judgment result. Specifically, the server lists overlapping predicted positions and checks whether they are at the same time.

[1010] Step 5:

[1011] The server generates a warning message and sends it to the device. If a collision risk is detected, the server generates a warning message and sends it to the device. The input is the collision risk judgment result obtained in step 4, and the output is a warning message to the device. The warning message contains the following content:

[1012] {

[1013] "id": "entity_1",

[1014] "warning": "collision_risk",

[1015] "message": "Attention! Possible collision ahead."

[1016] }

[1017] Step 6:

[1018] The device uses an emotion analysis engine to analyze the user's emotional state. The emotion analysis engine uses data acquired from the device's built-in camera and microphone as input and analyzes the user's emotional state. The output is the user's emotional state. Specifically, the data from the camera and microphone is analyzed using TensorFlow and OpenCV to evaluate the user's facial expressions and tone of voice.

[1019] Step 7:

[1020] The device dynamically adjusts the warning message based on the emotional state. Based on the results of the emotion analysis engine, the strength of the warning sound and visual signal is adjusted. The input is the emotional state obtained in step 6, and the output is the adjusted warning message. Specific actions include increasing the volume of the warning sound or the frequency of the flashing light to emphasize the notification of the audio or visual signal.

[1021] Step 8:

[1022] The user receives a warning message from the device and takes appropriate action. The user acknowledges the warning via audio or visual signals from the device and takes safe action. The input is the adjusted warning message and the output is the user's action.

[1023] This is the flow of processing in the program for the system that realizes this application example. Through this series of processing, it is possible to reduce the risk of collisions at intersections for autonomous vehicles and ensure the safety of users.

[1024] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1025] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1026] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1027] [Third embodiment]

[1028] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1029] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1031] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1032] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1033] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1035] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1036] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1038] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1039] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1040] The present invention provides a system for monitoring the movements of vehicles, pedestrians, and bicycles at intersections without traffic lights in real time to reduce the risk of collisions. The system includes a means for a terminal to acquire position information and speed information and transmit it to a server, a means for the server to calculate a predicted trajectory based on the received data and detect a collision risk, a means for the server to send a warning message to the terminal when a collision risk is detected, and a means for the terminal to issue an audio or visual signal to a user when it receives the warning message.

[1041] System operation example

[1042] The device acquires location and speed information using the built-in GPS sensor and speed sensor. For example, when pedestrian A is walking towards an intersection, pedestrian A's device acquires location information (latitude X1, longitude Y1) and speed information (speed V1), which are periodically sent to the server.

[1043] Sending data:

[1044] The device sends location and speed information to the server, which receives it. The data sent is in the following format:

[1045] {

[1046] "id": "entity_1",

[1047] "position": {"lat": X1, "lon": Y1},

[1048] "velocity": V1,

[1049] "timestamp": T1

[1050] }

[1051] The server stores the received data and uses only the most recent information for analysis based on the timestamp.

[1052] Collision risk analysis:

[1053] The server calculates the predicted trajectory of each entity. For example, if Pedestrian A and Bicycle B are heading towards the same intersection, the server calculates their predicted positions in the next 2 seconds and evaluates the likelihood that the entities will arrive at the same location.

[1054] Calculating the predicted trajectory:

[1055] The server calculates the predicted location based on the current location and velocity information as follows:

[1056] next_position = current_position + velocity time_interval

[1057] Based on this predicted position, the entity's future positional relationship is analyzed.

[1058] Sending a warning message:

[1059] If the server detects a collision risk, it generates and sends a warning message to the device of the relevant entity. For example, if there is a high possibility that pedestrian A and cyclist B will collide, the server will generate the following warning message:

[1060] {

[1061] "id": "entity_1",

[1062] "warning": "collision_risk",

[1063] "message": "Attention! Possible collision ahead."

[1064] }

[1065] This is sent to the device.

[1066] Receive alerts and notifications:

[1067] When the device receives a warning message from the server, it notifies the user of the warning. Notification methods include voice messages and flashing lights:

[1068] Display: "Attention! Possible collision ahead."

[1069] Sound: Play warning sound

[1070] Light: Start blinking

[1071] The user can then review this notification and take appropriate action.

[1072] Specific example processing:

[1073] For example, consider a situation where pedestrian A is walking towards an intersection and cyclist B is simultaneously heading towards the intersection. The system operates in the following steps:

[1074] 1. The device acquires location and speed information.

[1075] Pedestrian A's device acquires location information (X1, Y1) and speed (V1), and cyclist B's device also acquires location information (X2, Y2) and speed (V2).

[1076] 2. The device sends the data to the server.

[1077] Both terminals send data to the server, which receives it.

[1078] 3. The server calculates the predicted trajectory.

[1079] The server calculates the predicted trajectories of both vehicles and assesses the risk of collision at the intersection.

[1080] 4. The server generates and sends a warning message.

[1081] If a collision risk is detected, the server sends a warning message to the devices of pedestrian A and cyclist B.

[1082] 5. The device receives the warning message and notifies you.

[1083] The device displays a notification to the user and alerts them with an audio or visual signal.

[1084] In this way, the present invention reduces the risk of collisions between entities within an intersection, ensuring safety.

[1085] The processing flow will be explained below.

[1086] Step 1: The device acquires location and speed information

[1087] The device uses the built-in GPS sensor to obtain the current location (latitude, longitude) and the speed sensor to obtain the moving speed. For example, the device of pedestrian A obtains the location information (X1, Y1) and speed (V1).

[1088] Step 2: The device sends the data to the server

[1089] The device periodically sends the acquired location and speed information to the server. The data sent includes the device ID, location information, speed information, and timestamp. For example,

[1090] {

[1091] "id": "entity_1",

[1092] "position": {"lat": X1, "lon": Y1},

[1093] "velocity": V1,

[1094] "timestamp": T1

[1095] }

[1096] It is sent in the format:

[1097] Step 3: The server receives the data

[1098] The server receives location and speed information from the device via API and stores it in memory or in a database. The server overwrites older data based on the timestamp to keep only the most recent data.

[1099] Step 4: The server calculates the predicted trajectory

[1100] The server calculates the predicted location of each entity at regular intervals based on its current location and velocity. For example, the predicted location for the next 2 seconds is calculated using the following formula:

[1101] next_position = current_position + velocity time_interval

[1102] Based on this predicted position, a future trajectory for each entity is generated.

[1103] Step 5: Server detects collision risk

[1104] The server analyzes the trajectories of all generated entities and evaluates whether they will intersect at an intersection. For example, if pedestrian A and cyclist B are likely to arrive at the same location, it identifies the time and location and determines the collision risk.

[1105] Step 6: The server generates and sends a warning message

[1106] If the server detects a collision risk, it generates and sends a warning message to the device of the relevant entity. For example, it generates a warning message like this:

[1107] {

[1108] "id": "entity_1",

[1109] "warning": "collision_risk",

[1110] "message": "Attention! Possible collision ahead."

[1111] }

[1112] Send this message to the terminal.

[1113] Step 7: The device receives the warning message and notifies the user

[1114] When the terminal receives the warning message sent from the server, it issues a warning to the user with an audio or visual signal, such as displaying the warning message on the terminal screen, sounding an alarm, or flashing a light to alert the user.

[1115] Step 8: The user acts on the device warning

[1116] The user checks the warning from the device and takes safe action. For example, pedestrian A slows down as he approaches the intersection, or the driver of bicycle B slows down to delay entering the intersection.

[1117] Example 1

[1118] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1119] At intersections without traffic lights, the risk of collisions increases because many moving objects, including vehicles, pedestrians, and bicycles, cross the intersection. In such situations, a system is needed that can monitor the position and speed of each entity in real time, detect danger in advance, and issue a warning. However, conventional systems have difficulty accurately calculating the predicted trajectories of multiple moving objects and issuing a warning quickly.

[1120] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1121] In this invention, the server includes a means for the terminal to acquire position information and speed information and transmit it to the server, a means for the server to calculate a predicted trajectory of each entity using the data received and detect a collision risk, and a means for transmitting a warning message to the corresponding terminal when the server detects a collision risk. This makes it possible to simultaneously calculate the predicted trajectories of multiple entities and evaluate the collision risk at an intersection.

[1122] A "terminal" is a device for obtaining location and velocity information of an entity and transmitting it to a server.

[1123] "Location information" is latitude and longitude data that indicates the current location of an entity.

[1124] "Speed ​​information" is data that indicates the moving speed of an entity.

[1125] A "server" is a device or system that receives data transmitted from a terminal and calculates a predicted trajectory and evaluates a collision risk.

[1126] A "predicted trajectory" is a future movement path calculated based on the entity's current position and speed information.

[1127] "Collision risk" is an assessment that indicates the likelihood that multiple entities will arrive at the same location in the future, resulting in a collision accident.

[1128] A "warning message" is a warning communication that is sent to the relevant terminal when the server detects a collision risk.

[1129] The "audio signal" is an audio notification that alerts the user when a warning message is received by the terminal.

[1130] A "visual signal" is a light or screen display notification that alerts the user when a warning message is received by the terminal.

[1131] A "timestamp" is information that records the time when data is acquired or transmitted.

[1132] A "location sensor" is a device that can determine the current location of an entity and obtain that location information.

[1133] The present invention provides a system for monitoring the movements of vehicles, pedestrians, and bicycles at intersections without traffic lights in real time to reduce the risk of collision accidents. The system includes a means for a terminal to acquire position information and speed information and transmit it to a server, a means for the server to calculate a predicted trajectory based on the received data and detect a collision risk, a means for the server to transmit a warning message to the terminal when a collision risk is detected, and a means for the terminal to issue an audio or visual signal to a user when it receives the warning message.

[1134] The device acquires current location information (latitude and longitude) and speed information using the built-in location sensor and speed sensor. For example, if a pedestrian is walking towards an intersection, the device acquires location information (X1, Y1) and speed information (V1) and periodically sends this to the server. The data sent by the device includes the ID, location information (latitude and longitude), speed information, and the timestamp when the data was acquired.

[1135] The server receives data sent from the device and stores it in a database. Data sent from each entity is recorded with a timestamp, and the server uses only the most recent data for analysis. Based on the received data, the server calculates the predicted trajectory of each entity. The current position and speed information are used to calculate the predicted trajectory, using the formula "next position = current position + speed x time interval."

[1136] The server simultaneously calculates the predicted positions of multiple entities and analyzes their future positional relationships. For example, if pedestrian A and cyclist B are heading towards the same intersection, the server calculates their predicted positions and evaluates the probability that they will arrive at the intersection at the same time. Based on this evaluation, the server determines the collision risk.

[1137] If a collision risk is detected, the server generates and sends a warning message to the relevant entity. For example, if there is a high risk of a collision between pedestrian A and cyclist B, the server generates a warning message and sends it to both devices. The warning message includes a warning message and details of the warning.

[1138] When the device receives a warning message, it notifies the user with an audio or visual signal, such as by displaying the message "Attention! Possible collision ahead" on the display, emitting a warning sound, or flashing a light to alert the user.

[1139] For example, if pedestrian A is walking towards an intersection and cyclist B is heading towards the same intersection, the system will operate as follows:

[1140] 1. Pedestrian A's device acquires location information (X1, Y1) and speed (V1), and cyclist B's device also acquires location information (X2, Y2) and speed (V2).

[1141] 2. Both devices send data to the server.

[1142] 3. The server stores the received data and calculates the predicted trajectories of both vehicles.

[1143] 4. The server evaluates the collision risk based on the predicted trajectory.

[1144] 5. If a collision risk is detected, the server generates a warning message and sends it to the devices of pedestrian A and cyclist B.

[1145] 6. The terminal receives the warning message and notifies the user of the warning.

[1146] Such a system can reduce the risk of collisions between entities and ensure safety even at intersections without traffic lights. An example of a prompt sentence could be, "Watch out for the next intersection. Stop and check your surroundings."

[1147] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1148] System program processing flow

[1149] Step 1: The device obtains location and speed information

[1150] The device uses the built-in location and speed sensors to obtain current location information (latitude and longitude) and speed information. The input is the entity's sensor information, and the output is location information (X, Y) and speed information (V). This specific operation involves obtaining GPS data and calculating speed using the acceleration sensor.

[1151] Step 2: The device sends data to the server

[1152] The device sends the acquired location and speed information to the server. The input is the location and speed information acquired in step 1, and the output is the transmission of a data packet to the server. The transmitted data has the following format:

[1153] {

[1154] "id": "entity_1",

[1155] "position": {"lat": X, "lon": Y},

[1156] "velocity": V,

[1157] "timestamp": T

[1158] }

[1159] Specifically, the data packet is sent to the server using an HTTP request or the like.

[1160] Step 3: The server receives and stores the data

[1161] The server receives the data sent from the device and stores it in a database. The input is the data packet sent in step 2, and the output is the stored location and speed information. The specific operation is a write operation to the database.

[1162] Step 4: The server calculates the predicted trajectory

[1163] The server calculates the predicted trajectory of each entity based on the received position and velocity information. The input is the saved current position and velocity information, and the output is the predicted position (next_position) after the next time interval. The predicted position is calculated using the following formula:

[1164] next_position = current_position + velocity time_interval

[1165] The specific operation is to use this formula to predict the future position of each entity.

[1166] Step 5: Server evaluates collision risk

[1167] The server evaluates the collision risk of multiple entities based on the predicted trajectories. The input is the predicted position calculated in step 4, and the output is the collision risk assessment result. The specific operation is to compare the predicted positions and calculate the probability that the entities will arrive at the same point.

[1168] Step 6: The server generates a warning message

[1169] If a collision risk is detected, the server generates a warning message to be sent to the relevant entity. The input is the collision risk assessment result from step 5, and the output is a warning message. The specific format of the warning message is as follows:

[1170] {

[1171] "id": "entity_1",

[1172] "warning": "collision_risk",

[1173] "message": "Attention! Possible collision ahead."

[1174] }

[1175] Specifically, the system dynamically generates warning content based on the evaluation results.

[1176] Step 7: The server sends a warning message

[1177] The server sends the generated warning message to the corresponding terminal. The input is the warning message generated in step 6, and the output is the transmission of warning data to the terminal. Specifically, this message is sent to the corresponding terminal using an HTTP request, etc.

[1178] Step 8: Your device receives a warning message

[1179] The terminal receives the warning message sent from the server. The input is the warning message from the server, and the output is the internal processing of this message. The specific operation is to store the received message in memory and add it to the processing queue.

[1180] Step 9: The device notifies the user of the warning

[1181] When the device receives the warning message, it notifies the user of the warning with an audio or visual signal. The input is the warning message received in step 8, and the output is the notification to the user. Specifically, the device displays "Attention! Possible collision ahead." and plays a warning sound or flashes a light.

[1182] In this way, the risk of collisions between entities can be reduced and safety ensured even at intersections without traffic lights.

[1183] (Application example 1)

[1184] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1185] At unsignalized intersections and in congested urban environments, the risk of collisions between autonomous vehicles and entities such as other vehicles, pedestrians, and cyclists is increasing. In such situations, real-time dynamic prediction and immediate warnings are necessary for autonomous vehicles to operate safely and prevent accidents. However, current technology is unable to meet these needs, and there is a lack of systems that can adequately reduce the risk of collisions between entities. Accidents resulting from this pose a major problem that threatens the safety of passengers and those around them.

[1186] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1187] In this invention, the server includes: means for the terminal to acquire position information and speed information and transmit it to the server; means for the server to calculate a predicted trajectory of each entity using the data received and detect a collision risk; means for transmitting a warning message to the corresponding terminal when the server detects a collision risk; means for the terminal to issue an audio or visual signal to alert the user when it receives the warning message; and means for the terminal to be mounted in an autonomous vehicle and notify the driver of the warning message via an in-vehicle display and audio guidance system. This enables autonomous vehicles to detect collision risks with other entities in real time, even at intersections or in crowded environments, and ensure safe operation.

[1188] A "terminal" is a device that acquires location information and speed information and transmits it to a server.

[1189] "Predicted trajectory" is a calculation method that predicts the future position of each entity based on its current position and speed information.

[1190] "Collision risk" is an assessment that indicates the likelihood of contact or collision between entities using the predicted trajectory.

[1191] A "warning message" is a notification message that is sent to the relevant terminal when a collision risk is detected.

[1192] An "audio or visual signal" is a signal, such as a voice message or a flashing light, that is emitted to notify the user of a warning.

[1193] An "autonomous vehicle" is a vehicle that is equipped with GPS and speed sensors and operates autonomously.

[1194] An "in-vehicle display" is a screen installed inside an autonomous vehicle to visually display information.

[1195] A "voice guidance system" is a system installed inside an autonomous vehicle that provides information through voice messages.

[1196] The present invention is a system for reducing the risk of collisions between autonomous vehicles and entities such as other vehicles, pedestrians, and cyclists at unsignalized intersections and in congested urban environments. The system includes a terminal, a server, an in-vehicle display, and a voice guidance system.

[1197] System hardware configuration

[1198] Device: Installed in an autonomous vehicle, it continuously acquires location and speed information using built-in positioning sensors (e.g., GPS sensors) and speed sensors.

[1199] Server: Located on the cloud, it receives data sent from the device, calculates the predicted trajectory, and evaluates the collision risk.

[1200] In-vehicle displays: installed inside autonomous vehicles, they visually display warning messages and other necessary information.

[1201] Voice guidance system: A system that works in conjunction with the in-car display to provide warnings to the driver through voice messages.

[1202] Software configuration and processing flow

[1203] 1. Data Acquisition

[1204] The device uses built-in positioning and speed measurement sensors to obtain real-time position and speed information of the autonomous vehicle.

[1205] 2. Data Transmission

[1206] The terminal periodically transmits the acquired location and speed information to the server. The transmitted data includes location information (latitude and longitude), speed information, and a timestamp.

[1207] 3. Collision risk assessment

[1208] The server uses the received data to calculate the predicted trajectory of the autonomous vehicle and other entities, including calculating future positions based on current position and speed information.

[1209] 4. Generating Warning Messages

[1210] The server evaluates the collision risk based on the predicted trajectory, and if a collision risk is detected, it generates and sends a warning message to the relevant terminal.

[1211] 5. Warning Notification

[1212] When the terminal receives a warning message from the server, it issues a visual and audio warning to the driver through the in-car display and voice guidance system.

[1213] Prompt Sentence Examples

[1214] Below are some example prompts to input to a generative AI model:

[1215] "Design a system that acquires real-time location and speed information of an autonomous vehicle, sends it to a server, and evaluates the risk of collision. The server calculates a predicted trajectory based on data from other entities, and if there is a risk of collision, warns the driver on the vehicle's display and voice guidance system. Please provide Python code that runs in real time."

[1216] As a specific example, consider the following scenario: As autonomous vehicle A approaches an intersection, it detects the presence of pedestrian B walking toward the same intersection. The server calculates the predicted trajectories of both vehicles, and if it detects a high risk of collision, it sends a warning message to the device of autonomous vehicle A. The in-vehicle display will show a message saying, "Possible collision with pedestrian at intersection ahead. Please be careful," and the voice guidance system will give instructions such as, "Pay attention to what is ahead." This notification will alert the driver to be careful at the intersection, ensuring safe driving.

[1217] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1218] Step 1:

[1219] The device obtains location and speed information. The device uses a location sensor (e.g., GPS sensor) and a speed sensor to obtain the current latitude, longitude, and speed of the autonomous vehicle. In this case, the input is the raw data provided by the vehicle's sensors, and the output is the latitude, longitude, and speed values.

[1220] Step 2:

[1221] The device sends the acquired location and speed information to the server. The device sends data to the server, including the acquired location information (latitude, longitude), speed information, and timestamp. This data is encoded in JSON or other appropriate data format. The location and speed information obtained in step 1 is used as input, and the data sent to the server is generated as output.

[1222] Step 3:

[1223] The server stores the received data and uses it for analysis. The server stores the location and speed information received from the device in a database and uses only the most recent data for analysis based on the timestamp. This eliminates duplicate and old data. The input is the data sent from the device, and the output is the latest data set to use for analysis and calculations.

[1224] Step 4:

[1225] The server calculates the predicted trajectory of each entity. Based on the received position and velocity information, the server predicts the future position of each entity. This calculation is done by multiplying the current position by the velocity, for example. The input is the latest position and velocity information, and the output is the predicted position at a specific point in the future.

[1226] Step 5:

[1227] The server evaluates the collision risk. The server compares the predicted trajectories of each entity and determines whether there is a possibility of an intersection. If the intersection points are very close, it determines that there is a collision risk. The input is the predicted trajectory data, and the output is the collision risk assessment result (risk / no risk).

[1228] Step 6:

[1229] If the server detects a collision risk, it sends a warning message to the relevant terminal. The server generates and sends a warning message to the terminal of the entity where the collision risk was detected. This message indicates that a collision risk exists and calls for attention. The input is the collision risk evaluation result, and the output is the warning message to be sent to each terminal.

[1230] Step 7:

[1231] The terminal receives the warning message and alerts the user. The terminal notifies the driver of the received warning message via the in-car display or voice guidance system. For example, the display may show a message saying "There is a pedestrian ahead. Please be careful," and the voice guidance system may play a voice message saying "Pay attention to what is ahead." The input is the warning message received from the server, and the output is the warning notification to the user.

[1232] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1233] The present invention combines an emotion engine with a system for monitoring the movements of entities such as vehicles, pedestrians, and bicycles at intersections without traffic lights in real time and reducing the risk of collision accidents. This system includes means for a terminal to acquire position information and speed information and send it to a server, means for the server to calculate a predicted trajectory based on the received data and detect a collision risk, means for the server to send a warning message to the relevant terminal when a collision risk is detected, means for the terminal to issue an audio or visual signal to alert the user when it receives the warning message, and an emotion engine that recognizes the user's emotions.

[1234] System Overview

[1235] The device acquires location and speed information using the built-in GPS sensor and speed sensor. For example, when pedestrian A is walking towards an intersection, pedestrian A's device acquires location information (latitude X1, longitude Y1) and speed information (speed V1), which are periodically sent to the server.

[1236] Detailed System Operation

[1237] Sending data:

[1238] The device periodically sends the location and speed information it has acquired to the server. The data sent is in the following format:

[1239] {

[1240] "id": "entity_1",

[1241] "position": {"lat": X1, "lon": Y1},

[1242] "velocity": V1,

[1243] "timestamp": T1

[1244] }

[1245] The server stores the received data and uses only the most recent information for analysis based on the timestamp.

[1246] Collision risk analysis:

[1247] The server calculates the predicted location of each entity at regular intervals based on its current location and velocity. For example, the predicted location for the next 2 seconds is calculated using the following formula:

[1248] next_position = current_position + velocity time_interval

[1249] Based on this predicted position, a future trajectory for each entity is generated.

[1250] Calculating the predicted trajectory:

[1251] The server analyzes the trajectories of all entities and evaluates whether they intersect at an intersection. For example, if pedestrian A and cyclist B are likely to arrive at the same location, it identifies the time and location and determines the collision risk.

[1252] Generate and send a warning message:

[1253] If the server detects a collision risk, it generates and sends a warning message to the device of the relevant entity. For example, it generates a warning message like this:

[1254] {

[1255] "id": "entity_1",

[1256] "warning": "collision_risk",

[1257] "message": "Attention! Possible collision ahead."

[1258] }

[1259] This is sent to the device.

[1260] Using the Emotion Engine:

[1261] The emotion engine analyzes the user's emotions in real time and recognizes their emotional state. For example, it uses the device's camera and microphone to analyze emotions from changes in the user's facial expressions and voice.

[1262] Dynamically adjusting warnings based on emotions:

[1263] If the emotion engine determines that the user is in a negative emotional state, such as tension or anxiety, the server will issue a more emphatic warning message, for example by making the warning sound louder or flashing visual signals more frequently.

[1264] Receiving and notifying warning messages:

[1265] When the device receives a warning message from the server, it notifies the user of the warning. Notification methods include voice messages and flashing lights:

[1266] Display: "Attention! Possible collision ahead."

[1267] Sound: Play warning sound

[1268] Light: Start blinking

[1269] The user can then review this notification and take appropriate action.

[1270] Processing of specific examples

[1271] For example, consider a situation where pedestrian A is walking towards an intersection and cyclist B is simultaneously heading towards the intersection. The system operates in the following steps:

[1272] 1. The device acquires location and speed information.

[1273] Pedestrian A's device acquires location information (X1, Y1) and speed (V1), and cyclist B's device also acquires location information (X2, Y2) and speed (V2).

[1274] 2. The device sends the data to the server.

[1275] Both terminals send data to the server, which receives it.

[1276] 3. The server calculates the predicted trajectory.

[1277] The server calculates the predicted trajectories of both vehicles and assesses the risk of collision at the intersection.

[1278] 4. The server generates and sends a warning message.

[1279] If a collision risk is detected, the server sends a warning message to the devices of pedestrian A and cyclist B.

[1280] 5. The emotion engine analyzes the user's emotions.

[1281] The emotion engine analyzes the emotional state of pedestrian A and cyclist B and dynamically adjusts the content and format of the warning as needed.

[1282] 6. The device receives the warning message and notifies you.

[1283] The device displays a notification to the user and alerts them with an audio or visual signal.

[1284] 7. The user acts in accordance with the warning on the device.

[1285] The user acknowledges the warning and takes safe action.

[1286] In this way, the present invention reduces the risk of collisions between entities at intersections and ensures safety. By combining emotion engines, the effectiveness of warnings is maximized and users' safety awareness is enhanced.

[1287] The processing flow will be explained below.

[1288] Step 1: The device acquires location and speed information

[1289] The device uses the built-in GPS sensor to obtain the current location (latitude, longitude) and the speed sensor to obtain the moving speed. For example, the device of pedestrian A obtains the location information (X1, Y1) and speed (V1).

[1290] Step 2: The device sends the data to the server

[1291] The device periodically sends the acquired location and speed information to the server. The data sent includes the device ID, location information, speed information, and timestamp. For example,

[1292] {

[1293] "id": "entity_1",

[1294] "position": {"lat": X1, "lon": Y1},

[1295] "velocity": V1,

[1296] "timestamp": T1

[1297] }

[1298] It is sent in the format:

[1299] Step 3: The server receives the data

[1300] The server receives location and speed information from the device via API and stores it in memory or in a database. The server overwrites older data based on the timestamp to keep only the most recent data.

[1301] Step 4: The server calculates the predicted trajectory

[1302] The server calculates the predicted location of each entity at regular intervals based on its current location and velocity. For example, the predicted location for the next 2 seconds is calculated using the following formula:

[1303] next_position = current_position + velocity time_interval

[1304] Based on this predicted position, a future trajectory for each entity is generated.

[1305] Step 5: Server detects collision risk

[1306] The server analyzes the trajectories of all generated entities and evaluates whether they will intersect at an intersection. For example, if pedestrian A and cyclist B are likely to arrive at the same location, it identifies the time and location and determines the collision risk.

[1307] Step 6: The emotion engine analyzes the user's emotions

[1308] Using the device's built-in camera and microphone, the emotion engine analyzes the user's emotional state from their facial expressions and voice. For example, if the user is nervous, the engine will analyze their emotional state in real time.

[1309] Step 7: The server generates and sends a warning message

[1310] When the server detects a collision risk, it dynamically adjusts the warning message based on the analysis results of the emotion engine. For example, if the user is nervous, it generates a more emphasized warning message. Examples of generated warning messages are:

[1311] {

[1312] "id": "entity_1",

[1313] "warning": "collision_risk",

[1314] "message": "Attention! Possible collision ahead. Please be careful!"

[1315] }

[1316] This is sent to the device.

[1317] Step 8: The device receives the warning message and notifies the user.

[1318] When the device receives the warning message sent from the server, it will warn the user with audio or visual signals, for example, by displaying a warning message on the device screen, sounding an alarm, or flashing a light to alert the user:

[1319] Display: "Attention! Possible collision ahead. Please be careful!"

[1320] Sound: Play warning sound

[1321] Light: Start blinking

[1322] Step 9: User acts on the device warning

[1323] The user checks the warning from the device and takes safe action. For example, pedestrian A slows down as he approaches the intersection, or the driver of bicycle B slows down to delay entering the intersection.

[1324] This series of processes reduces the risk of collisions between entities at intersections without traffic lights, ensuring safety. In addition, the use of an emotion engine maximizes the effectiveness of warnings and increases user safety awareness.

[1325] Example 2

[1326] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1327] Traffic accidents frequently occur at intersections without traffic lights, and the risk increases significantly when different types of entities, such as pedestrians, vehicles, and bicycles, enter the intersection at the same time. Conventional systems have difficulty monitoring the movements of these entities in real time and providing users with appropriate warnings to reduce collision risks. Furthermore, they lack the means to appropriately adjust the content and format of warnings based on the user's emotional state. Therefore, a system that can more efficiently and effectively reduce collision risks in real-world usage environments is needed.

[1328] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1329] In this invention, the server includes a means for the terminal to acquire position information and speed information and transmit it to the server, a means for the server to calculate a predicted trajectory of each entity using the received data and detect a collision risk, a means for the server to transmit a warning message to the corresponding terminal when the server detects a collision risk, and a means for the terminal to analyze the emotional state of the user and dynamically adjust the content and format of the warning. This makes it possible to monitor the movements of entities at intersections without signals in real time, detect collision risks in a timely manner, and further improve the effectiveness of the warning according to the emotional state of the user, thereby promoting safe behavior.

[1330] A "terminal" is an electronic device that acquires location information and speed information and transmits it to a server.

[1331] The "server" is a device or system that receives data transmitted from the terminal, calculates the predicted trajectory of the entity based on the data, and detects the risk of collision.

[1332] "Location information" is the entity's current latitude and longitude data.

[1333] "Speed ​​information" is data about the speed at which an entity is moving.

[1334] An "entity" is a moving object such as a pedestrian, vehicle, or bicycle that moves around within an intersection.

[1335] A "predicted trajectory" is the path or position of an entity after a certain period of time, calculated by the server based on the entity's current location and speed information.

[1336] "Collision risk" refers to a situation where the predicted trajectories of different entities intersect, potentially resulting in a physical collision.

[1337] A "warning message" is a notification that is generated by the server and sent to the relevant terminal when a collision risk is detected.

[1338] An "audio or visual signal" is a sound or a visual alarm such as an indicator light that is emitted to alert the user when the terminal receives a warning message.

[1339] "Emotional state" refers to the user's psychological state, such as tension, anxiety, excitement, etc., and is analyzed using sensors such as a camera and microphone.

[1340] "Dynamic adjustment" refers to changing the content and notification method of the warning message in real time based on the user's emotional state.

[1341] This invention is a system for monitoring the movements of entities such as vehicles, pedestrians, and bicycles in real time at intersections without traffic lights to reduce the risk of collision accidents. The system includes means for acquiring position information and speed information of entities and transmitting it to a server, means for calculating a predicted trajectory based on the data received by the server and detecting a collision risk, means for transmitting a warning message to an appropriate terminal when a collision risk is detected, means for issuing an audio or visual signal to alert the user, and means for recognizing the user's emotions and dynamically adjusting the content and format of the warning.

[1342] Hardware and software used

[1343] Hardware:

[1344] Device: Smartphones and wearable devices are used. These devices have built-in GPS and speed sensors. Typical examples of smartphones include the Qualcomm Snapdragon GPS and Ublox M8.

[1345] Server: The server is built on a cloud service, for example, a web server using Flask on an AWS EC2 instance.

[1346] software:

[1347] GPS and speed data analysis: Location information is analyzed using the Python "Geopy" library, and speed information is obtained directly from the device's built-in sensors.

[1348] Data processing: The received data is processed and saved using the Python "Pandas" library, and the necessary analysis is performed.

[1349] Machine learning model: Sentiment analysis is performed using "Scikit-learn".

[1350] Example of a system

[1351] First, the device acquires location information (latitude X1, longitude Y1) and speed information (speed V1) using the built-in GPS sensor and speed sensor. For example, when pedestrian A is walking towards an intersection, pedestrian A's device acquires location information and speed information and sends it to the server in JSON format.

[1352] json

[1353] Topic: GPS data formatting

[1354] {

[1355] "id": "walk_A",

[1356] "position": {"lat": 35.6895, "lon": 139.6917},

[1357] "velocity": 1.2,

[1358] "timestamp": "2023-10-10T12:00:00Z"

[1359] }

[1360] The server then calculates the entity's predicted trajectory based on the received data to detect collision risk. This calculation uses the current position and speed to derive a predicted position for a certain time period. For example, the predicted position for the next 2 seconds is calculated using the following formula:

[1361] python

[1362] next_position = current_position + velocity time_interval

[1363] The server analyzes the trajectories of all entities and evaluates whether they intersect at an intersection. For example, if pedestrian A and cyclist B are likely to arrive at the same location, it identifies the time and location and determines the collision risk.

[1364] If a collision risk is detected, the server generates and sends a warning message to the device of the relevant entity. For example, the following warning message is sent to pedestrian A:

[1365] json

[1366] Note: Example of a warning message

[1367] {

[1368] "id": "walk_A",

[1369] "warning": "collision_risk",

[1370] "message": "Attention! Possible collision ahead."

[1371] }

[1372] When the device receives a warning message, it notifies the user through audio, visual signals (flashing lights), or vibration. Examples of notifications include:

[1373] json

[1374] Alert methods: sound notification, flashing light, vibration

[1375] Display: "Attention! Possible collision ahead."

[1376] Sound: Play warning sound

[1377] Light: Start blinking

[1378] An emotion engine is also used, which uses the device's camera and microphone to analyze the user's emotions and, if the user is in a negative emotional state such as tension or anxiety, will issue a more emphatic warning message, for example by increasing the volume of the warning sound or speeding up the blinking of the light signal.

[1379] Prompt Sentence Examples

[1380] "A pedestrian and a cyclist are approaching an intersection. Write Python code to calculate the collision risk and generate and send a warning message if necessary."

[1381] This allows the system to monitor entity movements at unsignalized intersections in real time, and improve safety by detecting and notifying collision risks early. Dynamic warning adjustment based on the user's emotional state maximizes the effectiveness of warning messages and increases safety awareness.

[1382] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1383] Step 1:

[1384] The device acquires location and speed information.

[1385] Input: Built-in GPS and speed sensors

[1386] Specific behavior:

[1387] The device uses the built-in GPS sensor to obtain latitude (X) and longitude (Y), and the speed sensor to measure speed (V).

[1388] Output: Position information (latitude X, longitude Y) and speed information (V)

[1389] Step 2:

[1390] The terminal transmits location information and speed information to the server.

[1391] Input: Obtained location information (latitude X, longitude Y) and speed information (V)

[1392] Specific behavior:

[1393] The device converts the location and speed information into JSON format and sends it to the server via an HTTP request.

[1394] Output: JSON data sent to the server

[1395] json

[1396] {

[1397] "id": "entity_1",

[1398] "position": {"lat": X, "lon": Y},

[1399] "velocity": V,

[1400] "timestamp": T

[1401] }

[1402] Step 3:

[1403] The server stores the received data.

[1404] Input: JSON data sent from the terminal

[1405] Specific behavior:

[1406] The server receives the HTTP request, parses the received JSON data, and saves it in the database. At this time, it saves only the latest data based on the timestamp (T).

[1407] Output: Location, speed, and timestamp stored in a database

[1408] Step 4:

[1409] The server calculates the predicted trajectories of all entities.

[1410] Input: Position and velocity information of all entities stored in the database

[1411] Specific behavior:

[1412] The server uses the position (X, Y) and velocity (V) information of each entity to calculate the predicted position at the next time step using the following formula:

[1413] python

[1414] next_position = current_position + velocity time_interval

[1415] For example, calculate the predicted location for the next 2 seconds and generate future trajectories for all entities.

[1416] Output: predicted trajectory (future_position)

[1417] Step 5:

[1418] The server determines the collision risk and generates a warning message.

[1419] Input: predicted trajectory (future_position)

[1420] Specific behavior:

[1421] The server analyzes the predicted trajectories of all entities and evaluates the likelihood of them intersecting with each other. If there is a risk of collision, it generates a warning message for the relevant entity.

[1422] Output: Warning message

[1423] json

[1424] {

[1425] "id": "entity_1",

[1426] "warning": "collision_risk",

[1427] "message": "Attention! Possible collision ahead."

[1428] }

[1429] Step 6:

[1430] The server sends a warning message to the relevant terminal.

[1431] Input: The generated warning message

[1432] Specific behavior:

[1433] The server sends the generated warning message to the relevant terminal via an HTTP request.

[1434] Output: Warning message sent to terminal

[1435] Step 7:

[1436] The terminal receives the warning message and notifies the user.

[1437] Input: The warning message sent by the server

[1438] Specific behavior:

[1439] The device receives the warning message and notifies the user of the warning through a voice message, flashing light, or vibration.

[1440] Output: Warnings notified to the user

[1441] json

[1442] Display: "Attention! Possible collision ahead."

[1443] Sound: Play warning sound

[1444] Light: Start blinking

[1445] Step 8:

[1446] The emotion engine analyzes the user's emotions and dynamically adjusts the notification method.

[1447] Input: User facial and voice data

[1448] Specific behavior:

[1449] The device's built-in camera and microphone are used to analyze the user's facial expressions and voice changes in real time, which allows the system to determine the user's psychological state and dynamically adjust the volume of the warning sound and the flashing speed according to the user's emotional state.

[1450] Output: Adjusted notification method

[1451] Through the above steps, the system can monitor the movements of entities at unsignalized intersections in real time and ensure safety by providing appropriate collision warnings.

[1452] (Application example 2)

[1453] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1454] At intersections without traffic lights, there is a high risk of collisions between entities such as vehicles, pedestrians, and cyclists, making safety particularly important for autonomous vehicles. However, conventional systems lack the means to monitor entities' movements in real time and provide appropriate warnings. Furthermore, they do not dynamically adjust warnings according to the user's emotional state, resulting in insufficient improvements in safety.

[1455] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1456] In this invention, the server includes means for the terminal to acquire position information and speed information and transmit it to the server, means for the server to calculate a predicted trajectory of each entity using the received data and detect a collision risk, means for the server to transmit a warning message to the corresponding terminal when the server detects a collision risk, and means for the terminal to analyze the emotional state of the user using a built-in emotion analysis engine and dynamically adjust the content and format of the warning message based on the analysis result. This makes it possible to reduce the collision risk of entities in real time and provide effective warnings according to the emotional state of the user.

[1457] A "terminal" is a device that acquires location and speed information and is equipped with an emotion analysis engine.

[1458] A "server" is a central processing unit that receives, analyzes, and processes data sent from a terminal.

[1459] "Location information" is data that indicates the current geographical location of a terminal.

[1460] "Speed ​​information" is data indicating the speed at which the terminal is moving.

[1461] A "predicted trajectory" is a calculated prediction of the trajectory that an entity will move in the future.

[1462] "Collision risk" refers to the likelihood that two or more entities will collide when reaching an intersection.

[1463] A "warning message" is a warning notification sent to a terminal when a collision risk is detected.

[1464] An "audio signal" is an audio notification that the terminal issues to alert the user.

[1465] A "visual signal" is a notification using light or display that a device emits to alert the user.

[1466] An "emotion analysis engine" is a system component that analyzes the user's emotional state in real time and utilizes the analysis results.

[1467] "Dynamic adjustment" refers to flexibly changing the content and format of the warning message depending on the user's emotional state.

[1468] MODE FOR CARRYING OUT THE INVENTION

[1469] The present invention relates to a system installed in an autonomous vehicle, which includes a terminal, a server, and an emotion analysis engine to reduce the risk of collision and ensure safety.

[1470] Hardware and Software Configuration

[1471] Terminal

[1472] It will be installed in self-driving vehicles.

[1473] Location and speed information is obtained using the built-in GPS sensor and speed sensor.

[1474] It is equipped with an emotion analysis engine and uses the camera and microphone to analyze the user's emotional state.

[1475] server

[1476] Receives and analyzes location and speed information sent from the device.

[1477] The latest information is stored based on timestamps and collision risk is predicted.

[1478] If a collision risk is detected, a warning message is sent to the device.

[1479] Sentiment Analysis Engine

[1480] Analyze the user's emotional state in real time.

[1481] Image analysis libraries such as TensorFlow and OpenCV are used to process data from the camera and microphone.

[1482] Data transmission and analysis

[1483] Terminal

[1484] The device periodically sends its location and speed information to the server. The data sent consists of the entity ID, location (latitude, longitude), speed, and timestamp. The format sent to the server is as follows:

[1485] {

[1486] "id": "vehicle_1",

[1487] "position": {"lat": X1, "lon": Y1},

[1488] "velocity": V1,

[1489] "timestamp": T1

[1490] }

[1491] server

[1492] The server calculates the predicted trajectory of each entity based on the received data, calculates the entity's position when it arrives at the intersection, and evaluates the collision risk with other entities. If a collision risk is detected, it sends a warning message to the relevant device.

[1493] Sentiment analysis and dynamic alert adjustment

[1494] Emotion analysis

[1495] The device's built-in emotion analysis engine analyzes data collected from the camera and microphone to recognize the user's emotional state. A machine learning model using TensorFlow analyzes facial expressions and voice data to detect negative emotional states such as tension or anxiety.

[1496] Adjustment of warning content

[1497] An emotion analysis engine recognizes the user's emotional state and dynamically adjusts the content and format of the alert message as needed - for example, if tension or anxiety is detected, the alert may sound louder or flash a visual signal more frequently.

[1498] Specific examples

[1499] For example, if autonomous vehicle "A" is located at latitude 35.6895 and longitude 139.6917 and is moving at a speed of 10 m / s, the following prompt sentence can be used to predict its location in the next 5 seconds and avoid the risk of collision:

[1500] If the current location of the vehicle is at latitude 35.6895, longitude 139.6917, and speed 10 m / s, predict the location of the vehicle in the next 5 seconds and generate a warning message to avoid a possible collision.

[1501] In this way, the present invention can reduce the risk of collisions between entities at intersections in autonomous vehicles and ensure user safety. The introduction of a sentiment analysis engine allows the content and format of warnings to be appropriately adjusted, enabling more effective warnings.

[1502] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1503] Step 1:

[1504] The device acquires location and speed information. The device uses the built-in GPS sensor and speed sensor to measure the current location (latitude, longitude) and speed of the autonomous vehicle. The input is real-time data from the GPS sensor and speed sensor, and the output is location and speed information.

[1505] Step 2:

[1506] The location and speed information acquired by the device is sent to the server. The acquired location information (latitude, longitude) and speed information are sent to the server along with a timestamp. The input is the location and speed information acquired in step 1, and the output is the data sent to the server.

[1507] Step 3:

[1508] The server uses the received data to calculate the predicted trajectory. The server receives the vehicle's current position and speed information as input, and uses this to calculate the predicted position for the next few seconds. The output is the calculated predicted position. The server predicts the position as follows:

[1509] Next position = Current position + Speed ​​Time interval

[1510] Step 4:

[1511] The server detects collision risk. The server analyzes the predicted trajectories of all entities and evaluates the collision risk at intersections. The input is the predicted trajectory of each entity, and the output is the collision risk judgment result. Specifically, the server lists overlapping predicted positions and checks whether they are at the same time.

[1512] Step 5:

[1513] The server generates a warning message and sends it to the device. If a collision risk is detected, the server generates a warning message and sends it to the device. The input is the collision risk judgment result obtained in step 4, and the output is a warning message to the device. The warning message contains the following content:

[1514] {

[1515] "id": "entity_1",

[1516] "warning": "collision_risk",

[1517] "message": "Attention! Possible collision ahead."

[1518] }

[1519] Step 6:

[1520] The device uses an emotion analysis engine to analyze the user's emotional state. The emotion analysis engine uses data acquired from the device's built-in camera and microphone as input and analyzes the user's emotional state. The output is the user's emotional state. Specifically, the data from the camera and microphone is analyzed using TensorFlow and OpenCV to evaluate the user's facial expressions and tone of voice.

[1521] Step 7:

[1522] The device dynamically adjusts the warning message based on the emotional state. Based on the results of the emotion analysis engine, the strength of the warning sound and visual signal is adjusted. The input is the emotional state obtained in step 6, and the output is the adjusted warning message. Specific actions include increasing the volume of the warning sound or the frequency of the flashing light to emphasize the notification of the audio or visual signal.

[1523] Step 8:

[1524] The user receives a warning message from the device and takes appropriate action. The user acknowledges the warning via audio or visual signals from the device and takes safe action. The input is the adjusted warning message and the output is the user's action.

[1525] This is the flow of processing in the program for the system that realizes this application example. Through this series of processing, it is possible to reduce the risk of collisions at intersections for autonomous vehicles and ensure the safety of users.

[1526] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1527] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1528] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1529] [Fourth embodiment]

[1530] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1531] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1532] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1533] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1534] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1535] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1536] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1537] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1538] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1539] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1540] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1541] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1542] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1543] The present invention provides a system for monitoring the movements of vehicles, pedestrians, and bicycles at intersections without traffic lights in real time to reduce the risk of collisions. The system includes a means for a terminal to acquire position information and speed information and transmit it to a server, a means for the server to calculate a predicted trajectory based on the received data and detect a collision risk, a means for the server to send a warning message to the terminal when a collision risk is detected, and a means for the terminal to issue an audio or visual signal to a user when it receives the warning message.

[1544] System operation example

[1545] The device acquires location and speed information using the built-in GPS sensor and speed sensor. For example, when pedestrian A is walking towards an intersection, pedestrian A's device acquires location information (latitude X1, longitude Y1) and speed information (speed V1), which are periodically sent to the server.

[1546] Sending data:

[1547] The device sends location and speed information to the server, which receives it. The data sent is in the following format:

[1548] {

[1549] "id": "entity_1",

[1550] "position": {"lat": X1, "lon": Y1},

[1551] "velocity": V1,

[1552] "timestamp": T1

[1553] }

[1554] The server stores the received data and uses only the most recent information for analysis based on the timestamp.

[1555] Collision risk analysis:

[1556] The server calculates the predicted trajectory of each entity. For example, if Pedestrian A and Bicycle B are heading towards the same intersection, the server calculates their predicted positions in the next 2 seconds and evaluates the likelihood that the entities will arrive at the same location.

[1557] Calculating the predicted trajectory:

[1558] The server calculates the predicted location based on the current location and velocity information as follows:

[1559] next_position = current_position + velocity time_interval

[1560] Based on this predicted position, the entity's future positional relationship is analyzed.

[1561] Sending a warning message:

[1562] If the server detects a collision risk, it generates and sends a warning message to the device of the relevant entity. For example, if there is a high possibility that pedestrian A and cyclist B will collide, the server will generate the following warning message:

[1563] {

[1564] "id": "entity_1",

[1565] "warning": "collision_risk",

[1566] "message": "Attention! Possible collision ahead."

[1567] }

[1568] This is sent to the device.

[1569] Receive alerts and notifications:

[1570] When the device receives a warning message from the server, it notifies the user of the warning. Notification methods include voice messages and flashing lights:

[1571] Display: "Attention! Possible collision ahead."

[1572] Sound: Play warning sound

[1573] Light: Start blinking

[1574] The user can then review this notification and take appropriate action.

[1575] Specific example processing:

[1576] For example, consider a situation where pedestrian A is walking towards an intersection and cyclist B is simultaneously heading towards the intersection. The system operates in the following steps:

[1577] 1. The device acquires location and speed information.

[1578] Pedestrian A's device acquires location information (X1, Y1) and speed (V1), and cyclist B's device also acquires location information (X2, Y2) and speed (V2).

[1579] 2. The device sends the data to the server.

[1580] Both terminals send data to the server, which receives it.

[1581] 3. The server calculates the predicted trajectory.

[1582] The server calculates the predicted trajectories of both vehicles and assesses the risk of collision at the intersection.

[1583] 4. The server generates and sends a warning message.

[1584] If a collision risk is detected, the server sends a warning message to the devices of pedestrian A and cyclist B.

[1585] 5. The device receives the warning message and notifies you.

[1586] The device displays a notification to the user and alerts them with an audio or visual signal.

[1587] In this way, the present invention reduces the risk of collisions between entities within an intersection, ensuring safety.

[1588] The processing flow will be explained below.

[1589] Step 1: The device acquires location and speed information

[1590] The device uses the built-in GPS sensor to obtain the current location (latitude, longitude) and the speed sensor to obtain the moving speed. For example, the device of pedestrian A obtains the location information (X1, Y1) and speed (V1).

[1591] Step 2: The device sends the data to the server

[1592] The device periodically sends the acquired location and speed information to the server. The data sent includes the device ID, location information, speed information, and timestamp. For example,

[1593] {

[1594] "id": "entity_1",

[1595] "position": {"lat": X1, "lon": Y1},

[1596] "velocity": V1,

[1597] "timestamp": T1

[1598] }

[1599] It is sent in the format:

[1600] Step 3: The server receives the data

[1601] The server receives location and speed information from the device via API and stores it in memory or in a database. The server overwrites older data based on the timestamp to keep only the most recent data.

[1602] Step 4: The server calculates the predicted trajectory

[1603] The server calculates the predicted location of each entity at regular intervals based on its current location and velocity. For example, the predicted location for the next 2 seconds is calculated using the following formula:

[1604] next_position = current_position + velocity time_interval

[1605] Based on this predicted position, a future trajectory for each entity is generated.

[1606] Step 5: Server detects collision risk

[1607] The server analyzes the trajectories of all generated entities and evaluates whether they will intersect at an intersection. For example, if pedestrian A and cyclist B are likely to arrive at the same location, it identifies the time and location and determines the collision risk.

[1608] Step 6: The server generates and sends a warning message

[1609] If the server detects a collision risk, it generates and sends a warning message to the device of the relevant entity. For example, it generates a warning message like this:

[1610] {

[1611] "id": "entity_1",

[1612] "warning": "collision_risk",

[1613] "message": "Attention! Possible collision ahead."

[1614] }

[1615] Send this message to the terminal.

[1616] Step 7: The device receives the warning message and notifies the user

[1617] When the terminal receives the warning message sent from the server, it issues a warning to the user with an audio or visual signal, such as displaying the warning message on the terminal screen, sounding an alarm, or flashing a light to alert the user.

[1618] Step 8: The user acts on the device warning

[1619] The user checks the warning from the device and takes safe action. For example, pedestrian A slows down as he approaches the intersection, or the driver of bicycle B slows down to delay entering the intersection.

[1620] Example 1

[1621] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1622] At intersections without traffic lights, the risk of collisions increases because many moving objects, including vehicles, pedestrians, and bicycles, cross the intersection. In such situations, a system is needed that can monitor the position and speed of each entity in real time, detect danger in advance, and issue a warning. However, conventional systems have difficulty accurately calculating the predicted trajectories of multiple moving objects and issuing a warning quickly.

[1623] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1624] In this invention, the server includes a means for the terminal to acquire position information and speed information and transmit it to the server, a means for the server to calculate a predicted trajectory of each entity using the data received and detect a collision risk, and a means for transmitting a warning message to the corresponding terminal when the server detects a collision risk. This makes it possible to simultaneously calculate the predicted trajectories of multiple entities and evaluate the collision risk at an intersection.

[1625] A "terminal" is a device for obtaining location and velocity information of an entity and transmitting it to a server.

[1626] "Location information" is latitude and longitude data that indicates the current location of an entity.

[1627] "Speed ​​information" is data that indicates the moving speed of an entity.

[1628] A "server" is a device or system that receives data transmitted from a terminal and calculates a predicted trajectory and evaluates a collision risk.

[1629] A "predicted trajectory" is a future movement path calculated based on the entity's current position and speed information.

[1630] "Collision risk" is an assessment that indicates the likelihood that multiple entities will arrive at the same location in the future, resulting in a collision accident.

[1631] A "warning message" is a warning communication that is sent to the relevant terminal when the server detects a collision risk.

[1632] The "audio signal" is an audio notification that alerts the user when a warning message is received by the terminal.

[1633] A "visual signal" is a light or screen display notification that alerts the user when a warning message is received by the terminal.

[1634] A "timestamp" is information that records the time when data is acquired or transmitted.

[1635] A "location sensor" is a device that can determine the current location of an entity and obtain that location information.

[1636] The present invention provides a system for monitoring the movements of vehicles, pedestrians, and bicycles at intersections without traffic lights in real time to reduce the risk of collision accidents. The system includes a means for a terminal to acquire position information and speed information and transmit it to a server, a means for the server to calculate a predicted trajectory based on the received data and detect a collision risk, a means for the server to transmit a warning message to the terminal when a collision risk is detected, and a means for the terminal to issue an audio or visual signal to a user when it receives the warning message.

[1637] The device acquires current location information (latitude and longitude) and speed information using the built-in location sensor and speed sensor. For example, if a pedestrian is walking towards an intersection, the device acquires location information (X1, Y1) and speed information (V1) and periodically sends this to the server. The data sent by the device includes the ID, location information (latitude and longitude), speed information, and the timestamp when the data was acquired.

[1638] The server receives data sent from the device and stores it in a database. Data sent from each entity is recorded with a timestamp, and the server uses only the most recent data for analysis. Based on the received data, the server calculates the predicted trajectory of each entity. The current position and speed information are used to calculate the predicted trajectory, using the formula "next position = current position + speed x time interval."

[1639] The server simultaneously calculates the predicted positions of multiple entities and analyzes their future positional relationships. For example, if pedestrian A and cyclist B are heading towards the same intersection, the server calculates their predicted positions and evaluates the probability that they will arrive at the intersection at the same time. Based on this evaluation, the server determines the collision risk.

[1640] If a collision risk is detected, the server generates and sends a warning message to the relevant entity. For example, if there is a high risk of a collision between pedestrian A and cyclist B, the server generates a warning message and sends it to both devices. The warning message includes a warning message and details of the warning.

[1641] When the device receives a warning message, it notifies the user with an audio or visual signal, such as by displaying the message "Attention! Possible collision ahead" on the display, emitting a warning sound, or flashing a light to alert the user.

[1642] For example, if pedestrian A is walking towards an intersection and cyclist B is heading towards the same intersection, the system will operate as follows:

[1643] 1. Pedestrian A's device acquires location information (X1, Y1) and speed (V1), and cyclist B's device also acquires location information (X2, Y2) and speed (V2).

[1644] 2. Both devices send data to the server.

[1645] 3. The server stores the received data and calculates the predicted trajectories of both vehicles.

[1646] 4. The server evaluates the collision risk based on the predicted trajectory.

[1647] 5. If a collision risk is detected, the server generates a warning message and sends it to the devices of pedestrian A and cyclist B.

[1648] 6. The terminal receives the warning message and notifies the user of the warning.

[1649] Such a system can reduce the risk of collisions between entities and ensure safety even at intersections without traffic lights. An example of a prompt sentence could be, "Watch out for the next intersection. Stop and check your surroundings."

[1650] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1651] System program processing flow

[1652] Step 1: The device obtains location and speed information

[1653] The device uses the built-in location and speed sensors to obtain current location information (latitude and longitude) and speed information. The input is the entity's sensor information, and the output is location information (X, Y) and speed information (V). This specific operation involves obtaining GPS data and calculating speed using the acceleration sensor.

[1654] Step 2: The device sends data to the server

[1655] The device sends the acquired location and speed information to the server. The input is the location and speed information acquired in step 1, and the output is the transmission of a data packet to the server. The transmitted data has the following format:

[1656] {

[1657] "id": "entity_1",

[1658] "position": {"lat": X, "lon": Y},

[1659] "velocity": V,

[1660] "timestamp": T

[1661] }

[1662] Specifically, the data packet is sent to the server using an HTTP request or the like.

[1663] Step 3: The server receives and stores the data

[1664] The server receives the data sent from the device and stores it in a database. The input is the data packet sent in step 2, and the output is the stored location and speed information. The specific operation is a write operation to the database.

[1665] Step 4: The server calculates the predicted trajectory

[1666] The server calculates the predicted trajectory of each entity based on the received position and velocity information. The input is the saved current position and velocity information, and the output is the predicted position (next_position) after the next time interval. The predicted position is calculated using the following formula:

[1667] next_position = current_position + velocity time_interval

[1668] The specific operation is to use this formula to predict the future position of each entity.

[1669] Step 5: Server evaluates collision risk

[1670] The server evaluates the collision risk of multiple entities based on the predicted trajectories. The input is the predicted position calculated in step 4, and the output is the collision risk assessment result. The specific operation is to compare the predicted positions and calculate the probability that the entities will arrive at the same point.

[1671] Step 6: The server generates a warning message

[1672] If a collision risk is detected, the server generates a warning message to be sent to the relevant entity. The input is the collision risk assessment result from step 5, and the output is a warning message. The specific format of the warning message is as follows:

[1673] {

[1674] "id": "entity_1",

[1675] "warning": "collision_risk",

[1676] "message": "Attention! Possible collision ahead."

[1677] }

[1678] Specifically, the system dynamically generates warning content based on the evaluation results.

[1679] Step 7: The server sends a warning message

[1680] The server sends the generated warning message to the corresponding terminal. The input is the warning message generated in step 6, and the output is the transmission of warning data to the terminal. Specifically, this message is sent to the corresponding terminal using an HTTP request, etc.

[1681] Step 8: Your device receives a warning message

[1682] The terminal receives the warning message sent from the server. The input is the warning message from the server, and the output is the internal processing of this message. The specific operation is to store the received message in memory and add it to the processing queue.

[1683] Step 9: The device notifies the user of the warning

[1684] When the device receives the warning message, it notifies the user of the warning with an audio or visual signal. The input is the warning message received in step 8, and the output is the notification to the user. Specifically, the device displays "Attention! Possible collision ahead." and plays a warning sound or flashes a light.

[1685] In this way, the risk of collisions between entities can be reduced and safety ensured even at intersections without traffic lights.

[1686] (Application example 1)

[1687] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1688] At unsignalized intersections and in congested urban environments, the risk of collisions between autonomous vehicles and entities such as other vehicles, pedestrians, and cyclists is increasing. In such situations, real-time dynamic prediction and immediate warnings are necessary for autonomous vehicles to operate safely and prevent accidents. However, current technology is unable to meet these needs, and there is a lack of systems that can adequately reduce the risk of collisions between entities. Accidents resulting from this pose a major problem that threatens the safety of passengers and those around them.

[1689] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1690] In this invention, the server includes: means for the terminal to acquire position information and speed information and transmit it to the server; means for the server to calculate a predicted trajectory of each entity using the data received and detect a collision risk; means for transmitting a warning message to the corresponding terminal when the server detects a collision risk; means for the terminal to issue an audio or visual signal to alert the user when it receives the warning message; and means for the terminal to be mounted in an autonomous vehicle and notify the driver of the warning message via an in-vehicle display and audio guidance system. This enables autonomous vehicles to detect collision risks with other entities in real time, even at intersections or in crowded environments, and ensure safe operation.

[1691] A "terminal" is a device that acquires location information and speed information and transmits it to a server.

[1692] "Predicted trajectory" is a calculation method that predicts the future position of each entity based on its current position and speed information.

[1693] "Collision risk" is an assessment that indicates the likelihood of contact or collision between entities using the predicted trajectory.

[1694] A "warning message" is a notification message that is sent to the relevant terminal when a collision risk is detected.

[1695] An "audio or visual signal" is a signal, such as a voice message or a flashing light, that is emitted to notify the user of a warning.

[1696] An "autonomous vehicle" is a vehicle that is equipped with GPS and speed sensors and operates autonomously.

[1697] An "in-vehicle display" is a screen installed inside an autonomous vehicle to visually display information.

[1698] A "voice guidance system" is a system installed inside an autonomous vehicle that provides information through voice messages.

[1699] The present invention is a system for reducing the risk of collisions between autonomous vehicles and entities such as other vehicles, pedestrians, and cyclists at unsignalized intersections and in congested urban environments. The system includes a terminal, a server, an in-vehicle display, and a voice guidance system.

[1700] System hardware configuration

[1701] Device: Installed in an autonomous vehicle, it continuously acquires location and speed information using built-in positioning sensors (e.g., GPS sensors) and speed sensors.

[1702] Server: Located on the cloud, it receives data sent from the device, calculates the predicted trajectory, and evaluates the collision risk.

[1703] In-vehicle displays: installed inside autonomous vehicles, they visually display warning messages and other necessary information.

[1704] Voice guidance system: A system that works in conjunction with the in-car display to provide warnings to the driver through voice messages.

[1705] Software configuration and processing flow

[1706] 1. Data Acquisition

[1707] The device uses built-in positioning and speed measurement sensors to obtain real-time position and speed information of the autonomous vehicle.

[1708] 2. Data Transmission

[1709] The terminal periodically transmits the acquired location and speed information to the server. The transmitted data includes location information (latitude and longitude), speed information, and a timestamp.

[1710] 3. Collision risk assessment

[1711] The server uses the received data to calculate the predicted trajectory of the autonomous vehicle and other entities, including calculating future positions based on current position and speed information.

[1712] 4. Generating Warning Messages

[1713] The server evaluates the collision risk based on the predicted trajectory, and if a collision risk is detected, it generates and sends a warning message to the relevant terminal.

[1714] 5. Warning Notification

[1715] When the terminal receives a warning message from the server, it issues a visual and audio warning to the driver through the in-car display and voice guidance system.

[1716] Prompt Sentence Examples

[1717] Below are some example prompts to input to a generative AI model:

[1718] "Design a system that acquires real-time location and speed information of an autonomous vehicle, sends it to a server, and evaluates the risk of collision. The server calculates a predicted trajectory based on data from other entities, and if there is a risk of collision, warns the driver on the vehicle's display and voice guidance system. Please provide Python code that runs in real time."

[1719] As a specific example, consider the following scenario: As autonomous vehicle A approaches an intersection, it detects the presence of pedestrian B walking toward the same intersection. The server calculates the predicted trajectories of both vehicles, and if it detects a high risk of collision, it sends a warning message to the device of autonomous vehicle A. The in-vehicle display will show a message saying, "Possible collision with pedestrian at intersection ahead. Please be careful," and the voice guidance system will give instructions such as, "Pay attention to what is ahead." This notification will alert the driver to be careful at the intersection, ensuring safe driving.

[1720] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1721] Step 1:

[1722] The device obtains location and speed information. The device uses a location sensor (e.g., GPS sensor) and a speed sensor to obtain the current latitude, longitude, and speed of the autonomous vehicle. In this case, the input is the raw data provided by the vehicle's sensors, and the output is the latitude, longitude, and speed values.

[1723] Step 2:

[1724] The device sends the acquired location and speed information to the server. The device sends data to the server, including the acquired location information (latitude, longitude), speed information, and timestamp. This data is encoded in JSON or other appropriate data format. The location and speed information obtained in step 1 is used as input, and the data sent to the server is generated as output.

[1725] Step 3:

[1726] The server stores the received data and uses it for analysis. The server stores the location and speed information received from the device in a database and uses only the most recent data for analysis based on the timestamp. This eliminates duplicate and old data. The input is the data sent from the device, and the output is the latest data set to use for analysis and calculations.

[1727] Step 4:

[1728] The server calculates the predicted trajectory of each entity. Based on the received position and velocity information, the server predicts the future position of each entity. This calculation is done by multiplying the current position by the velocity, for example. The input is the latest position and velocity information, and the output is the predicted position at a specific point in the future.

[1729] Step 5:

[1730] The server evaluates the collision risk. The server compares the predicted trajectories of each entity and determines whether there is a possibility of an intersection. If the intersection points are very close, it determines that there is a collision risk. The input is the predicted trajectory data, and the output is the collision risk assessment result (risk / no risk).

[1731] Step 6:

[1732] If the server detects a collision risk, it sends a warning message to the relevant terminal. The server generates and sends a warning message to the terminal of the entity where the collision risk was detected. This message indicates that a collision risk exists and calls for attention. The input is the collision risk evaluation result, and the output is the warning message to be sent to each terminal.

[1733] Step 7:

[1734] The terminal receives the warning message and alerts the user. The terminal notifies the driver of the received warning message via the in-car display or voice guidance system. For example, the display may show a message saying "There is a pedestrian ahead. Please be careful," and the voice guidance system may play a voice message saying "Pay attention to what is ahead." The input is the warning message received from the server, and the output is the warning notification to the user.

[1735] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1736] The present invention combines an emotion engine with a system for monitoring the movements of entities such as vehicles, pedestrians, and bicycles at intersections without traffic lights in real time and reducing the risk of collision accidents. This system includes means for a terminal to acquire position information and speed information and send it to a server, means for the server to calculate a predicted trajectory based on the received data and detect a collision risk, means for the server to send a warning message to the relevant terminal when a collision risk is detected, means for the terminal to issue an audio or visual signal to alert the user when it receives the warning message, and an emotion engine that recognizes the user's emotions.

[1737] System Overview

[1738] The device acquires location and speed information using the built-in GPS sensor and speed sensor. For example, when pedestrian A is walking towards an intersection, pedestrian A's device acquires location information (latitude X1, longitude Y1) and speed information (speed V1), which are periodically sent to the server.

[1739] Detailed System Operation

[1740] Sending data:

[1741] The device periodically sends the location and speed information it has acquired to the server. The data sent is in the following format:

[1742] {

[1743] "id": "entity_1",

[1744] "position": {"lat": X1, "lon": Y1},

[1745] "velocity": V1,

[1746] "timestamp": T1

[1747] }

[1748] The server stores the received data and uses only the most recent information for analysis based on the timestamp.

[1749] Collision risk analysis:

[1750] The server calculates the predicted location of each entity at regular intervals based on its current location and velocity. For example, the predicted location for the next 2 seconds is calculated using the following formula:

[1751] next_position = current_position + velocity time_interval

[1752] Based on this predicted position, a future trajectory for each entity is generated.

[1753] Calculating the predicted trajectory:

[1754] The server analyzes the trajectories of all entities and evaluates whether they intersect at an intersection. For example, if pedestrian A and cyclist B are likely to arrive at the same location, it identifies the time and location and determines the collision risk.

[1755] Generate and send a warning message:

[1756] If the server detects a collision risk, it generates and sends a warning message to the device of the relevant entity. For example, it generates a warning message like this:

[1757] {

[1758] "id": "entity_1",

[1759] "warning": "collision_risk",

[1760] "message": "Attention! Possible collision ahead."

[1761] }

[1762] This is sent to the device.

[1763] Using the Emotion Engine:

[1764] The emotion engine analyzes the user's emotions in real time and recognizes their emotional state. For example, it uses the device's camera and microphone to analyze emotions from changes in the user's facial expressions and voice.

[1765] Dynamically adjusting warnings based on emotions:

[1766] If the emotion engine determines that the user is in a negative emotional state, such as tension or anxiety, the server will issue a more emphatic warning message, for example by making the warning sound louder or flashing visual signals more frequently.

[1767] Receiving and notifying warning messages:

[1768] When the device receives a warning message from the server, it notifies the user of the warning. Notification methods include voice messages and flashing lights:

[1769] Display: "Attention! Possible collision ahead."

[1770] Sound: Play warning sound

[1771] Light: Start blinking

[1772] The user can then review this notification and take appropriate action.

[1773] Processing of specific examples

[1774] For example, consider a situation where pedestrian A is walking towards an intersection and cyclist B is simultaneously heading towards the intersection. The system operates in the following steps:

[1775] 1. The device acquires location and speed information.

[1776] Pedestrian A's device acquires location information (X1, Y1) and speed (V1), and cyclist B's device also acquires location information (X2, Y2) and speed (V2).

[1777] 2. The device sends the data to the server.

[1778] Both terminals send data to the server, which receives it.

[1779] 3. The server calculates the predicted trajectory.

[1780] The server calculates the predicted trajectories of both vehicles and assesses the risk of collision at the intersection.

[1781] 4. The server generates and sends a warning message.

[1782] If a collision risk is detected, the server sends a warning message to the devices of pedestrian A and cyclist B.

[1783] 5. The emotion engine analyzes the user's emotions.

[1784] The emotion engine analyzes the emotional state of pedestrian A and cyclist B and dynamically adjusts the content and format of the warning as needed.

[1785] 6. The device receives the warning message and notifies you.

[1786] The device displays a notification to the user and alerts them with an audio or visual signal.

[1787] 7. The user acts in accordance with the warning on the device.

[1788] The user acknowledges the warning and takes safe action.

[1789] In this way, the present invention reduces the risk of collisions between entities at intersections and ensures safety. By combining emotion engines, the effectiveness of warnings is maximized and users' safety awareness is enhanced.

[1790] The processing flow will be explained below.

[1791] Step 1: The device acquires location and speed information

[1792] The device uses the built-in GPS sensor to obtain the current location (latitude, longitude) and the speed sensor to obtain the moving speed. For example, the device of pedestrian A obtains the location information (X1, Y1) and speed (V1).

[1793] Step 2: The device sends the data to the server

[1794] The device periodically sends the acquired location and speed information to the server. The data sent includes the device ID, location information, speed information, and timestamp. For example,

[1795] {

[1796] "id": "entity_1",

[1797] "position": {"lat": X1, "lon": Y1},

[1798] "velocity": V1,

[1799] "timestamp": T1

[1800] }

[1801] It is sent in the format:

[1802] Step 3: The server receives the data

[1803] The server receives location and speed information from the device via API and stores it in memory or in a database. The server overwrites older data based on the timestamp to keep only the most recent data.

[1804] Step 4: The server calculates the predicted trajectory

[1805] The server calculates the predicted location of each entity at regular intervals based on its current location and velocity. For example, the predicted location for the next 2 seconds is calculated using the following formula:

[1806] next_position = current_position + velocity time_interval

[1807] Based on this predicted position, a future trajectory for each entity is generated.

[1808] Step 5: Server detects collision risk

[1809] The server analyzes the trajectories of all generated entities and evaluates whether they will intersect at an intersection. For example, if pedestrian A and cyclist B are likely to arrive at the same location, it identifies the time and location and determines the collision risk.

[1810] Step 6: The emotion engine analyzes the user's emotions

[1811] Using the device's built-in camera and microphone, the emotion engine analyzes the user's emotional state from their facial expressions and voice. For example, if the user is nervous, the engine will analyze their emotional state in real time.

[1812] Step 7: The server generates and sends a warning message

[1813] When the server detects a collision risk, it dynamically adjusts the warning message based on the analysis results of the emotion engine. For example, if the user is nervous, it generates a more emphasized warning message. Examples of generated warning messages are:

[1814] {

[1815] "id": "entity_1",

[1816] "warning": "collision_risk",

[1817] "message": "Attention! Possible collision ahead. Please be careful!"

[1818] }

[1819] This is sent to the device.

[1820] Step 8: The device receives the warning message and notifies the user.

[1821] When the device receives the warning message sent from the server, it will warn the user with audio or visual signals, for example, by displaying a warning message on the device screen, sounding an alarm, or flashing a light to alert the user:

[1822] Display: "Attention! Possible collision ahead. Please be careful!"

[1823] Sound: Play warning sound

[1824] Light: Start blinking

[1825] Step 9: User acts on the device warning

[1826] The user checks the warning from the device and takes safe action. For example, pedestrian A slows down as he approaches the intersection, or the driver of bicycle B slows down to delay entering the intersection.

[1827] This series of processes reduces the risk of collisions between entities at intersections without traffic lights, ensuring safety. In addition, the use of an emotion engine maximizes the effectiveness of warnings and increases user safety awareness.

[1828] Example 2

[1829] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1830] Traffic accidents frequently occur at intersections without traffic lights, and the risk increases significantly when different types of entities, such as pedestrians, vehicles, and bicycles, enter the intersection at the same time. Conventional systems have difficulty monitoring the movements of these entities in real time and providing users with appropriate warnings to reduce collision risks. Furthermore, they lack the means to appropriately adjust the content and format of warnings based on the user's emotional state. Therefore, a system that can more efficiently and effectively reduce collision risks in real-world usage environments is needed.

[1831] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1832] In this invention, the server includes a means for the terminal to acquire position information and speed information and transmit it to the server, a means for the server to calculate a predicted trajectory of each entity using the received data and detect a collision risk, a means for the server to transmit a warning message to the corresponding terminal when the server detects a collision risk, and a means for the terminal to analyze the emotional state of the user and dynamically adjust the content and format of the warning. This makes it possible to monitor the movements of entities at intersections without signals in real time, detect collision risks in a timely manner, and further improve the effectiveness of the warning according to the emotional state of the user, thereby promoting safe behavior.

[1833] A "terminal" is an electronic device that acquires location information and speed information and transmits it to a server.

[1834] The "server" is a device or system that receives data transmitted from the terminal, calculates the predicted trajectory of the entity based on the data, and detects the risk of collision.

[1835] "Location information" is the entity's current latitude and longitude data.

[1836] "Speed ​​information" is data about the speed at which an entity is moving.

[1837] An "entity" is a moving object such as a pedestrian, vehicle, or bicycle that moves around within an intersection.

[1838] A "predicted trajectory" is the path or position of an entity after a certain period of time, calculated by the server based on the entity's current location and speed information.

[1839] "Collision risk" refers to a situation where the predicted trajectories of different entities intersect, potentially resulting in a physical collision.

[1840] A "warning message" is a notification that is generated by the server and sent to the relevant terminal when a collision risk is detected.

[1841] An "audio or visual signal" is a sound or a visual alarm such as an indicator light that is emitted to alert the user when the terminal receives a warning message.

[1842] "Emotional state" refers to the user's psychological state, such as tension, anxiety, excitement, etc., and is analyzed using sensors such as a camera and microphone.

[1843] "Dynamic adjustment" refers to changing the content and notification method of the warning message in real time based on the user's emotional state.

[1844] This invention is a system for monitoring the movements of entities such as vehicles, pedestrians, and bicycles in real time at intersections without traffic lights to reduce the risk of collision accidents. The system includes means for acquiring position information and speed information of entities and transmitting it to a server, means for calculating a predicted trajectory based on the data received by the server and detecting a collision risk, means for transmitting a warning message to an appropriate terminal when a collision risk is detected, means for issuing an audio or visual signal to alert the user, and means for recognizing the user's emotions and dynamically adjusting the content and format of the warning.

[1845] Hardware and software used

[1846] Hardware:

[1847] Device: Smartphones and wearable devices are used. These devices have built-in GPS and speed sensors. Typical examples of smartphones include the Qualcomm Snapdragon GPS and Ublox M8.

[1848] Server: The server is built on a cloud service, for example, a web server using Flask on an AWS EC2 instance.

[1849] software:

[1850] GPS and speed data analysis: Location information is analyzed using the Python "Geopy" library, and speed information is obtained directly from the device's built-in sensors.

[1851] Data processing: The received data is processed and saved using the Python "Pandas" library, and the necessary analysis is performed.

[1852] Machine learning model: Sentiment analysis is performed using "Scikit-learn".

[1853] Example of a system

[1854] First, the device acquires location information (latitude X1, longitude Y1) and speed information (speed V1) using the built-in GPS sensor and speed sensor. For example, when pedestrian A is walking towards an intersection, pedestrian A's device acquires location information and speed information and sends it to the server in JSON format.

[1855] json

[1856] Topic: GPS data formatting

[1857] {

[1858] "id": "walk_A",

[1859] "position": {"lat": 35.6895, "lon": 139.6917},

[1860] "velocity": 1.2,

[1861] "timestamp": "2023-10-10T12:00:00Z"

[1862] }

[1863] The server then calculates the entity's predicted trajectory based on the received data to detect collision risk. This calculation uses the current position and speed to derive a predicted position for a certain time period. For example, the predicted position for the next 2 seconds is calculated using the following formula:

[1864] python

[1865] next_position = current_position + velocity time_interval

[1866] The server analyzes the trajectories of all entities and evaluates whether they intersect at an intersection. For example, if pedestrian A and cyclist B are likely to arrive at the same location, it identifies the time and location and determines the collision risk.

[1867] If a collision risk is detected, the server generates and sends a warning message to the device of the relevant entity. For example, the following warning message is sent to pedestrian A:

[1868] json

[1869] Note: Example of a warning message

[1870] {

[1871] "id": "walk_A",

[1872] "warning": "collision_risk",

[1873] "message": "Attention! Possible collision ahead."

[1874] }

[1875] When the device receives a warning message, it notifies the user through audio, visual signals (flashing lights), or vibration. Examples of notifications include:

[1876] json

[1877] Alert methods: sound notification, flashing light, vibration

[1878] Display: "Attention! Possible collision ahead."

[1879] Sound: Play warning sound

[1880] Light: Start blinking

[1881] An emotion engine is also used, which uses the device's camera and microphone to analyze the user's emotions and, if the user is in a negative emotional state such as tension or anxiety, will issue a more emphatic warning message, for example by increasing the volume of the warning sound or speeding up the blinking of the light signal.

[1882] Prompt Sentence Examples

[1883] "A pedestrian and a cyclist are approaching an intersection. Write Python code to calculate the collision risk and generate and send a warning message if necessary."

[1884] This allows the system to monitor entity movements at unsignalized intersections in real time, and improve safety by detecting and notifying collision risks early. Dynamic warning adjustment based on the user's emotional state maximizes the effectiveness of warning messages and increases safety awareness.

[1885] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1886] Step 1:

[1887] The device acquires location and speed information.

[1888] Input: Built-in GPS and speed sensors

[1889] Specific behavior:

[1890] The device uses the built-in GPS sensor to obtain latitude (X) and longitude (Y), and the speed sensor to measure speed (V).

[1891] Output: Position information (latitude X, longitude Y) and speed information (V)

[1892] Step 2:

[1893] The terminal transmits location information and speed information to the server.

[1894] Input: Obtained location information (latitude X, longitude Y) and speed information (V)

[1895] Specific behavior:

[1896] The device converts the location and speed information into JSON format and sends it to the server via an HTTP request.

[1897] Output: JSON data sent to the server

[1898] json

[1899] {

[1900] "id": "entity_1",

[1901] "position": {"lat": X, "lon": Y},

[1902] "velocity": V,

[1903] "timestamp": T

[1904] }

[1905] Step 3:

[1906] The server stores the received data.

[1907] Input: JSON data sent from the terminal

[1908] Specific behavior:

[1909] The server receives the HTTP request, parses the received JSON data, and saves it in the database. At this time, it saves only the latest data based on the timestamp (T).

[1910] Output: Location, speed, and timestamp stored in a database

[1911] Step 4:

[1912] The server calculates the predicted trajectories of all entities.

[1913] Input: Position and velocity information of all entities stored in the database

[1914] Specific behavior:

[1915] The server uses the position (X, Y) and velocity (V) information of each entity to calculate the predicted position at the next time step using the following formula:

[1916] python

[1917] next_position = current_position + velocity time_interval

[1918] For example, calculate the predicted location for the next 2 seconds and generate future trajectories for all entities.

[1919] Output: predicted trajectory (future_position)

[1920] Step 5:

[1921] The server determines the collision risk and generates a warning message.

[1922] Input: predicted trajectory (future_position)

[1923] Specific behavior:

[1924] The server analyzes the predicted trajectories of all entities and evaluates the likelihood of them intersecting with each other. If there is a risk of collision, it generates a warning message for the relevant entity.

[1925] Output: Warning message

[1926] json

[1927] {

[1928] "id": "entity_1",

[1929] "warning": "collision_risk",

[1930] "message": "Attention! Possible collision ahead."

[1931] }

[1932] Step 6:

[1933] The server sends a warning message to the relevant terminal.

[1934] Input: The generated warning message

[1935] Specific behavior:

[1936] The server sends the generated warning message to the relevant terminal via an HTTP request.

[1937] Output: Warning message sent to terminal

[1938] Step 7:

[1939] The terminal receives the warning message and notifies the user.

[1940] Input: The warning message sent by the server

[1941] Specific behavior:

[1942] The device receives the warning message and notifies the user of the warning through a voice message, flashing light, or vibration.

[1943] Output: Warnings notified to the user

[1944] json

[1945] Display: "Attention! Possible collision ahead."

[1946] Sound: Play warning sound

[1947] Light: Start blinking

[1948] Step 8:

[1949] The emotion engine analyzes the user's emotions and dynamically adjusts the notification method.

[1950] Input: User facial and voice data

[1951] Specific behavior:

[1952] The device's built-in camera and microphone are used to analyze the user's facial expressions and voice changes in real time, which allows the system to determine the user's psychological state and dynamically adjust the volume of the warning sound and the flashing speed according to the user's emotional state.

[1953] Output: Adjusted notification method

[1954] Through the above steps, the system can monitor the movements of entities at unsignalized intersections in real time and ensure safety by providing appropriate collision warnings.

[1955] (Application example 2)

[1956] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1957] At intersections without traffic lights, there is a high risk of collisions between entities such as vehicles, pedestrians, and cyclists, making safety particularly important for autonomous vehicles. However, conventional systems lack the means to monitor entities' movements in real time and provide appropriate warnings. Furthermore, they do not dynamically adjust warnings according to the user's emotional state, resulting in insufficient improvements in safety.

[1958] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1959] In this invention, the server includes means for the terminal to acquire position information and speed information and transmit it to the server, means for the server to calculate a predicted trajectory of each entity using the received data and detect a collision risk, means for the server to transmit a warning message to the corresponding terminal when the server detects a collision risk, and means for the terminal to analyze the emotional state of the user using a built-in emotion analysis engine and dynamically adjust the content and format of the warning message based on the analysis result. This makes it possible to reduce the collision risk of entities in real time and provide effective warnings according to the emotional state of the user.

[1960] A "terminal" is a device that acquires location and speed information and is equipped with an emotion analysis engine.

[1961] A "server" is a central processing unit that receives, analyzes, and processes data sent from a terminal.

[1962] "Location information" is data that indicates the current geographical location of a terminal.

[1963] "Speed ​​information" is data indicating the speed at which the terminal is moving.

[1964] A "predicted trajectory" is a calculated prediction of the trajectory that an entity will move in the future.

[1965] "Collision risk" refers to the likelihood that two or more entities will collide when reaching an intersection.

[1966] A "warning message" is a warning notification sent to a terminal when a collision risk is detected.

[1967] An "audio signal" is an audio notification that the terminal issues to alert the user.

[1968] A "visual signal" is a notification using light or display that a device emits to alert the user.

[1969] An "emotion analysis engine" is a system component that analyzes the user's emotional state in real time and utilizes the analysis results.

[1970] "Dynamic adjustment" refers to flexibly changing the content and format of the warning message depending on the user's emotional state.

[1971] MODE FOR CARRYING OUT THE INVENTION

[1972] The present invention relates to a system installed in an autonomous vehicle, which includes a terminal, a server, and an emotion analysis engine to reduce the risk of collision and ensure safety.

[1973] Hardware and Software Configuration

[1974] Terminal

[1975] It will be installed in self-driving vehicles.

[1976] Location and speed information is obtained using the built-in GPS sensor and speed sensor.

[1977] It is equipped with an emotion analysis engine and uses the camera and microphone to analyze the user's emotional state.

[1978] server

[1979] Receives and analyzes location and speed information sent from the device.

[1980] The latest information is stored based on timestamps and collision risk is predicted.

[1981] If a collision risk is detected, a warning message is sent to the device.

[1982] Sentiment Analysis Engine

[1983] Analyze the user's emotional state in real time.

[1984] Image analysis libraries such as TensorFlow and OpenCV are used to process data from the camera and microphone.

[1985] Data transmission and analysis

[1986] Terminal

[1987] The device periodically sends its location and speed information to the server. The data sent consists of the entity ID, location (latitude, longitude), speed, and timestamp. The format sent to the server is as follows:

[1988] {

[1989] "id": "vehicle_1",

[1990] "position": {"lat": X1, "lon": Y1},

[1991] "velocity": V1,

[1992] "timestamp": T1

[1993] }

[1994] server

[1995] The server calculates the predicted trajectory of each entity based on the received data, calculates the entity's position when it arrives at the intersection, and evaluates the collision risk with other entities. If a collision risk is detected, it sends a warning message to the relevant device.

[1996] Sentiment analysis and dynamic alert adjustment

[1997] Emotion analysis

[1998] The device's built-in emotion analysis engine analyzes data collected from the camera and microphone to recognize the user's emotional state. A machine learning model using TensorFlow analyzes facial expressions and voice data to detect negative emotional states such as tension or anxiety.

[1999] Adjustment of warning content

[2000] An emotion analysis engine recognizes the user's emotional state and dynamically adjusts the content and format of the alert message as needed - for example, if tension or anxiety is detected, the alert may sound louder or flash a visual signal more frequently.

[2001] Specific examples

[2002] For example, if autonomous vehicle "A" is located at latitude 35.6895 and longitude 139.6917 and is moving at a speed of 10 m / s, the following prompt sentence can be used to predict its location in the next 5 seconds and avoid the risk of collision:

[2003] If the current location of the vehicle is at latitude 35.6895, longitude 139.6917, and speed 10 m / s, predict the location of the vehicle in the next 5 seconds and generate a warning message to avoid a possible collision.

[2004] In this way, the present invention can reduce the risk of collisions between entities at intersections in autonomous vehicles and ensure user safety. The introduction of a sentiment analysis engine allows the content and format of warnings to be appropriately adjusted, enabling more effective warnings.

[2005] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2006] Step 1:

[2007] The device acquires location and speed information. The device uses the built-in GPS sensor and speed sensor to measure the current location (latitude, longitude) and speed of the autonomous vehicle. The input is real-time data from the GPS sensor and speed sensor, and the output is location and speed information.

[2008] Step 2:

[2009] The location and speed information acquired by the device is sent to the server. The acquired location information (latitude, longitude) and speed information are sent to the server along with a timestamp. The input is the location and speed information acquired in step 1, and the output is the data sent to the server.

[2010] Step 3:

[2011] The server uses the received data to calculate the predicted trajectory. The server receives the vehicle's current position and speed information as input, and uses this to calculate the predicted position for the next few seconds. The output is the calculated predicted position. The server predicts the position as follows:

[2012] Next position = Current position + Speed ​​Time interval

[2013] Step 4:

[2014] The server detects collision risk. The server analyzes the predicted trajectories of all entities and evaluates the collision risk at intersections. The input is the predicted trajectory of each entity, and the output is the collision risk judgment result. Specifically, the server lists overlapping predicted positions and checks whether they are at the same time.

[2015] Step 5:

[2016] The server generates a warning message and sends it to the device. If a collision risk is detected, the server generates a warning message and sends it to the device. The input is the collision risk judgment result obtained in step 4, and the output is a warning message to the device. The warning message contains the following content:

[2017] {

[2018] "id": "entity_1",

[2019] "warning": "collision_risk",

[2020] "message": "Attention! Possible collision ahead."

[2021] }

[2022] Step 6:

[2023] The device uses an emotion analysis engine to analyze the user's emotional state. The emotion analysis engine uses data acquired from the device's built-in camera and microphone as input and analyzes the user's emotional state. The output is the user's emotional state. Specifically, the data from the camera and microphone is analyzed using TensorFlow and OpenCV to evaluate the user's facial expressions and tone of voice.

[2024] Step 7:

[2025] The device dynamically adjusts the warning message based on the emotional state. Based on the results of the emotion analysis engine, the strength of the warning sound and visual signal is adjusted. The input is the emotional state obtained in step 6, and the output is the adjusted warning message. Specific actions include increasing the volume of the warning sound or the frequency of the flashing light to emphasize the notification of the audio or visual signal.

[2026] Step 8:

[2027] The user receives a warning message from the device and takes appropriate action. The user acknowledges the warning via audio or visual signals from the device and takes safe action. The input is the adjusted warning message and the output is the user's action.

[2028] This is the flow of processing in the program for the system that realizes this application example. Through this series of processing, it is possible to reduce the risk of collisions at intersections for autonomous vehicles and ensure the safety of users.

[2029] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2030] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2031] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2032] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2033] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2034] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2035] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2036] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2037] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2038] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2039] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2040] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2041] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2042] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2043] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2044] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2045] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2046] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2047] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2048] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2049] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2050] The following is further disclosed regarding the above embodiment.

[2051] (Claim 1)

[2052] A means for the terminal to acquire location information and speed information and transmit them to a server;

[2053] means for calculating a predicted trajectory of each entity using the data received by the server to detect a collision risk;

[2054] means for transmitting a warning message to a corresponding terminal when the server detects a collision risk;

[2055] The system includes means for issuing an audio or visual signal to alert the user when the terminal receives a warning message.

[2056] (Claim 2)

[2057] The system according to claim 1, characterized in that the server stores only the most recent location information and speed information based on a timestamp and uses it for analysis.

[2058] (Claim 3)

[2059] 2. The system according to claim 1, wherein the terminal acquires location information and speed information using a built-in GPS sensor and speed sensor.

[2060] "Example 1"

[2061] (Claim 1)

[2062] A means for the terminal to acquire location information and speed information and transmit them to a server;

[2063] means for calculating a predicted trajectory of each entity using the data received by the server to detect a collision risk;

[2064] means for transmitting a warning message to a corresponding terminal when the server detects a collision risk;

[2065] means for issuing an audio or visual signal to alert a user when the terminal receives a warning message;

[2066] a means for the server to simultaneously calculate predicted trajectories of multiple entities and assess collision risk at an intersection;

[2067] a means for the terminal to receive a warning message from the server and immediately notify the user;

[2068] A system including:

[2069] (Claim 2)

[2070] The system according to claim 1, characterized in that the server stores only the most recent location information and speed information based on a timestamp and uses it for analysis.

[2071] (Claim 3)

[2072] 2. The system of claim 1, wherein the terminal acquires location and speed information using built-in location and speed sensors.

[2073] "Application Example 1"

[2074] (Claim 1)

[2075] A means for the terminal to acquire location information and speed information and transmit them to a server;

[2076] means for calculating a predicted trajectory of each entity using the data received by the server to detect a collision risk;

[2077] means for transmitting a warning message to a corresponding terminal when the server detects a collision risk;

[2078] means for issuing an audio or visual signal to alert a user when the terminal receives a warning message;

[2079] The system includes the terminal mounted on an autonomous vehicle and means for notifying the driver of the warning message via an in-vehicle display and a voice guidance system.

[2080] (Claim 2)

[2081] The system according to claim 1, characterized in that the server stores only the most recent location information and speed information based on a timestamp and uses it for analysis.

[2082] (Claim 3)

[2083] 2. The system according to claim 1, wherein the terminal acquires location information and speed information using built-in location and speed measurement sensors.

[2084] "Example 2: Combining Emotion Engines"

[2085] (Claim 1)

[2086] A means for the terminal to acquire location information and speed information and transmit them to a server;

[2087] means for calculating a predicted trajectory of each entity using the data received by the server to detect a collision risk;

[2088] means for transmitting a warning message to a corresponding terminal when the server detects a collision risk;

[2089] means for issuing an audio or visual signal to alert a user when the terminal receives a warning message;

[2090] The system includes a means for the device to analyze the user's emotional state and dynamically adjust the content and format of the alert.

[2091] (Claim 2)

[2092] The system according to claim 1, characterized in that the server stores only the most recent location information and speed information based on a timestamp and uses it for analysis.

[2093] (Claim 3)

[2094] 2. The system according to claim 1, wherein the terminal acquires the location information and the speed information using an internal location sensor and a speed sensor.

[2095] "Application example 2 when combining emotion engines"

[2096] (Claim 1)

[2097] A means for the terminal to acquire location information and speed information and transmit them to a server;

[2098] means for calculating a predicted trajectory of each entity using the data received by the server to detect a collision risk;

[2099] means for transmitting a warning message to a corresponding terminal when the server detects a collision risk;

[2100] means for issuing an audio or visual signal to alert a user when the terminal receives a warning message;

[2101] The device uses its built-in emotion analysis engine to analyze the user's emotional state,

[2102] A system that includes a means for dynamically adjusting the content and format of warning messages based on the results of the analysis.

[2103] (Claim 2)

[2104] The system according to claim 1, characterized in that the server stores only the most recent location information and speed information based on a timestamp and uses it for analysis.

[2105] (Claim 3)

[2106] 2. The system according to claim 1, wherein the terminal acquires location information and speed information using a built-in GPS sensor and speed sensor. [Explanation of symbols]

[2107] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for the terminal to acquire location information and speed information and transmit them to a server; means for calculating a predicted trajectory of each entity using the data received by the server to detect a collision risk; means for transmitting a warning message to a corresponding terminal when the server detects a collision risk; The system includes means for issuing an audio or visual signal to alert the user when the terminal receives a warning message.

2. 2. The system according to claim 1, wherein the server stores only the most recent position information and speed information based on a timestamp and uses the information for analysis.

3. 2. The system according to claim 1, wherein the terminal acquires location information and speed information using a built-in GPS sensor and speed sensor.

Images