System

Abstract

To provide a system for relaxing traffic congestion, reducing traffic accidents and improving fuel efficiency by optimizing the timing of a traffic signal.SOLUTION: A system, comprising: means for obtaining a traffic picture; means for analyzing the obtained traffic picture to determine a traffic condition; means for optimizing timing of a traffic light based on the traffic condition; and means for applying the optimized signal timing to the traffic light.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The purpose of this invention is to mitigate traffic congestion, reduce traffic accidents, and improve fuel efficiency by optimizing traffic signal timing. Conventional traffic signals are under the jurisdiction of prefectural police, and their control and operation are often inefficient, which is one of the causes of traffic congestion in particular. To solve this problem, it is necessary to dynamically optimize traffic signal timing based on real-time traffic data. [Means for solving the problem]

[0005] The present invention solves the above problems by the following means. First, a means for acquiring traffic images is provided. This acquisition means acquires real-time video data using cameras installed at intersections and major roads. Next, a means for analyzing the acquired traffic images and determining traffic conditions is provided. This analysis means uses an AI-based image recognition algorithm to identify vehicles, pedestrians, bicycles, etc. and analyzes their respective positions, speeds, and directions. Furthermore, a means for optimizing traffic light timing based on traffic conditions is provided. This optimization means utilizes the analysis results and past data to calculate optimal signal timing for smooth traffic flow. Finally, a means for applying the optimized signal timing to traffic lights is provided. This application means transmits the calculated signal timing to each traffic light, causing the traffic light to operate based on the calculated signal timing. This enables traffic light timing to be optimized in real time, thereby alleviating traffic congestion, reducing traffic accidents, and improving fuel efficiency.

[0006] "Traffic video" refers to real-time video data acquired through cameras installed at intersections and major roads.

[0007] "Means for acquiring" refers to equipment or systems that capture traffic footage in real time using devices such as cameras or sensors, and store or transmit the footage.

[0008] "Means of analysis" refers to devices or systems that use AI or algorithms to identify vehicles, pedestrians, bicycles, etc. based on acquired traffic footage and determine traffic conditions.

[0009] "Optimization means" refers to a device or system that calculates traffic light timing based on traffic conditions and sets optimal signal timing.

[0010] "Applying means" refers to a device or system that transmits the calculated signal timing to a traffic light and ensures that the timing is actually reflected in the traffic light.

[0011] "AI" is an abbreviation for artificial intelligence, a technology that automatically makes judgments and predictions through machine learning and data analysis.

[0012] An "image recognition algorithm" is a mathematical algorithm for identifying objects such as vehicles, pedestrians, and bicycles from acquired video data. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] This invention relates to a traffic light timing optimization system that uses traffic video. This system acquires traffic video in real time, analyzes and predicts traffic conditions based on the video, and optimizes traffic light timing.

[0035] Overall system overview

[0036] The system consists of the following main components:

[0037] 1. Traffic image acquisition means (terminal)

[0038] 2. Data analysis method (server)

[0039] 3. Signal timing optimization means (server)

[0040] 4. Signal Timing Application Means (Terminal)

[0041] 5. Monitoring and Feedback Methods (Users)

[0042] Traffic image acquisition means (terminal)

[0043] The terminals are equipped with cameras that capture images of intersections and major roads in real time. These cameras are capable of capturing images day and night, and the terminals are equipped with communication functions to send the captured image data to a server.

[0044] Data analysis method (server)

[0045] The server receives traffic video data sent from the device. The received data is analyzed using an AI-based image recognition algorithm. This analysis identifies objects such as vehicles, pedestrians, and bicycles from the video data and obtains attribute data such as their location, speed, and direction.

[0046] Signal timing optimization means (server)

[0047] The server performs calculations to optimize traffic signal timing based on traffic condition data obtained from the data analysis means. It takes into account past data and current conditions to predict future traffic flow and congestion. It calculates the optimal duration for each phase of the traffic signal (green, yellow, red).

[0048] Signal timing application means (terminal)

[0049] The optimized signal timing calculated by the server is sent to the traffic light via the application means. The terminal (traffic light) changes the signal in real time according to the received signal timing. This will result in smoother traffic flow, and is expected to alleviate traffic congestion and reduce accidents.

[0050] Monitoring and feedback measures (users)

[0051] Users can use the system's monitoring function to monitor real-time traffic conditions. The dashboard displays information such as current traffic light timings, traffic flow, and congestion levels. Users can also manually adjust traffic light timings in case of emergencies or specific requirements.

[0052] Specific examples

[0053] Morning rush hour scenario

[0054] 1. Traffic image acquisition (terminal)

[0055] During the morning rush hour, a camera captures video in real time at an intersection.

[0056] The device sends this video data to the server every 5 seconds.

[0057] 2. Data analysis (server)

[0058] The server analyzes the received video and detects 30% more vehicles than usual.

[0059] The number of pedestrians was also confirmed to be more than double the usual amount.

[0060] 3. Signal timing optimization (server)

[0061] The server uses this data to predict that congestion will increase further over the next five minutes.

[0062] The green light will last 20 seconds longer than usual, and the timing of pedestrian signals will be extended by 10 seconds.

[0063] 4. Signal Timing Application (Terminal)

[0064] The server calculates the new signal timing and sends it to the traffic light, where it is applied immediately.

[0065] This will allow for smoother traffic flow and reduce congestion.

[0066] 5. Monitoring (User)

[0067] Managers can check the dashboard to see improvements in traffic conditions in real time.

[0068] You can also make manual adjustments if needed.

[0069] These measures make it possible to use traffic footage to optimize traffic light timing in real time, improving overall road efficiency.

[0070] The processing flow will be explained below.

[0071] Step 1:

[0072] Video shooting (device)

[0073] The device uses a camera to capture real-time images of intersections and major roads, capturing the images frame by frame and storing the data over time.

[0074] Step 2:

[0075] Data transmission (terminal)

[0076] The video data captured by the device is sent to the server at regular intervals using a streaming protocol, and is sent in real time.

[0077] Step 3:

[0078] Data reception (server)

[0079] The server receives traffic video data sent from the terminal, and the received data is temporarily stored in a buffer for subsequent analysis.

[0080] Step 4:

[0081] Image recognition (server)

[0082] The server uses AI to analyze the video data in the buffer, and image recognition algorithms analyze each frame to identify objects such as vehicles, pedestrians, and cyclists.

[0083] Step 5:

[0084] Traffic situation analysis (server)

[0085] The server obtains attribute data such as the location, speed, and direction of the identified objects and evaluates traffic flow and congestion. The analysis results are used to determine the current traffic situation in real time.

[0086] Step 6:

[0087] Pattern prediction (server)

[0088] The server uses past traffic data and current analysis results to predict future traffic patterns, using predictive models to calculate the likelihood of future congestion and jams.

[0089] Step 7:

[0090] Optimization calculation (server)

[0091] The server calculates the optimal timing for traffic lights based on traffic conditions and predictions, optimizing the duration of each phase (green, yellow, red) and setting the traffic light timing.

[0092] Step 8:

[0093] Command generation (server)

[0094] The server generates a command to send to the traffic light based on the calculation results, which includes the new traffic light timing.

[0095] Step 9:

[0096] Send command (server)

[0097] The server sends the generated command to the traffic light, which then transmits the command to the traffic light via the network.

[0098] Step 10:

[0099] Signal timing application (terminal)

[0100] The terminal (traffic light) applies the new signal timing according to the received command. The traffic light immediately starts operating based on the new settings.

[0101] Step 11:

[0102] Real-time monitoring (user)

[0103] Users monitor real-time traffic conditions using the system's dashboard, which displays information such as current traffic light timings, traffic flow, and congestion levels.

[0104] Step 12:

[0105] Feedback and manual adjustments (user)

[0106] If necessary, users can manually instruct the system to fine-tune signal timing, providing a means to respond to emergencies or specific requests.

[0107] Example 1

[0108] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0109] Current traffic management systems have fixed signal timing settings, making it difficult to optimize them to accommodate real-time traffic conditions, which can lead to traffic congestion and accidents. Furthermore, they lack the means to quickly respond to sudden changes in traffic conditions, making it difficult to alleviate sudden congestion and jams.

[0110] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0111] In this invention, the server includes a means for acquiring traffic images, a means for analyzing the acquired traffic images to determine traffic conditions, and a means for optimizing traffic light timing based on the traffic conditions. This makes it possible to grasp traffic conditions in real time and calculate and apply appropriate traffic light timing. Furthermore, by including a means for manually adjusting traffic light timing based on the analysis results, it is possible to quickly respond to sudden changes in traffic conditions.

[0112] "Traffic footage" is video data that records traffic flow and the movements of vehicles and pedestrians.

[0113] "Analysis" is the process of extracting specific information from the acquired video data and determining the traffic situation.

[0114] "Traffic conditions" refers to information that indicates the movement and status of vehicles, pedestrians, etc. in a specific location.

[0115] "Traffic light timing" refers to the duration and switching timing of each phase of a traffic light (green, yellow, red).

[0116] "Optimization" is the process of adjusting signal timing to achieve a specific goal (e.g., smooth traffic flow or fewer accidents).

[0117] "Apply" means to set the optimized signal timing to an actual traffic light.

[0118] The "means for manually adjusting signal timing" is a function that allows the user to manually change the signal timing as needed.

[0119] An "image recognition algorithm" is a computational method for identifying specific objects (e.g., vehicles, pedestrians) from video data.

[0120] "Pattern prediction" is the process of predicting future traffic conditions based on past data.

[0121] MODE FOR CARRYING OUT THE INVENTION

[0122] This invention relates to a traffic light timing optimization system that uses traffic video. This system acquires traffic video in real time, analyzes and predicts traffic conditions based on the video, and optimizes traffic light timing. Specifically, it consists of the following main components:

[0123] 1. Traffic image acquisition means (terminal)

[0124] 2. Data analysis method (server)

[0125] 3. Signal timing optimization means (server)

[0126] 4. Signal Timing Application Means (Terminal)

[0127] 5. Monitoring and Feedback Methods (Users)

[0128] Traffic image acquisition means (terminal)

[0129] The device is equipped with a high-resolution camera that captures images of intersections and major roads in real time. This camera operates 24 hours a day, capturing images day and night. The captured image data is compressed every five seconds and sent to a server. Communication is via an internet connection. For example, Sony's high-resolution security cameras can be used.

[0130] Data analysis method (server)

[0131] The server receives traffic video data sent from the device and temporarily stores it in a storage area. The server then analyzes the video data using AI image recognition algorithms such as YOLO and OpenCV. During this analysis, objects such as vehicles, pedestrians, and bicycles are identified and attribute data such as their location, speed, and direction are acquired. The acquired attribute data is stored on the server and used for subsequent analysis and prediction.

[0132] Signal timing optimization means (server)

[0133] The server uses traffic condition data obtained from the data analysis tool to perform calculations to optimize traffic light timing. It uses past data and a pattern prediction algorithm to predict traffic congestion over the next five minutes. For example, it can extend the green light duration by 20 seconds and the pedestrian signal timing by 10 seconds to smooth traffic flow.

[0134] Signal timing application means (terminal)

[0135] The optimized signal timing calculated by the server is sent to the traffic light via the terminal. The traffic light changes its signal in real time according to the received signal timing. The traffic light is controlled by sending instructions to the signal control unit using the TCP / IP stack.

[0136] Monitoring and feedback measures (users)

[0137] Users monitor traffic conditions in real time through the system's dashboard, which visually displays current traffic light timings, traffic volume at each intersection, and pedestrian flow. Users can manually adjust traffic light timings as needed. For example, they may temporarily change traffic light timings to respond to specific events or emergencies. System parameters can also be optimized based on real-time data feedback.

[0138] Specific examples

[0139] Morning rush hour scenario

[0140] 1. Traffic image acquisition (terminal)

[0141] During the morning rush hour, a camera captures video in real time at an intersection.

[0142] The device sends this video data to the server every 5 seconds.

[0143] 2. Data analysis (server)

[0144] The server analyzes the received video and detects 30% more vehicles than usual.

[0145] The server also verifies that the number of pedestrians is more than double the normal amount.

[0146] 3. Signal timing optimization (server)

[0147] The server uses this data to predict that congestion will increase further over the next five minutes.

[0148] The green light will last 20 seconds longer than usual, and the timing of pedestrian signals will be extended by 10 seconds.

[0149] 4. Signal Timing Application (Terminal)

[0150] The server calculates the new signal timing and sends it to the traffic light, where it is applied immediately.

[0151] The terminal adjusts traffic signals to ensure smooth vehicle flow and reduce congestion.

[0152] 5. Monitoring (User)

[0153] Managers can check the dashboard to see improvements in traffic conditions in real time.

[0154] The user manually fine-tunes as needed.

[0155] Example prompts for generative AI models

[0156] An example prompt for a traffic signal timing optimization system is as follows:

[0157] Prompt: Explain how to optimize traffic signals.

[0158]

[0159] Generative AI Model: Traffic signal optimization can be achieved by capturing traffic video in real time and analyzing it with AI algorithms. Based on the analyzed data, traffic signal timing can be calculated and applied to smooth traffic flow, reducing congestion and accidents.

[0160] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0161] Program processing flow

[0162] Each processing step will be explained by clearly showing the specific operation, input, and output.

[0163] Step 1: Acquiring and transmitting traffic images (terminal)

[0164] Input: Real-time camera footage monitoring traffic flow.

[0165] Specific operation: A high-resolution camera installed on the device captures images of intersections and major roads in real time. This camera operates 24 hours a day, capturing images day and night.

[0166] Data processing: The device compresses the acquired video data every 5 seconds (e.g., H.264 format) and converts it into a data size that can be transmitted.

[0167] Output: The compressed video data is sent to the server via the Internet.

[0168] Step 2: Receiving video data (server)

[0169] Input: Compressed video data sent from the device.

[0170] Specific operation: The server receives the video data transmitted from the terminal.

[0171] Data processing: After receiving, the data is checked for consistency and completeness, and temporarily stored in a database using a streaming API (e.g., FFmpeg).

[0172] Output: Saved video data.

[0173] Step 3: Data analysis and traffic situation evaluation (server)

[0174] Input: Stored video data.

[0175] Specific operation: The server uses YOLO (You Only Look Once) and OpenCV to analyze the received video data.

[0176] Data processing: Identify objects such as vehicles, pedestrians, and bicycles from video data and obtain their attribute data (position, speed, direction).

[0177] Output: Attribute data of identified objects.

[0178] Step 4: Calculate signal timing (server)

[0179] Input: Attribute data of the identified object.

[0180] How it works: The server uses the analysis results to calculate optimal traffic light timings. It uses historical data and pattern prediction algorithms to predict future traffic congestion.

[0181] Data processing: Calculations are made to adjust the duration of green and pedestrian signals based on congestion predictions.

[0182] Output: Optimized signal timing configuration data.

[0183] Step 5: Sending the calculation results (server)

[0184] Input: Optimized signal timing configuration data.

[0185] Specific operation: The server encodes the calculated optimized signal timing in JSON format and sends it to the terminal.

[0186] Data processing: Data is transmitted securely using the HTTPS protocol.

[0187] Output: Signal timing configuration data sent to the terminal.

[0188] Step 6: Apply signal timing (terminal)

[0189] Input: Signal timing configuration data sent from the server.

[0190] Specific operation: The terminal analyzes the received signal timing and applies it to the signal control unit.

[0191] Data processing: Send instructions to the signal control unit using the TCP / IP stack.

[0192] Output: Traffic light timing is optimized in real time.

[0193] Step 7: Monitoring and Feedback (Users)

[0194] Input: Real-time traffic and signal timing data.

[0195] Specific operation: The user monitors the traffic situation in real time through the system's dashboard.

[0196] Data processing: Visually display information such as current traffic light timings, traffic volume at each intersection, and pedestrian flow on the dashboard.

[0197] Output: The user manually adjusts the signal timing as needed to optimize the system settings.

[0198] (Application example 1)

[0199] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0200] To smooth traffic flow, alleviate congestion, and reduce accidents, it is necessary to analyze traffic conditions in real time with high accuracy and optimize traffic signal timing based on that analysis. Furthermore, with the spread of autonomous vehicles, coordination between vehicles and traffic signals is becoming even more important. However, current systems are not fully capable of achieving these goals, and the problems of traffic congestion and accidents remain unresolved. Furthermore, there is a lack of an information-sharing mechanism that allows autonomous vehicles to flexibly adapt to the traffic environment.

[0201] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0202] In this invention, the server includes means for acquiring traffic images, means for analyzing the acquired traffic images to determine traffic conditions, means for optimizing traffic light timings based on the traffic conditions, means for applying the optimized traffic light timings to the traffic lights, and means for communicating with autonomous vehicles and providing the optimized traffic light timings, thereby enabling autonomous vehicles to acquire optimized traffic light timing information in real time, making vehicle operation smoother and safer.

[0203] "Traffic video" refers to video data of intersections and major roads, captured in real time using cameras.

[0204] "Analysis" is the process of processing traffic video data and extracting attributes such as the location, speed, and direction of vehicles and pedestrians.

[0205] "Traffic conditions" refers to the movement and concentration of vehicles and pedestrians at a certain point, including the degree of congestion and flow in real time.

[0206] "Traffic light timing" refers to the duration of each green, yellow, and red phase of a traffic signal, and is adjusted to optimize traffic flow.

[0207] "Optimization" refers to adjusting the timing of traffic lights based on current traffic conditions, aiming to smooth traffic flow and alleviate congestion.

[0208] An "autonomous vehicle" is a vehicle that perceives its surroundings and drives autonomously without the intervention of a human driver.

[0209] "Communication" refers to the process of exchanging data between the autonomous vehicle and a server, sharing information about traffic signal timings and traffic conditions.

[0210] "Provision" refers to the act of the server sharing and transmitting necessary data and information to an autonomous vehicle.

[0211] This invention relates to a traffic light timing optimization system using traffic video. This system has the function of acquiring traffic video in real time, analyzing and predicting it, and optimizing traffic light timing. The optimized traffic light timing is also provided to autonomous vehicles, making vehicle operation smoother and safer.

[0212] Overall system overview

[0213] The system consists of the following main components:

[0214] 1. Traffic image acquisition means (terminal)

[0215] 2. Data analysis method (server)

[0216] 3. Signal timing optimization means (server)

[0217] 4. Signal Timing Application Means (Terminal)

[0218] 5. Monitoring and Feedback Methods (Users)

[0219] 6. Communication method with autonomous vehicles (server)

[0220] Traffic image acquisition means (terminal)

[0221] The terminals are equipped with high-resolution cameras that capture real-time images of intersections and major roads. The cameras can capture images day and night, and the terminals are equipped with communication functions to periodically send the captured image data to a server.

[0222] Data analysis method (server)

[0223] The server receives traffic video data sent from the device and analyzes it using the open-source image processing library "OpenCV." "TENSORFLOW (registered trademark)" is used as the AI ​​model to identify objects such as vehicles, pedestrians, and bicycles from the video data and obtain attribute data such as their position, speed, and direction.

[0224] Signal timing optimization means (server)

[0225] The server performs calculations to optimize traffic light timing based on traffic condition data obtained from the data analysis means. Based on past data and current conditions, it predicts future traffic flow and congestion and calculates the optimal duration of green, yellow, and red lights.

[0226] Signal timing application means (terminal)

[0227] The optimized signal timing calculated by the server is sent to the traffic light via the application means. The terminal (traffic light) changes the signal in real time according to the received signal timing. This will result in smoother traffic flow, and is expected to alleviate traffic congestion and reduce accidents.

[0228] Monitoring and feedback measures (users)

[0229] Users can use the system's monitoring function to monitor real-time traffic conditions. The dashboard displays information such as current traffic light timings, traffic flow, and congestion levels. Users can also manually adjust traffic light timings in case of emergencies or specific requirements.

[0230] Communication method with autonomous vehicles (server)

[0231] The server communicates with the autonomous vehicles and provides optimized traffic light timing in real time, allowing the autonomous vehicles to operate safely and smoothly with the latest traffic information.

[0232] Specific examples

[0233] Morning rush hour scenario

[0234] Traffic video capture: During the morning rush hour, a camera captures video in real time at an intersection. The device sends this video data to a server every 5 seconds.

[0235] Data analysis: The server analyzes the received video and detects 30% more vehicles than usual. The number of pedestrians is also confirmed to be more than double the usual number.

[0236] Traffic light timing optimization: Using this data, the server predicts that congestion will increase further over the next five minutes, so it extends the green light duration by 20 seconds and the pedestrian signal timing by 10 seconds.

[0237] Signal Timing Application: The server calculates new signal timings and sends them to the traffic lights, where they are applied immediately. This allows for smoother traffic flow and reduces congestion.

[0238] Monitoring: Administrators can view the dashboard to see improvements in traffic conditions in real time, and can make manual adjustments as needed.

[0239] Example prompt sentence:

[0240] Scenario:

[0241] 1. Traffic Video Capture: During peak hours, a vehicle's onboard camera captures video of an intersection.

[0242] The video data is transmitted to the server every 5 seconds.

[0243] 2. Data Analysis (Server): The server receives and analyzes the video.

[0244] It detects a 40% increase in vehicles and doubles the number of pedestrians compared to normal.

[0245] 3. Optimized Signal Timing Calculation (Server):

[0246] Predicting a further increase in congestion in the next 5 minutes, the server adjusts:

[0247] Green light duration is extended by 30 seconds and pedestrian signal timing by 15 seconds.

[0248] 4. Optimized Timing Application (Vehicle):

[0249] The optimized signal timing is sent back to the vehicle,

[0250] ensuring smooth traffic flow and reduced congestion.

[0251] 5. Monitoring (User): The vehicle's control system adapts in real-time, enhancing safety and efficiency.

[0252] These measures make it possible to use traffic footage to optimize traffic light timing in real time, improving overall road efficiency.

[0253] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0254] Step 1:

[0255] Traffic image acquisition (terminal):

[0256] A camera installed on the device captures images of intersections and major roads in real time. Images are captured day and night, and the image data from the camera is sent to the server periodically (e.g., every 5 seconds). The input is real-time traffic image data, and the output is the data transferred to the server.

[0257] Step 2:

[0258] Data reception (server):

[0259] The server receives traffic video data sent from the terminal. This data is added to a queue for image analysis and prepared for the next processing step. The input is video data from the terminal, and the output is data added to the analysis queue.

[0260] Step 3:

[0261] Data analysis (server):

[0262] The server analyzes the video data using the open-source image processing library "OpenCV." It uses "TensorFlow" as an AI model to identify objects such as vehicles, pedestrians, and bicycles from the video data and obtain attribute data such as their position, speed, and direction. The input is the received traffic video data, and the output is the analyzed traffic attribute data.

[0263] Step 4:

[0264] Signal timing optimization (server):

[0265] The server performs calculations to optimize traffic light timing based on the analyzed traffic attribute data. Based on past data and current conditions, it predicts future traffic flow and congestion and calculates the optimal durations of green, yellow, and red lights. The input is the analyzed traffic attribute data and past traffic data, and the output is optimized traffic light timing.

[0266] Step 5:

[0267] Signal Timing Application (Terminal):

[0268] The optimized signal timing calculated by the server is sent to the terminal (traffic light). The traffic light changes the signal in real time according to the received timing data. The input is the optimized signal timing, and the output is the traffic light timing change.

[0269] Step 6:

[0270] Monitoring (User):

[0271] Users can use the system's monitoring function to monitor real-time traffic conditions. The dashboard displays information such as current traffic light timing, traffic flow, and congestion level. Users can also manually adjust traffic light timing if necessary. The input is real-time traffic data and traffic light timing information, and the output is the user's monitoring results and feedback.

[0272] Step 7:

[0273] Providing optimized signal timing (server):

[0274] The server communicates with the autonomous vehicle and provides optimized signal timing in real time, allowing the autonomous vehicle to operate safely and smoothly using the latest traffic information. The input is optimized signal timing, and the output is data provided to the autonomous vehicle.

[0275] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0276] This invention combines an emotion engine with a traffic light timing optimization system that uses traffic video. This system acquires traffic video in real time, analyzes and predicts traffic conditions based on that data, and optimizes traffic light timing. It also recognizes user emotions and reflects them in traffic signal timing, achieving more advanced traffic management.

[0277] Overall system overview

[0278] The system consists of the following main components:

[0279] 1. Traffic image acquisition means (terminal)

[0280] 2. Data analysis method (server)

[0281] 3. Signal timing optimization means (server)

[0282] 4. Signal Timing Application Means (Terminal)

[0283] 5. Emotion Engine

[0284] 6. Monitoring and Feedback Methods (Users)

[0285] Traffic image acquisition means (terminal)

[0286] The terminals are equipped with cameras that capture images of intersections and major roads in real time. These cameras are capable of capturing images day and night, and the terminals are equipped with communication functions to send the captured image data to a server.

[0287] Data analysis method (server)

[0288] The server receives traffic video data sent from the device. The received data is analyzed using an AI-based image recognition algorithm. This analysis identifies objects such as vehicles, pedestrians, and bicycles from the video data and obtains attribute data such as their location, speed, and direction.

[0289] Signal timing optimization means (server)

[0290] The server performs calculations to optimize traffic signal timing based on traffic condition data obtained from the data analysis means. It takes into account past data and current conditions to predict future traffic flow and congestion. It calculates the optimal duration for each phase of the traffic signal (green, yellow, red).

[0291] Signal timing application means (terminal)

[0292] The optimized signal timing calculated by the server is sent to the traffic light via the application means. The terminal (traffic light) changes the signal in real time according to the received signal timing. This will result in smoother traffic flow, and is expected to alleviate traffic congestion and reduce accidents.

[0293] Emotion Engine

[0294] The emotion engine has the function of identifying user emotions and reflecting them in traffic management. Emotions are analyzed in real time, for example, from the user's voice and video. Based on the analysis results, the frustration and stress levels of road users are determined and reflected in adjustments to traffic signal timing.

[0295] Monitoring and feedback measures (users)

[0296] Users can use the system's monitoring function to monitor real-time traffic conditions. The dashboard displays information such as current traffic light timings, traffic flow, and congestion levels. Data from the emotion engine is also displayed to help users manually adjust traffic light timings. Users can also manually adjust traffic light timings in emergencies or according to specific requirements.

[0297] Specific examples

[0298] Morning rush hour scenario

[0299] 1. Traffic image acquisition (terminal)

[0300] During the morning rush hour, a camera captures video in real time at an intersection.

[0301] The device sends this video data to the server every 5 seconds.

[0302] 2. Data analysis (server)

[0303] The server analyzes the received video and detects 30% more vehicles than usual.

[0304] The number of pedestrians was also confirmed to be more than double the usual amount.

[0305] 3. Signal timing optimization (server)

[0306] The server uses this data to predict that congestion will increase further over the next five minutes.

[0307] The green light will last 20 seconds longer than usual, and the timing of pedestrian signals will be extended by 10 seconds.

[0308] 4. Signal Timing Application (Terminal)

[0309] The server calculates the new signal timing and sends it to the traffic light, where it is applied immediately.

[0310] This will allow for smoother traffic flow and reduce congestion.

[0311] 5. Utilizing the Emotion Engine (User)

[0312] Managers check the monitoring dashboard and view sentiment data along with traffic conditions.

[0313] The emotion engine detects high stress levels in the user and further fine-tunes signal timing to reflect this.

[0314] Road users' emotions are also monitored and, if necessary, traffic light adjustments are made to have a relaxing effect.

[0315] This will not only ease traffic congestion but also reduce the psychological burden on road users.

[0316] The processing flow will be explained below.

[0317] Step 1:

[0318] Video shooting (device)

[0319] The device uses a camera to capture real-time images of intersections and major roads, capturing the images frame by frame and storing the data over time.

[0320] Step 2:

[0321] Data transmission (terminal)

[0322] The video data captured by the device is sent to the server at regular intervals using a streaming protocol, and is sent in real time.

[0323] Step 3:

[0324] Data reception (server)

[0325] The server receives traffic video data sent from the terminal, and the received data is temporarily stored in a buffer for subsequent analysis.

[0326] Step 4:

[0327] Image recognition (server)

[0328] The server uses AI to analyze the video data in the buffer, and image recognition algorithms analyze each frame to identify objects such as vehicles, pedestrians, and cyclists.

[0329] Step 5:

[0330] Traffic situation analysis (server)

[0331] The server obtains attribute data such as the location, speed, and direction of the identified objects and evaluates traffic flow and congestion. The analysis results are used to determine the current traffic situation in real time.

[0332] Step 6:

[0333] Pattern prediction (server)

[0334] The server uses past traffic data and current analysis results to predict future traffic patterns, using predictive models to calculate the likelihood of future congestion and jams.

[0335] Step 7:

[0336] Optimization calculation (server)

[0337] The server calculates the optimal timing for traffic lights based on traffic conditions and predictions, optimizing the duration of each phase (green, yellow, red) and setting the traffic light timing.

[0338] Step 8:

[0339] Command generation (server)

[0340] The server generates a command to send to the traffic light based on the calculation results, which includes the new traffic light timing.

[0341] Step 9:

[0342] Send command (server)

[0343] The server sends the generated command to the traffic light, which then transmits the command to the traffic light via the network.

[0344] Step 10:

[0345] Signal timing application (terminal)

[0346] The terminal (traffic light) applies the new signal timing according to the received command. The traffic light immediately starts operating based on the new settings.

[0347] Step 11:

[0348] Emotion data collection (user)

[0349] Using an emotion engine, emotional data is obtained from the user's voice and video, analyzing the user's stress level and frustration and collecting emotional data in real time.

[0350] Step 12:

[0351] Emotion data analysis (server)

[0352] The server receives and analyzes the emotion data sent from the emotion engine, evaluates the user's emotional state indicated by the emotion data, and reflects this in traffic situation analysis.

[0353] Step 13:

[0354] Emotion-based signal timing adjustment (server)

[0355] The server analyzes the emotional data and fine-tunes the timing of the traffic lights. For example, if the user's stress level is high, the system will extend the duration of the green light.

[0356] Step 14:

[0357] Monitoring (users)

[0358] Users monitor real-time traffic conditions and emotion data using the system's dashboard, which displays information such as current traffic light timings, traffic flow, congestion levels, and the user's emotional state.

[0359] Step 15:

[0360] Feedback and manual adjustments (user)

[0361] If necessary, users can manually instruct the system to fine-tune signal timing, providing a means to respond to emergencies or specific requests.

[0362] Example 2

[0363] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0364] Optimization of traffic signals in transportation systems relies on analysis of traffic conditions and timing adjustment, but conventional systems do not take user emotions into consideration, resulting in problems such as user stress and frustration even when traffic flow is smooth.In addition, adjustment of traffic signals based on real-time traffic video and data analysis was limited, so there was a need for more precise and rapid optimization.

[0365] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0366] In this invention, the server includes means for acquiring traffic images, means for analyzing the acquired traffic images and determining traffic conditions, means for optimizing traffic light timing based on the traffic conditions, means for recognizing a user's emotions and adjusting traffic light timing based on the recognition results, and means for applying the optimized traffic light timing to the traffic lights. This enables real-time analysis and optimization of traffic conditions and also enables adjustments that take the user's emotions into consideration, thereby not only smoothing traffic flow but also reducing the psychological burden on the user.

[0367] "Traffic footage" refers to video data capturing traffic conditions, and records the movement of vehicles, pedestrians, bicycles, etc. at intersections and major roads in real time.

[0368] "Traffic conditions" refers to information that indicates the degree of congestion and movement of vehicles, pedestrians, bicycles, etc. on roads at a specific time and place.

[0369] "Traffic light timing" refers to the time it takes for a traffic light to change color (green, yellow, red), including the duration of the change to control traffic flow.

[0370] An "image recognition algorithm" is a computer algorithm that analyzes video data acquired from a camera and identifies specific objects (vehicles, pedestrians, bicycles, etc.).

[0371] "Pattern prediction" is a mathematical method for analyzing past traffic data and predicting current and future traffic conditions.

[0372] "Emotion recognition" is a technology that analyzes data such as a user's voice and facial expressions to identify their psychological state (stress, frustration, joy, etc.).

[0373] "Timing optimization" involves calculating the optimal duration of traffic light colors based on traffic conditions and user emotion data, and making adjustments to ensure the most efficient traffic flow.

[0374] "Monitoring" is the process of monitoring traffic conditions, traffic light timing, and user emotional data in real time and making adjustments as needed.

[0375] This invention not only optimizes traffic signal timing using traffic video, but also realizes advanced traffic management using user emotional data. This system captures traffic video in real time, analyzes the video data, and judges traffic conditions. Based on this judgment, it optimizes traffic signal timing and also reflects user emotional data, enabling efficient traffic management with reduced psychological burden.

[0376] Hardware and software used

[0377] Traffic image acquisition means (terminal)

[0378] The terminal is equipped with a camera that captures images of intersections and major roads in real time. This camera has the ability to capture images day and night, and sends the captured image data to a server. Specifically, it uses the H.264 codec to compress the image data and the SSL / TLS communication protocol for data communication.

[0379] Data analysis method (server)

[0380] The server receives traffic video data sent from the device, decodes the received data, and analyzes it using an AI-based image recognition algorithm (such as YOLO or OpenCV). This analysis identifies objects such as vehicles, pedestrians, and bicycles, and obtains attribute data such as location, speed, and direction.

[0381] Traffic condition data generation (server)

[0382] The server generates current traffic situation data based on the analysis results. The number of vehicles, number of pedestrians, and the speed and direction of each object are converted into statistical data and updated in real time. By storing this data in a database, it is possible to compare it with past data.

[0383] Signal timing optimization (server)

[0384] The server optimizes traffic signal timing based on traffic condition data. The optimization algorithm uses optimization theory and feedback control to calculate the duration of each phase (green, yellow, red) and update the timing in real time, including forecast data for the next five minutes.

[0385] Signal timing application means (terminal)

[0386] The optimized signal timing is sent from the server to the traffic lights, which then make real-time signal changes based on the received timing. To minimize delays, a low-latency communication protocol (e.g. MQTT) is used. The traffic light control software applies the new timing for each phase.

[0387] Emotion Engine

[0388] The emotion engine recognizes the user's emotions and reflects them in traffic management. Emotion data is acquired using a camera or microphone installed on the device, and the acquired audio and video data is analyzed on a server. Specifically, an emotion recognition algorithm (such as IBM Watson (registered trademark) or Microsoft (registered trademark) Azure (registered trademark) emotion recognition API) is used. Based on the emotion data, the level of stress or anger is determined and traffic light timing is adjusted.

[0389] Monitoring and Feedback (User)

[0390] Users can view real-time traffic conditions and sentiment data on a monitoring dashboard, which displays current traffic signal timings, traffic flow, congestion levels, and user sentiment data. Users can also manually adjust traffic signal timings in case of emergencies or specific requirements.

[0391] Examples of concrete examples and prompts

[0392] Specific examples

[0393] During the morning rush hour, cameras capture traffic footage in real time and send the acquired video data to a server. The server analyzes the data and detects an increase in vehicles and pedestrians compared to normal. It predicts that congestion will increase further over the next five minutes and makes adjustments such as extending the duration of green lights. User emotion data is also collected, and if high stress levels are detected, further fine-tuning of signal timing is made. Administrators can monitor this from a dashboard and make manual adjustments as needed.

[0394] Prompt Sentence Examples

[0395] An example prompt is:

[0396] Write a program that analyzes current traffic video data and optimizes traffic light timing. Specifically, for intersection A, with 50 vehicles and 30 pedestrians, extend the current green light by 10 seconds and shorten the pedestrian signal timing by 5 seconds. Also, further fine-tune the traffic light timing based on emotion data.

[0397] This system enables real-time analysis and optimization of traffic conditions, and more effective traffic management can be achieved by making adjustments that take into account user emotional data.

[0398] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0399] Step 1:

[0400] Traffic image acquisition (terminal)

[0401] The device is equipped with a camera that captures images of intersections and major roads in real time. Specifically, the camera captures images at 30 frames per second (fps) and continues to capture images at night using infrared functionality. The input is real-time images of intersections and roads, and the output is video data compressed using the H.264 codec.

[0402] Step 2:

[0403] Video data transmission (terminal)

[0404] The terminal transmits the acquired video data to the server. Specifically, the acquired video data is compressed every second and transmitted to the server using the SSL / TLS communication protocol. The input is video data compressed using H.264, and the output is encrypted data packets.

[0405] Step 3:

[0406] Traffic video analysis (server)

[0407] The server decodes the video data received from the device and analyzes traffic conditions by using AI for image recognition. YOLO and OpenCV are used as image recognition algorithms. Specifically, it identifies objects such as vehicles, pedestrians, and bicycles from the video data and determines their respective positions, speeds, and directions. The input is the video data sent to the server, and the output is attribute data for each object (position, speed, direction, etc.).

[0408] Step 4:

[0409] Traffic condition data generation (server)

[0410] The server generates current traffic situation data based on the analysis results. It converts the number of vehicles, the number of pedestrians, and the speed and direction of each object into statistical data and updates it in real time. Specifically, it saves the analysis results in list format and records them in a database. The input is the analysis results of the traffic video, and the output is statistical traffic situation data.

[0411] Step 5:

[0412] Signal timing optimization (server)

[0413] The server optimizes signal timing based on traffic condition data. The optimization algorithm uses optimization theory and feedback control. Specifically, it calculates the optimal duration of each phase (green, yellow, red). The input is traffic condition data updated in real time and past data, and the output is optimized signal timing data.

[0414] Step 6:

[0415] Optimized timing transmission (server)

[0416] The server transmits the optimized signal timing to the traffic lights using a low-latency communication protocol (e.g., MQTT). Specifically, it creates optimized timing data packets and sends them to the traffic lights. The input is the optimized signal timing data, and the output is the new signal timing that is applied to the traffic light control software.

[0417] Step 7:

[0418] Timing application to traffic lights (terminal)

[0419] The terminal (traffic light) changes the signal in real time according to the received signal timing. For example, the delay when changing from red to green is kept within a few milliseconds. The input is the new signal timing data sent from the server, and the output is the changed color of the traffic light.

[0420] Step 8:

[0421] Acquisition and analysis of emotion data (terminal / server)

[0422] The device (for example, an emotion-recognition camera or microphone) acquires the user's voice and video data. The server analyzes this data and recognizes the user's emotions using an emotion engine (for example, IBM Watson or Microsoft Azure's emotion recognition API). Specifically, it detects the level of stress or anger from the user's voice and facial expression. The input is the user's voice and video data, and the output is analyzed emotional data.

[0423] Step 9:

[0424] Adjustment of signal timing based on emotion data (server)

[0425] The server further adjusts the signal timing based on the acquired emotion data. For example, if the user's stress level is high, the algorithm adjusts to shorten the duration of the green light. The input is the analyzed emotion data in real time, and the output is fine-tuned signal timing data.

[0426] Step 10:

[0427] Monitoring and Feedback (User)

[0428] Users can check real-time traffic conditions and emotion data on a monitoring dashboard. The dashboard displays current traffic signal timings, traffic flow, congestion levels, and user emotion data. Users can also manually adjust traffic signal timings in emergencies or according to specific requirements. The inputs are traffic condition data and emotion data, and the output is the traffic signal timings manually adjusted by the user.

[0429] (Application example 2)

[0430] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0431] Conventional traffic management systems focus on analyzing traffic images and optimizing traffic signal timing, but real-time understanding and adaptation to traffic conditions is required to ensure smooth traffic flow. Furthermore, one of the factors that contribute to traffic accidents and congestion is the emotion and stress of drivers and pedestrians, yet few systems exist that take this into account. Therefore, new systems are needed that can optimize traffic flow while reducing the psychological burden on road users. Furthermore, with the spread of autonomous vehicles, new traffic management systems that can accommodate this are also required.

[0432] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0433] In this invention, the server includes means for acquiring traffic video, means for analyzing the traffic video and determining traffic conditions, means for optimizing traffic light timing based on the traffic conditions, means for applying the optimized traffic light timing to the traffic lights, means for analyzing user emotions and adjusting traffic light timing based on the emotions, and means for monitoring surrounding traffic conditions in real time using video acquired by a camera in the autonomous vehicle and optimizing the driving pattern of the autonomous vehicle based on the analysis results, thereby enabling optimization of traffic flow, flexible traffic management in response to the emotions of road users, and safe and efficient operation of the autonomous vehicle.

[0434] The "means for acquiring traffic images" refers to a device that uses cameras installed at locations such as intersections and major roads to acquire images of traffic conditions in real time.

[0435] The "means for analyzing traffic images and determining traffic conditions" is a device that uses an AI-based image recognition algorithm to analyze the status of vehicles, pedestrians, and traffic lights based on acquired traffic images, and determines the current traffic conditions.

[0436] A "means for optimizing traffic signal timing based on traffic conditions" is a device that uses analyzed traffic condition data to calculate the duration of green, yellow, and red signals at a traffic signal and optimize traffic flow.

[0437] The "means for applying optimized signal timing to traffic lights" is a device that transmits the calculated optimal signal timing to actual traffic lights and changes the signals in real time.

[0438] The "means for analyzing the user's emotions and adjusting the timing of traffic lights based on those emotions" refers to a device that analyzes emotions from the user's voice and video, and reflects that data in a traffic management system, thereby adjusting the timing of traffic lights based on the user's emotions.

[0439] "Means for monitoring surrounding traffic conditions in real time using images captured by a camera on an autonomous vehicle" refers to a device that uses a camera mounted on an autonomous vehicle to capture images of traffic conditions, acquires the image data in real time, and monitors surrounding traffic.

[0440] The "means for optimizing the driving pattern of an autonomous vehicle based on the analysis results" refers to a device that adjusts the speed and route of an autonomous vehicle to the optimum based on the analysis results of the acquired video data.

[0441] This invention combines an emotion engine with a function for linking with autonomous vehicles to optimize traffic light timing using traffic video. This system acquires traffic video in real time, analyzes and predicts the data, and optimizes traffic light timing. Furthermore, by analyzing user emotions and reflecting them in traffic light timing, it is possible to enhance traffic management and optimize the operation of autonomous vehicles.

[0442] Overall system overview

[0443] The system consists of the following main components:

[0444] 1. Means for acquiring traffic images (devices including cameras)

[0445] 2. A means of analyzing data and determining traffic conditions (server)

[0446] 3. Means for optimizing traffic light timing (server)

[0447] 4. Means (terminal) for applying optimized signal timing to traffic lights

[0448] 5. A means to analyze user emotions and adjust traffic light timing based on those emotions (emotion engine)

[0449] 6. Means of monitoring surrounding traffic conditions using images captured by cameras in autonomous vehicles (cameras and systems for autonomous vehicles)

[0450] 7. Means for optimizing the driving patterns of autonomous vehicles based on analysis results (autonomous driving systems)

[0451] Hardware and software used

[0452] The following hardware and software are used to implement this system.

[0453] Hardware: Cameras, autonomous vehicle cameras, servers, traffic light control terminals

[0454] Software: OpenCV (image analysis), TensorFlow (AI modeling), Python scripts (data analysis), ROS (Robot Operating System, integration with autonomous driving systems)

[0455] Program processing explanation

[0456] The server first receives real-time video data from cameras to capture traffic images. It then analyzes the received video data using OpenCV and TensorFlow to determine traffic conditions. Traffic data includes the position, speed, and direction of objects such as vehicles, pedestrians, and bicycles. It then performs calculations to optimize traffic light timing based on the traffic conditions and sends the optimal signal timing to the traffic light control terminal.

[0457] Meanwhile, an emotion engine is used to analyze user emotions. This engine collects passenger emotional data from audio and video and analyzes it in real time. This emotional data is used to fine-tune signal timing, setting optimal signal timing to reduce passenger stress.

[0458] Finally, the autonomous vehicle's camera continuously monitors the surrounding traffic conditions and analyzes the acquired data in real time. Using ROS, the system instructs the autonomous driving system on optimal driving patterns, helping to ensure smooth traffic flow. In particular, taking into account passenger emotional data can help ensure a comfortable and safe operation.

[0459] Adding concrete examples and prompt sentences for the generative AI model

[0460] For example, consider the following system situation:

[0461] 1. Traffic video acquisition: Cameras on autonomous vehicles capture real-time traffic conditions at intersections and transmit the video data to a server inside the vehicle.

[0462] 2. Data analysis: The server analyzes the received video and determines traffic flow based on information such as the location, speed, and direction of identified vehicles and pedestrians.

[0463] 3. Optimization: Based on the analysis results, the speed and route of autonomous vehicles are optimized and traffic light timing is adjusted.

[0464] 4. Emotion analysis: Analyzes passengers' emotions in real time from their voice data and fine-tunes traffic signal timing to ensure stress-free driving.

[0465] Example prompt for a generative AI model:

[0466] Analyze traffic video data in real time to identify vehicles, pedestrians, and bicycles. Also, analyze passengers' emotions based on their voice data and select driving patterns according to their stress levels. Use TensorFlow and OpenCV for video data analysis, and an emotion engine for emotion analysis. Use ROS for integration with the autonomous driving system.

[0467] According to the above aspects, the present invention can optimize traffic flow, reduce the psychological burden on traffic users, and realize safe and efficient operation of autonomous vehicles.

[0468] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0469] Step 1:

[0470] Traffic footage acquisition

[0471] The device captures real-time traffic images using cameras installed at intersections and major roads. This image data is also captured by cameras on autonomous vehicles. The captured image data is sent directly to the server. The input data is the image data from the cameras, and the output data is the image data sent to the server.

[0472] Step 2:

[0473] Traffic video analysis

[0474] The server processes the received traffic video data. Specifically, it performs image analysis using OpenCV and TensorFlow. Image analysis identifies objects such as vehicles, pedestrians, and bicycles, and calculates their respective positions, speeds, and directions. The input data is the traffic video data, and the output data is attribute data of the identified objects.

[0475] Step 3:

[0476] Judging traffic conditions

[0477] The server determines the current traffic situation based on the image analysis results. Specifically, it calculates traffic flow and congestion levels based on the position, speed, and direction of each object. This makes it possible to identify traffic bottlenecks and dangerous areas. The input data is the attribute data of the identified objects, and the output data is the traffic situation assessment result.

[0478] Step 4:

[0479] Traffic light timing optimization

[0480] The server optimizes traffic light timing based on the traffic conditions it determines. It also takes into account past data and current conditions to calculate the duration of each traffic light phase (green, yellow, red). The input data is the traffic condition determination result, and the output data is the optimized traffic light timing.

[0481] Step 5:

[0482] Signal Timing Enforcement

[0483] The terminal applies the optimized signal timing received from the server to the actual traffic light, which changes the signal in real time and ensures smooth traffic flow. The input data is the optimized signal timing, and the output data is the result of the traffic light timing change.

[0484] Step 6:

[0485] User sentiment analysis

[0486] The emotion engine analyzes the user's (passenger's) voice and video data in real time to determine emotions such as stress and frustration, thereby understanding the user's psychological state. The input data is the user's voice and video data, and the output data is the analyzed emotion data.

[0487] Step 7:

[0488] Fine-tuning signal timing

[0489] The server fine-tunes the optimized signal timing based on the emotion data obtained from the emotion engine. This sets the optimal signal timing to reduce the user's stress. The input data is the analyzed emotion data, and the output data is the fine-tuned signal timing.

[0490] Step 8:

[0491] Optimizing driving patterns for autonomous vehicles

[0492] The autonomous vehicle system optimizes driving patterns based on acquired traffic video data and emotion data. Specifically, it uses ROS to adjust the vehicle's speed and route. The input data is traffic video data and emotion data, and the output data is the optimized driving pattern.

[0493] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0494] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0495] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0496] [Second embodiment]

[0497] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0498] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0499] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0500] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0501] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0502] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0503] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0504] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0505] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0506] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0507] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0508] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0509] This invention relates to a traffic light timing optimization system that uses traffic video. This system acquires traffic video in real time, analyzes and predicts traffic conditions based on the video, and optimizes traffic light timing.

[0510] Overall system overview

[0511] The system consists of the following main components:

[0512] 1. Traffic image acquisition means (terminal)

[0513] 2. Data analysis method (server)

[0514] 3. Signal timing optimization means (server)

[0515] 4. Signal Timing Application Means (Terminal)

[0516] 5. Monitoring and Feedback Methods (Users)

[0517] Traffic image acquisition means (terminal)

[0518] The terminals are equipped with cameras that capture images of intersections and major roads in real time. These cameras are capable of capturing images day and night, and the terminals are equipped with communication functions to send the captured image data to a server.

[0519] Data analysis method (server)

[0520] The server receives traffic video data sent from the device. The received data is analyzed using an AI-based image recognition algorithm. This analysis identifies objects such as vehicles, pedestrians, and bicycles from the video data and obtains attribute data such as their location, speed, and direction.

[0521] Signal timing optimization means (server)

[0522] The server performs calculations to optimize traffic signal timing based on traffic condition data obtained from the data analysis means. It takes into account past data and current conditions to predict future traffic flow and congestion. It calculates the optimal duration for each phase of the traffic signal (green, yellow, red).

[0523] Signal timing application means (terminal)

[0524] The optimized signal timing calculated by the server is sent to the traffic light via the application means. The terminal (traffic light) changes the signal in real time according to the received signal timing. This will result in smoother traffic flow, and is expected to alleviate traffic congestion and reduce accidents.

[0525] Monitoring and feedback measures (users)

[0526] Users can use the system's monitoring function to monitor real-time traffic conditions. The dashboard displays information such as current traffic light timings, traffic flow, and congestion levels. Users can also manually adjust traffic light timings in case of emergencies or specific requirements.

[0527] Specific examples

[0528] Morning rush hour scenario

[0529] 1. Traffic image acquisition (terminal)

[0530] During the morning rush hour, a camera captures video in real time at an intersection.

[0531] The device sends this video data to the server every 5 seconds.

[0532] 2. Data analysis (server)

[0533] The server analyzes the received video and detects 30% more vehicles than usual.

[0534] The number of pedestrians was also confirmed to be more than double the usual amount.

[0535] 3. Signal timing optimization (server)

[0536] The server uses this data to predict that congestion will increase further over the next five minutes.

[0537] The green light will last 20 seconds longer than usual, and the timing of pedestrian signals will be extended by 10 seconds.

[0538] 4. Signal Timing Application (Terminal)

[0539] The server calculates the new signal timing and sends it to the traffic light, where it is applied immediately.

[0540] This will allow for smoother traffic flow and reduce congestion.

[0541] 5. Monitoring (User)

[0542] Managers can check the dashboard to see improvements in traffic conditions in real time.

[0543] You can also make manual adjustments if needed.

[0544] These measures make it possible to use traffic footage to optimize traffic light timing in real time, improving overall road efficiency.

[0545] The processing flow will be explained below.

[0546] Step 1:

[0547] Video shooting (device)

[0548] The device uses a camera to capture real-time images of intersections and major roads, capturing the images frame by frame and storing the data over time.

[0549] Step 2:

[0550] Data transmission (terminal)

[0551] The video data captured by the device is sent to the server at regular intervals using a streaming protocol, and is sent in real time.

[0552] Step 3:

[0553] Data reception (server)

[0554] The server receives traffic video data sent from the terminal, and the received data is temporarily stored in a buffer for subsequent analysis.

[0555] Step 4:

[0556] Image recognition (server)

[0557] The server uses AI to analyze the video data in the buffer, and image recognition algorithms analyze each frame to identify objects such as vehicles, pedestrians, and cyclists.

[0558] Step 5:

[0559] Traffic situation analysis (server)

[0560] The server obtains attribute data such as the location, speed, and direction of the identified objects and evaluates traffic flow and congestion. The analysis results are used to determine the current traffic situation in real time.

[0561] Step 6:

[0562] Pattern prediction (server)

[0563] The server uses past traffic data and current analysis results to predict future traffic patterns, using predictive models to calculate the likelihood of future congestion and jams.

[0564] Step 7:

[0565] Optimization calculation (server)

[0566] The server calculates the optimal timing for traffic lights based on traffic conditions and predictions, optimizing the duration of each phase (green, yellow, red) and setting the traffic light timing.

[0567] Step 8:

[0568] Command generation (server)

[0569] The server generates a command to send to the traffic light based on the calculation results, which includes the new traffic light timing.

[0570] Step 9:

[0571] Send command (server)

[0572] The server sends the generated command to the traffic light, which then transmits the command to the traffic light via the network.

[0573] Step 10:

[0574] Signal timing application (terminal)

[0575] The terminal (traffic light) applies the new signal timing according to the received command. The traffic light immediately starts operating based on the new settings.

[0576] Step 11:

[0577] Real-time monitoring (user)

[0578] Users monitor real-time traffic conditions using the system's dashboard, which displays information such as current traffic light timings, traffic flow, and congestion levels.

[0579] Step 12:

[0580] Feedback and manual adjustments (user)

[0581] If necessary, users can manually instruct the system to fine-tune signal timing, providing a means to respond to emergencies or specific requests.

[0582] Example 1

[0583] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0584] Current traffic management systems have fixed signal timing settings, making it difficult to optimize them to accommodate real-time traffic conditions, which can lead to traffic congestion and accidents. Furthermore, they lack the means to quickly respond to sudden changes in traffic conditions, making it difficult to alleviate sudden congestion and jams.

[0585] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0586] In this invention, the server includes a means for acquiring traffic images, a means for analyzing the acquired traffic images to determine traffic conditions, and a means for optimizing traffic light timing based on the traffic conditions. This makes it possible to grasp traffic conditions in real time and calculate and apply appropriate traffic light timing. Furthermore, by including a means for manually adjusting traffic light timing based on the analysis results, it is possible to quickly respond to sudden changes in traffic conditions.

[0587] "Traffic footage" is video data that records traffic flow and the movements of vehicles and pedestrians.

[0588] "Analysis" is the process of extracting specific information from the acquired video data and determining the traffic situation.

[0589] "Traffic conditions" refers to information that indicates the movement and status of vehicles, pedestrians, etc. in a specific location.

[0590] "Traffic light timing" refers to the duration and switching timing of each phase of a traffic light (green, yellow, red).

[0591] "Optimization" is the process of adjusting signal timing to achieve a specific goal (e.g., smooth traffic flow or fewer accidents).

[0592] "Apply" means to set the optimized signal timing to an actual traffic light.

[0593] The "means for manually adjusting signal timing" is a function that allows the user to manually change the signal timing as needed.

[0594] An "image recognition algorithm" is a computational method for identifying specific objects (e.g., vehicles, pedestrians) from video data.

[0595] "Pattern prediction" is the process of predicting future traffic conditions based on past data.

[0596] MODE FOR CARRYING OUT THE INVENTION

[0597] This invention relates to a traffic light timing optimization system that uses traffic video. This system acquires traffic video in real time, analyzes and predicts traffic conditions based on the video, and optimizes traffic light timing. Specifically, it consists of the following main components:

[0598] 1. Traffic image acquisition means (terminal)

[0599] 2. Data analysis method (server)

[0600] 3. Signal timing optimization means (server)

[0601] 4. Signal Timing Application Means (Terminal)

[0602] 5. Monitoring and Feedback Methods (Users)

[0603] Traffic image acquisition means (terminal)

[0604] The device is equipped with a high-resolution camera that captures images of intersections and major roads in real time. This camera operates 24 hours a day, capturing images day and night. The captured image data is compressed every five seconds and sent to a server. Communication is via an internet connection. For example, Sony's high-resolution security cameras can be used.

[0605] Data analysis method (server)

[0606] The server receives traffic video data sent from the device and temporarily stores it in a storage area. The server then analyzes the video data using AI image recognition algorithms such as YOLO and OpenCV. During this analysis, objects such as vehicles, pedestrians, and bicycles are identified and attribute data such as their location, speed, and direction are acquired. The acquired attribute data is stored on the server and used for subsequent analysis and prediction.

[0607] Signal timing optimization means (server)

[0608] The server uses traffic condition data obtained from the data analysis tool to perform calculations to optimize traffic light timing. It uses past data and a pattern prediction algorithm to predict traffic congestion over the next five minutes. For example, it can extend the green light duration by 20 seconds and the pedestrian signal timing by 10 seconds to smooth traffic flow.

[0609] Signal timing application means (terminal)

[0610] The optimized signal timing calculated by the server is sent to the traffic light via the terminal. The traffic light changes its signal in real time according to the received signal timing. The traffic light is controlled by sending instructions to the signal control unit using the TCP / IP stack.

[0611] Monitoring and feedback measures (users)

[0612] Users monitor traffic conditions in real time through the system's dashboard, which visually displays current traffic light timings, traffic volume at each intersection, and pedestrian flow. Users can manually adjust traffic light timings as needed. For example, they may temporarily change traffic light timings to respond to specific events or emergencies. System parameters can also be optimized based on real-time data feedback.

[0613] Specific examples

[0614] Morning rush hour scenario

[0615] 1. Traffic image acquisition (terminal)

[0616] During the morning rush hour, a camera captures video in real time at an intersection.

[0617] The device sends this video data to the server every 5 seconds.

[0618] 2. Data analysis (server)

[0619] The server analyzes the received video and detects 30% more vehicles than usual.

[0620] The server also verifies that the number of pedestrians is more than double the normal amount.

[0621] 3. Signal timing optimization (server)

[0622] The server uses this data to predict that congestion will increase further over the next five minutes.

[0623] The green light will last 20 seconds longer than usual, and the timing of pedestrian signals will be extended by 10 seconds.

[0624] 4. Signal Timing Application (Terminal)

[0625] The server calculates the new signal timing and sends it to the traffic light, where it is applied immediately.

[0626] The terminal adjusts traffic signals to ensure smooth vehicle flow and reduce congestion.

[0627] 5. Monitoring (User)

[0628] Managers can check the dashboard to see improvements in traffic conditions in real time.

[0629] The user manually fine-tunes as needed.

[0630] Example prompts for generative AI models

[0631] An example prompt for a traffic signal timing optimization system is as follows:

[0632] Prompt: Explain how to optimize traffic signals.

[0633]

[0634] Generative AI Model: Traffic signal optimization can be achieved by capturing traffic video in real time and analyzing it with AI algorithms. Based on the analyzed data, traffic signal timing can be calculated and applied to smooth traffic flow, reducing congestion and accidents.

[0635] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0636] Program processing flow

[0637] Each processing step will be explained by clearly showing the specific operation, input, and output.

[0638] Step 1: Acquiring and transmitting traffic images (terminal)

[0639] Input: Real-time camera footage monitoring traffic flow.

[0640] Specific operation: A high-resolution camera installed on the device captures images of intersections and major roads in real time. This camera operates 24 hours a day, capturing images day and night.

[0641] Data processing: The device compresses the acquired video data every 5 seconds (e.g., H.264 format) and converts it into a data size that can be transmitted.

[0642] Output: The compressed video data is sent to the server via the Internet.

[0643] Step 2: Receiving video data (server)

[0644] Input: Compressed video data sent from the device.

[0645] Specific operation: The server receives the video data transmitted from the terminal.

[0646] Data processing: After receiving, the data is checked for consistency and completeness, and temporarily stored in a database using a streaming API (e.g., FFmpeg).

[0647] Output: Saved video data.

[0648] Step 3: Data analysis and traffic situation evaluation (server)

[0649] Input: Stored video data.

[0650] Specific operation: The server uses YOLO (You Only Look Once) and OpenCV to analyze the received video data.

[0651] Data processing: Identify objects such as vehicles, pedestrians, and bicycles from video data and obtain their attribute data (position, speed, direction).

[0652] Output: Attribute data of identified objects.

[0653] Step 4: Calculate signal timing (server)

[0654] Input: Attribute data of the identified object.

[0655] How it works: The server uses the analysis results to calculate optimal traffic light timings. It uses historical data and pattern prediction algorithms to predict future traffic congestion.

[0656] Data processing: Calculations are made to adjust the duration of green and pedestrian signals based on congestion predictions.

[0657] Output: Optimized signal timing configuration data.

[0658] Step 5: Sending the calculation results (server)

[0659] Input: Optimized signal timing configuration data.

[0660] Specific operation: The server encodes the calculated optimized signal timing in JSON format and sends it to the terminal.

[0661] Data processing: Data is transmitted securely using the HTTPS protocol.

[0662] Output: Signal timing configuration data sent to the terminal.

[0663] Step 6: Apply signal timing (terminal)

[0664] Input: Signal timing configuration data sent from the server.

[0665] Specific operation: The terminal analyzes the received signal timing and applies it to the signal control unit.

[0666] Data processing: Send instructions to the signal control unit using the TCP / IP stack.

[0667] Output: Traffic light timing is optimized in real time.

[0668] Step 7: Monitoring and Feedback (Users)

[0669] Input: Real-time traffic and signal timing data.

[0670] Specific operation: The user monitors the traffic situation in real time through the system's dashboard.

[0671] Data processing: Visually display information such as current traffic light timings, traffic volume at each intersection, and pedestrian flow on the dashboard.

[0672] Output: The user manually adjusts the signal timing as needed to optimize the system settings.

[0673] (Application example 1)

[0674] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0675] To smooth traffic flow, alleviate congestion, and reduce accidents, it is necessary to analyze traffic conditions in real time with high accuracy and optimize traffic signal timing based on that analysis. Furthermore, with the spread of autonomous vehicles, coordination between vehicles and traffic signals is becoming even more important. However, current systems are not fully capable of achieving these goals, and the problems of traffic congestion and accidents remain unresolved. Furthermore, there is a lack of an information-sharing mechanism that allows autonomous vehicles to flexibly adapt to the traffic environment.

[0676] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0677] In this invention, the server includes means for acquiring traffic images, means for analyzing the acquired traffic images to determine traffic conditions, means for optimizing traffic light timings based on the traffic conditions, means for applying the optimized traffic light timings to the traffic lights, and means for communicating with autonomous vehicles and providing the optimized traffic light timings, thereby enabling autonomous vehicles to acquire optimized traffic light timing information in real time, making vehicle operation smoother and safer.

[0678] "Traffic video" refers to video data of intersections and major roads, captured in real time using cameras.

[0679] "Analysis" is the process of processing traffic video data and extracting attributes such as the location, speed, and direction of vehicles and pedestrians.

[0680] "Traffic conditions" refers to the movement and concentration of vehicles and pedestrians at a certain point, including the degree of congestion and flow in real time.

[0681] "Traffic light timing" refers to the duration of each green, yellow, and red phase of a traffic signal, and is adjusted to optimize traffic flow.

[0682] "Optimization" refers to adjusting the timing of traffic lights based on current traffic conditions, aiming to smooth traffic flow and alleviate congestion.

[0683] An "autonomous vehicle" is a vehicle that perceives its surroundings and drives autonomously without the intervention of a human driver.

[0684] "Communication" refers to the process of exchanging data between the autonomous vehicle and a server, sharing information about traffic signal timings and traffic conditions.

[0685] "Provision" refers to the act of the server sharing and transmitting necessary data and information to an autonomous vehicle.

[0686] This invention relates to a traffic light timing optimization system using traffic video. This system has the function of acquiring traffic video in real time, analyzing and predicting it, and optimizing traffic light timing. The optimized traffic light timing is also provided to autonomous vehicles, making vehicle operation smoother and safer.

[0687] Overall system overview

[0688] The system consists of the following main components:

[0689] 1. Traffic image acquisition means (terminal)

[0690] 2. Data analysis method (server)

[0691] 3. Signal timing optimization means (server)

[0692] 4. Signal Timing Application Means (Terminal)

[0693] 5. Monitoring and Feedback Methods (Users)

[0694] 6. Communication method with autonomous vehicles (server)

[0695] Traffic image acquisition means (terminal)

[0696] The terminals are equipped with high-resolution cameras that capture real-time images of intersections and major roads. The cameras can capture images day and night, and the terminals are equipped with communication functions to periodically send the captured image data to a server.

[0697] Data analysis method (server)

[0698] The server receives traffic video data sent from the devices and analyzes it using the open-source image processing library "OpenCV." "TensorFlow" is used as the AI ​​model to identify objects such as vehicles, pedestrians, and bicycles from the video data and obtain attribute data such as their position, speed, and direction.

[0699] Signal timing optimization means (server)

[0700] The server performs calculations to optimize traffic light timing based on traffic condition data obtained from the data analysis means. Based on past data and current conditions, it predicts future traffic flow and congestion and calculates the optimal duration of green, yellow, and red lights.

[0701] Signal timing application means (terminal)

[0702] The optimized signal timing calculated by the server is sent to the traffic light via the application means. The terminal (traffic light) changes the signal in real time according to the received signal timing. This will result in smoother traffic flow, and is expected to alleviate traffic congestion and reduce accidents.

[0703] Monitoring and feedback measures (users)

[0704] Users can use the system's monitoring function to monitor real-time traffic conditions. The dashboard displays information such as current traffic light timings, traffic flow, and congestion levels. Users can also manually adjust traffic light timings in case of emergencies or specific requirements.

[0705] Communication method with autonomous vehicles (server)

[0706] The server communicates with the autonomous vehicles and provides optimized traffic light timing in real time, allowing the autonomous vehicles to operate safely and smoothly with the latest traffic information.

[0707] Specific examples

[0708] Morning rush hour scenario

[0709] Traffic video capture: During the morning rush hour, a camera captures video in real time at an intersection. The device sends this video data to a server every 5 seconds.

[0710] Data analysis: The server analyzes the received video and detects 30% more vehicles than usual. The number of pedestrians is also confirmed to be more than double the usual number.

[0711] Traffic light timing optimization: Using this data, the server predicts that congestion will increase further over the next five minutes, so it extends the green light duration by 20 seconds and the pedestrian signal timing by 10 seconds.

[0712] Signal Timing Application: The server calculates new signal timings and sends them to the traffic lights, where they are applied immediately. This allows for smoother traffic flow and reduces congestion.

[0713] Monitoring: Administrators can view the dashboard to see improvements in traffic conditions in real time, and can make manual adjustments as needed.

[0714] Example prompt sentence:

[0715] Scenario:

[0716] 1. Traffic Video Capture: During peak hours, a vehicle's onboard camera captures video of an intersection.

[0717] The video data is transmitted to the server every 5 seconds.

[0718] 2. Data Analysis (Server): The server receives and analyzes the video.

[0719] It detects a 40% increase in vehicles and doubles the number of pedestrians compared to normal.

[0720] 3. Optimized Signal Timing Calculation (Server):

[0721] Predicting a further increase in congestion in the next 5 minutes, the server adjusts:

[0722] Green light duration is extended by 30 seconds and pedestrian signal timing by 15 seconds.

[0723] 4. Optimized Timing Application (Vehicle):

[0724] The optimized signal timing is sent back to the vehicle,

[0725] ensuring smooth traffic flow and reduced congestion.

[0726] 5. Monitoring (User): The vehicle's control system adapts in real-time, enhancing safety and efficiency.

[0727] These measures make it possible to use traffic footage to optimize traffic light timing in real time, improving overall road efficiency.

[0728] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0729] Step 1:

[0730] Traffic image acquisition (terminal):

[0731] A camera installed on the device captures images of intersections and major roads in real time. Images are captured day and night, and the image data from the camera is sent to the server periodically (e.g., every 5 seconds). The input is real-time traffic image data, and the output is the data transferred to the server.

[0732] Step 2:

[0733] Data reception (server):

[0734] The server receives traffic video data sent from the terminal. This data is added to a queue for image analysis and prepared for the next processing step. The input is video data from the terminal, and the output is data added to the analysis queue.

[0735] Step 3:

[0736] Data analysis (server):

[0737] The server analyzes the video data using the open-source image processing library "OpenCV." It uses "TensorFlow" as an AI model to identify objects such as vehicles, pedestrians, and bicycles from the video data and obtain attribute data such as their position, speed, and direction. The input is the received traffic video data, and the output is the analyzed traffic attribute data.

[0738] Step 4:

[0739] Signal timing optimization (server):

[0740] The server performs calculations to optimize traffic light timing based on the analyzed traffic attribute data. Based on past data and current conditions, it predicts future traffic flow and congestion and calculates the optimal durations of green, yellow, and red lights. The input is the analyzed traffic attribute data and past traffic data, and the output is optimized traffic light timing.

[0741] Step 5:

[0742] Signal Timing Application (Terminal):

[0743] The optimized signal timing calculated by the server is sent to the terminal (traffic light). The traffic light changes the signal in real time according to the received timing data. The input is the optimized signal timing, and the output is the traffic light timing change.

[0744] Step 6:

[0745] Monitoring (User):

[0746] Users can use the system's monitoring function to monitor real-time traffic conditions. The dashboard displays information such as current traffic light timing, traffic flow, and congestion level. Users can also manually adjust traffic light timing if necessary. The input is real-time traffic data and traffic light timing information, and the output is the user's monitoring results and feedback.

[0747] Step 7:

[0748] Providing optimized signal timing (server):

[0749] The server communicates with the autonomous vehicle and provides optimized signal timing in real time, allowing the autonomous vehicle to operate safely and smoothly using the latest traffic information. The input is optimized signal timing, and the output is data provided to the autonomous vehicle.

[0750] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0751] This invention combines an emotion engine with a traffic light timing optimization system that uses traffic video. This system acquires traffic video in real time, analyzes and predicts traffic conditions based on that data, and optimizes traffic light timing. It also recognizes user emotions and reflects them in traffic signal timing, achieving more advanced traffic management.

[0752] Overall system overview

[0753] The system consists of the following main components:

[0754] 1. Traffic image acquisition means (terminal)

[0755] 2. Data analysis method (server)

[0756] 3. Signal timing optimization means (server)

[0757] 4. Signal Timing Application Means (Terminal)

[0758] 5. Emotion Engine

[0759] 6. Monitoring and Feedback Methods (Users)

[0760] Traffic image acquisition means (terminal)

[0761] The terminals are equipped with cameras that capture images of intersections and major roads in real time. These cameras are capable of capturing images day and night, and the terminals are equipped with communication functions to send the captured image data to a server.

[0762] Data analysis method (server)

[0763] The server receives traffic video data sent from the device. The received data is analyzed using an AI-based image recognition algorithm. This analysis identifies objects such as vehicles, pedestrians, and bicycles from the video data and obtains attribute data such as their location, speed, and direction.

[0764] Signal timing optimization means (server)

[0765] The server performs calculations to optimize traffic signal timing based on traffic condition data obtained from the data analysis means. It takes into account past data and current conditions to predict future traffic flow and congestion. It calculates the optimal duration for each phase of the traffic signal (green, yellow, red).

[0766] Signal timing application means (terminal)

[0767] The optimized signal timing calculated by the server is sent to the traffic light via the application means. The terminal (traffic light) changes the signal in real time according to the received signal timing. This will result in smoother traffic flow, and is expected to alleviate traffic congestion and reduce accidents.

[0768] Emotion Engine

[0769] The emotion engine has the function of identifying user emotions and reflecting them in traffic management. Emotions are analyzed in real time, for example, from the user's voice and video. Based on the analysis results, the frustration and stress levels of road users are determined and reflected in adjustments to traffic signal timing.

[0770] Monitoring and feedback measures (users)

[0771] Users can use the system's monitoring function to monitor real-time traffic conditions. The dashboard displays information such as current traffic light timings, traffic flow, and congestion levels. Data from the emotion engine is also displayed to help users manually adjust traffic light timings. Users can also manually adjust traffic light timings in emergencies or according to specific requirements.

[0772] Specific examples

[0773] Morning rush hour scenario

[0774] 1. Traffic image acquisition (terminal)

[0775] During the morning rush hour, a camera captures video in real time at an intersection.

[0776] The device sends this video data to the server every 5 seconds.

[0777] 2. Data analysis (server)

[0778] The server analyzes the received video and detects 30% more vehicles than usual.

[0779] The number of pedestrians was also confirmed to be more than double the usual amount.

[0780] 3. Signal timing optimization (server)

[0781] The server uses this data to predict that congestion will increase further over the next five minutes.

[0782] The green light will last 20 seconds longer than usual, and the timing of pedestrian signals will be extended by 10 seconds.

[0783] 4. Signal Timing Application (Terminal)

[0784] The server calculates the new signal timing and sends it to the traffic light, where it is applied immediately.

[0785] This will allow for smoother traffic flow and reduce congestion.

[0786] 5. Utilizing the Emotion Engine (User)

[0787] Managers check the monitoring dashboard and view sentiment data along with traffic conditions.

[0788] The emotion engine detects high stress levels in the user and further fine-tunes signal timing to reflect this.

[0789] Road users' emotions are also monitored and, if necessary, traffic light adjustments are made to have a relaxing effect.

[0790] This will not only ease traffic congestion but also reduce the psychological burden on road users.

[0791] The processing flow will be explained below.

[0792] Step 1:

[0793] Video shooting (device)

[0794] The device uses a camera to capture real-time images of intersections and major roads, capturing the images frame by frame and storing the data over time.

[0795] Step 2:

[0796] Data transmission (terminal)

[0797] The video data captured by the device is sent to the server at regular intervals using a streaming protocol, and is sent in real time.

[0798] Step 3:

[0799] Data reception (server)

[0800] The server receives traffic video data sent from the terminal, and the received data is temporarily stored in a buffer for subsequent analysis.

[0801] Step 4:

[0802] Image recognition (server)

[0803] The server uses AI to analyze the video data in the buffer, and image recognition algorithms analyze each frame to identify objects such as vehicles, pedestrians, and cyclists.

[0804] Step 5:

[0805] Traffic situation analysis (server)

[0806] The server obtains attribute data such as the location, speed, and direction of the identified objects and evaluates traffic flow and congestion. The analysis results are used to determine the current traffic situation in real time.

[0807] Step 6:

[0808] Pattern prediction (server)

[0809] The server uses past traffic data and current analysis results to predict future traffic patterns, using predictive models to calculate the likelihood of future congestion and jams.

[0810] Step 7:

[0811] Optimization calculation (server)

[0812] The server calculates the optimal timing for traffic lights based on traffic conditions and predictions, optimizing the duration of each phase (green, yellow, red) and setting the traffic light timing.

[0813] Step 8:

[0814] Command generation (server)

[0815] The server generates a command to send to the traffic light based on the calculation results, which includes the new traffic light timing.

[0816] Step 9:

[0817] Send command (server)

[0818] The server sends the generated command to the traffic light, which then transmits the command to the traffic light via the network.

[0819] Step 10:

[0820] Signal timing application (terminal)

[0821] The terminal (traffic light) applies the new signal timing according to the received command. The traffic light immediately starts operating based on the new settings.

[0822] Step 11:

[0823] Emotion data collection (user)

[0824] Using an emotion engine, emotional data is obtained from the user's voice and video, analyzing the user's stress level and frustration and collecting emotional data in real time.

[0825] Step 12:

[0826] Emotion data analysis (server)

[0827] The server receives and analyzes the emotion data sent from the emotion engine, evaluates the user's emotional state indicated by the emotion data, and reflects this in traffic situation analysis.

[0828] Step 13:

[0829] Emotion-based signal timing adjustment (server)

[0830] The server analyzes the emotional data and fine-tunes the timing of the traffic lights. For example, if the user's stress level is high, the system will extend the duration of the green light.

[0831] Step 14:

[0832] Monitoring (users)

[0833] Users monitor real-time traffic conditions and emotion data using the system's dashboard, which displays information such as current traffic light timings, traffic flow, congestion levels, and the user's emotional state.

[0834] Step 15:

[0835] Feedback and manual adjustments (user)

[0836] If necessary, users can manually instruct the system to fine-tune signal timing, providing a means to respond to emergencies or specific requests.

[0837] Example 2

[0838] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0839] Optimization of traffic signals in transportation systems relies on analysis of traffic conditions and timing adjustment, but conventional systems do not take user emotions into consideration, resulting in problems such as user stress and frustration even when traffic flow is smooth.In addition, adjustment of traffic signals based on real-time traffic video and data analysis was limited, so there was a need for more precise and rapid optimization.

[0840] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0841] In this invention, the server includes means for acquiring traffic images, means for analyzing the acquired traffic images and determining traffic conditions, means for optimizing traffic light timing based on the traffic conditions, means for recognizing a user's emotions and adjusting traffic light timing based on the recognition results, and means for applying the optimized traffic light timing to the traffic lights. This enables real-time analysis and optimization of traffic conditions and also enables adjustments that take the user's emotions into consideration, thereby not only smoothing traffic flow but also reducing the psychological burden on the user.

[0842] "Traffic footage" refers to video data capturing traffic conditions, and records the movement of vehicles, pedestrians, bicycles, etc. at intersections and major roads in real time.

[0843] "Traffic conditions" refers to information that indicates the degree of congestion and movement of vehicles, pedestrians, bicycles, etc. on roads at a specific time and place.

[0844] "Traffic light timing" refers to the time it takes for a traffic light to change color (green, yellow, red), including the duration of the change to control traffic flow.

[0845] An "image recognition algorithm" is a computer algorithm that analyzes video data acquired from a camera and identifies specific objects (vehicles, pedestrians, bicycles, etc.).

[0846] "Pattern prediction" is a mathematical method for analyzing past traffic data and predicting current and future traffic conditions.

[0847] "Emotion recognition" is a technology that analyzes data such as a user's voice and facial expressions to identify their psychological state (stress, frustration, joy, etc.).

[0848] "Timing optimization" involves calculating the optimal duration of traffic light colors based on traffic conditions and user emotion data, and making adjustments to ensure the most efficient traffic flow.

[0849] "Monitoring" is the process of monitoring traffic conditions, traffic light timing, and user emotional data in real time and making adjustments as needed.

[0850] This invention not only optimizes traffic signal timing using traffic video, but also realizes advanced traffic management using user emotional data. This system captures traffic video in real time, analyzes the video data, and judges traffic conditions. Based on this judgment, it optimizes traffic signal timing and also reflects user emotional data, enabling efficient traffic management with reduced psychological burden.

[0851] Hardware and software used

[0852] Traffic image acquisition means (terminal)

[0853] The terminal is equipped with a camera that captures images of intersections and major roads in real time. This camera has the ability to capture images day and night, and sends the captured image data to a server. Specifically, it uses the H.264 codec to compress the image data and the SSL / TLS communication protocol for data communication.

[0854] Data analysis method (server)

[0855] The server receives traffic video data sent from the device, decodes the received data, and analyzes it using an AI-based image recognition algorithm (such as YOLO or OpenCV). This analysis identifies objects such as vehicles, pedestrians, and bicycles, and obtains attribute data such as location, speed, and direction.

[0856] Traffic condition data generation (server)

[0857] The server generates current traffic situation data based on the analysis results. The number of vehicles, number of pedestrians, and the speed and direction of each object are converted into statistical data and updated in real time. By storing this data in a database, it is possible to compare it with past data.

[0858] Signal timing optimization (server)

[0859] The server optimizes traffic signal timing based on traffic condition data. The optimization algorithm uses optimization theory and feedback control to calculate the duration of each phase (green, yellow, red) and update the timing in real time, including forecast data for the next five minutes.

[0860] Signal timing application means (terminal)

[0861] The optimized signal timing is sent from the server to the traffic lights, which then make real-time signal changes based on the received timing. To minimize delays, a low-latency communication protocol (e.g. MQTT) is used. The traffic light control software applies the new timing for each phase.

[0862] Emotion Engine

[0863] The emotion engine recognizes the user's emotions and reflects them in traffic management. Emotion data is acquired using a camera or microphone installed on the device, and the acquired audio and video data is analyzed on a server. Specifically, an emotion recognition algorithm (such as IBM Watson or Microsoft Azure's emotion recognition API) is used. Based on the emotion data, the level of stress or anger is determined and traffic light timing is adjusted.

[0864] Monitoring and Feedback (User)

[0865] Users can view real-time traffic conditions and sentiment data on a monitoring dashboard, which displays current traffic signal timings, traffic flow, congestion levels, and user sentiment data. Users can also manually adjust traffic signal timings in case of emergencies or specific requirements.

[0866] Examples of concrete examples and prompts

[0867] Specific examples

[0868] During the morning rush hour, cameras capture traffic footage in real time and send the acquired video data to a server. The server analyzes the data and detects an increase in vehicles and pedestrians compared to normal. It predicts that congestion will increase further over the next five minutes and makes adjustments such as extending the duration of green lights. User emotion data is also collected, and if high stress levels are detected, further fine-tuning of signal timing is made. Administrators can monitor this from a dashboard and make manual adjustments as needed.

[0869] Prompt Sentence Examples

[0870] An example prompt is:

[0871] Write a program that analyzes current traffic video data and optimizes traffic light timing. Specifically, for intersection A, with 50 vehicles and 30 pedestrians, extend the current green light by 10 seconds and shorten the pedestrian signal timing by 5 seconds. Also, further fine-tune the traffic light timing based on emotion data.

[0872] This system enables real-time analysis and optimization of traffic conditions, and more effective traffic management can be achieved by making adjustments that take into account user emotional data.

[0873] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0874] Step 1:

[0875] Traffic image acquisition (terminal)

[0876] The device is equipped with a camera that captures images of intersections and major roads in real time. Specifically, the camera captures images at 30 frames per second (fps) and continues to capture images at night using infrared functionality. The input is real-time images of intersections and roads, and the output is video data compressed using the H.264 codec.

[0877] Step 2:

[0878] Video data transmission (terminal)

[0879] The terminal transmits the acquired video data to the server. Specifically, the acquired video data is compressed every second and transmitted to the server using the SSL / TLS communication protocol. The input is video data compressed using H.264, and the output is encrypted data packets.

[0880] Step 3:

[0881] Traffic video analysis (server)

[0882] The server decodes the video data received from the device and analyzes traffic conditions by using AI for image recognition. YOLO and OpenCV are used as image recognition algorithms. Specifically, it identifies objects such as vehicles, pedestrians, and bicycles from the video data and determines their respective positions, speeds, and directions. The input is the video data sent to the server, and the output is attribute data for each object (position, speed, direction, etc.).

[0883] Step 4:

[0884] Traffic condition data generation (server)

[0885] The server generates current traffic situation data based on the analysis results. It converts the number of vehicles, the number of pedestrians, and the speed and direction of each object into statistical data and updates it in real time. Specifically, it saves the analysis results in list format and records them in a database. The input is the analysis results of the traffic video, and the output is statistical traffic situation data.

[0886] Step 5:

[0887] Signal timing optimization (server)

[0888] The server optimizes signal timing based on traffic condition data. The optimization algorithm uses optimization theory and feedback control. Specifically, it calculates the optimal duration of each phase (green, yellow, red). The input is traffic condition data updated in real time and past data, and the output is optimized signal timing data.

[0889] Step 6:

[0890] Optimized timing transmission (server)

[0891] The server transmits the optimized signal timing to the traffic lights using a low-latency communication protocol (e.g., MQTT). Specifically, it creates optimized timing data packets and sends them to the traffic lights. The input is the optimized signal timing data, and the output is the new signal timing that is applied to the traffic light control software.

[0892] Step 7:

[0893] Timing application to traffic lights (terminal)

[0894] The terminal (traffic light) changes the signal in real time according to the received signal timing. For example, the delay when changing from red to green is kept within a few milliseconds. The input is the new signal timing data sent from the server, and the output is the changed color of the traffic light.

[0895] Step 8:

[0896] Acquisition and analysis of emotion data (terminal / server)

[0897] The device (for example, an emotion-recognition camera or microphone) acquires the user's voice and video data. The server analyzes this data and recognizes the user's emotions using an emotion engine (for example, IBM Watson or Microsoft Azure's emotion recognition API). Specifically, it detects the level of stress or anger from the user's voice and facial expression. The input is the user's voice and video data, and the output is analyzed emotional data.

[0898] Step 9:

[0899] Adjustment of signal timing based on emotion data (server)

[0900] The server further adjusts the signal timing based on the acquired emotion data. For example, if the user's stress level is high, the algorithm adjusts to shorten the duration of the green light. The input is the analyzed emotion data in real time, and the output is fine-tuned signal timing data.

[0901] Step 10:

[0902] Monitoring and Feedback (User)

[0903] Users can check real-time traffic conditions and emotion data on a monitoring dashboard. The dashboard displays current traffic signal timings, traffic flow, congestion levels, and user emotion data. Users can also manually adjust traffic signal timings in emergencies or according to specific requirements. The inputs are traffic condition data and emotion data, and the output is the traffic signal timings manually adjusted by the user.

[0904] (Application example 2)

[0905] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0906] Conventional traffic management systems focus on analyzing traffic images and optimizing traffic signal timing, but real-time understanding and adaptation to traffic conditions is required to ensure smooth traffic flow. Furthermore, one of the factors that contribute to traffic accidents and congestion is the emotion and stress of drivers and pedestrians, yet few systems exist that take this into account. Therefore, new systems are needed that can optimize traffic flow while reducing the psychological burden on road users. Furthermore, with the spread of autonomous vehicles, new traffic management systems that can accommodate this are also required.

[0907] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0908] In this invention, the server includes means for acquiring traffic video, means for analyzing the traffic video and determining traffic conditions, means for optimizing traffic light timing based on the traffic conditions, means for applying the optimized traffic light timing to the traffic lights, means for analyzing user emotions and adjusting traffic light timing based on the emotions, and means for monitoring surrounding traffic conditions in real time using video acquired by a camera in the autonomous vehicle and optimizing the driving pattern of the autonomous vehicle based on the analysis results, thereby enabling optimization of traffic flow, flexible traffic management in response to the emotions of road users, and safe and efficient operation of the autonomous vehicle.

[0909] The "means for acquiring traffic images" refers to a device that uses cameras installed at locations such as intersections and major roads to acquire images of traffic conditions in real time.

[0910] The "means for analyzing traffic images and determining traffic conditions" is a device that uses an AI-based image recognition algorithm to analyze the status of vehicles, pedestrians, and traffic lights based on acquired traffic images, and determines the current traffic conditions.

[0911] A "means for optimizing traffic signal timing based on traffic conditions" is a device that uses analyzed traffic condition data to calculate the duration of green, yellow, and red signals at a traffic signal and optimize traffic flow.

[0912] The "means for applying optimized signal timing to traffic lights" is a device that transmits the calculated optimal signal timing to actual traffic lights and changes the signals in real time.

[0913] The "means for analyzing the user's emotions and adjusting the timing of traffic lights based on those emotions" refers to a device that analyzes emotions from the user's voice and video, and reflects that data in a traffic management system, thereby adjusting the timing of traffic lights based on the user's emotions.

[0914] "Means for monitoring surrounding traffic conditions in real time using images captured by a camera on an autonomous vehicle" refers to a device that uses a camera mounted on an autonomous vehicle to capture images of traffic conditions, acquires the image data in real time, and monitors surrounding traffic.

[0915] The "means for optimizing the driving pattern of an autonomous vehicle based on the analysis results" refers to a device that adjusts the speed and route of an autonomous vehicle to the optimum based on the analysis results of the acquired video data.

[0916] This invention combines an emotion engine with a function for linking with autonomous vehicles to optimize traffic light timing using traffic video. This system acquires traffic video in real time, analyzes and predicts the data, and optimizes traffic light timing. Furthermore, by analyzing user emotions and reflecting them in traffic light timing, it is possible to enhance traffic management and optimize the operation of autonomous vehicles.

[0917] Overall system overview

[0918] The system consists of the following main components:

[0919] 1. Means for acquiring traffic images (devices including cameras)

[0920] 2. A means of analyzing data and determining traffic conditions (server)

[0921] 3. Means for optimizing traffic light timing (server)

[0922] 4. Means (terminal) for applying optimized signal timing to traffic lights

[0923] 5. A means to analyze user emotions and adjust traffic light timing based on those emotions (emotion engine)

[0924] 6. Means of monitoring surrounding traffic conditions using images captured by cameras in autonomous vehicles (cameras and systems for autonomous vehicles)

[0925] 7. Means for optimizing the driving patterns of autonomous vehicles based on analysis results (autonomous driving systems)

[0926] Hardware and software used

[0927] The following hardware and software are used to implement this system.

[0928] Hardware: Cameras, autonomous vehicle cameras, servers, traffic light control terminals

[0929] Software: OpenCV (image analysis), TensorFlow (AI modeling), Python scripts (data analysis), ROS (Robot Operating System, integration with autonomous driving systems)

[0930] Program processing explanation

[0931] The server first receives real-time video data from cameras to capture traffic images. It then analyzes the received video data using OpenCV and TensorFlow to determine traffic conditions. Traffic data includes the position, speed, and direction of objects such as vehicles, pedestrians, and bicycles. It then performs calculations to optimize traffic light timing based on the traffic conditions and sends the optimal signal timing to the traffic light control terminal.

[0932] Meanwhile, an emotion engine is used to analyze user emotions. This engine collects passenger emotional data from audio and video and analyzes it in real time. This emotional data is used to fine-tune signal timing, setting optimal signal timing to reduce passenger stress.

[0933] Finally, the autonomous vehicle's camera continuously monitors the surrounding traffic conditions and analyzes the acquired data in real time. Using ROS, the system instructs the autonomous driving system on optimal driving patterns, helping to ensure smooth traffic flow. In particular, taking into account passenger emotional data can help ensure a comfortable and safe operation.

[0934] Adding concrete examples and prompt sentences for the generative AI model

[0935] For example, consider the following system situation:

[0936] 1. Traffic video acquisition: Cameras on autonomous vehicles capture real-time traffic conditions at intersections and transmit the video data to a server inside the vehicle.

[0937] 2. Data analysis: The server analyzes the received video and determines traffic flow based on information such as the location, speed, and direction of identified vehicles and pedestrians.

[0938] 3. Optimization: Based on the analysis results, the speed and route of autonomous vehicles are optimized and traffic light timing is adjusted.

[0939] 4. Emotion analysis: Analyzes passengers' emotions in real time from their voice data and fine-tunes traffic signal timing to ensure stress-free driving.

[0940] Example prompt for a generative AI model:

[0941] Analyze traffic video data in real time to identify vehicles, pedestrians, and bicycles. Also, analyze passengers' emotions based on their voice data and select driving patterns according to their stress levels. Use TensorFlow and OpenCV for video data analysis, and an emotion engine for emotion analysis. Use ROS for integration with the autonomous driving system.

[0942] According to the above aspects, the present invention can optimize traffic flow, reduce the psychological burden on traffic users, and realize safe and efficient operation of autonomous vehicles.

[0943] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0944] Step 1:

[0945] Traffic footage acquisition

[0946] The device captures real-time traffic images using cameras installed at intersections and major roads. This image data is also captured by cameras on autonomous vehicles. The captured image data is sent directly to the server. The input data is the image data from the cameras, and the output data is the image data sent to the server.

[0947] Step 2:

[0948] Traffic video analysis

[0949] The server processes the received traffic video data. Specifically, it performs image analysis using OpenCV and TensorFlow. Image analysis identifies objects such as vehicles, pedestrians, and bicycles, and calculates their respective positions, speeds, and directions. The input data is the traffic video data, and the output data is attribute data of the identified objects.

[0950] Step 3:

[0951] Judging traffic conditions

[0952] The server determines the current traffic situation based on the image analysis results. Specifically, it calculates traffic flow and congestion levels based on the position, speed, and direction of each object. This makes it possible to identify traffic bottlenecks and dangerous areas. The input data is the attribute data of the identified objects, and the output data is the traffic situation assessment result.

[0953] Step 4:

[0954] Traffic light timing optimization

[0955] The server optimizes traffic light timing based on the traffic conditions it determines. It also takes into account past data and current conditions to calculate the duration of each traffic light phase (green, yellow, red). The input data is the traffic condition determination result, and the output data is the optimized traffic light timing.

[0956] Step 5:

[0957] Signal Timing Enforcement

[0958] The terminal applies the optimized signal timing received from the server to the actual traffic light, which changes the signal in real time and ensures smooth traffic flow. The input data is the optimized signal timing, and the output data is the result of the traffic light timing change.

[0959] Step 6:

[0960] User sentiment analysis

[0961] The emotion engine analyzes the user's (passenger's) voice and video data in real time to determine emotions such as stress and frustration, thereby understanding the user's psychological state. The input data is the user's voice and video data, and the output data is the analyzed emotion data.

[0962] Step 7:

[0963] Fine-tuning signal timing

[0964] The server fine-tunes the optimized signal timing based on the emotion data obtained from the emotion engine. This sets the optimal signal timing to reduce the user's stress. The input data is the analyzed emotion data, and the output data is the fine-tuned signal timing.

[0965] Step 8:

[0966] Optimizing driving patterns for autonomous vehicles

[0967] The autonomous vehicle system optimizes driving patterns based on acquired traffic video data and emotion data. Specifically, it uses ROS to adjust the vehicle's speed and route. The input data is traffic video data and emotion data, and the output data is the optimized driving pattern.

[0968] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0969] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0970] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0971] [Third embodiment]

[0972] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0973] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0974] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0975] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0976] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0977] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0978] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0979] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0980] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0981] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0982] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0983] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0984] This invention relates to a traffic light timing optimization system that uses traffic video. This system acquires traffic video in real time, analyzes and predicts traffic conditions based on the video, and optimizes traffic light timing.

[0985] Overall system overview

[0986] The system consists of the following main components:

[0987] 1. Traffic image acquisition means (terminal)

[0988] 2. Data analysis method (server)

[0989] 3. Signal timing optimization means (server)

[0990] 4. Signal Timing Application Means (Terminal)

[0991] 5. Monitoring and Feedback Methods (Users)

[0992] Traffic image acquisition means (terminal)

[0993] The terminals are equipped with cameras that capture images of intersections and major roads in real time. These cameras are capable of capturing images day and night, and the terminals are equipped with communication functions to send the captured image data to a server.

[0994] Data analysis method (server)

[0995] The server receives traffic video data sent from the device. The received data is analyzed using an AI-based image recognition algorithm. This analysis identifies objects such as vehicles, pedestrians, and bicycles from the video data and obtains attribute data such as their location, speed, and direction.

[0996] Signal timing optimization means (server)

[0997] The server performs calculations to optimize traffic signal timing based on traffic condition data obtained from the data analysis means. It takes into account past data and current conditions to predict future traffic flow and congestion. It calculates the optimal duration for each phase of the traffic signal (green, yellow, red).

[0998] Signal timing application means (terminal)

[0999] The optimized signal timing calculated by the server is sent to the traffic light via the application means. The terminal (traffic light) changes the signal in real time according to the received signal timing. This will result in smoother traffic flow, and is expected to alleviate traffic congestion and reduce accidents.

[1000] Monitoring and feedback measures (users)

[1001] Users can use the system's monitoring function to monitor real-time traffic conditions. The dashboard displays information such as current traffic light timings, traffic flow, and congestion levels. Users can also manually adjust traffic light timings in case of emergencies or specific requirements.

[1002] Specific examples

[1003] Morning rush hour scenario

[1004] 1. Traffic image acquisition (terminal)

[1005] During the morning rush hour, a camera captures video in real time at an intersection.

[1006] The device sends this video data to the server every 5 seconds.

[1007] 2. Data analysis (server)

[1008] The server analyzes the received video and detects 30% more vehicles than usual.

[1009] The number of pedestrians was also confirmed to be more than double the usual amount.

[1010] 3. Signal timing optimization (server)

[1011] The server uses this data to predict that congestion will increase further over the next five minutes.

[1012] The green light will last 20 seconds longer than usual, and the timing of pedestrian signals will be extended by 10 seconds.

[1013] 4. Signal Timing Application (Terminal)

[1014] The server calculates the new signal timing and sends it to the traffic light, where it is applied immediately.

[1015] This will allow for smoother traffic flow and reduce congestion.

[1016] 5. Monitoring (User)

[1017] Managers can check the dashboard to see improvements in traffic conditions in real time.

[1018] You can also make manual adjustments if needed.

[1019] These measures make it possible to use traffic footage to optimize traffic light timing in real time, improving overall road efficiency.

[1020] The processing flow will be explained below.

[1021] Step 1:

[1022] Video shooting (device)

[1023] The device uses a camera to capture real-time images of intersections and major roads, capturing the images frame by frame and storing the data over time.

[1024] Step 2:

[1025] Data transmission (terminal)

[1026] The video data captured by the device is sent to the server at regular intervals using a streaming protocol, and is sent in real time.

[1027] Step 3:

[1028] Data reception (server)

[1029] The server receives traffic video data sent from the terminal, and the received data is temporarily stored in a buffer for subsequent analysis.

[1030] Step 4:

[1031] Image recognition (server)

[1032] The server uses AI to analyze the video data in the buffer, and image recognition algorithms analyze each frame to identify objects such as vehicles, pedestrians, and cyclists.

[1033] Step 5:

[1034] Traffic situation analysis (server)

[1035] The server obtains attribute data such as the location, speed, and direction of the identified objects and evaluates traffic flow and congestion. The analysis results are used to determine the current traffic situation in real time.

[1036] Step 6:

[1037] Pattern prediction (server)

[1038] The server uses past traffic data and current analysis results to predict future traffic patterns, using predictive models to calculate the likelihood of future congestion and jams.

[1039] Step 7:

[1040] Optimization calculation (server)

[1041] The server calculates the optimal timing for traffic lights based on traffic conditions and predictions, optimizing the duration of each phase (green, yellow, red) and setting the traffic light timing.

[1042] Step 8:

[1043] Command generation (server)

[1044] The server generates a command to send to the traffic light based on the calculation results, which includes the new traffic light timing.

[1045] Step 9:

[1046] Send command (server)

[1047] The server sends the generated command to the traffic light, which then transmits the command to the traffic light via the network.

[1048] Step 10:

[1049] Signal timing application (terminal)

[1050] The terminal (traffic light) applies the new signal timing according to the received command. The traffic light immediately starts operating based on the new settings.

[1051] Step 11:

[1052] Real-time monitoring (user)

[1053] Users monitor real-time traffic conditions using the system's dashboard, which displays information such as current traffic light timings, traffic flow, and congestion levels.

[1054] Step 12:

[1055] Feedback and manual adjustments (user)

[1056] If necessary, users can manually instruct the system to fine-tune signal timing, providing a means to respond to emergencies or specific requests.

[1057] Example 1

[1058] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1059] Current traffic management systems have fixed signal timing settings, making it difficult to optimize them to accommodate real-time traffic conditions, which can lead to traffic congestion and accidents. Furthermore, they lack the means to quickly respond to sudden changes in traffic conditions, making it difficult to alleviate sudden congestion and jams.

[1060] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1061] In this invention, the server includes a means for acquiring traffic images, a means for analyzing the acquired traffic images to determine traffic conditions, and a means for optimizing traffic light timing based on the traffic conditions. This makes it possible to grasp traffic conditions in real time and calculate and apply appropriate traffic light timing. Furthermore, by including a means for manually adjusting traffic light timing based on the analysis results, it is possible to quickly respond to sudden changes in traffic conditions.

[1062] "Traffic footage" is video data that records traffic flow and the movements of vehicles and pedestrians.

[1063] "Analysis" is the process of extracting specific information from the acquired video data and determining the traffic situation.

[1064] "Traffic conditions" refers to information that indicates the movement and status of vehicles, pedestrians, etc. in a specific location.

[1065] "Traffic light timing" refers to the duration and switching timing of each phase of a traffic light (green, yellow, red).

[1066] "Optimization" is the process of adjusting signal timing to achieve a specific goal (e.g., smooth traffic flow or fewer accidents).

[1067] "Apply" means to set the optimized signal timing to an actual traffic light.

[1068] The "means for manually adjusting signal timing" is a function that allows the user to manually change the signal timing as needed.

[1069] An "image recognition algorithm" is a computational method for identifying specific objects (e.g., vehicles, pedestrians) from video data.

[1070] "Pattern prediction" is the process of predicting future traffic conditions based on past data.

[1071] MODE FOR CARRYING OUT THE INVENTION

[1072] This invention relates to a traffic light timing optimization system that uses traffic video. This system acquires traffic video in real time, analyzes and predicts traffic conditions based on the video, and optimizes traffic light timing. Specifically, it consists of the following main components:

[1073] 1. Traffic image acquisition means (terminal)

[1074] 2. Data analysis method (server)

[1075] 3. Signal timing optimization means (server)

[1076] 4. Signal Timing Application Means (Terminal)

[1077] 5. Monitoring and Feedback Methods (Users)

[1078] Traffic image acquisition means (terminal)

[1079] The device is equipped with a high-resolution camera that captures images of intersections and major roads in real time. This camera operates 24 hours a day, capturing images day and night. The captured image data is compressed every five seconds and sent to a server. Communication is via an internet connection. For example, Sony's high-resolution security cameras can be used.

[1080] Data analysis method (server)

[1081] The server receives traffic video data sent from the device and temporarily stores it in a storage area. The server then analyzes the video data using AI image recognition algorithms such as YOLO and OpenCV. During this analysis, objects such as vehicles, pedestrians, and bicycles are identified and attribute data such as their location, speed, and direction are acquired. The acquired attribute data is stored on the server and used for subsequent analysis and prediction.

[1082] Signal timing optimization means (server)

[1083] The server uses traffic condition data obtained from the data analysis tool to perform calculations to optimize traffic light timing. It uses past data and a pattern prediction algorithm to predict traffic congestion over the next five minutes. For example, it can extend the green light duration by 20 seconds and the pedestrian signal timing by 10 seconds to smooth traffic flow.

[1084] Signal timing application means (terminal)

[1085] The optimized signal timing calculated by the server is sent to the traffic light via the terminal. The traffic light changes its signal in real time according to the received signal timing. The traffic light is controlled by sending instructions to the signal control unit using the TCP / IP stack.

[1086] Monitoring and feedback measures (users)

[1087] Users monitor traffic conditions in real time through the system's dashboard, which visually displays current traffic light timings, traffic volume at each intersection, and pedestrian flow. Users can manually adjust traffic light timings as needed. For example, they may temporarily change traffic light timings to respond to specific events or emergencies. System parameters can also be optimized based on real-time data feedback.

[1088] Specific examples

[1089] Morning rush hour scenario

[1090] 1. Traffic image acquisition (terminal)

[1091] During the morning rush hour, a camera captures video in real time at an intersection.

[1092] The device sends this video data to the server every 5 seconds.

[1093] 2. Data analysis (server)

[1094] The server analyzes the received video and detects 30% more vehicles than usual.

[1095] The server also verifies that the number of pedestrians is more than double the normal amount.

[1096] 3. Signal timing optimization (server)

[1097] The server uses this data to predict that congestion will increase further over the next five minutes.

[1098] The green light will last 20 seconds longer than usual, and the timing of pedestrian signals will be extended by 10 seconds.

[1099] 4. Signal Timing Application (Terminal)

[1100] The server calculates the new signal timing and sends it to the traffic light, where it is applied immediately.

[1101] The terminal adjusts traffic signals to ensure smooth vehicle flow and reduce congestion.

[1102] 5. Monitoring (User)

[1103] Managers can check the dashboard to see improvements in traffic conditions in real time.

[1104] The user manually fine-tunes as needed.

[1105] Example prompts for generative AI models

[1106] An example prompt for a traffic signal timing optimization system is as follows:

[1107] Prompt: Explain how to optimize traffic signals.

[1108]

[1109] Generative AI Model: Traffic signal optimization can be achieved by capturing traffic video in real time and analyzing it with AI algorithms. Based on the analyzed data, traffic signal timing can be calculated and applied to smooth traffic flow, reducing congestion and accidents.

[1110] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1111] Program processing flow

[1112] Each processing step will be explained by clearly showing the specific operation, input, and output.

[1113] Step 1: Acquiring and transmitting traffic images (terminal)

[1114] Input: Real-time camera footage monitoring traffic flow.

[1115] Specific operation: A high-resolution camera installed on the device captures images of intersections and major roads in real time. This camera operates 24 hours a day, capturing images day and night.

[1116] Data processing: The device compresses the acquired video data every 5 seconds (e.g., H.264 format) and converts it into a data size that can be transmitted.

[1117] Output: The compressed video data is sent to the server via the Internet.

[1118] Step 2: Receiving video data (server)

[1119] Input: Compressed video data sent from the device.

[1120] Specific operation: The server receives the video data transmitted from the terminal.

[1121] Data processing: After receiving, the data is checked for consistency and completeness, and temporarily stored in a database using a streaming API (e.g., FFmpeg).

[1122] Output: Saved video data.

[1123] Step 3: Data analysis and traffic situation evaluation (server)

[1124] Input: Stored video data.

[1125] Specific operation: The server uses YOLO (You Only Look Once) and OpenCV to analyze the received video data.

[1126] Data processing: Identify objects such as vehicles, pedestrians, and bicycles from video data and obtain their attribute data (position, speed, direction).

[1127] Output: Attribute data of identified objects.

[1128] Step 4: Calculate signal timing (server)

[1129] Input: Attribute data of the identified object.

[1130] How it works: The server uses the analysis results to calculate optimal traffic light timings. It uses historical data and pattern prediction algorithms to predict future traffic congestion.

[1131] Data processing: Calculations are made to adjust the duration of green and pedestrian signals based on congestion predictions.

[1132] Output: Optimized signal timing configuration data.

[1133] Step 5: Sending the calculation results (server)

[1134] Input: Optimized signal timing configuration data.

[1135] Specific operation: The server encodes the calculated optimized signal timing in JSON format and sends it to the terminal.

[1136] Data processing: Data is transmitted securely using the HTTPS protocol.

[1137] Output: Signal timing configuration data sent to the terminal.

[1138] Step 6: Apply signal timing (terminal)

[1139] Input: Signal timing configuration data sent from the server.

[1140] Specific operation: The terminal analyzes the received signal timing and applies it to the signal control unit.

[1141] Data processing: Send instructions to the signal control unit using the TCP / IP stack.

[1142] Output: Traffic light timing is optimized in real time.

[1143] Step 7: Monitoring and Feedback (Users)

[1144] Input: Real-time traffic and signal timing data.

[1145] Specific operation: The user monitors the traffic situation in real time through the system's dashboard.

[1146] Data processing: Visually display information such as current traffic light timings, traffic volume at each intersection, and pedestrian flow on the dashboard.

[1147] Output: The user manually adjusts the signal timing as needed to optimize the system settings.

[1148] (Application example 1)

[1149] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1150] To smooth traffic flow, alleviate congestion, and reduce accidents, it is necessary to analyze traffic conditions in real time with high accuracy and optimize traffic signal timing based on that analysis. Furthermore, with the spread of autonomous vehicles, coordination between vehicles and traffic signals is becoming even more important. However, current systems are not fully capable of achieving these goals, and the problems of traffic congestion and accidents remain unresolved. Furthermore, there is a lack of an information-sharing mechanism that allows autonomous vehicles to flexibly adapt to the traffic environment.

[1151] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1152] In this invention, the server includes means for acquiring traffic images, means for analyzing the acquired traffic images to determine traffic conditions, means for optimizing traffic light timings based on the traffic conditions, means for applying the optimized traffic light timings to the traffic lights, and means for communicating with autonomous vehicles and providing the optimized traffic light timings, thereby enabling autonomous vehicles to acquire optimized traffic light timing information in real time, making vehicle operation smoother and safer.

[1153] "Traffic video" refers to video data of intersections and major roads, captured in real time using cameras.

[1154] "Analysis" is the process of processing traffic video data and extracting attributes such as the location, speed, and direction of vehicles and pedestrians.

[1155] "Traffic conditions" refers to the movement and concentration of vehicles and pedestrians at a certain point, including the degree of congestion and flow in real time.

[1156] "Traffic light timing" refers to the duration of each green, yellow, and red phase of a traffic signal, and is adjusted to optimize traffic flow.

[1157] "Optimization" refers to adjusting the timing of traffic lights based on current traffic conditions, aiming to smooth traffic flow and alleviate congestion.

[1158] An "autonomous vehicle" is a vehicle that perceives its surroundings and drives autonomously without the intervention of a human driver.

[1159] "Communication" refers to the process of exchanging data between the autonomous vehicle and a server, sharing information about traffic signal timings and traffic conditions.

[1160] "Provision" refers to the act of the server sharing and transmitting necessary data and information to an autonomous vehicle.

[1161] This invention relates to a traffic light timing optimization system using traffic video. This system has the function of acquiring traffic video in real time, analyzing and predicting it, and optimizing traffic light timing. The optimized traffic light timing is also provided to autonomous vehicles, making vehicle operation smoother and safer.

[1162] Overall system overview

[1163] The system consists of the following main components:

[1164] 1. Traffic image acquisition means (terminal)

[1165] 2. Data analysis method (server)

[1166] 3. Signal timing optimization means (server)

[1167] 4. Signal Timing Application Means (Terminal)

[1168] 5. Monitoring and Feedback Methods (Users)

[1169] 6. Communication method with autonomous vehicles (server)

[1170] Traffic image acquisition means (terminal)

[1171] The terminals are equipped with high-resolution cameras that capture real-time images of intersections and major roads. The cameras can capture images day and night, and the terminals are equipped with communication functions to periodically send the captured image data to a server.

[1172] Data analysis method (server)

[1173] The server receives traffic video data sent from the devices and analyzes it using the open-source image processing library "OpenCV." "TensorFlow" is used as the AI ​​model to identify objects such as vehicles, pedestrians, and bicycles from the video data and obtain attribute data such as their position, speed, and direction.

[1174] Signal timing optimization means (server)

[1175] The server performs calculations to optimize traffic light timing based on traffic condition data obtained from the data analysis means. Based on past data and current conditions, it predicts future traffic flow and congestion and calculates the optimal duration of green, yellow, and red lights.

[1176] Signal timing application means (terminal)

[1177] The optimized signal timing calculated by the server is sent to the traffic light via the application means. The terminal (traffic light) changes the signal in real time according to the received signal timing. This will result in smoother traffic flow, and is expected to alleviate traffic congestion and reduce accidents.

[1178] Monitoring and feedback measures (users)

[1179] Users can use the system's monitoring function to monitor real-time traffic conditions. The dashboard displays information such as current traffic light timings, traffic flow, and congestion levels. Users can also manually adjust traffic light timings in case of emergencies or specific requirements.

[1180] Communication method with autonomous vehicles (server)

[1181] The server communicates with the autonomous vehicles and provides optimized traffic light timing in real time, allowing the autonomous vehicles to operate safely and smoothly with the latest traffic information.

[1182] Specific examples

[1183] Morning rush hour scenario

[1184] Traffic video capture: During the morning rush hour, a camera captures video in real time at an intersection. The device sends this video data to a server every 5 seconds.

[1185] Data analysis: The server analyzes the received video and detects 30% more vehicles than usual. The number of pedestrians is also confirmed to be more than double the usual number.

[1186] Traffic light timing optimization: Using this data, the server predicts that congestion will increase further over the next five minutes, so it extends the green light duration by 20 seconds and the pedestrian signal timing by 10 seconds.

[1187] Signal Timing Application: The server calculates new signal timings and sends them to the traffic lights, where they are applied immediately. This allows for smoother traffic flow and reduces congestion.

[1188] Monitoring: Administrators can view the dashboard to see improvements in traffic conditions in real time, and can make manual adjustments as needed.

[1189] Example prompt sentence:

[1190] Scenario:

[1191] 1. Traffic Video Capture: During peak hours, a vehicle's onboard camera captures video of an intersection.

[1192] The video data is transmitted to the server every 5 seconds.

[1193] 2. Data Analysis (Server): The server receives and analyzes the video.

[1194] It detects a 40% increase in vehicles and doubles the number of pedestrians compared to normal.

[1195] 3. Optimized Signal Timing Calculation (Server):

[1196] Predicting a further increase in congestion in the next 5 minutes, the server adjusts:

[1197] Green light duration is extended by 30 seconds and pedestrian signal timing by 15 seconds.

[1198] 4. Optimized Timing Application (Vehicle):

[1199] The optimized signal timing is sent back to the vehicle,

[1200] ensuring smooth traffic flow and reduced congestion.

[1201] 5. Monitoring (User): The vehicle's control system adapts in real-time, enhancing safety and efficiency.

[1202] These measures make it possible to use traffic footage to optimize traffic light timing in real time, improving overall road efficiency.

[1203] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1204] Step 1:

[1205] Traffic image acquisition (terminal):

[1206] A camera installed on the device captures images of intersections and major roads in real time. Images are captured day and night, and the image data from the camera is sent to the server periodically (e.g., every 5 seconds). The input is real-time traffic image data, and the output is the data transferred to the server.

[1207] Step 2:

[1208] Data reception (server):

[1209] The server receives traffic video data sent from the terminal. This data is added to a queue for image analysis and prepared for the next processing step. The input is video data from the terminal, and the output is data added to the analysis queue.

[1210] Step 3:

[1211] Data analysis (server):

[1212] The server analyzes the video data using the open-source image processing library "OpenCV." It uses "TensorFlow" as an AI model to identify objects such as vehicles, pedestrians, and bicycles from the video data and obtain attribute data such as their position, speed, and direction. The input is the received traffic video data, and the output is the analyzed traffic attribute data.

[1213] Step 4:

[1214] Signal timing optimization (server):

[1215] The server performs calculations to optimize traffic light timing based on the analyzed traffic attribute data. Based on past data and current conditions, it predicts future traffic flow and congestion and calculates the optimal durations of green, yellow, and red lights. The input is the analyzed traffic attribute data and past traffic data, and the output is optimized traffic light timing.

[1216] Step 5:

[1217] Signal Timing Application (Terminal):

[1218] The optimized signal timing calculated by the server is sent to the terminal (traffic light). The traffic light changes the signal in real time according to the received timing data. The input is the optimized signal timing, and the output is the traffic light timing change.

[1219] Step 6:

[1220] Monitoring (User):

[1221] Users can use the system's monitoring function to monitor real-time traffic conditions. The dashboard displays information such as current traffic light timing, traffic flow, and congestion level. Users can also manually adjust traffic light timing if necessary. The input is real-time traffic data and traffic light timing information, and the output is the user's monitoring results and feedback.

[1222] Step 7:

[1223] Providing optimized signal timing (server):

[1224] The server communicates with the autonomous vehicle and provides optimized signal timing in real time, allowing the autonomous vehicle to operate safely and smoothly using the latest traffic information. The input is optimized signal timing, and the output is data provided to the autonomous vehicle.

[1225] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1226] This invention combines an emotion engine with a traffic light timing optimization system that uses traffic video. This system acquires traffic video in real time, analyzes and predicts traffic conditions based on that data, and optimizes traffic light timing. It also recognizes user emotions and reflects them in traffic signal timing, achieving more advanced traffic management.

[1227] Overall system overview

[1228] The system consists of the following main components:

[1229] 1. Traffic image acquisition means (terminal)

[1230] 2. Data analysis method (server)

[1231] 3. Signal timing optimization means (server)

[1232] 4. Signal Timing Application Means (Terminal)

[1233] 5. Emotion Engine

[1234] 6. Monitoring and Feedback Methods (Users)

[1235] Traffic image acquisition means (terminal)

[1236] The terminals are equipped with cameras that capture images of intersections and major roads in real time. These cameras are capable of capturing images day and night, and the terminals are equipped with communication functions to send the captured image data to a server.

[1237] Data analysis method (server)

[1238] The server receives traffic video data sent from the device. The received data is analyzed using an AI-based image recognition algorithm. This analysis identifies objects such as vehicles, pedestrians, and bicycles from the video data and obtains attribute data such as their location, speed, and direction.

[1239] Signal timing optimization means (server)

[1240] The server performs calculations to optimize traffic signal timing based on traffic condition data obtained from the data analysis means. It takes into account past data and current conditions to predict future traffic flow and congestion. It calculates the optimal duration for each phase of the traffic signal (green, yellow, red).

[1241] Signal timing application means (terminal)

[1242] The optimized signal timing calculated by the server is sent to the traffic light via the application means. The terminal (traffic light) changes the signal in real time according to the received signal timing. This will result in smoother traffic flow, and is expected to alleviate traffic congestion and reduce accidents.

[1243] Emotion Engine

[1244] The emotion engine has the function of identifying user emotions and reflecting them in traffic management. Emotions are analyzed in real time, for example, from the user's voice and video. Based on the analysis results, the frustration and stress levels of road users are determined and reflected in adjustments to traffic signal timing.

[1245] Monitoring and feedback measures (users)

[1246] Users can use the system's monitoring function to monitor real-time traffic conditions. The dashboard displays information such as current traffic light timings, traffic flow, and congestion levels. Data from the emotion engine is also displayed to help users manually adjust traffic light timings. Users can also manually adjust traffic light timings in emergencies or according to specific requirements.

[1247] Specific examples

[1248] Morning rush hour scenario

[1249] 1. Traffic image acquisition (terminal)

[1250] During the morning rush hour, a camera captures video in real time at an intersection.

[1251] The device sends this video data to the server every 5 seconds.

[1252] 2. Data analysis (server)

[1253] The server analyzes the received video and detects 30% more vehicles than usual.

[1254] The number of pedestrians was also confirmed to be more than double the usual amount.

[1255] 3. Signal timing optimization (server)

[1256] The server uses this data to predict that congestion will increase further over the next five minutes.

[1257] The green light will last 20 seconds longer than usual, and the timing of pedestrian signals will be extended by 10 seconds.

[1258] 4. Signal Timing Application (Terminal)

[1259] The server calculates the new signal timing and sends it to the traffic light, where it is applied immediately.

[1260] This will allow for smoother traffic flow and reduce congestion.

[1261] 5. Utilizing the Emotion Engine (User)

[1262] Managers check the monitoring dashboard and view sentiment data along with traffic conditions.

[1263] The emotion engine detects high stress levels in the user and further fine-tunes signal timing to reflect this.

[1264] Road users' emotions are also monitored and, if necessary, traffic light adjustments are made to have a relaxing effect.

[1265] This will not only ease traffic congestion but also reduce the psychological burden on road users.

[1266] The processing flow will be explained below.

[1267] Step 1:

[1268] Video shooting (device)

[1269] The device uses a camera to capture real-time images of intersections and major roads, capturing the images frame by frame and storing the data over time.

[1270] Step 2:

[1271] Data transmission (terminal)

[1272] The video data captured by the device is sent to the server at regular intervals using a streaming protocol, and is sent in real time.

[1273] Step 3:

[1274] Data reception (server)

[1275] The server receives traffic video data sent from the terminal, and the received data is temporarily stored in a buffer for subsequent analysis.

[1276] Step 4:

[1277] Image recognition (server)

[1278] The server uses AI to analyze the video data in the buffer, and image recognition algorithms analyze each frame to identify objects such as vehicles, pedestrians, and cyclists.

[1279] Step 5:

[1280] Traffic situation analysis (server)

[1281] The server obtains attribute data such as the location, speed, and direction of the identified objects and evaluates traffic flow and congestion. The analysis results are used to determine the current traffic situation in real time.

[1282] Step 6:

[1283] Pattern prediction (server)

[1284] The server uses past traffic data and current analysis results to predict future traffic patterns, using predictive models to calculate the likelihood of future congestion and jams.

[1285] Step 7:

[1286] Optimization calculation (server)

[1287] The server calculates the optimal timing for traffic lights based on traffic conditions and predictions, optimizing the duration of each phase (green, yellow, red) and setting the traffic light timing.

[1288] Step 8:

[1289] Command generation (server)

[1290] The server generates a command to send to the traffic light based on the calculation results, which includes the new traffic light timing.

[1291] Step 9:

[1292] Send command (server)

[1293] The server sends the generated command to the traffic light, which then transmits the command to the traffic light via the network.

[1294] Step 10:

[1295] Signal timing application (terminal)

[1296] The terminal (traffic light) applies the new signal timing according to the received command. The traffic light immediately starts operating based on the new settings.

[1297] Step 11:

[1298] Emotion data collection (user)

[1299] Using an emotion engine, emotional data is obtained from the user's voice and video, analyzing the user's stress level and frustration and collecting emotional data in real time.

[1300] Step 12:

[1301] Emotion data analysis (server)

[1302] The server receives and analyzes the emotion data sent from the emotion engine, evaluates the user's emotional state indicated by the emotion data, and reflects this in traffic situation analysis.

[1303] Step 13:

[1304] Emotion-based signal timing adjustment (server)

[1305] The server analyzes the emotional data and fine-tunes the timing of the traffic lights. For example, if the user's stress level is high, the system will extend the duration of the green light.

[1306] Step 14:

[1307] Monitoring (users)

[1308] Users monitor real-time traffic conditions and emotion data using the system's dashboard, which displays information such as current traffic light timings, traffic flow, congestion levels, and the user's emotional state.

[1309] Step 15:

[1310] Feedback and manual adjustments (user)

[1311] If necessary, users can manually instruct the system to fine-tune signal timing, providing a means to respond to emergencies or specific requests.

[1312] Example 2

[1313] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1314] Optimization of traffic signals in transportation systems relies on analysis of traffic conditions and timing adjustment, but conventional systems do not take user emotions into consideration, resulting in problems such as user stress and frustration even when traffic flow is smooth.In addition, adjustment of traffic signals based on real-time traffic video and data analysis was limited, so there was a need for more precise and rapid optimization.

[1315] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1316] In this invention, the server includes means for acquiring traffic images, means for analyzing the acquired traffic images and determining traffic conditions, means for optimizing traffic light timing based on the traffic conditions, means for recognizing a user's emotions and adjusting traffic light timing based on the recognition results, and means for applying the optimized traffic light timing to the traffic lights. This enables real-time analysis and optimization of traffic conditions and also enables adjustments that take the user's emotions into consideration, thereby not only smoothing traffic flow but also reducing the psychological burden on the user.

[1317] "Traffic footage" refers to video data capturing traffic conditions, and records the movement of vehicles, pedestrians, bicycles, etc. at intersections and major roads in real time.

[1318] "Traffic conditions" refers to information that indicates the degree of congestion and movement of vehicles, pedestrians, bicycles, etc. on roads at a specific time and place.

[1319] "Traffic light timing" refers to the time it takes for a traffic light to change color (green, yellow, red), including the duration of the change to control traffic flow.

[1320] An "image recognition algorithm" is a computer algorithm that analyzes video data acquired from a camera and identifies specific objects (vehicles, pedestrians, bicycles, etc.).

[1321] "Pattern prediction" is a mathematical method for analyzing past traffic data and predicting current and future traffic conditions.

[1322] "Emotion recognition" is a technology that analyzes data such as a user's voice and facial expressions to identify their psychological state (stress, frustration, joy, etc.).

[1323] "Timing optimization" involves calculating the optimal duration of traffic light colors based on traffic conditions and user emotion data, and making adjustments to ensure the most efficient traffic flow.

[1324] "Monitoring" is the process of monitoring traffic conditions, traffic light timing, and user emotional data in real time and making adjustments as needed.

[1325] This invention not only optimizes traffic signal timing using traffic video, but also realizes advanced traffic management using user emotional data. This system captures traffic video in real time, analyzes the video data, and judges traffic conditions. Based on this judgment, it optimizes traffic signal timing and also reflects user emotional data, enabling efficient traffic management with reduced psychological burden.

[1326] Hardware and software used

[1327] Traffic image acquisition means (terminal)

[1328] The terminal is equipped with a camera that captures images of intersections and major roads in real time. This camera has the ability to capture images day and night, and sends the captured image data to a server. Specifically, it uses the H.264 codec to compress the image data and the SSL / TLS communication protocol for data communication.

[1329] Data analysis method (server)

[1330] The server receives traffic video data sent from the device, decodes the received data, and analyzes it using an AI-based image recognition algorithm (such as YOLO or OpenCV). This analysis identifies objects such as vehicles, pedestrians, and bicycles, and obtains attribute data such as location, speed, and direction.

[1331] Traffic condition data generation (server)

[1332] The server generates current traffic situation data based on the analysis results. The number of vehicles, number of pedestrians, and the speed and direction of each object are converted into statistical data and updated in real time. By storing this data in a database, it is possible to compare it with past data.

[1333] Signal timing optimization (server)

[1334] The server optimizes traffic signal timing based on traffic condition data. The optimization algorithm uses optimization theory and feedback control to calculate the duration of each phase (green, yellow, red) and update the timing in real time, including forecast data for the next five minutes.

[1335] Signal timing application means (terminal)

[1336] The optimized signal timing is sent from the server to the traffic lights, which then make real-time signal changes based on the received timing. To minimize delays, a low-latency communication protocol (e.g. MQTT) is used. The traffic light control software applies the new timing for each phase.

[1337] Emotion Engine

[1338] The emotion engine recognizes the user's emotions and reflects them in traffic management. Emotion data is acquired using a camera or microphone installed on the device, and the acquired audio and video data is analyzed on a server. Specifically, an emotion recognition algorithm (such as IBM Watson or Microsoft Azure's emotion recognition API) is used. Based on the emotion data, the level of stress or anger is determined and traffic light timing is adjusted.

[1339] Monitoring and Feedback (User)

[1340] Users can view real-time traffic conditions and sentiment data on a monitoring dashboard, which displays current traffic signal timings, traffic flow, congestion levels, and user sentiment data. Users can also manually adjust traffic signal timings in case of emergencies or specific requirements.

[1341] Examples of concrete examples and prompts

[1342] Specific examples

[1343] During the morning rush hour, cameras capture traffic footage in real time and send the acquired video data to a server. The server analyzes the data and detects an increase in vehicles and pedestrians compared to normal. It predicts that congestion will increase further over the next five minutes and makes adjustments such as extending the duration of green lights. User emotion data is also collected, and if high stress levels are detected, further fine-tuning of signal timing is made. Administrators can monitor this from a dashboard and make manual adjustments as needed.

[1344] Prompt Sentence Examples

[1345] An example prompt is:

[1346] Write a program that analyzes current traffic video data and optimizes traffic light timing. Specifically, for intersection A, with 50 vehicles and 30 pedestrians, extend the current green light by 10 seconds and shorten the pedestrian signal timing by 5 seconds. Also, further fine-tune the traffic light timing based on emotion data.

[1347] This system enables real-time analysis and optimization of traffic conditions, and more effective traffic management can be achieved by making adjustments that take into account user emotional data.

[1348] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1349] Step 1:

[1350] Traffic image acquisition (terminal)

[1351] The device is equipped with a camera that captures images of intersections and major roads in real time. Specifically, the camera captures images at 30 frames per second (fps) and continues to capture images at night using infrared functionality. The input is real-time images of intersections and roads, and the output is video data compressed using the H.264 codec.

[1352] Step 2:

[1353] Video data transmission (terminal)

[1354] The terminal transmits the acquired video data to the server. Specifically, the acquired video data is compressed every second and transmitted to the server using the SSL / TLS communication protocol. The input is video data compressed using H.264, and the output is encrypted data packets.

[1355] Step 3:

[1356] Traffic video analysis (server)

[1357] The server decodes the video data received from the device and analyzes traffic conditions by using AI for image recognition. YOLO and OpenCV are used as image recognition algorithms. Specifically, it identifies objects such as vehicles, pedestrians, and bicycles from the video data and determines their respective positions, speeds, and directions. The input is the video data sent to the server, and the output is attribute data for each object (position, speed, direction, etc.).

[1358] Step 4:

[1359] Traffic condition data generation (server)

[1360] The server generates current traffic situation data based on the analysis results. It converts the number of vehicles, the number of pedestrians, and the speed and direction of each object into statistical data and updates it in real time. Specifically, it saves the analysis results in list format and records them in a database. The input is the analysis results of the traffic video, and the output is statistical traffic situation data.

[1361] Step 5:

[1362] Signal timing optimization (server)

[1363] The server optimizes signal timing based on traffic condition data. The optimization algorithm uses optimization theory and feedback control. Specifically, it calculates the optimal duration of each phase (green, yellow, red). The input is traffic condition data updated in real time and past data, and the output is optimized signal timing data.

[1364] Step 6:

[1365] Optimized timing transmission (server)

[1366] The server transmits the optimized signal timing to the traffic lights using a low-latency communication protocol (e.g., MQTT). Specifically, it creates optimized timing data packets and sends them to the traffic lights. The input is the optimized signal timing data, and the output is the new signal timing that is applied to the traffic light control software.

[1367] Step 7:

[1368] Timing application to traffic lights (terminal)

[1369] The terminal (traffic light) changes the signal in real time according to the received signal timing. For example, the delay when changing from red to green is kept within a few milliseconds. The input is the new signal timing data sent from the server, and the output is the changed color of the traffic light.

[1370] Step 8:

[1371] Acquisition and analysis of emotion data (terminal / server)

[1372] The device (for example, an emotion-recognition camera or microphone) acquires the user's voice and video data. The server analyzes this data and recognizes the user's emotions using an emotion engine (for example, IBM Watson or Microsoft Azure's emotion recognition API). Specifically, it detects the level of stress or anger from the user's voice and facial expression. The input is the user's voice and video data, and the output is analyzed emotional data.

[1373] Step 9:

[1374] Adjustment of signal timing based on emotion data (server)

[1375] The server further adjusts the signal timing based on the acquired emotion data. For example, if the user's stress level is high, the algorithm adjusts to shorten the duration of the green light. The input is the analyzed emotion data in real time, and the output is fine-tuned signal timing data.

[1376] Step 10:

[1377] Monitoring and Feedback (User)

[1378] Users can check real-time traffic conditions and emotion data on a monitoring dashboard. The dashboard displays current traffic signal timings, traffic flow, congestion levels, and user emotion data. Users can also manually adjust traffic signal timings in emergencies or according to specific requirements. The inputs are traffic condition data and emotion data, and the output is the traffic signal timings manually adjusted by the user.

[1379] (Application example 2)

[1380] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1381] Conventional traffic management systems focus on analyzing traffic images and optimizing traffic signal timing, but real-time understanding and adaptation to traffic conditions is required to ensure smooth traffic flow. Furthermore, one of the factors that contribute to traffic accidents and congestion is the emotion and stress of drivers and pedestrians, yet few systems exist that take this into account. Therefore, new systems are needed that can optimize traffic flow while reducing the psychological burden on road users. Furthermore, with the spread of autonomous vehicles, new traffic management systems that can accommodate this are also required.

[1382] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1383] In this invention, the server includes means for acquiring traffic video, means for analyzing the traffic video and determining traffic conditions, means for optimizing traffic light timing based on the traffic conditions, means for applying the optimized traffic light timing to the traffic lights, means for analyzing user emotions and adjusting traffic light timing based on the emotions, and means for monitoring surrounding traffic conditions in real time using video acquired by a camera in the autonomous vehicle and optimizing the driving pattern of the autonomous vehicle based on the analysis results, thereby enabling optimization of traffic flow, flexible traffic management in response to the emotions of road users, and safe and efficient operation of the autonomous vehicle.

[1384] The "means for acquiring traffic images" refers to a device that uses cameras installed at locations such as intersections and major roads to acquire images of traffic conditions in real time.

[1385] The "means for analyzing traffic images and determining traffic conditions" is a device that uses an AI-based image recognition algorithm to analyze the status of vehicles, pedestrians, and traffic lights based on acquired traffic images, and determines the current traffic conditions.

[1386] A "means for optimizing traffic signal timing based on traffic conditions" is a device that uses analyzed traffic condition data to calculate the duration of green, yellow, and red signals at a traffic signal and optimize traffic flow.

[1387] The "means for applying optimized signal timing to traffic lights" is a device that transmits the calculated optimal signal timing to actual traffic lights and changes the signals in real time.

[1388] The "means for analyzing the user's emotions and adjusting the timing of traffic lights based on those emotions" refers to a device that analyzes emotions from the user's voice and video, and reflects that data in a traffic management system, thereby adjusting the timing of traffic lights based on the user's emotions.

[1389] "Means for monitoring surrounding traffic conditions in real time using images captured by a camera on an autonomous vehicle" refers to a device that uses a camera mounted on an autonomous vehicle to capture images of traffic conditions, acquires the image data in real time, and monitors surrounding traffic.

[1390] The "means for optimizing the driving pattern of an autonomous vehicle based on the analysis results" refers to a device that adjusts the speed and route of an autonomous vehicle to the optimum based on the analysis results of the acquired video data.

[1391] This invention combines an emotion engine with a function for linking with autonomous vehicles to optimize traffic light timing using traffic video. This system acquires traffic video in real time, analyzes and predicts the data, and optimizes traffic light timing. Furthermore, by analyzing user emotions and reflecting them in traffic light timing, it is possible to enhance traffic management and optimize the operation of autonomous vehicles.

[1392] Overall system overview

[1393] The system consists of the following main components:

[1394] 1. Means for acquiring traffic images (devices including cameras)

[1395] 2. A means of analyzing data and determining traffic conditions (server)

[1396] 3. Means for optimizing traffic light timing (server)

[1397] 4. Means (terminal) for applying optimized signal timing to traffic lights

[1398] 5. A means to analyze user emotions and adjust traffic light timing based on those emotions (emotion engine)

[1399] 6. Means of monitoring surrounding traffic conditions using images captured by cameras in autonomous vehicles (cameras and systems for autonomous vehicles)

[1400] 7. Means for optimizing the driving patterns of autonomous vehicles based on analysis results (autonomous driving systems)

[1401] Hardware and software used

[1402] The following hardware and software are used to implement this system.

[1403] Hardware: Cameras, autonomous vehicle cameras, servers, traffic light control terminals

[1404] Software: OpenCV (image analysis), TensorFlow (AI modeling), Python scripts (data analysis), ROS (Robot Operating System, integration with autonomous driving systems)

[1405] Program processing explanation

[1406] The server first receives real-time video data from cameras to capture traffic images. It then analyzes the received video data using OpenCV and TensorFlow to determine traffic conditions. Traffic data includes the position, speed, and direction of objects such as vehicles, pedestrians, and bicycles. It then performs calculations to optimize traffic light timing based on the traffic conditions and sends the optimal signal timing to the traffic light control terminal.

[1407] Meanwhile, an emotion engine is used to analyze user emotions. This engine collects passenger emotional data from audio and video and analyzes it in real time. This emotional data is used to fine-tune signal timing, setting optimal signal timing to reduce passenger stress.

[1408] Finally, the autonomous vehicle's camera continuously monitors the surrounding traffic conditions and analyzes the acquired data in real time. Using ROS, the system instructs the autonomous driving system on optimal driving patterns, helping to ensure smooth traffic flow. In particular, taking into account passenger emotional data can help ensure a comfortable and safe operation.

[1409] Adding concrete examples and prompt sentences for the generative AI model

[1410] For example, consider the following system situation:

[1411] 1. Traffic video acquisition: Cameras on autonomous vehicles capture real-time traffic conditions at intersections and transmit the video data to a server inside the vehicle.

[1412] 2. Data analysis: The server analyzes the received video and determines traffic flow based on information such as the location, speed, and direction of identified vehicles and pedestrians.

[1413] 3. Optimization: Based on the analysis results, the speed and route of autonomous vehicles are optimized and traffic light timing is adjusted.

[1414] 4. Emotion analysis: Analyzes passengers' emotions in real time from their voice data and fine-tunes traffic signal timing to ensure stress-free driving.

[1415] Example prompt for a generative AI model:

[1416] Analyze traffic video data in real time to identify vehicles, pedestrians, and bicycles. Also, analyze passengers' emotions based on their voice data and select driving patterns according to their stress levels. Use TensorFlow and OpenCV for video data analysis, and an emotion engine for emotion analysis. Use ROS for integration with the autonomous driving system.

[1417] According to the above aspects, the present invention can optimize traffic flow, reduce the psychological burden on traffic users, and realize safe and efficient operation of autonomous vehicles.

[1418] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1419] Step 1:

[1420] Traffic footage acquisition

[1421] The device captures real-time traffic images using cameras installed at intersections and major roads. This image data is also captured by cameras on autonomous vehicles. The captured image data is sent directly to the server. The input data is the image data from the cameras, and the output data is the image data sent to the server.

[1422] Step 2:

[1423] Traffic video analysis

[1424] The server processes the received traffic video data. Specifically, it performs image analysis using OpenCV and TensorFlow. Image analysis identifies objects such as vehicles, pedestrians, and bicycles, and calculates their respective positions, speeds, and directions. The input data is the traffic video data, and the output data is attribute data of the identified objects.

[1425] Step 3:

[1426] Judging traffic conditions

[1427] The server determines the current traffic situation based on the image analysis results. Specifically, it calculates traffic flow and congestion levels based on the position, speed, and direction of each object. This makes it possible to identify traffic bottlenecks and dangerous areas. The input data is the attribute data of the identified objects, and the output data is the traffic situation assessment result.

[1428] Step 4:

[1429] Traffic light timing optimization

[1430] The server optimizes traffic light timing based on the traffic conditions it determines. It also takes into account past data and current conditions to calculate the duration of each traffic light phase (green, yellow, red). The input data is the traffic condition determination result, and the output data is the optimized traffic light timing.

[1431] Step 5:

[1432] Signal Timing Enforcement

[1433] The terminal applies the optimized signal timing received from the server to the actual traffic light, which changes the signal in real time and ensures smooth traffic flow. The input data is the optimized signal timing, and the output data is the result of the traffic light timing change.

[1434] Step 6:

[1435] User sentiment analysis

[1436] The emotion engine analyzes the user's (passenger's) voice and video data in real time to determine emotions such as stress and frustration, thereby understanding the user's psychological state. The input data is the user's voice and video data, and the output data is the analyzed emotion data.

[1437] Step 7:

[1438] Fine-tuning signal timing

[1439] The server fine-tunes the optimized signal timing based on the emotion data obtained from the emotion engine. This sets the optimal signal timing to reduce the user's stress. The input data is the analyzed emotion data, and the output data is the fine-tuned signal timing.

[1440] Step 8:

[1441] Optimizing driving patterns for autonomous vehicles

[1442] The autonomous vehicle system optimizes driving patterns based on acquired traffic video data and emotion data. Specifically, it uses ROS to adjust the vehicle's speed and route. The input data is traffic video data and emotion data, and the output data is the optimized driving pattern.

[1443] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1444] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1445] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1446] [Fourth embodiment]

[1447] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1448] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1449] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1450] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1451] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1452] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1453] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1454] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1455] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1456] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1457] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1458] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1459] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1460] This invention relates to a traffic light timing optimization system that uses traffic video. This system acquires traffic video in real time, analyzes and predicts traffic conditions based on the video, and optimizes traffic light timing.

[1461] Overall system overview

[1462] The system consists of the following main components:

[1463] 1. Traffic image acquisition means (terminal)

[1464] 2. Data analysis method (server)

[1465] 3. Signal timing optimization means (server)

[1466] 4. Signal Timing Application Means (Terminal)

[1467] 5. Monitoring and Feedback Methods (Users)

[1468] Traffic image acquisition means (terminal)

[1469] The terminals are equipped with cameras that capture images of intersections and major roads in real time. These cameras are capable of capturing images day and night, and the terminals are equipped with communication functions to send the captured image data to a server.

[1470] Data analysis method (server)

[1471] The server receives traffic video data sent from the device. The received data is analyzed using an AI-based image recognition algorithm. This analysis identifies objects such as vehicles, pedestrians, and bicycles from the video data and obtains attribute data such as their location, speed, and direction.

[1472] Signal timing optimization means (server)

[1473] The server performs calculations to optimize traffic signal timing based on traffic condition data obtained from the data analysis means. It takes into account past data and current conditions to predict future traffic flow and congestion. It calculates the optimal duration for each phase of the traffic signal (green, yellow, red).

[1474] Signal timing application means (terminal)

[1475] The optimized signal timing calculated by the server is sent to the traffic light via the application means. The terminal (traffic light) changes the signal in real time according to the received signal timing. This will result in smoother traffic flow, and is expected to alleviate traffic congestion and reduce accidents.

[1476] Monitoring and feedback measures (users)

[1477] Users can use the system's monitoring function to monitor real-time traffic conditions. The dashboard displays information such as current traffic light timings, traffic flow, and congestion levels. Users can also manually adjust traffic light timings in case of emergencies or specific requirements.

[1478] Specific examples

[1479] Morning rush hour scenario

[1480] 1. Traffic image acquisition (terminal)

[1481] During the morning rush hour, a camera captures video in real time at an intersection.

[1482] The device sends this video data to the server every 5 seconds.

[1483] 2. Data analysis (server)

[1484] The server analyzes the received video and detects 30% more vehicles than usual.

[1485] The number of pedestrians was also confirmed to be more than double the usual amount.

[1486] 3. Signal timing optimization (server)

[1487] The server uses this data to predict that congestion will increase further over the next five minutes.

[1488] The green light will last 20 seconds longer than usual, and the timing of pedestrian signals will be extended by 10 seconds.

[1489] 4. Signal Timing Application (Terminal)

[1490] The server calculates the new signal timing and sends it to the traffic light, where it is applied immediately.

[1491] This will allow for smoother traffic flow and reduce congestion.

[1492] 5. Monitoring (User)

[1493] Managers can check the dashboard to see improvements in traffic conditions in real time.

[1494] You can also make manual adjustments if needed.

[1495] These measures make it possible to use traffic footage to optimize traffic light timing in real time, improving overall road efficiency.

[1496] The processing flow will be explained below.

[1497] Step 1:

[1498] Video shooting (device)

[1499] The device uses a camera to capture real-time images of intersections and major roads, capturing the images frame by frame and storing the data over time.

[1500] Step 2:

[1501] Data transmission (terminal)

[1502] The video data captured by the device is sent to the server at regular intervals using a streaming protocol, and is sent in real time.

[1503] Step 3:

[1504] Data reception (server)

[1505] The server receives traffic video data sent from the terminal, and the received data is temporarily stored in a buffer for subsequent analysis.

[1506] Step 4:

[1507] Image recognition (server)

[1508] The server uses AI to analyze the video data in the buffer, and image recognition algorithms analyze each frame to identify objects such as vehicles, pedestrians, and cyclists.

[1509] Step 5:

[1510] Traffic situation analysis (server)

[1511] The server obtains attribute data such as the location, speed, and direction of the identified objects and evaluates traffic flow and congestion. The analysis results are used to determine the current traffic situation in real time.

[1512] Step 6:

[1513] Pattern prediction (server)

[1514] The server uses past traffic data and current analysis results to predict future traffic patterns, using predictive models to calculate the likelihood of future congestion and jams.

[1515] Step 7:

[1516] Optimization calculation (server)

[1517] The server calculates the optimal timing for traffic lights based on traffic conditions and predictions, optimizing the duration of each phase (green, yellow, red) and setting the traffic light timing.

[1518] Step 8:

[1519] Command generation (server)

[1520] The server generates a command to send to the traffic light based on the calculation results, which includes the new traffic light timing.

[1521] Step 9:

[1522] Send command (server)

[1523] The server sends the generated command to the traffic light, which then transmits the command to the traffic light via the network.

[1524] Step 10:

[1525] Signal timing application (terminal)

[1526] The terminal (traffic light) applies the new signal timing according to the received command. The traffic light immediately starts operating based on the new settings.

[1527] Step 11:

[1528] Real-time monitoring (user)

[1529] Users monitor real-time traffic conditions using the system's dashboard, which displays information such as current traffic light timings, traffic flow, and congestion levels.

[1530] Step 12:

[1531] Feedback and manual adjustments (user)

[1532] If necessary, users can manually instruct the system to fine-tune signal timing, providing a means to respond to emergencies or specific requests.

[1533] Example 1

[1534] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1535] Current traffic management systems have fixed signal timing settings, making it difficult to optimize them to accommodate real-time traffic conditions, which can lead to traffic congestion and accidents. Furthermore, they lack the means to quickly respond to sudden changes in traffic conditions, making it difficult to alleviate sudden congestion and jams.

[1536] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1537] In this invention, the server includes a means for acquiring traffic images, a means for analyzing the acquired traffic images to determine traffic conditions, and a means for optimizing traffic light timing based on the traffic conditions. This makes it possible to grasp traffic conditions in real time and calculate and apply appropriate traffic light timing. Furthermore, by including a means for manually adjusting traffic light timing based on the analysis results, it is possible to quickly respond to sudden changes in traffic conditions.

[1538] "Traffic footage" is video data that records traffic flow and the movements of vehicles and pedestrians.

[1539] "Analysis" is the process of extracting specific information from the acquired video data and determining the traffic situation.

[1540] "Traffic conditions" refers to information that indicates the movement and status of vehicles, pedestrians, etc. in a specific location.

[1541] "Traffic light timing" refers to the duration and switching timing of each phase of a traffic light (green, yellow, red).

[1542] "Optimization" is the process of adjusting signal timing to achieve a specific goal (e.g., smooth traffic flow or fewer accidents).

[1543] "Apply" means to set the optimized signal timing to an actual traffic light.

[1544] The "means for manually adjusting signal timing" is a function that allows the user to manually change the signal timing as needed.

[1545] An "image recognition algorithm" is a computational method for identifying specific objects (e.g., vehicles, pedestrians) from video data.

[1546] "Pattern prediction" is the process of predicting future traffic conditions based on past data.

[1547] MODE FOR CARRYING OUT THE INVENTION

[1548] This invention relates to a traffic light timing optimization system that uses traffic video. This system acquires traffic video in real time, analyzes and predicts traffic conditions based on the video, and optimizes traffic light timing. Specifically, it consists of the following main components:

[1549] 1. Traffic image acquisition means (terminal)

[1550] 2. Data analysis method (server)

[1551] 3. Signal timing optimization means (server)

[1552] 4. Signal Timing Application Means (Terminal)

[1553] 5. Monitoring and Feedback Methods (Users)

[1554] Traffic image acquisition means (terminal)

[1555] The device is equipped with a high-resolution camera that captures images of intersections and major roads in real time. This camera operates 24 hours a day, capturing images day and night. The captured image data is compressed every five seconds and sent to a server. Communication is via an internet connection. For example, Sony's high-resolution security cameras can be used.

[1556] Data analysis method (server)

[1557] The server receives traffic video data sent from the device and temporarily stores it in a storage area. The server then analyzes the video data using AI image recognition algorithms such as YOLO and OpenCV. During this analysis, objects such as vehicles, pedestrians, and bicycles are identified and attribute data such as their location, speed, and direction are acquired. The acquired attribute data is stored on the server and used for subsequent analysis and prediction.

[1558] Signal timing optimization means (server)

[1559] The server uses traffic condition data obtained from the data analysis tool to perform calculations to optimize traffic light timing. It uses past data and a pattern prediction algorithm to predict traffic congestion over the next five minutes. For example, it can extend the green light duration by 20 seconds and the pedestrian signal timing by 10 seconds to smooth traffic flow.

[1560] Signal timing application means (terminal)

[1561] The optimized signal timing calculated by the server is sent to the traffic light via the terminal. The traffic light changes its signal in real time according to the received signal timing. The traffic light is controlled by sending instructions to the signal control unit using the TCP / IP stack.

[1562] Monitoring and feedback measures (users)

[1563] Users monitor traffic conditions in real time through the system's dashboard, which visually displays current traffic light timings, traffic volume at each intersection, and pedestrian flow. Users can manually adjust traffic light timings as needed. For example, they may temporarily change traffic light timings to respond to specific events or emergencies. System parameters can also be optimized based on real-time data feedback.

[1564] Specific examples

[1565] Morning rush hour scenario

[1566] 1. Traffic image acquisition (terminal)

[1567] During the morning rush hour, a camera captures video in real time at an intersection.

[1568] The device sends this video data to the server every 5 seconds.

[1569] 2. Data analysis (server)

[1570] The server analyzes the received video and detects 30% more vehicles than usual.

[1571] The server also verifies that the number of pedestrians is more than double the normal amount.

[1572] 3. Signal timing optimization (server)

[1573] The server uses this data to predict that congestion will increase further over the next five minutes.

[1574] The green light will last 20 seconds longer than usual, and the timing of pedestrian signals will be extended by 10 seconds.

[1575] 4. Signal Timing Application (Terminal)

[1576] The server calculates the new signal timing and sends it to the traffic light, where it is applied immediately.

[1577] The terminal adjusts traffic signals to ensure smooth vehicle flow and reduce congestion.

[1578] 5. Monitoring (User)

[1579] Managers can check the dashboard to see improvements in traffic conditions in real time.

[1580] The user manually fine-tunes as needed.

[1581] Example prompts for generative AI models

[1582] An example prompt for a traffic signal timing optimization system is as follows:

[1583] Prompt: Explain how to optimize traffic signals.

[1584]

[1585] Generative AI Model: Traffic signal optimization can be achieved by capturing traffic video in real time and analyzing it with AI algorithms. Based on the analyzed data, traffic signal timing can be calculated and applied to smooth traffic flow, reducing congestion and accidents.

[1586] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1587] Program processing flow

[1588] Each processing step will be explained by clearly showing the specific operation, input, and output.

[1589] Step 1: Acquiring and transmitting traffic images (terminal)

[1590] Input: Real-time camera footage monitoring traffic flow.

[1591] Specific operation: A high-resolution camera installed on the device captures images of intersections and major roads in real time. This camera operates 24 hours a day, capturing images day and night.

[1592] Data processing: The device compresses the acquired video data every 5 seconds (e.g., H.264 format) and converts it into a data size that can be transmitted.

[1593] Output: The compressed video data is sent to the server via the Internet.

[1594] Step 2: Receiving video data (server)

[1595] Input: Compressed video data sent from the device.

[1596] Specific operation: The server receives the video data transmitted from the terminal.

[1597] Data processing: After receiving, the data is checked for consistency and completeness, and temporarily stored in a database using a streaming API (e.g., FFmpeg).

[1598] Output: Saved video data.

[1599] Step 3: Data analysis and traffic situation evaluation (server)

[1600] Input: Stored video data.

[1601] Specific operation: The server uses YOLO (You Only Look Once) and OpenCV to analyze the received video data.

[1602] Data processing: Identify objects such as vehicles, pedestrians, and bicycles from video data and obtain their attribute data (position, speed, direction).

[1603] Output: Attribute data of identified objects.

[1604] Step 4: Calculate signal timing (server)

[1605] Input: Attribute data of the identified object.

[1606] How it works: The server uses the analysis results to calculate optimal traffic light timings. It uses historical data and pattern prediction algorithms to predict future traffic congestion.

[1607] Data processing: Calculations are made to adjust the duration of green and pedestrian signals based on congestion predictions.

[1608] Output: Optimized signal timing configuration data.

[1609] Step 5: Sending the calculation results (server)

[1610] Input: Optimized signal timing configuration data.

[1611] Specific operation: The server encodes the calculated optimized signal timing in JSON format and sends it to the terminal.

[1612] Data processing: Data is transmitted securely using the HTTPS protocol.

[1613] Output: Signal timing configuration data sent to the terminal.

[1614] Step 6: Apply signal timing (terminal)

[1615] Input: Signal timing configuration data sent from the server.

[1616] Specific operation: The terminal analyzes the received signal timing and applies it to the signal control unit.

[1617] Data processing: Send instructions to the signal control unit using the TCP / IP stack.

[1618] Output: Traffic light timing is optimized in real time.

[1619] Step 7: Monitoring and Feedback (Users)

[1620] Input: Real-time traffic and signal timing data.

[1621] Specific operation: The user monitors the traffic situation in real time through the system's dashboard.

[1622] Data processing: Visually display information such as current traffic light timings, traffic volume at each intersection, and pedestrian flow on the dashboard.

[1623] Output: The user manually adjusts the signal timing as needed to optimize the system settings.

[1624] (Application example 1)

[1625] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1626] To smooth traffic flow, alleviate congestion, and reduce accidents, it is necessary to analyze traffic conditions in real time with high accuracy and optimize traffic signal timing based on that analysis. Furthermore, with the spread of autonomous vehicles, coordination between vehicles and traffic signals is becoming even more important. However, current systems are not fully capable of achieving these goals, and the problems of traffic congestion and accidents remain unresolved. Furthermore, there is a lack of an information-sharing mechanism that allows autonomous vehicles to flexibly adapt to the traffic environment.

[1627] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1628] In this invention, the server includes means for acquiring traffic images, means for analyzing the acquired traffic images to determine traffic conditions, means for optimizing traffic light timings based on the traffic conditions, means for applying the optimized traffic light timings to the traffic lights, and means for communicating with autonomous vehicles and providing the optimized traffic light timings, thereby enabling autonomous vehicles to acquire optimized traffic light timing information in real time, making vehicle operation smoother and safer.

[1629] "Traffic video" refers to video data of intersections and major roads, captured in real time using cameras.

[1630] "Analysis" is the process of processing traffic video data and extracting attributes such as the location, speed, and direction of vehicles and pedestrians.

[1631] "Traffic conditions" refers to the movement and concentration of vehicles and pedestrians at a certain point, including the degree of congestion and flow in real time.

[1632] "Traffic light timing" refers to the duration of each green, yellow, and red phase of a traffic signal, and is adjusted to optimize traffic flow.

[1633] "Optimization" refers to adjusting the timing of traffic lights based on current traffic conditions, aiming to smooth traffic flow and alleviate congestion.

[1634] An "autonomous vehicle" is a vehicle that perceives its surroundings and drives autonomously without the intervention of a human driver.

[1635] "Communication" refers to the process of exchanging data between the autonomous vehicle and a server, sharing information about traffic signal timings and traffic conditions.

[1636] "Provision" refers to the act of the server sharing and transmitting necessary data and information to an autonomous vehicle.

[1637] This invention relates to a traffic light timing optimization system using traffic video. This system has the function of acquiring traffic video in real time, analyzing and predicting it, and optimizing traffic light timing. The optimized traffic light timing is also provided to autonomous vehicles, making vehicle operation smoother and safer.

[1638] Overall system overview

[1639] The system consists of the following main components:

[1640] 1. Traffic image acquisition means (terminal)

[1641] 2. Data analysis method (server)

[1642] 3. Signal timing optimization means (server)

[1643] 4. Signal Timing Application Means (Terminal)

[1644] 5. Monitoring and Feedback Methods (Users)

[1645] 6. Communication method with autonomous vehicles (server)

[1646] Traffic image acquisition means (terminal)

[1647] The terminals are equipped with high-resolution cameras that capture real-time images of intersections and major roads. The cameras can capture images day and night, and the terminals are equipped with communication functions to periodically send the captured image data to a server.

[1648] Data analysis method (server)

[1649] The server receives traffic video data sent from the devices and analyzes it using the open-source image processing library "OpenCV." "TensorFlow" is used as the AI ​​model to identify objects such as vehicles, pedestrians, and bicycles from the video data and obtain attribute data such as their position, speed, and direction.

[1650] Signal timing optimization means (server)

[1651] The server performs calculations to optimize traffic light timing based on traffic condition data obtained from the data analysis means. Based on past data and current conditions, it predicts future traffic flow and congestion and calculates the optimal duration of green, yellow, and red lights.

[1652] Signal timing application means (terminal)

[1653] The optimized signal timing calculated by the server is sent to the traffic light via the application means. The terminal (traffic light) changes the signal in real time according to the received signal timing. This will result in smoother traffic flow, and is expected to alleviate traffic congestion and reduce accidents.

[1654] Monitoring and feedback measures (users)

[1655] Users can use the system's monitoring function to monitor real-time traffic conditions. The dashboard displays information such as current traffic light timings, traffic flow, and congestion levels. Users can also manually adjust traffic light timings in case of emergencies or specific requirements.

[1656] Communication method with autonomous vehicles (server)

[1657] The server communicates with the autonomous vehicles and provides optimized traffic light timing in real time, allowing the autonomous vehicles to operate safely and smoothly with the latest traffic information.

[1658] Specific examples

[1659] Morning rush hour scenario

[1660] Traffic video capture: During the morning rush hour, a camera captures video in real time at an intersection. The device sends this video data to a server every 5 seconds.

[1661] Data analysis: The server analyzes the received video and detects 30% more vehicles than usual. The number of pedestrians is also confirmed to be more than double the usual number.

[1662] Traffic light timing optimization: Using this data, the server predicts that congestion will increase further over the next five minutes, so it extends the green light duration by 20 seconds and the pedestrian signal timing by 10 seconds.

[1663] Signal Timing Application: The server calculates new signal timings and sends them to the traffic lights, where they are applied immediately. This allows for smoother traffic flow and reduces congestion.

[1664] Monitoring: Administrators can view the dashboard to see improvements in traffic conditions in real time, and can make manual adjustments as needed.

[1665] Example prompt sentence:

[1666] Scenario:

[1667] 1. Traffic Video Capture: During peak hours, a vehicle's onboard camera captures video of an intersection.

[1668] The video data is transmitted to the server every 5 seconds.

[1669] 2. Data Analysis (Server): The server receives and analyzes the video.

[1670] It detects a 40% increase in vehicles and doubles the number of pedestrians compared to normal.

[1671] 3. Optimized Signal Timing Calculation (Server):

[1672] Predicting a further increase in congestion in the next 5 minutes, the server adjusts:

[1673] Green light duration is extended by 30 seconds and pedestrian signal timing by 15 seconds.

[1674] 4. Optimized Timing Application (Vehicle):

[1675] The optimized signal timing is sent back to the vehicle,

[1676] ensuring smooth traffic flow and reduced congestion.

[1677] 5. Monitoring (User): The vehicle's control system adapts in real-time, enhancing safety and efficiency.

[1678] These measures make it possible to use traffic footage to optimize traffic light timing in real time, improving overall road efficiency.

[1679] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1680] Step 1:

[1681] Traffic image acquisition (terminal):

[1682] A camera installed on the device captures images of intersections and major roads in real time. Images are captured day and night, and the image data from the camera is sent to the server periodically (e.g., every 5 seconds). The input is real-time traffic image data, and the output is the data transferred to the server.

[1683] Step 2:

[1684] Data reception (server):

[1685] The server receives traffic video data sent from the terminal. This data is added to a queue for image analysis and prepared for the next processing step. The input is video data from the terminal, and the output is data added to the analysis queue.

[1686] Step 3:

[1687] Data analysis (server):

[1688] The server analyzes the video data using the open-source image processing library "OpenCV." It uses "TensorFlow" as an AI model to identify objects such as vehicles, pedestrians, and bicycles from the video data and obtain attribute data such as their position, speed, and direction. The input is the received traffic video data, and the output is the analyzed traffic attribute data.

[1689] Step 4:

[1690] Signal timing optimization (server):

[1691] The server performs calculations to optimize traffic light timing based on the analyzed traffic attribute data. Based on past data and current conditions, it predicts future traffic flow and congestion and calculates the optimal durations of green, yellow, and red lights. The input is the analyzed traffic attribute data and past traffic data, and the output is optimized traffic light timing.

[1692] Step 5:

[1693] Signal Timing Application (Terminal):

[1694] The optimized signal timing calculated by the server is sent to the terminal (traffic light). The traffic light changes the signal in real time according to the received timing data. The input is the optimized signal timing, and the output is the traffic light timing change.

[1695] Step 6:

[1696] Monitoring (User):

[1697] Users can use the system's monitoring function to monitor real-time traffic conditions. The dashboard displays information such as current traffic light timing, traffic flow, and congestion level. Users can also manually adjust traffic light timing if necessary. The input is real-time traffic data and traffic light timing information, and the output is the user's monitoring results and feedback.

[1698] Step 7:

[1699] Providing optimized signal timing (server):

[1700] The server communicates with the autonomous vehicle and provides optimized signal timing in real time, allowing the autonomous vehicle to operate safely and smoothly using the latest traffic information. The input is optimized signal timing, and the output is data provided to the autonomous vehicle.

[1701] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1702] This invention combines an emotion engine with a traffic light timing optimization system that uses traffic video. This system acquires traffic video in real time, analyzes and predicts traffic conditions based on that data, and optimizes traffic light timing. It also recognizes user emotions and reflects them in traffic signal timing, achieving more advanced traffic management.

[1703] Overall system overview

[1704] The system consists of the following main components:

[1705] 1. Traffic image acquisition means (terminal)

[1706] 2. Data analysis method (server)

[1707] 3. Signal timing optimization means (server)

[1708] 4. Signal Timing Application Means (Terminal)

[1709] 5. Emotion Engine

[1710] 6. Monitoring and Feedback Methods (Users)

[1711] Traffic image acquisition means (terminal)

[1712] The terminals are equipped with cameras that capture images of intersections and major roads in real time. These cameras are capable of capturing images day and night, and the terminals are equipped with communication functions to send the captured image data to a server.

[1713] Data analysis method (server)

[1714] The server receives traffic video data sent from the device. The received data is analyzed using an AI-based image recognition algorithm. This analysis identifies objects such as vehicles, pedestrians, and bicycles from the video data and obtains attribute data such as their location, speed, and direction.

[1715] Signal timing optimization means (server)

[1716] The server performs calculations to optimize traffic signal timing based on traffic condition data obtained from the data analysis means. It takes into account past data and current conditions to predict future traffic flow and congestion. It calculates the optimal duration for each phase of the traffic signal (green, yellow, red).

[1717] Signal timing application means (terminal)

[1718] The optimized signal timing calculated by the server is sent to the traffic light via the application means. The terminal (traffic light) changes the signal in real time according to the received signal timing. This will result in smoother traffic flow, and is expected to alleviate traffic congestion and reduce accidents.

[1719] Emotion Engine

[1720] The emotion engine has the function of identifying user emotions and reflecting them in traffic management. Emotions are analyzed in real time, for example, from the user's voice and video. Based on the analysis results, the frustration and stress levels of road users are determined and reflected in adjustments to traffic signal timing.

[1721] Monitoring and feedback measures (users)

[1722] Users can use the system's monitoring function to monitor real-time traffic conditions. The dashboard displays information such as current traffic light timings, traffic flow, and congestion levels. Data from the emotion engine is also displayed to help users manually adjust traffic light timings. Users can also manually adjust traffic light timings in emergencies or according to specific requirements.

[1723] Specific examples

[1724] Morning rush hour scenario

[1725] 1. Traffic image acquisition (terminal)

[1726] During the morning rush hour, a camera captures video in real time at an intersection.

[1727] The device sends this video data to the server every 5 seconds.

[1728] 2. Data analysis (server)

[1729] The server analyzes the received video and detects 30% more vehicles than usual.

[1730] The number of pedestrians was also confirmed to be more than double the usual amount.

[1731] 3. Signal timing optimization (server)

[1732] The server uses this data to predict that congestion will increase further over the next five minutes.

[1733] The green light will last 20 seconds longer than usual, and the timing of pedestrian signals will be extended by 10 seconds.

[1734] 4. Signal Timing Application (Terminal)

[1735] The server calculates the new signal timing and sends it to the traffic light, where it is applied immediately.

[1736] This will allow for smoother traffic flow and reduce congestion.

[1737] 5. Utilizing the Emotion Engine (User)

[1738] Managers check the monitoring dashboard and view sentiment data along with traffic conditions.

[1739] The emotion engine detects high stress levels in the user and further fine-tunes signal timing to reflect this.

[1740] Road users' emotions are also monitored and, if necessary, traffic light adjustments are made to have a relaxing effect.

[1741] This will not only ease traffic congestion but also reduce the psychological burden on road users.

[1742] The processing flow will be explained below.

[1743] Step 1:

[1744] Video shooting (device)

[1745] The device uses a camera to capture real-time images of intersections and major roads, capturing the images frame by frame and storing the data over time.

[1746] Step 2:

[1747] Data transmission (terminal)

[1748] The video data captured by the device is sent to the server at regular intervals using a streaming protocol, and is sent in real time.

[1749] Step 3:

[1750] Data reception (server)

[1751] The server receives traffic video data sent from the terminal, and the received data is temporarily stored in a buffer for subsequent analysis.

[1752] Step 4:

[1753] Image recognition (server)

[1754] The server uses AI to analyze the video data in the buffer, and image recognition algorithms analyze each frame to identify objects such as vehicles, pedestrians, and cyclists.

[1755] Step 5:

[1756] Traffic situation analysis (server)

[1757] The server obtains attribute data such as the location, speed, and direction of the identified objects and evaluates traffic flow and congestion. The analysis results are used to determine the current traffic situation in real time.

[1758] Step 6:

[1759] Pattern prediction (server)

[1760] The server uses past traffic data and current analysis results to predict future traffic patterns, using predictive models to calculate the likelihood of future congestion and jams.

[1761] Step 7:

[1762] Optimization calculation (server)

[1763] The server calculates the optimal timing for traffic lights based on traffic conditions and predictions, optimizing the duration of each phase (green, yellow, red) and setting the traffic light timing.

[1764] Step 8:

[1765] Command generation (server)

[1766] The server generates a command to send to the traffic light based on the calculation results, which includes the new traffic light timing.

[1767] Step 9:

[1768] Send command (server)

[1769] The server sends the generated command to the traffic light, which then transmits the command to the traffic light via the network.

[1770] Step 10:

[1771] Signal timing application (terminal)

[1772] The terminal (traffic light) applies the new signal timing according to the received command. The traffic light immediately starts operating based on the new settings.

[1773] Step 11:

[1774] Emotion data collection (user)

[1775] Using an emotion engine, emotional data is obtained from the user's voice and video, analyzing the user's stress level and frustration and collecting emotional data in real time.

[1776] Step 12:

[1777] Emotion data analysis (server)

[1778] The server receives and analyzes the emotion data sent from the emotion engine, evaluates the user's emotional state indicated by the emotion data, and reflects this in traffic situation analysis.

[1779] Step 13:

[1780] Emotion-based signal timing adjustment (server)

[1781] The server analyzes the emotional data and fine-tunes the timing of the traffic lights. For example, if the user's stress level is high, the system will extend the duration of the green light.

[1782] Step 14:

[1783] Monitoring (users)

[1784] Users monitor real-time traffic conditions and emotion data using the system's dashboard, which displays information such as current traffic light timings, traffic flow, congestion levels, and the user's emotional state.

[1785] Step 15:

[1786] Feedback and manual adjustments (user)

[1787] If necessary, users can manually instruct the system to fine-tune signal timing, providing a means to respond to emergencies or specific requests.

[1788] Example 2

[1789] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1790] Optimization of traffic signals in transportation systems relies on analysis of traffic conditions and timing adjustment, but conventional systems do not take user emotions into consideration, resulting in problems such as user stress and frustration even when traffic flow is smooth.In addition, adjustment of traffic signals based on real-time traffic video and data analysis was limited, so there was a need for more precise and rapid optimization.

[1791] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1792] In this invention, the server includes means for acquiring traffic images, means for analyzing the acquired traffic images and determining traffic conditions, means for optimizing traffic light timing based on the traffic conditions, means for recognizing a user's emotions and adjusting traffic light timing based on the recognition results, and means for applying the optimized traffic light timing to the traffic lights. This enables real-time analysis and optimization of traffic conditions and also enables adjustments that take the user's emotions into consideration, thereby not only smoothing traffic flow but also reducing the psychological burden on the user.

[1793] "Traffic footage" refers to video data capturing traffic conditions, and records the movement of vehicles, pedestrians, bicycles, etc. at intersections and major roads in real time.

[1794] "Traffic conditions" refers to information that indicates the degree of congestion and movement of vehicles, pedestrians, bicycles, etc. on roads at a specific time and place.

[1795] "Traffic light timing" refers to the time it takes for a traffic light to change color (green, yellow, red), including the duration of the change to control traffic flow.

[1796] An "image recognition algorithm" is a computer algorithm that analyzes video data acquired from a camera and identifies specific objects (vehicles, pedestrians, bicycles, etc.).

[1797] "Pattern prediction" is a mathematical method for analyzing past traffic data and predicting current and future traffic conditions.

[1798] "Emotion recognition" is a technology that analyzes data such as a user's voice and facial expressions to identify their psychological state (stress, frustration, joy, etc.).

[1799] "Timing optimization" involves calculating the optimal duration of traffic light colors based on traffic conditions and user emotion data, and making adjustments to ensure the most efficient traffic flow.

[1800] "Monitoring" is the process of monitoring traffic conditions, traffic light timing, and user emotional data in real time and making adjustments as needed.

[1801] This invention not only optimizes traffic signal timing using traffic video, but also realizes advanced traffic management using user emotional data. This system captures traffic video in real time, analyzes the video data, and judges traffic conditions. Based on this judgment, it optimizes traffic signal timing and also reflects user emotional data, enabling efficient traffic management with reduced psychological burden.

[1802] Hardware and software used

[1803] Traffic image acquisition means (terminal)

[1804] The terminal is equipped with a camera that captures images of intersections and major roads in real time. This camera has the ability to capture images day and night, and sends the captured image data to a server. Specifically, it uses the H.264 codec to compress the image data and the SSL / TLS communication protocol for data communication.

[1805] Data analysis method (server)

[1806] The server receives traffic video data sent from the device, decodes the received data, and analyzes it using an AI-based image recognition algorithm (such as YOLO or OpenCV). This analysis identifies objects such as vehicles, pedestrians, and bicycles, and obtains attribute data such as location, speed, and direction.

[1807] Traffic condition data generation (server)

[1808] The server generates current traffic situation data based on the analysis results. The number of vehicles, number of pedestrians, and the speed and direction of each object are converted into statistical data and updated in real time. By storing this data in a database, it is possible to compare it with past data.

[1809] Signal timing optimization (server)

[1810] The server optimizes traffic signal timing based on traffic condition data. The optimization algorithm uses optimization theory and feedback control to calculate the duration of each phase (green, yellow, red) and update the timing in real time, including forecast data for the next five minutes.

[1811] Signal timing application means (terminal)

[1812] The optimized signal timing is sent from the server to the traffic lights, which then make real-time signal changes based on the received timing. To minimize delays, a low-latency communication protocol (e.g. MQTT) is used. The traffic light control software applies the new timing for each phase.

[1813] Emotion Engine

[1814] The emotion engine recognizes the user's emotions and reflects them in traffic management. Emotion data is acquired using a camera or microphone installed on the device, and the acquired audio and video data is analyzed on a server. Specifically, an emotion recognition algorithm (such as IBM Watson or Microsoft Azure's emotion recognition API) is used. Based on the emotion data, the level of stress or anger is determined and traffic light timing is adjusted.

[1815] Monitoring and Feedback (User)

[1816] Users can view real-time traffic conditions and sentiment data on a monitoring dashboard, which displays current traffic signal timings, traffic flow, congestion levels, and user sentiment data. Users can also manually adjust traffic signal timings in case of emergencies or specific requirements.

[1817] Examples of concrete examples and prompts

[1818] Specific examples

[1819] During the morning rush hour, cameras capture traffic footage in real time and send the acquired video data to a server. The server analyzes the data and detects an increase in vehicles and pedestrians compared to normal. It predicts that congestion will increase further over the next five minutes and makes adjustments such as extending the duration of green lights. User emotion data is also collected, and if high stress levels are detected, further fine-tuning of signal timing is made. Administrators can monitor this from a dashboard and make manual adjustments as needed.

[1820] Prompt Sentence Examples

[1821] An example prompt is:

[1822] Write a program that analyzes current traffic video data and optimizes traffic light timing. Specifically, for intersection A, with 50 vehicles and 30 pedestrians, extend the current green light by 10 seconds and shorten the pedestrian signal timing by 5 seconds. Also, further fine-tune the traffic light timing based on emotion data.

[1823] This system enables real-time analysis and optimization of traffic conditions, and more effective traffic management can be achieved by making adjustments that take into account user emotional data.

[1824] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1825] Step 1:

[1826] Traffic image acquisition (terminal)

[1827] The device is equipped with a camera that captures images of intersections and major roads in real time. Specifically, the camera captures images at 30 frames per second (fps) and continues to capture images at night using infrared functionality. The input is real-time images of intersections and roads, and the output is video data compressed using the H.264 codec.

[1828] Step 2:

[1829] Video data transmission (terminal)

[1830] The terminal transmits the acquired video data to the server. Specifically, the acquired video data is compressed every second and transmitted to the server using the SSL / TLS communication protocol. The input is video data compressed using H.264, and the output is encrypted data packets.

[1831] Step 3:

[1832] Traffic video analysis (server)

[1833] The server decodes the video data received from the device and analyzes traffic conditions by using AI for image recognition. YOLO and OpenCV are used as image recognition algorithms. Specifically, it identifies objects such as vehicles, pedestrians, and bicycles from the video data and determines their respective positions, speeds, and directions. The input is the video data sent to the server, and the output is attribute data for each object (position, speed, direction, etc.).

[1834] Step 4:

[1835] Traffic condition data generation (server)

[1836] The server generates current traffic situation data based on the analysis results. It converts the number of vehicles, the number of pedestrians, and the speed and direction of each object into statistical data and updates it in real time. Specifically, it saves the analysis results in list format and records them in a database. The input is the analysis results of the traffic video, and the output is statistical traffic situation data.

[1837] Step 5:

[1838] Signal timing optimization (server)

[1839] The server optimizes signal timing based on traffic condition data. The optimization algorithm uses optimization theory and feedback control. Specifically, it calculates the optimal duration of each phase (green, yellow, red). The input is traffic condition data updated in real time and past data, and the output is optimized signal timing data.

[1840] Step 6:

[1841] Optimized timing transmission (server)

[1842] The server transmits the optimized signal timing to the traffic lights using a low-latency communication protocol (e.g., MQTT). Specifically, it creates optimized timing data packets and sends them to the traffic lights. The input is the optimized signal timing data, and the output is the new signal timing that is applied to the traffic light control software.

[1843] Step 7:

[1844] Timing application to traffic lights (terminal)

[1845] The terminal (traffic light) changes the signal in real time according to the received signal timing. For example, the delay when changing from red to green is kept within a few milliseconds. The input is the new signal timing data sent from the server, and the output is the changed color of the traffic light.

[1846] Step 8:

[1847] Acquisition and analysis of emotion data (terminal / server)

[1848] The device (for example, an emotion-recognition camera or microphone) acquires the user's voice and video data. The server analyzes this data and recognizes the user's emotions using an emotion engine (for example, IBM Watson or Microsoft Azure's emotion recognition API). Specifically, it detects the level of stress or anger from the user's voice and facial expression. The input is the user's voice and video data, and the output is analyzed emotional data.

[1849] Step 9:

[1850] Adjustment of signal timing based on emotion data (server)

[1851] The server further adjusts the signal timing based on the acquired emotion data. For example, if the user's stress level is high, the algorithm adjusts to shorten the duration of the green light. The input is the analyzed emotion data in real time, and the output is fine-tuned signal timing data.

[1852] Step 10:

[1853] Monitoring and Feedback (User)

[1854] Users can check real-time traffic conditions and emotion data on a monitoring dashboard. The dashboard displays current traffic signal timings, traffic flow, congestion levels, and user emotion data. Users can also manually adjust traffic signal timings in emergencies or according to specific requirements. The inputs are traffic condition data and emotion data, and the output is the traffic signal timings manually adjusted by the user.

[1855] (Application example 2)

[1856] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1857] Conventional traffic management systems focus on analyzing traffic images and optimizing traffic signal timing, but real-time understanding and adaptation to traffic conditions is required to ensure smooth traffic flow. Furthermore, one of the factors that contribute to traffic accidents and congestion is the emotion and stress of drivers and pedestrians, yet few systems exist that take this into account. Therefore, new systems are needed that can optimize traffic flow while reducing the psychological burden on road users. Furthermore, with the spread of autonomous vehicles, new traffic management systems that can accommodate this are also required.

[1858] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1859] In this invention, the server includes means for acquiring traffic video, means for analyzing the traffic video and determining traffic conditions, means for optimizing traffic light timing based on the traffic conditions, means for applying the optimized traffic light timing to the traffic lights, means for analyzing user emotions and adjusting traffic light timing based on the emotions, and means for monitoring surrounding traffic conditions in real time using video acquired by a camera in the autonomous vehicle and optimizing the driving pattern of the autonomous vehicle based on the analysis results, thereby enabling optimization of traffic flow, flexible traffic management in response to the emotions of road users, and safe and efficient operation of the autonomous vehicle.

[1860] The "means for acquiring traffic images" refers to a device that uses cameras installed at locations such as intersections and major roads to acquire images of traffic conditions in real time.

[1861] The "means for analyzing traffic images and determining traffic conditions" is a device that uses an AI-based image recognition algorithm to analyze the status of vehicles, pedestrians, and traffic lights based on acquired traffic images, and determines the current traffic conditions.

[1862] A "means for optimizing traffic signal timing based on traffic conditions" is a device that uses analyzed traffic condition data to calculate the duration of green, yellow, and red signals at a traffic signal and optimize traffic flow.

[1863] The "means for applying optimized signal timing to traffic lights" is a device that transmits the calculated optimal signal timing to actual traffic lights and changes the signals in real time.

[1864] The "means for analyzing the user's emotions and adjusting the timing of traffic lights based on those emotions" refers to a device that analyzes emotions from the user's voice and video, and reflects that data in a traffic management system, thereby adjusting the timing of traffic lights based on the user's emotions.

[1865] "Means for monitoring surrounding traffic conditions in real time using images captured by a camera on an autonomous vehicle" refers to a device that uses a camera mounted on an autonomous vehicle to capture images of traffic conditions, acquires the image data in real time, and monitors surrounding traffic.

[1866] The "means for optimizing the driving pattern of an autonomous vehicle based on the analysis results" refers to a device that adjusts the speed and route of an autonomous vehicle to the optimum based on the analysis results of the acquired video data.

[1867] This invention combines an emotion engine with a function for linking with autonomous vehicles to optimize traffic light timing using traffic video. This system acquires traffic video in real time, analyzes and predicts the data, and optimizes traffic light timing. Furthermore, by analyzing user emotions and reflecting them in traffic light timing, it is possible to enhance traffic management and optimize the operation of autonomous vehicles.

[1868] Overall system overview

[1869] The system consists of the following main components:

[1870] 1. Means for acquiring traffic images (devices including cameras)

[1871] 2. A means of analyzing data and determining traffic conditions (server)

[1872] 3. Means for optimizing traffic light timing (server)

[1873] 4. Means (terminal) for applying optimized signal timing to traffic lights

[1874] 5. A means to analyze user emotions and adjust traffic light timing based on those emotions (emotion engine)

[1875] 6. Means of monitoring surrounding traffic conditions using images captured by cameras in autonomous vehicles (cameras and systems for autonomous vehicles)

[1876] 7. Means for optimizing the driving patterns of autonomous vehicles based on analysis results (autonomous driving systems)

[1877] Hardware and software used

[1878] The following hardware and software are used to implement this system.

[1879] Hardware: Cameras, autonomous vehicle cameras, servers, traffic light control terminals

[1880] Software: OpenCV (image analysis), TensorFlow (AI modeling), Python scripts (data analysis), ROS (Robot Operating System, integration with autonomous driving systems)

[1881] Program processing explanation

[1882] The server first receives real-time video data from cameras to capture traffic images. It then analyzes the received video data using OpenCV and TensorFlow to determine traffic conditions. Traffic data includes the position, speed, and direction of objects such as vehicles, pedestrians, and bicycles. It then performs calculations to optimize traffic light timing based on the traffic conditions and sends the optimal signal timing to the traffic light control terminal.

[1883] Meanwhile, an emotion engine is used to analyze user emotions. This engine collects passenger emotional data from audio and video and analyzes it in real time. This emotional data is used to fine-tune signal timing, setting optimal signal timing to reduce passenger stress.

[1884] Finally, the autonomous vehicle's camera continuously monitors the surrounding traffic conditions and analyzes the acquired data in real time. Using ROS, the system instructs the autonomous driving system on optimal driving patterns, helping to ensure smooth traffic flow. In particular, taking into account passenger emotional data can help ensure a comfortable and safe operation.

[1885] Adding concrete examples and prompt sentences for the generative AI model

[1886] For example, consider the following system situation:

[1887] 1. Traffic video acquisition: Cameras on autonomous vehicles capture real-time traffic conditions at intersections and transmit the video data to a server inside the vehicle.

[1888] 2. Data analysis: The server analyzes the received video and determines traffic flow based on information such as the location, speed, and direction of identified vehicles and pedestrians.

[1889] 3. Optimization: Based on the analysis results, the speed and route of autonomous vehicles are optimized and traffic light timing is adjusted.

[1890] 4. Emotion analysis: Analyzes passengers' emotions in real time from their voice data and fine-tunes traffic signal timing to ensure stress-free driving.

[1891] Example prompt for a generative AI model:

[1892] Analyze traffic video data in real time to identify vehicles, pedestrians, and bicycles. Also, analyze passengers' emotions based on their voice data and select driving patterns according to their stress levels. Use TensorFlow and OpenCV for video data analysis, and an emotion engine for emotion analysis. Use ROS for integration with the autonomous driving system.

[1893] According to the above aspects, the present invention can optimize traffic flow, reduce the psychological burden on traffic users, and realize safe and efficient operation of autonomous vehicles.

[1894] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1895] Step 1:

[1896] Traffic footage acquisition

[1897] The device captures real-time traffic images using cameras installed at intersections and major roads. This image data is also captured by cameras on autonomous vehicles. The captured image data is sent directly to the server. The input data is the image data from the cameras, and the output data is the image data sent to the server.

[1898] Step 2:

[1899] Traffic video analysis

[1900] The server processes the received traffic video data. Specifically, it performs image analysis using OpenCV and TensorFlow. Image analysis identifies objects such as vehicles, pedestrians, and bicycles, and calculates their respective positions, speeds, and directions. The input data is the traffic video data, and the output data is attribute data of the identified objects.

[1901] Step 3:

[1902] Judging traffic conditions

[1903] The server determines the current traffic situation based on the image analysis results. Specifically, it calculates traffic flow and congestion levels based on the position, speed, and direction of each object. This makes it possible to identify traffic bottlenecks and dangerous areas. The input data is the attribute data of the identified objects, and the output data is the traffic situation assessment result.

[1904] Step 4:

[1905] Traffic light timing optimization

[1906] The server optimizes traffic light timing based on the traffic conditions it determines. It also takes into account past data and current conditions to calculate the duration of each traffic light phase (green, yellow, red). The input data is the traffic condition determination result, and the output data is the optimized traffic light timing.

[1907] Step 5:

[1908] Signal Timing Enforcement

[1909] The terminal applies the optimized signal timing received from the server to the actual traffic light, which changes the signal in real time and ensures smooth traffic flow. The input data is the optimized signal timing, and the output data is the result of the traffic light timing change.

[1910] Step 6:

[1911] User sentiment analysis

[1912] The emotion engine analyzes the user's (passenger's) voice and video data in real time to determine emotions such as stress and frustration, thereby understanding the user's psychological state. The input data is the user's voice and video data, and the output data is the analyzed emotion data.

[1913] Step 7:

[1914] Fine-tuning signal timing

[1915] The server fine-tunes the optimized signal timing based on the emotion data obtained from the emotion engine. This sets the optimal signal timing to reduce the user's stress. The input data is the analyzed emotion data, and the output data is the fine-tuned signal timing.

[1916] Step 8:

[1917] Optimizing driving patterns for autonomous vehicles

[1918] The autonomous vehicle system optimizes driving patterns based on acquired traffic video data and emotion data. Specifically, it uses ROS to adjust the vehicle's speed and route. The input data is traffic video data and emotion data, and the output data is the optimized driving pattern.

[1919] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1920] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1921] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1922] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1923] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1924] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1925] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1926] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1927] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1928] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1929] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1930] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1931] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1932] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1933] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1934] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1935] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1936] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1937] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1938] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1939] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1940] The following is further disclosed regarding the above embodiment.

[1941] (Claim 1)

[1942] a means for acquiring traffic footage;

[1943] A means for analyzing the acquired traffic video and determining the traffic situation;

[1944] means for optimizing traffic light timing based on traffic conditions;

[1945] means for applying the optimized signal timing to a traffic light;

[1946] A system including:

[1947] (Claim 2)

[1948] 10. The system of claim 1, further comprising means for using image recognition algorithms to analyze traffic footage.

[1949] (Claim 3)

[1950] 2. The system according to claim 1, further comprising means for performing pattern prediction based on past data to predict traffic conditions.

[1951] "Example 1"

[1952] (Claim 1)

[1953] a means for acquiring traffic footage;

[1954] A means for analyzing the acquired traffic video and determining the traffic situation;

[1955] means for optimizing traffic light timing based on traffic conditions;

[1956] means for applying the optimized signal timing to a traffic light;

[1957] A system that includes a means to manually adjust signal timing based on the analysis results.

[1958] (Claim 2)

[1959] 10. The system of claim 1, further comprising means for using image recognition algorithms to analyze traffic footage.

[1960] (Claim 3)

[1961] 2. The system according to claim 1, further comprising means for performing pattern prediction based on past data to predict traffic conditions.

[1962] "Application Example 1"

[1963] (Claim 1)

[1964] a means for acquiring traffic footage;

[1965] A means for analyzing the acquired traffic video and determining the traffic situation;

[1966] means for optimizing traffic light timing based on traffic conditions;

[1967] means for applying the optimized signal timing to a traffic light;

[1968] means for communicating with the automated vehicle to provide optimized signal timing;

[1969] A system including:

[1970] (Claim 2)

[1971] 10. The system of claim 1, further comprising means for using image recognition algorithms to analyze traffic footage.

[1972] (Claim 3)

[1973] 2. The system according to claim 1, further comprising means for performing pattern prediction based on past data to predict traffic conditions.

[1974] "Example 2: Combining Emotion Engines"

[1975] (Claim 1)

[1976] a means for acquiring traffic footage;

[1977] A means for analyzing the acquired traffic video and determining the traffic situation;

[1978] means for optimizing traffic light timing based on traffic conditions;

[1979] means for recognizing a user's emotion and adjusting the timing of a traffic light based on the recognition result;

[1980] means for applying the optimized signal timing to a traffic light;

[1981] A system including:

[1982] (Claim 2)

[1983] 10. The system of claim 1, further comprising means for using image recognition algorithms to analyze traffic footage.

[1984] (Claim 3)

[1985] 2. The system according to claim 1, further comprising means for performing pattern prediction based on past data to predict traffic conditions.

[1986] "Application example 2 when combining emotion engines"

[1987] (Claim 1)

[1988] a means for acquiring traffic footage;

[1989] A means for analyzing the acquired traffic video and determining the traffic situation;

[1990] means for optimizing traffic light timing based on traffic conditions;

[1991] means for applying the optimized signal timing to a traffic light;

[1992] means for analyzing a user's emotions and adjusting traffic light timing based on the emotions;

[1993] Using images captured by cameras in autonomous vehicles, surrounding traffic conditions are monitored in real time.

[1994] a means for optimizing the driving pattern of the autonomous vehicle based on the analysis results;

[1995] A system including:

[1996] (Claim 2)

[1997] 10. The system of claim 1, further comprising means for using image recognition algorithms to analyze traffic footage.

[1998] (Claim 3)

[1999] 2. The system according to claim 1, further comprising means for performing pattern prediction based on past data to predict traffic conditions. [Explanation of symbols]

[2000] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for acquiring traffic footage; A means for analyzing the acquired traffic video and determining the traffic situation; means for optimizing traffic light timing based on traffic conditions; means for applying the optimized signal timing to a traffic light; A system including:

2. 10. The system of claim 1, further comprising means for using image recognition algorithms to analyze traffic footage.

3. 2. The system according to claim 1, further comprising means for performing pattern prediction based on past data to predict traffic conditions.

Images