Information processing system

Abstract

The invention provides an information processing system. The information processing system comprises a device for acquiring a traffic image; the device is used for analyzing the acquired traffic image and judging the traffic condition; means for optimizing traffic signal timing based on traffic conditions; and means for applying the optimized signal timing to the annunciator.

Description

Technical Field

[0001] The technology disclosed herein relates to an information processing system. Background Technology

[0002] Japanese Patent Application Publication No. 2022-180282 discloses a method for controlling a role-based chatbot executed by at least one processor. The method includes the following steps: receiving a user's speech; adding the user's speech to a prompt word, the prompt word containing instruction statements associated with an explanation of the chatbot's role; encoding the prompt word; and inputting the encoded prompt word into a language model to generate a chatbot response to the user's speech. Summary of the Invention

[0003] To address the aforementioned problems, this invention provides an information processing system, comprising: a device for acquiring traffic images; a device for analyzing the acquired traffic images and determining traffic conditions; a device for optimizing signal timing based on traffic conditions; and a device for applying the optimized signal timing to the signal controllers. By employing image recognition algorithms to automatically analyze traffic flow, dynamic signal optimization based on real traffic conditions and historical data is achieved, effectively improving road traffic efficiency and realizing intelligent management.

[0004] "Traffic images" refer to video or image data acquired through camera and other image acquisition devices that can reflect the real-time traffic conditions in a certain area.

[0005] "Analysis" refers to the process of processing the acquired traffic images and using algorithms to identify various traffic participants and their status information in the images.

[0006] "Traffic conditions" refers to the dynamic state of traffic at a specific intersection or road at a given moment, including vehicle flow, number of pedestrians, speed, direction of travel, and degree of congestion.

[0007] "Traffic signal timing" refers to the setting method and switching sequence of the duration of each phase of a traffic signal light (such as red light, green light, and yellow light).

[0008] "Optimization" refers to the process of adjusting and improving signal timing based on traffic conditions and other parameters to improve traffic efficiency.

[0009] "Image recognition algorithm" refers to a computational method that uses artificial intelligence and other technologies to automatically detect and classify objects in an image.

[0010] "Historical data" refers to traffic-related data such as traffic flow, signal switching, and congestion conditions recorded in previous time periods.

[0011] "Traffic signal controller" refers to a signal control device used to control the traffic flow order at intersections, mainly including display modules for red lights, green lights, yellow lights, etc. Attached Figure Description

[0012] Figure 1 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the first embodiment.

[0013] Figure 2 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and smart device according to the first embodiment.

[0014] Figure 3 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the second embodiment.

[0015] Figure 4 This is a conceptual diagram illustrating an example of the main functions of the data processing device and smart glasses according to the second embodiment.

[0016] Figure 5 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the third embodiment.

[0017] Figure 6 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and head-mounted terminal according to the third embodiment.

[0018] Figure 7 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the fourth embodiment.

[0019] Figure 8 This is a conceptual diagram illustrating an example of the main functions of the data processing device and robot according to the fourth embodiment.

[0020] Figure 9 This represents an emotion map that maps multiple emotions.

[0021] Figure 10 This represents an emotion map that maps multiple emotions.

[0022] Figure 11 This is a sequence diagram illustrating the processing flow of the data processing system of the first embodiment.

[0023] Figure 12 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 1.

[0024] Figure 13 This is a sequence diagram illustrating the processing flow of the data processing system of the second embodiment.

[0025] Figure 14 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 2. Detailed Implementation

[0026] Hereinafter, an example of an implementation of the system according to the present disclosure will be described with reference to the accompanying drawings.

[0027] First, let me explain the terminology used in the following instructions.

[0028] In the following embodiments, the processor (hereinafter referred to as "processor") with reference numerals may be a single computing device or a combination of multiple computing devices. Furthermore, the processor may be a single computing device or a combination of multiple computing devices. Examples of computing devices include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), etc.

[0029] In the following embodiments, RAM (Random Access Memory), as indicated in the figures, is a memory that temporarily stores information and is used as working memory by the processor.

[0030] In the following embodiments, the memory, as indicated by the reference numerals, is one or more non-volatile storage devices that store various programs and parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), disks (e.g., hard disks), or magnetic tapes.

[0031] In the following embodiments, the communication I / F (Interface) with reference numerals is an interface that includes a communication processor and an antenna, etc. The communication I / F is responsible for communication between multiple computers. As an example of a communication specification applicable to the communication I / F, wireless communication specifications such as 5G (5th Generation Mobile Communication System), Wi-Fi (wireless fidelity) (registered trademark), or Bluetooth (registered trademark) can be listed.

[0032] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. Furthermore, in this specification, when "and / or" connects to express more than three items, the same interpretation as "A and / or B" applies.

[0033] First Implementation Method

[0034] Figure 1 An example of the configuration of the data processing system 10 according to the first embodiment is shown.

[0035] like Figure 1 As shown, the data processing system 10 includes a data processing device 12 and an intelligent device 14. A server can be cited as an example of the data processing device 12.

[0036] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0037] The smart device 14 includes a computer 36, a receiving device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. In addition, the receiving device 38, output device 40, camera 42, and communication I / F 44 are also connected to the bus 52.

[0038] The receiving device 38 includes a touchscreen 38A and a microphone 38B, and receives user input. The touchscreen 38A receives user input via touch by detecting contact with an indicator (e.g., a pen or finger). The microphone 38B receives user input via sound by detecting the user's voice. The control unit 46A in the processor 46 sends data representing the user input received by the touchscreen 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data representing the user input.

[0039] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting data in a form perceptible to the user 20 (e.g., sound and / or text). The display 40A displays visual information such as text and images according to instructions from the processor 46. The speaker 40B outputs sound according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0040] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for sending and receiving various information between processor 46 and processor 28 via network 54.

[0041] Figure 2 The diagram shows an example of the main functions of the data processing device 12 and the smart device 14.

[0042] like Figure 2 As shown, in the data processing apparatus 12, specific processing is performed by the processor 28. A specific processing program 56 is stored in the memory 32. The specific processing program 56 is an example of a "program" as understood in this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0043] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0044] In the smart device 14, the processor 46 performs the acceptance output processing. The memory 50 stores the acceptance output program 60. The acceptance output program 60 is used in conjunction with the data processing system 10 and the specific processing program 56. The processor 46 reads the acceptance output program 60 from the memory 50 and executes the read acceptance output program 60 on the RAM 48. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48. Furthermore, the smart device 14 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48.

[0045] Alternatively, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains the processing results (prediction results, etc.) using the data generation model 58 by communicating with the server device that has the data generation model 58. Furthermore, the data processing device 12 may be a server device or a user-held terminal device (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of the processing of the data processing system 10 of the first embodiment will be described.

[0046] Example 1

[0047] The flow of a specific process in Example 1 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. Furthermore, the data processing device 12 is referred to as the "server," and the smart device 14 is referred to as the "terminal."

[0048] Existing traffic signal control systems mostly use fixed timing sequences, making it difficult to reflect real-time road traffic flow and dynamic conditions. This leads to increased road congestion and low traffic efficiency during peak hours, and an inability to quickly respond to emergencies or traffic anomalies, affecting the safety and intelligence of urban traffic operations. Therefore, there is an urgent need for a highly efficient and intelligent traffic signal management system that can automatically optimize signal control timing based on real-time traffic conditions, supplemented by human intervention capabilities.

[0049] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 1 is achieved by the following means.

[0050] In this invention, the server includes an information acquisition device for acquiring traffic scene image data, a device for transmitting image data via a communication network, a device for analyzing the acquired image data using machine learning algorithms to identify and extract attribute information of various traffic objects, a device for inferring the future traffic environment based on historical data and using prediction algorithms, a device for automatically generating and adjusting signal control parameters according to the analysis and prediction results, and a device capable of communicating with the signal control device and changing its output settings in real time, combined with visualization and manual intervention. This enables automatic and dynamic optimization of traffic signal timing, improving road traffic efficiency and safety, while also achieving adaptive response and manual intervention capabilities.

[0051] "Information acquisition device" refers to equipment used to acquire image or video data in traffic scenarios, including but not limited to cameras, sensors, etc.

[0052] "Image data" refers to pictures or video files obtained by information collection devices that can reflect the dynamic traffic situation.

[0053] A "communication network" refers to a wired or wireless network system used to transmit data between information acquisition devices, information processing devices, and signal control devices.

[0054] "Information processing device" refers to a computer system that analyzes, processes, stores, and outputs control parameters for received traffic image data.

[0055] "Machine learning algorithms" refer to artificial intelligence computing methods that automatically learn from large amounts of data samples to achieve recognition, classification, or prediction, including deep learning.

[0056] "Moving targets" refers to moving objects appearing in traffic images, including traffic participants such as vehicles, pedestrians, and bicycles.

[0057] "Attribute information" refers to relevant data about a moving target, such as its location, speed, direction, and quantity.

[0058] "Historical traffic-related data" refers to the collection of information such as past traffic flow, signal timing, and road conditions accumulated during the operation of the system.

[0059] "Prediction algorithms" refer to mathematical methods that use historical data to infer and calculate future traffic flow, congestion, etc., including time series prediction models.

[0060] "Signal control device" refers to hardware or software equipment that switches, modifies, and adjusts traffic lights, electronic traffic signs, etc.

[0061] "Control communication protocol" refers to a standardized communication method used to transmit instructions and data between information processing devices and signal control devices.

[0062] "Visual display device" refers to terminal equipment, such as a display screen or user interface, used to intuitively display traffic conditions and control information to users.

[0063] "User" refers to the manager or operator who uses the system for monitoring, control or manual intervention.

[0064] This invention relates to a system based on intelligent traffic image analysis and signal optimization control, specifically including modules such as an information acquisition device, a terminal, a server, a signal control device, and a visual user interface. The following description illustrates specific embodiments of this invention.

[0065] Terminals can be equipped with high-resolution cameras (such as commercially available surveillance cameras) or industrial computing units (such as embedded hardware platforms), and are deployed at road intersections or main roads. The terminal is responsible for capturing traffic scenes in real time and encoding and compressing the image or video data using built-in software (such as the FFmpeg video processing tool running on a Linux system). To facilitate data transmission, the terminal periodically sends the collected information to the server via wired or wireless communication networks (such as Ethernet, 4G / 5G networks, etc.).

[0066] The server is typically a high-performance computer or a cloud server. After receiving image data from the terminal, the server uses machine learning algorithms (such as YOLO or deep learning models based on OpenCV) to perform target recognition on the traffic images, extracting attribute information of moving targets such as vehicles, pedestrians, and bicycles, including data such as position, speed, direction of travel, and target number. The server also calls upon historical databases and applies time-series prediction algorithms (such as LSTM neural networks) to predict future traffic flow and congestion levels. Based on the analysis and prediction results, the server automatically generates parameters for optimizing signal control, such as the duration of traffic lights at each stage. Then, the server sends the optimized control commands to the signal control device through standardized communication protocols (such as HTTPS, TCP / IP, etc.).

[0067] After receiving control signals from the server, the traffic signal control device adjusts the duration of traffic lights in real time for different time periods, thereby achieving intelligent management and optimization of road signals. This process effectively alleviates traffic congestion, improves traffic efficiency, and enables rapid response to abnormal events.

[0068] Users can view the real-time traffic status of each intersection through a visual display terminal (such as a web dashboard or touchscreen) connected to the system, including traffic light timing, traffic flow, and object distribution. In case of emergency or specific needs, users can manually adjust the signal control parameters through the interface. The server and terminal will respond promptly and complete the parameter changes, ensuring the flexibility and reliability of intelligent dispatching.

[0069] This invention can be deployed and implemented in environments such as traffic management and smart transportation hubs in large and medium-sized cities. The number of data collection devices, server performance, and complexity of the recognition model can also be flexibly adjusted according to different city sizes and application needs.

[0070] For example, during the morning rush hour, the terminal camera captures traffic images every 5 seconds and uploads them to the server. When the server detects a surge in traffic, it uses a predictive model to decide to extend the green light at the current intersection by 20 seconds. Upon receiving the instruction, the signal control device immediately adjusts the signal light duration. Users can monitor and evaluate the effect in real time through the dashboard. If further traffic management is needed, parameters can also be manually adjusted to adapt to temporary situations.

[0071] Example of prompts for generative artificial intelligence models:

[0072] "Please explain how to achieve dynamic optimization scheduling of urban traffic lights through traffic video analysis and AI image recognition."

[0073] "Please describe the principle and process of a traffic signal control system based on intelligent image recognition and real-time prediction."

[0074] use Figure 11 The processing flow is explained.

[0075] Step 1:

[0076] The terminal uses a high-resolution camera to collect real-time traffic image data from intersections or roads. The input is the actual traffic scene, and the terminal obtains the raw video stream from the camera. Specifically, the terminal calls the camera every 5 seconds to capture a short video or image segment, and uses tools such as FFmpeg to compress the video data into H.264 format. The output is the compressed traffic video or image data file.

[0077] Step 2:

[0078] The terminal sends compressed traffic data to the server via wired or wireless communication networks. The input is a compressed video or image file. The terminal uses its built-in network communication module to upload the data stream to the specified server IP and port. The server, as the receiver, listens on the corresponding communication port. The output is the traffic image data packet successfully received by the server.

[0079] Step 3:

[0080] The server receives traffic image data uploaded by the client, performs integrity verification on the data, and temporarily stores it in local storage or a database. The input is the received compressed traffic video data file. The server uses FFmpeg or a self-developed interface to perform packet verification and frame segmentation, storing the data in a data directory or database. The output is the raw traffic image data stored locally.

[0081] Step 4:

[0082] The server uses machine learning algorithms (such as YOLO or OpenCV deep learning models) to identify and analyze traffic image data. The input is stored traffic image or video frame data. The server calls object detection algorithms to identify targets such as vehicles, pedestrians, and bicycles, and extracts attributes such as category, location, speed, and quantity for each target. Data processing steps include image preprocessing, model inference, and target attribute mapping. The output is a structured data table or JSON data containing various target attributes.

[0083] Step 5:

[0084] The server combines historical traffic data with time-series prediction models (such as LSTM) to predict traffic flow and congestion levels over a future period. The input consists of current and historical traffic attribute data. The server calls the prediction module to perform data integration and time-series modeling, and the output is a set of traffic target change trends and congestion predictions for the next few minutes.

[0085] Step 6:

[0086] Based on target detection and prediction results, the server generates and optimizes signal control parameters. The input consists of multi-target attribute structured data and predicted traffic trends. The server executes a signal timing calculation module to adjust the duration of red lights, green lights, and pedestrian lights at each intersection. The output is a set of parameters for the desired signal timing, such as the duration in seconds for each phase.

[0087] Step 7:

[0088] The server sends optimized signal timing parameters to the endpoint or signal control device via a communication protocol (such as HTTPS or TCP / IP). The input is the optimized signal parameter data; after encoding the data, the server issues a command. The endpoint or signal control device, acting as the receiving end, immediately parses the command and prepares to switch signal states. The output is the new control command received by the signal control device.

[0089] Step 8:

[0090] The terminal or signal control device parses the received signal parameters and adjusts the physical traffic light status in real time. The input is the signal timing parameters sent by the server; the terminal control hardware switches the traffic lights via relays or interface circuits. Data processing includes command parsing, parameter verification, and electrical drive. The output is the traffic lights automatically switching operation according to the new parameters.

[0091] Step 9:

[0092] Users can view real-time traffic conditions and traffic light operation information through a visual display terminal. Input consists of signal timing and traffic flow data from the server; users retrieve charts and data information on a web dashboard or display screen. Output is a real-time monitoring interface for traffic conditions and signal timing at various intersections.

[0093] Step 10:

[0094] Users can manually adjust signal control parameters through the interface in case of special events. The input is the signal adjustment command set by the user, who can input temporary traffic light adjustment requests via sliders, buttons, etc. After receiving the command, the server or terminal regenerates and sends out the signal parameters, enabling manual intervention. The output is the manually corrected signal timing and the real-time response of the corresponding traffic lights at the intersection.

[0095] Application Example 1

[0096] The process flow corresponding to the specific processing in Use Case 1 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. Furthermore, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0097] Current traffic signal control systems often rely solely on preset times or simple real-time traffic flow data, making it difficult to respond promptly and accurately to complex and ever-changing traffic conditions. With increasing urban traffic pressure and the development of autonomous driving technology, existing systems show significant shortcomings in alleviating congestion, reducing accidents, improving road efficiency, and ensuring a better user experience. Furthermore, existing systems generally lack real-time data collection and response mechanisms for road users' emotional states, hindering the achievement of more humane and intelligent traffic management.

[0098] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 1 is achieved by the following means.

[0099] In this invention, the server includes a device for acquiring traffic video information, a device for parsing the video information and extracting the movement attributes of traffic participants, a device for calculating and predicting traffic conditions based on time-series information, a device for optimizing the timing of signal control equipment, a device for applying the optimization results to the signal control equipment, a device for displaying the control status and traffic conditions, a device for manually adjusting the signal control timing according to the operation of management personnel, a device for further optimizing signal control by estimating and utilizing the emotional state of traffic participants, and a device for providing optimization information to the mobile entities through a mobile communication network. This enables real-time intelligent traffic control based on traffic video and emotional data, effectively alleviating traffic congestion, improving road safety and traffic efficiency, and also considering the experience of road users.

[0100] "Traffic video information" refers to image or video data collected in real time from traffic scenes such as roads and intersections using camera devices.

[0101] "Traffic participants" refers to all dynamic entities moving in the traffic environment, such as vehicles, pedestrians, and bicycles.

[0102] "Mobility attributes" refer to the dynamic characteristic data of traffic participants, such as their real-time location, speed, and direction of movement.

[0103] "Time series information" refers to a continuous set of traffic attribute data collected at different points in time, used to describe the changing trends of traffic conditions.

[0104] "Traffic conditions" refers to the overall state of traffic flow, congestion level, and traffic efficiency at a specific time and place.

[0105] "Signal control equipment" refers to electronic equipment used to manage and control changes in the status of road traffic lights.

[0106] "Signal control moment" refers to the specific duration and switching time of each phase change (such as red light, green light, etc.) in the signal control equipment.

[0107] "Management device" refers to a human-machine interface terminal used by traffic management personnel to monitor and adjust traffic signals and view traffic conditions.

[0108] "Traffic managers" refers to the staff or management entities responsible for operating and supervising traffic signal control systems.

[0109] "Monitoring data" refers to the collection of information collected in real time by various sensors or terminals that reflects the behavior and status of road users.

[0110] “Voice information” refers to voice signals that reflect the emotions and state of road users, obtained through audio acquisition devices.

[0111] "Emotional state" refers to the emotional response exhibited by road users in a specific traffic environment, such as psychological states like stress, anxiety, and pleasure.

[0112] "Mobile vehicle" refers to a means of transportation that can move on roads, including autonomous vehicles.

[0113] "Mobile communication network" refers to a wireless communication system that supports high-speed data exchange between terminals, servers, and mobile devices.

[0114] "Inference model" refers to an algorithmic model that uses methods such as machine learning to analyze data and predict traffic trends or patterns.

[0115] The embodiments of the present invention are described below.

[0116] This invention provides an intelligent traffic signal control system based on traffic video information and emotion data. The system mainly comprises three parts: terminals (front-end acquisition devices), servers (back-end data processing centers), and users (traffic managers or autonomous driving vehicles, etc.).

[0117] The terminal uses high-resolution cameras (such as webcams) to collect real-time traffic video information from roads and intersections. Equipped with a local processing module, the terminal performs preliminary processing and encoding of the collected video data. It then transmits the video data to the server in real-time at preset intervals (e.g., every 5 seconds) via a communication module (such as 4G, 5G, or Wi-Fi). The terminal can also integrate an audio acquisition module to collect voice information from road users or administrators, serving as one source of emotion inference data.

[0118] As the core of data processing and control decision-making, the server's key functions are as follows:

[0119] First, the server uses dedicated video processing hardware or a general-purpose high-performance server (such as an x86-based server) to run data analysis programs under an operating system (such as Linux). The server utilizes open-source image processing libraries (such as OpenCV) and machine learning platforms (such as TensorFlow) to perform frame segmentation and object detection on the video data from the terminal, automatically extracting the movement attributes of traffic participants (such as position, speed, and direction).

[0120] Subsequently, the server analyzes historical traffic attribute data and current time-series data, and uses predictive models (such as LSTM and other time-series neural networks) to infer future traffic conditions and flow trends, providing early warnings of congestion at intersections or main roads. Simultaneously, based on decision optimization algorithms (such as linear programming or genetic algorithms), the server automatically calculates and generates optimal control timing schemes for signal control equipment.

[0121] The server encapsulates the optimized signal control timings into instructions and transmits them to the terminal via the network. The terminal then controls the signal control equipment (such as PLC, signal light control board, etc.) to adjust the duration of the red light, green light, and other signal phases in real time.

[0122] To enhance the intelligence and humanization of traffic management, an emotion engine module is integrated into the server. This module utilizes data collected from audio and video recordings, calls emotion recognition models (such as emotion analysis APIs), infers the emotional state of road users or managers (such as stress, anxiety, etc.), and feeds the results back to the main server process. Under special circumstances such as high emotional stress, the server can automatically further adjust signal control timing to achieve traffic management guided by road experience.

[0123] The system also includes an information display and interaction module. Users can view real-time information such as traffic flow, traffic light status, congestion index, and mood status through management terminals (such as computers, tablets, or mobile web interfaces). Users can manually adjust traffic light durations based on system recommendations, and the system automatically records all adjustments, feeding them back into the data model for subsequent self-optimization. The system can also push optimized signal timings and traffic warnings to mobile terminals such as autonomous vehicles via vehicle-to-everything (V2X) protocols, improving the safety and smoothness of autonomous driving.

[0124] This invention is applicable to various scenarios such as urban main roads, intersections, and intelligent driving test zones. It can effectively improve traffic flow, reduce congestion, optimize road experience, and continuously improve the overall intelligent decision-making capability of the system based on the ever-accumulating traffic and emotion data.

[0125] Specific examples illustrate:

[0126] During the morning rush hour, a terminal camera continuously captures video of an intersection, automatically uploading the video to the server every 5 seconds. The server uses OpenCV and TensorFlow to automatically detect the current number of vehicles and pedestrians. If the number significantly exceeds normal levels, an LSTM model predicts that congestion will worsen in five minutes. The server automatically extends the main road green light by 30 seconds and the pedestrian signal by 15 seconds, pushing the decision to the signal control system. An emotion engine detects high stress levels among administrators, prompting the server to further fine-tune the signal timing; all changes are displayed in real-time on the user's dashboard. Upon receiving the new signal cycle information, autonomous vehicles adjust their routes to improve throughput. Ultimately, administrators can view the traffic improvement on the dashboard and fine-tune the duration of each phase light as needed.

[0127] Example of prompts for generative artificial intelligence models:

[0128] Context description:

[0129] 1. Traffic cameras capture real-time images of intersections during the morning rush hour, transmitting them every 5 seconds.

[0130] 2. The server used OpenCV and TensorFlow to detect that the number of vehicles had increased by 40% compared to normal, and the number of pedestrians had doubled.

[0131] 3. If the server predicts that congestion will worsen in the next 5 minutes, it will automatically extend the green light by 30 seconds and the pedestrian signal by 15 seconds.

[0132] 4. The optimized timing sequence is immediately sent to the signal controllers via the network and applied.

[0133] 5. Administrators can monitor traffic flow and driver mood in real time on the dashboard. If the driver's stress level is too high, the system will automatically readjust the signal light duration to alleviate the stress.

[0134] use Figure 12 The processing flow is explained.

[0135] Step 1:

[0136] The terminal uses a high-resolution camera to capture real-time video of traffic intersections or main roads, and uses a local processing module to perform preliminary encoding and packaging of the video stream. The input is real-time intersection images, and the output is a compressed video stream. The terminal automatically sends a video stream to the server every 5 seconds. Specific actions include adjusting the camera angle, enabling night mode, and uploading data to the network via a 4G / 5G module.

[0137] Step 2:

[0138] The server receives and stores video data from the terminal. The input is the video stream transmitted from the terminal, and the output is the raw video file stored in the server's buffer. The server first performs integrity checks and format recognition on the received data packets, adding valid data to the processing queue. Specific actions include data reception, storage, and automatic error packet alarms.

[0139] Step 3:

[0140] The server uses image processing tools such as OpenCV to decompose the received video data frame by frame, and uses generative artificial intelligence models (such as a TensorFlow-based object detection network) to identify and classify objects such as vehicles, pedestrians, and bicycles in each frame. The input is cached video frame data; after feature extraction and object detection, the output is a dataset containing the movement attributes of multiple traffic participants (such as location coordinates, speed, and direction). Specific actions include loading the trained model, batch image inference, and recording the detection results.

[0141] Step 4:

[0142] The server uses historical traffic databases and currently extracted traffic attribute information to predict traffic flow and congestion trends for the next time period using time-series prediction models (such as LSTM). The inputs are the movement attributes of traffic participants and historical time-series data. Through time-series modeling and pattern recognition, the outputs are traffic condition assessments and future trend predictions. Specific actions include accessing the database, batch data cleaning, and time-series model inference.

[0143] Step 5:

[0144] Based on current and predicted traffic conditions, the server calculates the optimal control cycle and duration for each direction's traffic lights using optimization algorithms (such as linear programming or genetic algorithms). The inputs are traffic condition assessments and control parameters; after optimization, the output is the new traffic light phase duration and switching commands. Specific actions include setting algorithm parameters, finding the optimal solution, and encapsulating the results into a control command format (such as JSON).

[0145] Step 6:

[0146] The server sends the optimized signal control commands to the terminal via the network. The input is the optimized signal command, and the output is the terminal control command trigger. The terminal parses the received commands and adjusts the red, green, and other phase cycles of the traffic lights in real time through a local PLC or signal control module. Specific actions include local script execution, traffic light state switching, and status uploading to the server.

[0147] Step 7:

[0148] Based on data uploaded from endpoints and management terminals, the server integrates an emotion engine to perform sentiment analysis on audio and video data from managers or road participants. The input consists of audio and video signals; after emotion feature extraction and inference, the output is emotional state data (such as stress scores). If a high-stress state is detected, the server can dynamically adjust the signal control cycle to alleviate stress. Specific actions include emotion recognition API calls, result analysis, and feedback control.

[0149] Step 8:

[0150] Users can view real-time traffic light status, traffic flow, congestion, and sentiment index at various intersections through the management dashboard terminal. Input is status information transmitted from the server, and output is a visual interface and management commands. Users can manually modify signal control parameters through the terminal as needed. Specific actions include interactive interface operations, parameter input, and immediate system notification.

[0151] Step 9:

[0152] The server pushes the latest signal control decisions and traffic status information to autonomous vehicles and other mobile entities via mobile communication networks according to predetermined protocols (such as V2X or MQTT). The input is the optimized control result, and the output is the traffic data packets pushed to the mobile entities. Autonomous vehicles adjust their autonomous driving strategies based on the received data. Specific actions include batch data packaging, protocol encapsulation, and timed broadcasting.

[0153] Alternatively, an emotion engine for inferring user emotions can be combined. That is, the specific processing unit 290 can also use the emotion-specific model 59 to infer user emotions and perform specific processing using user emotions.

[0154] Example 2

[0155] The flow of a specific process in Example 2 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. The data processing device 12 will be referred to as the "server," and the smart device 14 as the "terminal."

[0156] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 2 is achieved by the following means.

[0157] In this invention, the server includes a spatial information acquisition device for collecting traffic scene image information, an information processing device for deep learning image recognition processing of the image data to generate moving body attribute information, a combination of numerical calculations and temporal information to determine traffic flow status and calculate signal control plans, and a computational processing device for integrating and analyzing user personal information and psychological state data inferred from audio and video. Finally, based on the psychological state, the optimized signal control plan is dynamically corrected and output to the physical signal control device. This enables high-precision intelligent signal optimization control driven by both traffic flow and user emotions, based on multi-source data fusion, reducing the psychological burden on road users and improving the intelligence level of the traffic management system and road traffic efficiency.

[0158] "Spatial information acquisition device" refers to hardware or system used to collect image information or video data of the surrounding environment (including traffic conditions, target objects, etc.) through sensing devices within a predetermined area.

[0159] "Information processing device" refers to a computing device or system used to receive, store, analyze, identify, filter, or preprocess collected data.

[0160] Image analysis and processing refers to the process of using computer vision algorithms (such as image recognition and feature extraction) to analyze acquired image or video data in order to identify target objects and their attributes.

[0161] "Moving object attribute information" refers to the characteristic data describing moving objects (such as vehicles, pedestrians, bicycles, etc.) in a traffic scene, including parameters such as their type, location, speed, and direction of movement.

[0162] "Time-series information" refers to a set of data collected and recorded in chronological order, used to reflect the patterns of traffic conditions changing over time.

[0163] "Numerical processing device" refers to hardware or system used to perform various data processing, statistical analysis and mathematical operations.

[0164] "Traffic flow status" refers to data that comprehensively describes the state of traffic objects on a road under specific time and space conditions, including their quantity, distribution, speed, and efficiency.

[0165] "Controlled object device" refers to a device that can execute control commands and change its own state through electronic or mechanical means, such as signal controllers and actuators.

[0166] "Operation plan" refers to a specific action or control scheme for traffic signals and other equipment, generated based on current traffic conditions and forecasts.

[0167] "Personal information data" refers to various types of data that can be used to identify or recognize an individual's identity or attributes, including but not limited to age, gender, and physiological status.

[0168] "Audio and video information" refers to data containing sound and image content, used to analyze a user's emotions, moods, or other states.

[0169] "Psychological state information" refers to the results inferred about psychological states such as stress, emotions, and anxiety through the analysis of data such as individual behavior, voice, and facial expressions.

[0170] "Physical control device" refers to a device that actually exists and can emit physical signals or actions to directly control the corresponding equipment.

[0171] "Signal control device" refers to a device or system that can adjust its own display content (such as the status of traffic lights) according to received control commands.

[0172] "Optimized operation plan" refers to a signal control scheme that improves overall traffic efficiency and user experience by analyzing multi-source data such as traffic conditions and user psychology.

[0173] The embodiments of the present invention are described below.

[0174] This invention relates to an intelligent traffic signal optimization system based on multi-source data fusion and artificial intelligence algorithms. The system comprises at least three main components: terminals, servers, and users. Each component has a clearly defined role and works collaboratively to achieve efficient perception, analysis, decision-making, and feedback of traffic flow and user psychological states at road intersections or major road sections.

[0175] The terminal typically consists of a spatial information acquisition device with image capture capabilities (such as a network camera) and an audio / video acquisition module. Typical hardware includes a high-resolution digital camera (with infrared night vision capabilities) and integrated audio acquisition equipment. The terminal can continuously capture on-site traffic images at 30 frames per second via a camera installed at traffic intersections, recording daytime and nighttime traffic conditions without interruption. Simultaneously, the terminal periodically collects voice data from pedestrians and drivers, as well as some facial images, via a microphone for subsequent emotion recognition. The acquired image and audio data are compressed using a local H.264 codec and then uploaded to the server in real time via a highly secure SSL / TLS protocol. The terminal is equipped with necessary local data caching and preprocessing capabilities to ensure data accuracy and timeliness.

[0176] The server, serving as the core processing center of this system, includes information processing devices, numerical computation devices, and the necessary software systems. The server receives real-time traffic data from multiple terminals via a communication interface. Using open-source or commercial image recognition algorithms (such as YOLO, OpenCV, and other machine learning software modules), the server performs object detection and feature extraction on each uploaded frame of traffic image, identifying the type, quantity, speed, and direction of movement of various moving objects (such as vehicles, pedestrians, and bicycles). All identified moving object attribute information is stored in a relational database and associated with the collection timestamp for use in traffic flow statistics and trend analysis.

[0177] The server calculates signal control plans using optimization theory methods based on past and current time-series information. It can utilize basic numerical computing libraries (such as NumPy and Pandas) for historical data comparison and statistical prediction. For traffic conditions in the next 5 minutes or other settable time periods, the server dynamically adjusts the control scheme (such as the duration of each phase) of the traffic lights (including main traffic lights and pedestrian traffic lights).

[0178] The server also integrates AI-powered emotion recognition technology. By calling general emotion recognition services through API interfaces (such as using general cloud-based cognitive service APIs or locally trained emotion recognition models), it analyzes the audio and video data collected from terminals and outputs labels and intensities of user psychological states such as stress, anger, and anxiety. The server jointly optimizes the emotion analysis results with traffic flow conditions. If it detects an increase in user psychological stress during a certain period, the server automatically adjusts signal timing to reduce negative experiences during waiting.

[0179] Ultimately, the server sends the optimized operation plan to the terminal signal control device via a low-latency protocol (such as MQTT). The terminal physical control unit then directly applies the latest timing to the field signal equipment, achieving real-time response. The terminal can also report the execution status in real time, allowing the server to further revise and optimize the strategy, thus achieving closed-loop control.

[0180] Users, as managers or traffic participants, can view the traffic conditions, signal timing, and user sentiment information calculated by the system in real time through a dedicated monitoring interface (such as a web visualization dashboard or mobile terminal), and have the authority to manually fine-tune the signal timing, enhancing the system's flexibility and emergency response capabilities.

[0181] For example:

[0182] During the morning rush hour, the terminal continuously collects traffic video and audio samples from pedestrians at the intersection, uploading them after SSL encryption. The server uses the YOLO algorithm to identify that there are currently 50 vehicles and 30 pedestrians at the intersection. Historical trends indicate that traffic flow will continue to rise, so the server extends the green light time by 10 seconds and shortens the pedestrian signal time by 5 seconds. At the same time, the emotion recognition module detects that the on-site stress index is high, so the server makes further minor adjustments to the signal control strategy. All adjustments are issued in real time to ensure smooth traffic and user comfort. The user management terminal can view these dynamic data and system adjustments at any time on the monitoring interface and issue new signal adjustment commands directly when necessary.

[0183] Example of prompts for generative artificial intelligence models:

[0184] The current intersection video data shows 50 vehicles and 30 pedestrians. Please use Python to optimize the traffic light timing, extending the green light by 10 seconds and shortening the pedestrian light by 5 seconds, and further fine-tuning the signal scheme based on real-time sentiment data (high stress index).

[0185] Please describe a system algorithm framework that combines traffic video analysis with user emotion recognition to achieve adaptive traffic signal control.

[0186] Input: YOLO recognition data + sentiment analysis results; Output: Optimal traffic light timing control scheme.

[0187] The present invention is flexible in its implementation and is applicable to road traffic environments and scenarios with varying pedestrian flow, as well as different scales and structures.

[0188] use Figure 13 The processing flow is explained.

[0189] Step 1:

[0190] The terminal uses traffic cameras and microphones to collect real-time traffic video data and environmental audio information from intersections or road segments. The input is the raw image and audio streams of the scene. The terminal compresses the video using H.264 and uploads it to the server every second via SSL / TLS protocol. The output is encrypted compressed audio and video data packets.

[0191] Step 2:

[0192] The server receives encrypted audio and video data packets from various endpoints. It first uses a decoding module (such as FFmpeg) to decrypt and restore the data, recovering each frame of the image and audio stream. The input is an encrypted compressed data packet, and the output is a structured original image frame sequence and an audio file.

[0193] Step 3:

[0194] The server uses image recognition algorithms (such as YOLO and OpenCV) to perform object detection on all video frames, identifying specific objects in the scene (such as vehicles, pedestrians, bicycles, etc.) and extracting their attributes such as position, speed, and direction of motion. The input is a structured sequence of image frames, and the server obtains the recognition results through model inference. The output is attribute data for each object (such as type, position, speed, direction, etc.).

[0195] Step 4:

[0196] The server performs statistical analysis on all object attribute data in chronological order, generating comprehensive traffic status descriptions such as traffic flow, average speed, and congestion level. The input is a set of object attribute data; after data aggregation, calculation, and statistical analysis, the output is traffic flow status data and corresponding time-series statistical reports.

[0197] Step 5:

[0198] Based on the latest and historical traffic status data, the server uses optimization algorithms (such as feedback control and dynamic programming) to calculate the optimal duration of each signal phase (green light, yellow light, red light) and generate a signal control plan. The inputs are traffic flow status data and historical time-series information; after optimization model calculations, the output is the optimal signal timing control parameters.

[0199] Step 6:

[0200] The terminal uses a microphone and camera to capture facial expressions and voice samples of users (pedestrians, drivers) in real time and uploads them to the server periodically. The input consists of raw facial images and live audio. After basic preprocessing, the terminal pushes the audio and video data packets to the server, and the output is a tagged audio and video data stream.

[0201] Step 7:

[0202] The server uses sentiment recognition algorithms (such as those based on machine learning cloud service APIs) to process the aforementioned user audio and video samples, identifying and quantifying the user's psychological state, such as stress, anger, and anxiety levels. The input is user audio and video data; the server analyzes and classifies it, and the output is structured sentiment state labels and quantified numerical values.

[0203] Step 8:

[0204] The server combines current traffic flow status with user psychological state labels to further fine-tune the traffic light control plan. For example, it reduces waiting time for some people during periods of high stress. The inputs are sentiment labels and the original signal control plan, which are then refined and optimized through a decision model, with the output being the final signal timing control parameters.

[0205] Step 9:

[0206] The server packages the latest optimized signal control parameters and sends them to the terminal signal controller via low-latency protocols such as MQTT. The input is the final signal timing parameters, and the output is the signal timing instructions that the signal controller can execute. After receiving the instructions, the terminal controller directly controls each signal light to switch states according to the new parameters and provides real-time feedback on the execution status.

[0207] Step 10:

[0208] Users can view real-time traffic conditions, signal timing, and user sentiment data at each intersection through a monitoring interface (such as a web dashboard). Inputs include parameters such as traffic flow, signal status, and sentiment values. After visualization rendering, the output is an operable management interface. Users can manually adjust specified signal light parameters as needed; commands are approved by the server and then queued for execution by the signal controller.

[0209] Application Example 2

[0210] The process flow corresponding to the specific processing in Use Case 2 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. In addition, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0211] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 2 is achieved by the following means.

[0212] In this invention, the server includes: a device for acquiring image information related to the operating status through an information acquisition device; a device for analyzing the acquired image information and determining the operating status; a device for optimizing the control timing of the controlled object device based on the information processing results; a device for applying the optimized control timing to the controlled object device; a device for acquiring and analyzing the user's voice information or image information and determining their psychological state; a device for further adjusting the control timing based on the user's psychological state; and a device for acquiring the operating status of the surrounding area of ​​the autonomous driving mobile vehicle and optimizing the autonomous driving control mode based on traffic and emotional information. This enables dynamic optimization of signal control and mobile vehicle driving strategies based on real-time traffic conditions and user psychological states, not only improving traffic flow and safety but also effectively reducing the psychological burden on road users, promoting the deep integration and development of intelligent traffic management and autonomous driving technology.

[0213] "Information acquisition device" refers to a hardware device used to collect data related to the system's operating status, such as sensing devices like cameras and microphones.

[0214] "Information processing device" refers to a device that analyzes, judges, and calculates collected data. It can be a server, computer, or other equipment with data computing and artificial intelligence processing capabilities.

[0215] "Image information" refers to digital image or video data that includes traffic flow, object status, etc., acquired through sensing devices (such as cameras).

[0216] "Operating status" refers to the real-time status information of key system objects (such as traffic roads, moving vehicles, etc.), including traffic flow, speed, location, and congestion level.

[0217] "Controlled object device" refers to hardware equipment that aims to control the system and adjusts its state according to the output control signals, such as traffic lights, traffic control terminals, or autonomous vehicles.

[0218] "Operation control time" refers to the specific time parameters for executing actions or switching states of the controlled device, such as the time for signal lights to change color or the time for vehicle acceleration and deceleration.

[0219] "Users" refers to individuals who use a transportation system or related services, including drivers, passengers, pedestrians, and other participants in transportation.

[0220] "Psychological state" refers to the psychological and emotional characteristics such as feelings, stress, and anxiety exhibited by the user in a specific time and space environment, which are judged through signals such as voice and facial expressions.

[0221] "Artificial intelligence processing device" refers to a hardware and software system that has the ability to intelligently analyze, recognize and infer complex data (such as image information, voice information and behavioral data) based on algorithm models.

[0222] "Mobile vehicle" refers to a mobile device or vehicle that moves in traffic environments such as roads, including autonomous vehicles and public transportation vehicles.

[0223] "Driving control mode" refers to the automatic adjustment of the driving strategy and behavior of a moving body, including the setting of parameters such as speed, route, acceleration and deceleration.

[0224] "Historical data" refers to a collection of data representing the state of a system or object that was collected and stored during its past operation, and is used for subsequent comparison, pattern analysis, or trend prediction.

[0225] This invention relates to a system for optimizing traffic management and mobile vehicle control using an information acquisition device, an information processing device, and an artificial intelligence processing device. The system mainly includes the following components and processing steps:

[0226] First, the terminal is equipped with information acquisition devices, including cameras and microphones, to collect image and sound information in traffic scenarios in real time. Cameras are typically installed on main roads, intersections, or moving objects (such as autonomous vehicles) to efficiently collect real-time status information of targets such as vehicles, pedestrians, and traffic lights on the road. Voice acquisition devices can be installed in the cockpit or passenger compartment to collect user voice data and ambient sound.

[0227] Secondly, the terminal transmits the collected data to the server in real time via wired network, wireless network, or dedicated communication channel. The server, as an information processing device, is equipped with high-performance computing chips and storage modules, and has open-source artificial intelligence software such as OpenCV and TensorFlow installed, along with Python scripts for data processing.

[0228] After receiving the image information, the server first uses OpenCV for image preprocessing, including image segmentation and object detection. Then, it calls a pre-trained TensorFlow AI model to classify and identify various traffic participants in the image, extracting structured attribute information such as vehicle type, location, speed, and direction. For historical data, the server also calls the storage module to perform historical comparison and trend analysis, thereby assessing the current traffic conditions and predicting potential trends. The data results are stored in the database, and the parameters are iteratively optimized.

[0229] The server synchronously receives user audio or video signals and performs multimodal emotion analysis using artificial intelligence emotion models (such as emotion recognition engines). Based on the analyzed psychological states (such as anxiety, stress, anger, etc.), the server judges the user's psychological burden and further incorporates the analysis results into traffic light strategy optimization or autonomous driving decision-making mechanisms.

[0230] In terms of traffic light control, the server calculates and optimizes the timing of the controlled devices based on real-time traffic conditions and user emotional states. For example, the system automatically extends the green light time on main roads to alleviate congestion or reduces waiting time to reduce user anxiety. The server sends signal control commands to the traffic signal terminals via standard protocols, and the traffic signal terminals adjust the actual display status of the traffic lights in real time accordingly, and provide feedback and alarms for abnormal situations.

[0231] In terms of mobile vehicle control, the server, based on traffic environment data and user psychological state, utilizes control software such as ROS (Robot Operating System) to form an autonomous driving cooperative system. The server dynamically issues parameters such as driving speed and path, and the autonomous vehicle terminal receives the instructions, executes the actual driving operations, and reports the current status to the server in real time, achieving closed-loop control. If abnormal congestion occurs or the user's emotions change suddenly, the system can autonomously optimize the control strategy to ensure high safety and a superior user experience.

[0232] The system implementation of this invention can be applied to scenarios such as urban intelligent transportation and autonomous driving mobility services. By organically combining information acquisition, artificial intelligence analysis, and signal and autonomous driving control, it can effectively regulate the psychological burden on users while ensuring road traffic safety and smoothness, thereby enhancing the humanization and intelligence level of the transportation system.

[0233] Specific examples:

[0234] During the morning rush hour, endpoints (such as intersection cameras) continuously collect and transmit video streams. The server, using a TensorFlow model, detects a significant increase in traffic flow at a particular intersection, while passengers express impatience via voice. The system incorporates the psychological state analysis results along with traffic conditions into the signal control model, automatically adjusting green light durations. Simultaneously, ROS automatically adjusts the speed and route of autonomous vehicles, thereby reducing congestion and alleviating passenger dissatisfaction.

[0235] Example of prompts for generative artificial intelligence models:

[0236] Please use OpenCV and TensorFlow to perform target recognition and analysis on real-time traffic images, and output the number and status of vehicles, pedestrians, and bicycles; at the same time, perform psychological state recognition based on passenger voice, and combine traffic flow conditions to recommend the optimal traffic light timing and autonomous vehicle driving strategy.

[0237] Please perform multimodal sentiment analysis on the uploaded video and audio data to determine the user's stress, anxiety, and other levels, and use the analysis report for the generation of intelligent signal control and autonomous driving commands.

[0238] use Figure 14 The processing flow is explained.

[0239] Step 1:

[0240] The terminal uses cameras and microphones installed at intersections, roads, and mobile devices (such as autonomous vehicles) to collect real-time image and audio information of traffic scenes. The input is real-time video and audio signals from the traffic scene, and the output is raw video streams and audio data files. This raw data is transmitted to the server via wired or wireless networks.

[0241] Step 2:

[0242] After receiving the raw video stream and audio data from the terminal, the server first uses OpenCV to perform frame extraction and basic preprocessing on the video data, followed by object detection and region segmentation. The input is the raw video stream, and the processing includes image denoising, frame segmentation, and region filtering. The output is a structured set of image frames.

[0243] Step 3:

[0244] The server uses generative AI models such as TensorFlow to perform target recognition and classification on pre-processed image frames, identifying objects such as vehicles, pedestrians, and bicycles, and analyzing their position, speed, and direction of movement. The input is a set of structured image frames, which, after being processed by the deep learning model, outputs structured labeled data containing object categories and motion parameters, such as lists or database entries.

[0245] Step 4:

[0246] The server analyzes all identified target data and combines it with traffic flow data from a historical database to comprehensively assess the current road congestion level and traffic flow status, and predict traffic trends for the next few minutes. Inputs include structured target labels and historical data; processing includes data statistics and trend modeling; outputs include the current traffic status and prediction results.

[0247] Step 5:

[0248] The terminal acquires user voice and facial expression information, converts it into a digital audio and video stream, and reports it to the server's emotion recognition module. The input is the user's real-time voice and image data, which is packaged and sent to the server. The output is the raw multimodal audio and video stream.

[0249] Step 6:

[0250] The server analyzes user audio and video data using an AI-powered sentiment recognition model to identify and quantify the user's emotional state (such as stress, anxiety, and irritability). The input is the user's multimodal audio and video stream; after deep sentiment analysis, the output is a specific sentiment category and score (e.g., "High stress, 0.8").

[0251] Step 7:

[0252] The server integrates traffic condition analysis results and user sentiment data, and dynamically adjusts traffic light control parameters, including the duration of red and green lights, based on intelligent optimization algorithms. Inputs include traffic flow status and user sentiment scores; processing involves multi-objective optimization algorithms; and outputs the latest traffic light timing parameters.

[0253] Step 8:

[0254] The terminal (signal controller terminal) receives optimized traffic light parameters from the server and controls the actual color switching of the traffic lights in real time. The input is the traffic light control parameters, which are controlled by the terminal's hardware logic. The output is the changed traffic light lighting status, and the execution result can be fed back to the server.

[0255] Step 9:

[0256] The server integrates traffic conditions and user emotional states, and uses ROS or other control systems to generate optimal operating instructions for autonomous vehicles, including speed, route, and acceleration. Inputs include traffic predictions, user emotions, and current vehicle information; processing results in control commands; and output is the real-time action decision the vehicle should execute.

[0257] Step 10:

[0258] The terminal (autonomous vehicle) receives operation control commands from the server and directly executes operations such as acceleration, deceleration, steering, and lane changing. It also feeds back the current execution status and perceived environment to the server through the onboard system. The input is the control commands issued by the server, which are executed by the autonomous driving module, and the output is the vehicle's actions and feedback data.

[0259] The specific processing unit 290 sends the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires sound representing user input regarding the result of the specific processing. The control unit 46A sends the sound data representing user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0260] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0261] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart device 14 or external devices, and the smart device 14 acquires or collects information required for processing from the data processing device 12 or external devices.

[0262] For example, the collection unit is implemented by the control unit 46A of the smart device 14 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart device 14 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the output device 40 of the smart device 14 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0263] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart device 14.

[0264] Second Implementation Method

[0265] Figure 3 An example of the configuration of the data processing system 210 according to the second embodiment is shown.

[0266] like Figure 3 As shown, the data processing system 210 includes a data processing device 12 and smart glasses 214. A server can be cited as an example of the data processing device 12.

[0267] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0268] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, and communication I / F 44 are also connected to the bus 52.

[0269] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0270] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to capture images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0271] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0272] Figure 4 This illustrates an example of the main functions of the data processing device 12 and the smart glasses 214. For example... Figure 4 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0273] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0274] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0275] In the smart glasses 214, the processor 46 performs reception and output processing. The memory 50 stores the reception and output program 60. The processor 46 reads the reception and output program 60 from the memory 50 and executes the read reception and output program 60 on the RAM 48. The reception and output processing is implemented by the processor 46 operating as a control unit 46A according to the reception and output program 60 executed on the RAM 48. Furthermore, the smart glasses 214 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290.

[0276] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart glasses 214. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0277] Example 1

[0278] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0279] Application Example 1

[0280] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0281] Example 2

[0282] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0283] Application Example 2

[0284] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0285] The specific processing unit 290 sends the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A outputs the result of the specific processing to the speaker 240. The microphone 238 acquires sound input representing the user's input regarding the result of the specific processing. The control unit 46A sends the sound data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0286] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0287] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or external devices, and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or external devices.

[0288] For example, the collection unit is implemented by the control unit 46A of the smart glasses 214 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart glasses 214 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the smart glasses 214 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0289] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart glasses 214.

[0290] Third Implementation Method

[0291] Figure 5 An example of the configuration of the data processing system 310 according to the third embodiment is shown.

[0292] like Figure 5 As shown, the data processing system 310 includes a data processing device 12 and a head-mounted terminal 314. A server can be cited as an example of the data processing device 12.

[0293] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0294] The head-mounted terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, display 343, and communication I / F 44 are also connected to the bus 52.

[0295] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0296] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to capture images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0297] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0298] Figure 6 This illustrates an example of the main functions of the data processing device 12 and the head-mounted terminal 314. For example... Figure 6 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0299] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0300] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0301] In the head-mounted terminal 314, the processor 46 performs the acceptance / output processing. The memory 50 stores the acceptance / output program 60. The processor 46 reads the acceptance / output program 60 from the memory 50 and executes the read acceptance / output program 60 on the RAM 48. The acceptance / output processing is implemented by the processor 46 operating as a control unit 46A according to the acceptance / output program 60 executed on the RAM 48.

[0302] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the head-mounted terminal 314. In the following description, the data processing device 12 will be referred to as the "server" and the head-mounted terminal 314 will be referred to as the "terminal".

[0303] Example 1

[0304] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0305] Application Example 1

[0306] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0307] Example 2

[0308] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0309] Application Example 2

[0310] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0311] The specific processing unit 290 sends the result of the specific processing to the head-mounted terminal 314. In the head-mounted terminal 314, the control unit 46A outputs the result of the specific processing to the speaker 240 and the display 343. The microphone 238 acquires sound input representing the user's input regarding the result of the specific processing. The control unit 46A sends the sound data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0312] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 includes prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0313] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the head-mounted terminal 314, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the head-mounted terminal 314. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the head-mounted terminal 314 or external devices, and the head-mounted terminal 314 acquires or collects information required for processing from the data processing device 12 or external devices.

[0314] For example, the collection unit is implemented by the control unit 46A of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the head-mounted terminal 314 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 to analyze the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12 to generate a menu using a generation AI. For example, the serving unit is implemented by the speaker 240 and display 343 of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12 to provide the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0315] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the head-mounted terminal 314.

[0316] Fourth Implementation Method

[0317] Figure 7 An example of the configuration of the data processing system 410 according to the fourth embodiment is shown.

[0318] like Figure 7 As shown, the data processing system 410 includes a data processing device 12 and a robot 414. A server can be cited as an example of the data processing device 12.

[0319] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0320] Robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, controlled object 443, and communication I / F 44 are also connected to the bus 52.

[0321] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0322] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to photograph the area around robot 414 (e.g., the field of view defined by a perspective equivalent to the field of vision of an average healthy person).

[0323] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0324] The controlled object 443 includes a display device, LEDs (light-emitting diodes) for the eyes, and motors for driving the arms, hands, and feet. The posture or movement of the robot 414 is controlled by controlling the motors in the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. In addition, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0325] Figure 8 This illustrates an example of the main functions of the data processing device 12 and the robot 414. For example... Figure 8 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0326] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0327] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0328] In robot 414, the processor 46 performs the acceptance and output processing. The memory 50 stores the acceptance and output program 60. The processor 46 reads the acceptance and output program 60 from the memory 50 and executes the read acceptance and output program 60 on RAM 48. The acceptance and output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance and output program 60 executed on RAM 48.

[0329] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the robot 414. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 will be referred to as the "terminal".

[0330] Example 1

[0331] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0332] Application Example 1

[0333] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0334] Example 2

[0335] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0336] Application Example 2

[0337] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0338] The specific processing unit 290 sends the result of the specific processing to the robot 414. In the robot 414, the control unit 46A outputs the result of the specific processing to the speaker 240 and the controlled object 443. The microphone 238 acquires sound input representing the result of the specific processing. The control unit 46A sends the sound data representing the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0339] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0340] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or external devices, and the robot 414 acquires or collects information required for processing from the data processing device 12 or external devices.

[0341] For example, the collection unit is implemented by the control unit 46A of the robot 414 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the robot 414 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the robot 414 and the control object 443 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0342] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the robot 414.

[0343] Furthermore, the emotion-specific model 59, acting as an emotion engine, can determine a user's emotion based on a specific mapping. Specifically, the emotion-specific model 59 can determine a user's emotion based on an emotion graph that serves as a specific mapping (see [reference]). Figure 9 The system determines the user's emotions. Furthermore, the emotion-specific model 59 can similarly determine the robot's emotions, and the specific processing unit 290 performs specific processing based on the robot's emotions.

[0344] Figure 9 This is a diagram representing an emotion map 400 that maps multiple emotions. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotion is. On the outer side of the concentric circles, emotions representing states or behaviors arising from mood are arranged. Emotions are concepts that include feelings and mental states. Emotions generated by reactions occurring in the brain are arranged roughly to the left of the concentric circles. Emotions derived from situational judgments are arranged roughly to the right of the concentric circles. Emotions generated by reactions occurring in the brain and derived from situational judgments are arranged roughly above and below the concentric circles. Furthermore, "pleasant" emotions are arranged above the concentric circles, and "unpleasant" emotions are arranged below them. Thus, in the emotion map 400, multiple emotions are mapped based on the structure that generates emotions, and emotions that are likely to occur simultaneously are mapped close to each other.

[0345] These emotions are distributed at the three o'clock position of the emotion map 400, typically fluctuating between peace and anxiety. In the right half of the emotion map 400, situational awareness dominates over internal sensation, thus resulting in an impression of calm.

[0346] The inner side of the emotion map 400 represents the inner state, while the outer side represents behavior. Therefore, the further outward you are from the emotion map 400, the more visible the emotion becomes (manifested in behavior).

[0347] Here, human emotions are based on various balances such as posture and blood sugar levels. When these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotions in robots, cars, motorcycles, etc., can also be created in the following way: based on various balances such as posture and remaining battery power, when these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotion maps can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a Brain Physiological Signal Analysis System for Voice Emotion Recognition and Emotion, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). In the left half of the emotion map, emotions belonging to the sensory-dominated region, called "response," are arranged. Furthermore, in the right half of the emotion map, emotions belonging to the situational cognition-dominated region, called "situation," are arranged.

[0348] In the emotion map, two types of emotions that promote learning are defined. One is a negative emotion on the situational side, in the middle or peripheral region of "repentance" or "reflection." This occurs when the robot experiences negative emotions such as "I don't want to experience this feeling again" or "I don't want to be blamed again." The other is a positive emotion on the response side, near the "desire" region. This occurs when there are positive feelings such as "wanting more" or "wanting to know more."

[0349] The emotion-specific model 59 inputs user input into a pre-trained neural network to obtain emotion values ​​representing each emotion shown in the emotion map 400, thereby determining the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network... Figure 10 As shown in the sentiment graph 900, it was trained in a way that sentiments that are configured close to each other have similar values. Figure 10 The text shows examples of emotions such as "peace of mind", "stability", and "reassurance" that have similar emotion values.

[0350] The above description focuses on the functions of the data processing device 12, but the system of this disclosure is not necessarily installed on a server. The system of this disclosure can also be installed as a general information processing system. This disclosure can also be installed, for example, as a software program running on a personal computer, an application running on a smartphone, etc. The method of this disclosure can also be provided to users in the form of SaaS (Software as a Service).

[0351] In the above embodiments, an example of a specific process being performed by a single computer 22 is given. However, the technology disclosed herein is not limited to this, and the specific process can also be distributed among multiple computers, including computer 22. For example, the data generation model 58 can be located on an external device of the data processing apparatus 12, where data is generated based on the input data.

[0352] In the above embodiments, examples of storing a specific processing program 56 in the memory 32 have been described, but the technology disclosed herein is not limited thereto. For example, the specific processing program 56 may also be stored in a portable computer-readable non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed into the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0353] Alternatively, a specific processing program 56 may be pre-stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 according to the requirements of the data processing device 12.

[0354] In addition, it is not necessary to store all the specific processing program 56 in the storage device such as the server connected to the data processing device 12 via the network 54 or in the memory 32; a portion of the specific processing program 56 may be stored in advance.

[0355] As hardware resources for performing specific processes, various processors, as shown below, can be used. For example, a CPU can be listed as a processor, which functions as a general-purpose processor that performs specific processes by executing software, i.e., a program. Furthermore, processors can be listed as special-purpose circuits such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application-Specific Integrated Circuits), which are processors with circuitry specifically designed to perform specific processes. Each processor has built-in or connected memory, and each processor executes specific processes using that memory.

[0356] The hardware resources for performing a specific process can consist of one of these various processors, or a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resources for performing a specific process can be a single processor.

[0357] As an example of a single processor, there are two approaches: First, a processor is composed of a combination of one or more CPUs and software, which functions as a hardware resource to perform a specific process; second, as represented by a SoC (System-on-a-chip), a processor is used to implement the functionality of the entire system, which includes multiple hardware resources for performing a specific process, using a single IC (Integrated Circuit) chip. In this way, the specific process is implemented by using one or more of the aforementioned processors as hardware resources.

[0358] Furthermore, the hardware architecture of these various processors, more specifically, can utilize circuits that combine semiconductor elements and other circuit components. Moreover, the specific process described above is just one example. Therefore, without departing from the main point, unnecessary steps can certainly be deleted, new steps added, or the processing order changed.

[0359] The descriptions and illustrations above are detailed explanations of a portion of the technology disclosed herein, and are merely one example of the technology disclosed herein. For example, the above descriptions of the structure, function, effect, and results are just one example of the structure, function, effect, and results of a portion of the technology disclosed herein. Therefore, without departing from the spirit of the technology disclosed herein, unnecessary parts may be deleted, new elements added, or replacements may be made to the descriptions and illustrations above. Furthermore, to avoid confusion and facilitate understanding of a portion of the technology disclosed herein, explanations of common technical knowledge that do not require special explanation under the premise of being able to implement the technology disclosed herein have been omitted from the descriptions and illustrations above.

[0360] All documents, patent applications and technical specifications set forth in this specification are incorporated herein by reference to the same extent that each document, patent application and technical specification is specifically and individually described therein and referenced by reference.

[0361] In addition, the following notes are provided in response to the above explanation.

[0362] Example 1

[0363] (Note 1)

[0364] An information processing system includes: a device for periodically acquiring image data representing traffic conditions using an information acquisition device; a device for transmitting the image data to an information processing device via a communication network; a device for parsing the image data using a machine learning algorithm in the information processing device to obtain attribute information of multiple moving targets; a device for inferring future traffic conditions based on accumulated historical traffic-related data using a prediction algorithm; a device for determining control parameters for optimizing the output of a signal control device based on the parsed attribute information and the inference results; a device for transmitting the control parameters to the signal control device via a control communication protocol and changing its operating settings; and a device for enabling users to obtain traffic condition data and signal control information through a visual display device and manually adjust the control parameters when needed.

[0365] (Note 2)

[0366] According to the information processing system described in Appendix 1, the information processing device uses a deep learning-based image analysis algorithm to identify moving targets.

[0367] (Note 3)

[0368] According to the information processing system described in Note 1, the information processing device uses a time-series prediction model to predict future values ​​of traffic volume or congestion level.

[0369] Application Example 1

[0370] (Note 1)

[0371] An information processing system includes: a device for acquiring traffic video information; a device for parsing the acquired traffic video information and extracting the movement attributes of traffic participants; a device for calculating traffic conditions based on the movement attributes and time-series information of traffic participants; a device for optimizing the signal control timing of signal control equipment based on traffic conditions; a device for applying the optimized signal control timing to the signal control equipment; a device for displaying the control status of the signal control equipment and traffic conditions to a management device; a device for manually adjusting the signal control timing according to the operation of traffic managers; a device for inferring the emotional state of traffic participants based on monitoring data or voice information, and further adjusting the signal control timing based on the inference result; and a device for providing the optimized signal control timing and traffic conditions to the mobile participants through a mobile communication network.

[0372] (Note 2)

[0373] According to the information processing system described in Note 1, a recognition model based on image processing computing devices and machine learning technology is used in the parsing process.

[0374] (Note 3)

[0375] According to the information processing system described in Note 1, when calculating traffic conditions, historical traffic attribute information or time series data from multiple times is used to perform pattern prediction through an inference model.

[0376] Example 2

[0377] (Note 1)

[0378] An information processing system includes: a device for acquiring image information of target objects containing traffic conditions via a spatial information acquisition device; a device for performing image analysis processing on the acquired image information via an information processing device and generating moving object attribute information; a device for acquiring traffic flow status and calculating an operation plan for a controlled object device based on the attribute information and stored time-series information via a numerical calculation device; a device for acquiring and analyzing user personal information data and psychological state information inferred based on audio and video information via a calculation processing device, and revising the operation plan based on the psychological state information; and a device for outputting the optimized operation plan via a physical control device and applying it to a signal control device in the real world.

[0379] (Note 2)

[0380] According to the information processing system described in Appendix 1, the image analysis and processing employs an image recognition algorithm that includes a machine learning model.

[0381] (Note 3)

[0382] According to the information processing system described in Note 1, when calculating the operation plan, a method is used to predict future traffic conditions by using time-series information from multiple points in time stored in the storage device for mathematical prediction.

[0383] Application Example 2

[0384] (Note 1)

[0385] An information processing system includes: a device for acquiring image information related to operating status via an information acquisition device; a device for analyzing the acquired image information and determining the operating status via an information processing device; a device for optimizing the operating control timing of a controlled object device based on the operating status determined by the information processing device; a device for applying the optimized operating control timing to the controlled object device; a device for acquiring voice or image information of a user and analyzing it via an artificial intelligence processing device to determine the user's psychological state; a device for further adjusting the operating control timing based on the determined psychological state; a device for acquiring the operating status of the surrounding area of ​​the mobile body via an information acquisition device installed on the mobile body; and a device for optimizing the driving control mode of the mobile body based on the operating status and psychological state acquired and analyzed by the information processing device.

[0386] (Note 2)

[0387] According to the information processing system described in Appendix 1, the information processing device uses an artificial intelligence model to analyze image information and the user's voice or image information.

[0388] (Note 3)

[0389] According to the information processing system described in Appendix 1, the information processing device makes historical operational status predictions based on previously stored data.

Claims

1. An information processing system, characterized in that, include: Devices used to acquire traffic images; A device for analyzing acquired traffic images and determining traffic conditions; A device for optimizing signal timing based on traffic conditions; A device for applying optimized signal timing to signal controllers.

2. The information processing system according to claim 1, characterized in that, The apparatus for parsing traffic images includes a means for analyzing traffic images using image recognition algorithms.

3. The information processing system according to claim 1, characterized in that, The device for determining traffic conditions includes a device for predicting traffic condition patterns based on historical data.

Images