Information processing system

Abstract

The invention provides an information processing system. The information processing system comprises a device for acquiring a crossroad video in real time; a device for analyzing the acquired video and detecting passing vehicles and pedestrians; the device is used for judging the traffic volume based on the detection result; means for changing the signal light when it is determined that the traffic volume is lower than the predetermined threshold; and means for confirming the changed signal state.

Description

Technical Field

[0001] The technology disclosed herein relates to an information processing system. Background Technology

[0002] Japanese Patent Application Publication No. 2022-180282 discloses a method for controlling a role-based chatbot executed by at least one processor. The method includes the following steps: receiving a user's speech; adding the user's speech to a prompt word, the prompt word containing instruction statements associated with an explanation of the chatbot's role; encoding the prompt word; and inputting the encoded prompt word into a language model to generate a chatbot response to the user's speech.

[0003] Existing traffic signal control systems typically employ timed control, which cannot dynamically adjust based on actual road conditions. This often leads to signal light malfunctions and waiting times at night or when traffic volume is low, reducing road efficiency and increasing energy waste. Therefore, there is an urgent need for a system capable of real-time monitoring of intersection traffic conditions and automatically optimizing signal control to address these issues. Summary of the Invention

[0004] This invention provides an information processing system, comprising: a device for real-time acquisition of video of an intersection; a device for analyzing the acquired video and detecting passing vehicles and pedestrians; a device for determining traffic volume based on the detection results; a device for changing traffic lights when the determined traffic volume is below a predetermined threshold; and a device for confirming the status of the changed signals. Through the coordinated operation of the above devices, dynamic traffic signal control based on real-time traffic volume assessment is achieved, thereby optimizing intersection traffic efficiency and reducing unnecessary waiting time.

[0005] "A device for real-time acquisition of video at intersections" refers to a hardware or software component that can continuously capture video footage from cameras located at road intersections.

[0006] "A device that analyzes acquired video and detects passing vehicles and pedestrians" refers to a hardware or software module that uses image processing or artificial intelligence technology to analyze real-time acquired video data and identify vehicles and pedestrians appearing in the video footage.

[0007] "A device for determining traffic volume based on detection results" refers to a calculation or logic module that calculates the number of vehicles and pedestrians passing through an intersection within a certain time period based on vehicle and pedestrian detection results, and compares the results with a preset threshold to determine the current traffic situation.

[0008] "A device that changes traffic lights when traffic volume is below a predetermined threshold" refers to a control module that can automatically issue a command to switch traffic lights (such as from red to green or from green to red) when it determines that the total number of vehicles and pedestrians at an intersection is below a preset threshold.

[0009] "A device for confirming the status of a signal after a change" refers to a device or module that can detect and determine whether the actual status of a signal light has been successfully switched to a new status after a traffic signal command is issued and executed.

[0010] "A device for acquiring the current state of a traffic light" refers to a detection tool or communication module used to actively acquire and obtain the current display color of a traffic light (such as red light, green light, etc.).

[0011] "A device that records the changed signal status to a log" refers to a hardware or software device that stores information such as the actual status of the traffic light after switching, the switching time, and the reason for switching in the form of data in a system log file. Attached Figure Description

[0012] Figure 1 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the first embodiment.

[0013] Figure 2 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and smart device according to the first embodiment.

[0014] Figure 3 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the second embodiment.

[0015] Figure 4 This is a conceptual diagram illustrating an example of the main functions of the data processing device and smart glasses according to the second embodiment.

[0016] Figure 5 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the third embodiment.

[0017] Figure 6 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and head-mounted terminal according to the third embodiment.

[0018] Figure 7 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the fourth embodiment.

[0019] Figure 8 This is a conceptual diagram illustrating an example of the main functions of the data processing device and robot according to the fourth embodiment.

[0020] Figure 9 This represents an emotion map that maps multiple emotions.

[0021] Figure 10 This represents an emotion map that maps multiple emotions.

[0022] Figure 11 This is a sequence diagram illustrating the processing flow of the data processing system of the first embodiment.

[0023] Figure 12 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 1.

[0024] Figure 13 This is a sequence diagram illustrating the processing flow of the data processing system of the second embodiment.

[0025] Figure 14 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 2. Detailed Implementation

[0026] Hereinafter, an example of an implementation of the system according to the present disclosure will be described with reference to the accompanying drawings.

[0027] First, let me explain the terminology used in the following instructions.

[0028] In the following embodiments, the processor (hereinafter referred to as "processor") with reference numerals may be a single computing device or a combination of multiple computing devices. Furthermore, the processor may be a single computing device or a combination of multiple computing devices. Examples of computing devices include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), etc.

[0029] In the following embodiments, RAM (Random Access Memory), as indicated in the figures, is a memory that temporarily stores information and is used as working memory by the processor.

[0030] In the following embodiments, the memory, as indicated by the reference numerals, is one or more non-volatile storage devices that store various programs and parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), disks (e.g., hard disks), or magnetic tapes.

[0031] In the following embodiments, the communication I / F (Interface) with reference numerals is an interface that includes a communication processor and an antenna, etc. The communication I / F is responsible for communication between multiple computers. As an example of a communication specification applicable to the communication I / F, wireless communication specifications such as 5G (5th Generation Mobile Communication System), Wi-Fi (wireless fidelity) (registered trademark), or Bluetooth (registered trademark) can be listed.

[0032] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. Furthermore, in this specification, when "and / or" connects to express more than three items, the same interpretation as "A and / or B" applies.

[0033] First Implementation Method

[0034] Figure 1 An example of the configuration of the data processing system 10 according to the first embodiment is shown.

[0035] like Figure 1 As shown, the data processing system 10 includes a data processing device 12 and an intelligent device 14. A server can be cited as an example of the data processing device 12.

[0036] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0037] The smart device 14 includes a computer 36, a receiving device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. In addition, the receiving device 38, output device 40, camera 42, and communication I / F 44 are also connected to the bus 52.

[0038] The receiving device 38 includes a touchscreen 38A and a microphone 38B, and receives user input. The touchscreen 38A receives user input via touch by detecting contact with an indicator (e.g., a pen or finger). The microphone 38B receives user input via sound by detecting the user's voice. The control unit 46A in the processor 46 sends data representing the user input received by the touchscreen 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data representing the user input.

[0039] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting data in a form perceptible to the user 20 (e.g., sound and / or text). The display 40A displays visual information such as text and images according to instructions from the processor 46. The speaker 40B outputs sound according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0040] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for sending and receiving various information between processor 46 and processor 28 via network 54.

[0041] Figure 2 The diagram shows an example of the main functions of the data processing device 12 and the smart device 14.

[0042] like Figure 2 As shown, in the data processing apparatus 12, specific processing is performed by the processor 28. A specific processing program 56 is stored in the memory 32. The specific processing program 56 is an example of a "program" as understood in this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0043] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0044] In the smart device 14, the processor 46 performs the acceptance output processing. The memory 50 stores the acceptance output program 60. The acceptance output program 60 is used in conjunction with the data processing system 10 and the specific processing program 56. The processor 46 reads the acceptance output program 60 from the memory 50 and executes the read acceptance output program 60 on the RAM 48. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48. Furthermore, the smart device 14 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48.

[0045] Alternatively, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains the processing results (prediction results, etc.) using the data generation model 58 by communicating with the server device that has the data generation model 58. Furthermore, the data processing device 12 may be a server device or a user-held terminal device (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of the processing of the data processing system 10 of the first embodiment will be described.

[0046] Example 1

[0047] The flow of a specific process in Example 1 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. Furthermore, the data processing device 12 is referred to as the "server," and the smart device 14 is referred to as the "terminal."

[0048] Most existing traffic signal control systems are based on preset timed control, making it difficult to reflect changes in actual traffic flow in real time, especially at night or during low-flow periods, which can easily lead to unnecessary traffic congestion and low traffic efficiency. Furthermore, existing technologies have weak management and post-event traceability capabilities for signal status changes, making it difficult to adjust and optimize traffic control strategies in a timely manner. Therefore, there is an urgent need for a traffic signal control system that can dynamically detect and judge traffic conditions based on real-time image information, automatically and flexibly adjust signal status, and record and trace the signal status change process.

[0049] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 1 is achieved by the following means.

[0050] In this invention, the server includes a device for real-time acquisition of image information around the intersection, a device for temporarily storing the image information in a storage device, a device for detecting moving objects using image recognition algorithms and machine learning models, a device for recording the data of the detected moving objects, a device for determining traffic volume based on the detection results and comparing it with a threshold, a device for automatically sending a signal status change command to the traffic control device when the traffic volume is below the threshold, a device for confirming the signal status after the change, and an information management device for recording the entire processing and the reasons for the judgment. This allows for automatic and efficient adjustment of the signal status according to actual traffic conditions, improving traffic flow efficiency, while simultaneously recording and tracing the entire process of signal changes, facilitating subsequent traffic management and optimization.

[0051] "Image information acquisition device" refers to a device that can acquire real-time image data of the surrounding environment in designated areas such as intersections, including but not limited to video camera equipment and imaging sensors.

[0052] "Information processing device" refers to a device that processes, analyzes, stores, and manages acquired image information, including computer systems, edge computing devices, etc.

[0053] "Temporary storage device" refers to a data storage device used to temporarily store acquired image data for a certain period of time, including memory, cache unit, etc.

[0054] "Image recognition algorithm" refers to a computational method that can analyze input image information and identify specific targets or features, including traditional image processing, deep learning algorithms, etc.

[0055] "Machine learning model" refers to an artificial intelligence model that is trained on a large amount of data and can automatically extract and classify the features of objects in an image, such as object detection model and classification model.

[0056] "Moving objects" refer to movable objects detected in image information, including traffic participants such as vehicles, pedestrians, and bicycles.

[0057] "Structured data" refers to a data type with a fixed format that can be split by fields, used to record specific attribute information of detected objects, such as position, direction, and speed.

[0058] "Traffic volume" refers to the number or frequency of moving objects passing through an intersection within a certain time and space.

[0059] "Setting threshold" refers to a pre-set baseline value for traffic volume to determine traffic conditions and trigger signal changes.

[0060] "Information and communication devices" refer to communication hardware and software used to enable information exchange between servers and traffic control devices, including wired or wireless network interfaces, communication protocol modules, etc.

[0061] "Traffic control devices" refer to control equipment used to control and execute changes in traffic signal status, such as traffic light controllers.

[0062] "Signal status change instruction" refers to a control instruction generated by the server based on the judgment result and sent to the traffic control device to realize the signal status change.

[0063] "Information Record Management System" refers to a data management platform or software system used to store, manage, query, and trace information related to changes in traffic signal status.

[0064] This invention provides a traffic signal management system based on image acquisition, artificial intelligence analysis, and automatic control. A preferred embodiment of this invention is described in detail below, including its hardware configuration, software platform application, and specific implementation.

[0065] This system mainly consists of the following units:

[0066] 1. Image information acquisition device: A high-resolution digital camera can be used as the terminal, installed at the road intersection, to collect image information of the intersection and its surroundings in real time. For example, a general-purpose high-definition camera can be selected as the image acquisition hardware.

[0067] 2. Information Processing Unit: The server serves as the core data processing node of the system. The server can be an edge intelligent hardware device with high-performance image processing capabilities, such as a general-purpose embedded AI computing platform. The server has storage units (such as DRAM) and computing units, and is responsible for collecting, storing, analyzing, and transmitting control signals for image data.

[0068] 3. Image Recognition and Analysis Unit: The server uses artificial intelligence-based image recognition algorithms, such as convolutional neural networks and object detection algorithms (e.g., widely used object detection neural network models), combined with general open-source computer vision libraries to perform image analysis, automatically detect moving objects such as vehicles and pedestrians, and extract key data such as their location information, motion trajectory, and speed.

[0069] 4. Judgment and Decision Unit: The server analyzes the number of moving objects (i.e., traffic volume) counted within each time period and compares the traffic volume with a preset threshold. When the current traffic volume is found to be lower than the set threshold, the system will automatically determine that signal status adjustment is required.

[0070] 5. Communication and Control Unit: This system employs information communication devices to send signal status change commands to the traffic control unit in real time via wired or wireless networks and standard communication protocols (such as MQTT, HTTP, etc.). Upon receiving the commands, the traffic control unit automatically changes the signal light status, achieving dynamic adjustment of the intersection signals.

[0071] 6. Signal Status Feedback and Log Management Unit: After a signal change is completed, the server obtains the latest signal status through the communication device and uses an information recording and management system (such as a log database, visualization analysis platform, etc.) to record the entire processing flow, the reason for the judgment, and the final control result, so as to facilitate retrospective review and management.

[0072] The advantages of this invention are that it enables flexible and efficient signal control based on actual road traffic conditions, without relying on fixed time periods or manual adjustments, reducing traffic congestion and improving road traffic efficiency. The system can also track the entire signal change process, achieving transparent management and optimization.

[0073] Example of a real-world use case: At night, a camera at an intersection captures multiple consecutive frames of footage. The server, using a target detection model, determines that there are very few vehicles and pedestrians passing through during this time. At this point, the server automatically issues a change signal command, switching the red light to green ahead of schedule. Once the change is confirmed, the operation is automatically recorded in the management system. Administrators can retrieve the change log at any time through a front-end visual interface and adjust traffic strategies accordingly.

[0074] Example of prompts for generative artificial intelligence models:

[0075] Please describe in detail how to use an object detection model and traffic signal control device to achieve automatic adjustment of intelligent traffic lights at night, considering the following hardware and software environment: camera, edge AI device, AI framework, and log management system. Please describe the data acquisition, detection, communication, and logging process for each step.

[0076] Through the above-described embodiments, those skilled in the art can accurately implement the invention based on this description, making it applicable to various real-world road scenarios requiring dynamic traffic signal management.

[0077] use Figure 11 The processing flow is explained.

[0078] Step 1:

[0079] The server periodically acquires real-time video data streams from the endpoints (cameras) at the intersection via a network interface. The input consists of video frames captured in real-time by the cameras, and the output is continuous raw image data. The server parses and buffers each frame according to a preset acquisition frequency, providing the data foundation for subsequent processing.

[0080] Step 2:

[0081] The server temporarily stores the received image data in a memory buffer. The input is the raw image frames obtained in step 1, and the output is a queue of video frames arranged in chronological order in memory. The server maintains a circular buffer queue to ensure that new frames overwrite old frames in a timely manner, and inserts the corresponding timestamps to achieve efficient access and real-time performance.

[0082] Step 3:

[0083] The server utilizes deployed object detection models (such as deep learning models) and image processing software to analyze each frame of image data in memory. The input is video frames in a memory buffer, and the output is a list of attribute data for each detected moving object (such as a vehicle or pedestrian), including location, category, confidence level, direction of movement, and speed. The server combines the target trajectories between consecutive frames to perform structured processing on the data for each moving object.

[0084] Step 4:

[0085] The server counts the number of all detected moving objects within a certain time window, thus calculating the traffic volume for that period. The input is the list of moving object attribute data output from step 3. Data processing operations include tracking, deduplication, and summarizing the identities of unique moving objects. The output is a comparison between the current traffic volume and a preset threshold, used for subsequent decision-making.

[0086] Step 5:

[0087] The server determines whether the traffic volume is below a preset threshold. If it is below the threshold, the server sends a signal status change command to the signal control terminal via the communication interface. The input is the traffic volume assessment result obtained in step 4 and the current signal status; the output is the sent signal change command. The server will determine the change target (e.g., switching from red to green) based on the current traffic light status.

[0088] Step 6:

[0089] The server receives signal status feedback from the signal control terminal and confirms that the signal has been switched as expected. The input is the feedback information from the signal control terminal, and the output is the confirmation result of the signal change. The server verifies the received status data to prevent commands from being executed incorrectly.

[0090] Step 7:

[0091] The server records the process data, determination reasons, operation time, and other information related to this signal status change into the information recording management system. The input consists of the processing logs, determinations, and execution results of all the aforementioned steps, and the output consists of structured log entries, facilitating later data traceability and management.

[0092] Application Example 1

[0093] The process flow corresponding to the specific processing in Use Case 1 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. Furthermore, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0094] Existing traffic signal control systems struggle to achieve high-precision real-time perception and intelligent dynamic control of traffic conditions at intersections, particularly in their inadequate integration of autonomous vehicles and user emotional states. This leads to reduced traffic flow and safety, and the inability to flexibly adjust traffic signals based on actual traffic volume and user emotions increases the risk of congestion and accidents.

[0095] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 1 is achieved by the following means.

[0096] In this invention, the server includes a device for acquiring dynamic status data and user attribute data in real time from multiple information acquisition devices; a device for processing the dynamic status data through a data processing device to identify target objects and moving bodies; a device for calculating the amount of movement in a spatial area; a device for threshold determination based on user attributes and movement results; a device for generating and sending control signals to a control device based on the determination results; a device for acquiring and confirming the status of the controlled object; a device for recording operations to a status recording device; a device for providing relevant information to external moving bodies or terminal devices; and a device for optimizing the movement path of external moving bodies or terminal devices. This enables intelligent, automated, and efficient traffic signal control based on actual traffic dynamic data and user emotional information, improving traffic flow and safety.

[0097] "Information acquisition device" refers to hardware equipment used to collect environmental status, moving body information and user attribute information in the target space in real time, such as cameras and sensors.

[0098] "Dynamic status data" refers to data that reflects the current status, position, and behavioral changes of target objects and moving bodies within a spatial area.

[0099] "User attribute data" refers to data that reflects the physiological or psychological state, behavioral characteristics, and emotional tendencies of traffic participants.

[0100] "Data processing device" refers to a computing unit used to analyze, identify and process various raw data collected, including servers, edge computing devices, etc.

[0101] "Target object" refers to a specific entity such as a vehicle, pedestrian, or bicycle detected by the system within a spatial area.

[0102] "Mobile body" refers to an object that has the ability to move within a spatial area, including but not limited to vehicles and pedestrians.

[0103] "Spatial area" refers to a specific physical area that is monitored, analyzed, and controlled by traffic signals, such as intersections and their surrounding road sections.

[0104] "Threshold determination" refers to the process of comparing actual detection data with preset quantitative standards to determine whether the critical conditions for triggering the control signal have been met.

[0105] "Control signal" refers to the instruction information generated by the system based on the judgment result and used to control the state changes of controlled objects such as traffic lights.

[0106] "Control device" refers to the hardware and software system that can receive control signals and adjust the state of traffic signals accordingly, such as a traffic light controller.

[0107] "Controlled object" refers to an object whose state is controlled in real time within a signal control system, such as a traffic light.

[0108] "Status recording device" refers to a data storage and retrieval system, such as a database, used to store the execution status of control signals, the reasons for the determination, and related historical information.

[0109] "External mobile bodies" refer to vehicles outside the system that can receive and adjust their own operating status according to traffic signal information, such as autonomous vehicles.

[0110] "Terminal device" refers to an electronic device that can communicate with a server, receive system information, and display or utilize that information, such as a smartphone or vehicle terminal.

[0111] "Route optimization" refers to the process of automatically adjusting and optimizing one's own behavior route or through strategies based on received traffic signals and status information.

[0112] "Emotion recognition device" refers to a device used to collect and analyze information such as a user's facial expressions and tone of voice in real time in order to determine the user's emotional state.

[0113] This invention relates to an intelligent traffic signal control system based on dynamic status data and user attribute data. The system includes basic units such as a server, information acquisition device, data processing device, status recording device, control device, external moving body, and terminal device, and the devices communicate with each other through wired or wireless networks.

[0114] The server can consist of a general-purpose computer, an industrial server, or edge computing hardware with an AI inference module. Information acquisition devices, such as high-definition video cameras, thermal sensors, and emotion recognition cameras, can be installed at traffic intersections or lane edges to collect real-time data on the dynamics, spatial states, and physiological or emotional attributes of traffic participants. The server processes video stream information using image processing software libraries such as OpenCV and combines this with deep learning models such as YOLOv4 for automatic detection and counting of targets such as vehicles, pedestrians, and non-motorized vehicles. Furthermore, the server employs emotion recognition models (such as convolutional neural networks and integrated emotion analysis engines based on speech and images) to analyze user facial expressions and speech features, enabling intelligent determination of the user's emotional state.

[0115] The data processing unit is responsible for integrating the collected data and recording traffic objects in a structured manner, including type, quantity, spatial distribution, and movement trends. The server determines traffic flow and user attribute status within a time window based on set threshold rules. For example, when the system detects that the number of vehicles waiting on the road is less than a set threshold, or that multiple users exhibit negative emotions such as anxiety or anger, the server automatically generates a control signal and sends it to the traffic signal control terminal via API or a dedicated transmission protocol, instructing it to change the traffic lights earlier or later.

[0116] Status recording devices (such as local databases or remote cloud storage) automatically record the execution time, status type, cause analysis, etc. of each signal operation, which facilitates subsequent traceability and management.

[0117] The server also pushes signal change information and traffic status to external mobile entities (such as autonomous vehicles) or passenger terminal devices (such as smartphone apps and in-vehicle navigation devices) via V2X communication technology, Wi-Fi, and cellular mobile networks. The terminal system of autonomous vehicles (which may use platforms such as ROS and Apollo) dynamically optimizes the travel route based on real-time information, improving the safety and efficiency of passing through intersections.

[0118] Users can view current traffic status information, remaining waiting time at traffic lights, and emotional feedback analyzed by the system through the terminal. Some terminals (such as dashcams and smart wearable devices) support real-time uploading of multimodal information such as the user's physiological and emotional state, enhancing the relevance of interaction and system response.

[0119] Specific example

[0120] For example, when traffic is extremely low at night, the server determines, based on images captured by cameras and facial expression data uploaded by terminals, that only two vehicles are stationary at the intersection, with one driver appearing anxious. The server immediately generates a control signal to switch the green light early based on the image and emotion recognition results. The signal control device then changes the light color accordingly, and the record is simultaneously transmitted to the status database. Furthermore, the signal data is sent in real-time to the two autonomous vehicles via V2X, achieving the optimal traffic experience.

[0121] Examples of prompts for generative artificial intelligence models

[0122] Please complete the implementation of traffic object detection and recognition based on YOLOv4 in the following Python source code.

[0123] "Please explain how the intelligent traffic signal control system integrates emotion recognition data and dynamic state data to make a comprehensive judgment and issue signal control commands."

[0124] "Assuming that an autonomous vehicle on the road has received signal status data pushed by the traffic signal system, please give an example of how to dynamically change the driving plan within the local route planning system."

[0125] This invention deeply integrates various information acquisition devices, multiple AI analysis technologies, and real-time signal control methods. Through periodic and automated data collection and decision-making, it not only improves traffic safety and efficiency but also enables personalized user perception and scenario adaptation, providing an innovative solution for the field of intelligent transportation.

[0126] use Figure 12 The processing flow is explained.

[0127] Step 1:

[0128] The server periodically sends data requests over the network to cameras and emotion recognition sensors deployed at intersections, collecting real-time video streams and multimodal data such as user facial expressions and voice. The input consists of video frame streams from the cameras and raw user attribute data collected by the emotion sensors. The server temporarily caches the acquired data in local memory as the foundation for subsequent analysis. Specific actions include establishing Socket connections with each data collection device, timing-controlling the data collection frequency, and formatting the raw data.

[0129] Step 2:

[0130] The server utilizes software components such as OpenCV and YOLOv4 to preprocess the input video frame stream, performing denoising and resolution normalization. Then, it performs deep learning object detection on each frame, intelligently identifying targets such as vehicles and pedestrians, and recording their type, location, and motion trajectory. Simultaneously, it calls the sentiment analysis module to analyze and recognize facial expressions and speech signals, inferring the user's emotional state (such as anxiety or anger). The output is a structured list of objects and a user sentiment tag table. This step involves data processing including neural network inference, feature extraction, and classification.

[0131] Step 3:

[0132] The server calculates the number of vehicles and pedestrians passing through the target object list generated in the previous step within a set time window; simultaneously, it calculates the sentiment label distribution for all users. The input is the structured detection results (target type, quantity, user sentiment), and the output is the traffic flow value and sentiment statistics for the current window. The server performs data aggregation and rule comparison operations, such as comparing the number of vehicles passing through with thresholds, and performing hierarchical statistics on the sentiment results.

[0133] Step 4:

[0134] The server comprehensively determines signal change conditions based on traffic flow statistics and user sentiment. The input is the traffic flow data and sentiment analysis conclusions from the previous step, and the output is the decision signal for the traffic light change. Specific actions include finding the set threshold, determining whether the light-changing conditions are met, and generating the light-changing command signal data structure.

[0135] Step 5:

[0136] The server sends the generated control commands to the traffic signal control device via API or a dedicated bus protocol and obtains the feedback status of the signal controller in real time. The inputs are the signal change command and the current signal status; the output is the actual execution result of the signal light status change. The server records the operation results and the reasons for the change, such as the change time, type, and decision factor.

[0137] Step 6:

[0138] The server pushes key information such as the latest traffic light status, traffic flow, and user sentiment to autonomous vehicles or mobile terminals passing through the intersection via V2X or other wireless communication methods. The input is comprehensive status information, and the output is message push to external devices. The server actively monitors the data channel with the target terminal to ensure real-time data delivery.

[0139] Step 7:

[0140] Terminals (such as autonomous vehicles, smartphone apps, and in-vehicle navigation systems) receive information pushed by the server and automatically adjust their local route planning or navigation strategies. The input is real-time traffic and signal data pushed by the server, and the output is optimized driving routes or navigation changes displayed on the terminal device. Specific actions include parsing data packets, updating the route planning module, and displaying the latest light status and traffic suggestions on the interface.

[0141] Step 8:

[0142] Users can view traffic light information, waiting times, anxiety alerts, and other feedback information through their terminal devices. The input is the data displayed on the terminal, and the output is the user's behavioral decisions (such as whether to wait, change routes, etc.). Specific actions include the user actively viewing the app interface, obtaining intersection traffic information, and selecting the appropriate next step.

[0143] Alternatively, an emotion engine for inferring user emotions can be combined. That is, the specific processing unit 290 can also use the emotion-specific model 59 to infer user emotions and perform specific processing using user emotions.

[0144] Example 2

[0145] The flow of a specific process in Example 2 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. The data processing device 12 will be referred to as the "server," and the smart device 14 as the "terminal."

[0146] Existing traffic signal control systems typically rely solely on simple traffic flow assessments and signal switching, failing to reflect the psychological state of users at intersections and the ever-changing real-time situation. This makes it difficult to achieve optimal signal control, easily leading to traffic congestion, pedestrian inconvenience, and reduced safety. Therefore, how to simultaneously combine environmental information, the attributes and dynamic behavior of target objects, and the psychological state of users to achieve more intelligent and flexible traffic signal control is a pressing technical challenge in this field.

[0147] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 2 is achieved by the following means.

[0148] In this invention, the server includes devices for acquiring various information from an environmental information acquisition device based on time information, extracting target object attributes and location information through an analysis device and an identification device, parsing target object psychological state information, calculating traffic flow and psychological state evaluation values, and automatically controlling signal devices and performing state confirmation and information recording when set conditions are met. This allows for dynamic and flexible adjustment of traffic signals not only based on traffic flow but also considering user psychological states, effectively improving traffic efficiency and safety, and enhancing the travel experience for pedestrians and vehicles.

[0149] "Information acquisition device" refers to a hardware or system device that can collect environmental information around an intersection in real time and send the information to a processing device.

[0150] "Analysis device" refers to a device that processes and analyzes acquired environmental information and outputs results, including computing devices or software modules with analysis functions.

[0151] "Identification device" refers to a hardware or software device that can identify target objects from environmental information and extract their attribute and location information.

[0152] "Target object" refers to vehicles, pedestrians or other designated objects that are detected and identified in environmental information.

[0153] "Attribute information" refers to various information used to describe the characteristics of a target object, including but not limited to parameters such as category, status, and size.

[0154] "Location information" refers to data used to represent the spatial location of a target object, which usually includes coordinates, region, or orientation.

[0155] "Action information" refers to the behavioral characteristics exhibited by a target object within a certain period of time, such as moving, staying, or approaching.

[0156] "Psychological state information" refers to the psychological and emotional state of a target object obtained through analysis or identification methods, including but not limited to feelings of pleasure, anxiety, and anger.

[0157] "Traffic flow evaluation value" refers to an evaluation index obtained based on the quantity and status of target objects statistically analyzed within a certain period, used to reflect the busyness of intersections.

[0158] "Psychological state evaluation score" refers to the comprehensive evaluation result obtained after analyzing the psychological state of the target object, which is used to measure the overall emotional trend of users.

[0159] "Signal control device" refers to a terminal device used to automatically control the switching of traffic lights according to input instructions and can receive external control information.

[0160] "Signal display status" refers to the current state of a traffic light, such as green, red, or yellow.

[0161] "Information recording device" refers to a storage device or system used to save information on changes in signal display status, reasons for changes, and relevant time points.

[0162] The embodiments of the present invention are described below:

[0163] This system is applicable to the field of automatic traffic signal control. It primarily achieves comprehensive analysis of real-time environmental information at intersections and user psychological states, thereby intelligently adjusting signal display status. The system includes multiple functional modules such as information acquisition devices, analysis devices, recognition devices, signal control devices, and information recording devices.

[0164] In this embodiment, the terminal (such as a camera and emotion recognition sensor installed at an intersection) is responsible for continuously collecting environmental information. The camera can be a standard surveillance camera, capable of uploading images in JPEG format at regular intervals; the emotion recognition sensor can integrate a facial expression analysis module and a voice feature analysis module, outputting relevant data such as user facial expressions and voice emotions, and transmitting them in JSON format.

[0165] The server, serving as the core analysis platform, is equipped with a high-performance processor and graphics accelerator card (such as an NVIDIA GPU), and runs deep learning model software such as YOLOv5, as well as psychological state analysis APIs (such as cloud-based emotion recognition API interfaces). The server receives environmental images and psychological state data from the aforementioned terminals through periodic requests, and then caches and archives these data.

[0166] The server first inputs image data into the YOLOv5 model to identify and locate target objects such as vehicles and pedestrians, extracting attribute information such as category, quantity, and coordinates. Then, the server analyzes psychological state data, identifying each user's current emotional state (such as pleasure, anxiety, anger, etc.) and generating a psychological state evaluation value. The server then statistically analyzes traffic volume and psychological state evaluation values ​​in set time intervals (e.g., every minute), and determines whether to adjust the traffic signal display status based on preset judgment rules and thresholds.

[0167] When the signal switching conditions are met, the server sends a light switching command to the signal control device via a REST API or similar method, controlling the signal display status (e.g., switching from red to green). After the signal status changes, the server retrieves the status information again for confirmation and saves detailed information such as the entire process time and the reason for the change to an information recording device (e.g., a log database based on MySQL or MongoDB).

[0168] This system not only focuses on traffic flow and object distribution, but also introduces for the first time an automatic perception and analysis of users' actual psychological state and a signal adaptive adjustment scheme based on this information, achieving more efficient and safer intelligent traffic signal control.

[0169] Specific application example: During the day, the server automatically acquires video images and real-time sentiment data from a certain intersection's cameras. Analysis reveals that while the number of pedestrians is relatively low, the number of people exhibiting anxiety is high. The server triggers an early green light switch, automatically recording in the log as "Early switch to green light due to increased pedestrian anxiety." This process improves user experience and traffic flow.

[0170] Example of prompts for generative artificial intelligence models:

[0171] For intersections containing cameras and emotion sensors, design a complete algorithm to intelligently adjust traffic lights based on real-time traffic volume and emotion feedback.

[0172] How can I call the YOLOv5 emotion recognition API and apply it to intelligent traffic light decision-making? Please provide detailed steps.

[0173] This paper presents a method to count the number of anxious and angry people at an intersection every minute and automatically issue signal control commands when a threshold is reached.

[0174] use Figure 13 The processing flow is explained.

[0175] Step 1:

[0176] The terminal collects data periodically. Every 5 seconds, the terminal (camera and emotion recognition sensor) automatically captures real-time footage of the intersection and detects facial expressions and vocal emotions of passing pedestrians. The input is the intersection environment, and the output is JPEG images and JSON-formatted emotion data. The terminal packages the latest image and emotion data for uploading and can temporarily cache it in local memory.

[0177] Step 2:

[0178] The terminal uploads data to the server. The terminal sends an HTTP POST request to the server's designated interface via Wi-Fi or Ethernet, uploading a JPEG image and JSON sentiment data as the request body. The input is the JPEG image and JSON sentiment data; the output is a copy of the data received by the server. If the terminal detects a network anomaly, it will retry the upload operation to ensure data real-time performance.

[0179] Step 3:

[0180] The server receives and caches data. It listens on an interface to continuously receive all data uploaded by terminals, adding a timestamp to each data entry and storing it in a memory buffer. Input consists of uploaded images and sentiment data; output is a dataset with a time index. The server also verifies data integrity, logging any missing or abnormal data in an error log.

[0181] Step 4:

[0182] The server performs target recognition and sentiment analysis. It uses a YOLOv5 deep learning model to analyze received images, detecting and locating vehicles and pedestrians, and extracting category, quantity, and location information. Simultaneously, the server reads sentiment data and uses a sentiment analysis API to identify the specific emotions (such as anxiety, anger, etc.) of each target object. The input consists of JPEG images and JSON sentiment data; the output is a structured list of data objects (including target category, quantity, coordinates, and sentiment labels).

[0183] Step 5:

[0184] The server statistically analyzes and determines traffic flow and psychological states. It uses tools such as Python to count the number of vehicles and pedestrians and the frequency of various emotional states within a specific time period. By comparing these statistics with preset thresholds, it determines whether signal switching conditions have been met. The input is a list of object and emotion data; the output is a report of traffic flow and psychological state evaluation values. The server automatically archives the statistical process.

[0185] Step 6:

[0186] The server issues a signal switching command. If the traffic is below a threshold or the number of anxious / angry individuals reaches a certain threshold, the server calls the REST API to send a control command to the signal control device, requesting that the red light be switched to green or vice versa. The input is a judgment report, and the output is the sent signal control command. The server records the command sending time and waits for feedback.

[0187] Step 7:

[0188] The server confirms the signal change and logs it. After receiving feedback from the signal control device, the server queries the current signal status for comparison. Once the switch is confirmed to be valid, the server saves detailed information such as the signal change action, reason, and time to the log database or file system. The input is the feedback information from the signal control device, and the output is a complete log record including detailed time, switch type, and switch reason.

[0189] Application Example 2

[0190] The process flow corresponding to the specific processing in Use Case 2 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. In addition, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0191] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 2 is achieved by the following means.

[0192] In this invention, the server includes a device for acquiring information from an acquisition device in real time, a device for parsing the acquired information to detect moving objects and users of those moving objects, a device for inferring the psychological state of users of those moving objects based on the detection results, a device for determining traffic volume and psychological state and changing the controlled signal when predetermined conditions are met, a device for confirming the changed signal state, a device for displaying the changed signal state and the reason for the change on an information display device, and a device for recording the changed signal state and the reason for the change. This allows for intelligent judgment and adaptive signal adjustment based on actual traffic flow and the emotional state of road users, improving traffic efficiency while enhancing the human experience and road safety of traffic participants, and achieving efficient response to complex traffic situations.

[0193] "Acquisition device" refers to equipment that can acquire traffic location data in real time, including but not limited to cameras, sensors, etc.

[0194] "Information" refers to the data content collected by the acquisition device, including images, audio, behavioral data, emotional data, etc.

[0195] "Analysis" refers to the process of processing and analyzing acquired information to extract the required data such as traffic objects or psychological states.

[0196] "Mobile objects" refer to objects that can move in traffic locations, including vehicles, pedestrians, bicycles, etc.

[0197] "Mobile users" refers to people who operate or ride in mobile vehicles, including drivers, passengers, pedestrians, etc.

[0198] "Mental state" refers to the emotional state inferred by detecting the facial expressions or behaviors of mobile users, such as anxiety, anger, calmness, etc.

[0199] "Traffic volume" refers to the number of moving bodies and their users passing through a traffic location within a set time window.

[0200] "Signal" refers to the information status of control devices used to indicate whether traffic is passing or stopping, including traffic lights.

[0201] "Controlled signal" refers to a signal device that is controlled and modified by this system, including traffic lights, etc.

[0202] "Reason for change" refers to the basis for changing the traffic signal, including judgment conditions such as traffic volume and psychological state.

[0203] "Information notification device" refers to a terminal device used to display the signal status and reasons for changes to mobile users in real time, such as smart glasses and displays.

[0204] "Records" refer to the process and data of storing information such as the status and reasons for changes in traffic signals in a storage medium.

[0205] "Output prompt statements" refer to input instructions or text fragments used to guide information processing models in generating content.

[0206] This invention, based on artificial intelligence and various sensing technologies, enables intelligent adaptive control of traffic signals, taking into account both traffic flow and the psychological state of road users, effectively improving road safety and traffic efficiency. The following provides a detailed description of the system's overall structure, usage, and operation.

[0207] This system comprises several main components, including a server, acquisition devices, emotion recognition devices, signal control devices, and information display terminals. The server can utilize a general-purpose computing platform, such as an industrial control server, or it can consist of multiple distributed computing nodes. The acquisition devices are monitoring equipment such as cameras installed at intersections to collect images or video streams in real time. The emotion recognition devices may include facial recognition cameras integrated into vehicles or road sections, voice analysis modules, etc., to acquire facial expressions or behavioral characteristics of road users. The signal control devices can be standard traffic light electronic control units. The information display terminals can be display devices such as smart glasses or in-vehicle multimedia screens.

[0208] In the system, the server acquires real-time video streams and emotion recognition sensor data from cameras via communication protocols (such as Ethernet or Wi-Fi), and temporarily stores this raw data in a database (such as MySQL or MongoDB) or a cache module (such as Redis). The server uses video analysis software (such as OpenCV libraries and YOLOv5 deep learning models) to perform real-time object detection and tracking on the acquired video streams, automatically identifying moving objects such as vehicles and pedestrians in the lanes and recording their number and trajectories. Simultaneously, the server calls an emotion recognition engine (such as one based on TensorFlow or other deep learning models) to analyze user facial expressions and actions, accurately inferring the user's psychological state, such as anxiety, anger, or calmness.

[0209] After completing a dual analysis of road traffic volume and user psychological state, the server makes a judgment based on preset rules (e.g., changing the traffic light in advance when traffic volume is low and user anxiety is detected). If a traffic light change is needed, the server sends a light-changing command to the signal control device via standard protocols (e.g., TCP / IP, MODBUS). Upon receiving the command, the signal control device controls the corresponding traffic light to switch between red, green, and yellow signals and promptly feeds back the status to the server.

[0210] The server stores all traffic light change records (including change time, status, and reason) in a database and pushes the changed signal status and reason to the in-vehicle information terminal via wireless communication methods such as Bluetooth and Wi-Fi. After receiving the push information, the terminal displays the detailed signal status (such as "Signal has turned green ahead of schedule, reason: user anxiety detected") in text or icon form on smart glasses or multimedia screens, allowing users to know the decision-making logic of the traffic system and the current traffic light status in real time, thereby improving safety and traffic efficiency.

[0211] For example, at an intersection, a camera detects few vehicles and pedestrians crossing the road, while an emotion recognition device identifies a driver as anxious. Based on this data, the server determines it's appropriate to change the traffic light from red to green ahead of schedule, then notifies the signal control device to complete the color change and alerts the driver on their smart glasses to the reason for the change. All actions are fully recorded in a log table for easy tracing and analysis later.

[0212] Generative artificial intelligence models can also be integrated into this system to automatically generate or optimize prompts, providing users, administrators, or system maintenance personnel with a more user-friendly and intelligent interaction. For example, the following prompts can be used:

[0213] "How can traffic lights be automatically adjusted when an autonomous vehicle passes through an intersection if the user is in a state of anxiety?"

[0214] "Please describe the entire process of how the server controls traffic lights based on traffic flow and user emotion recognition results."

[0215] How can the terminal display signal status changes from the server and their reasons?

[0216] Through the above structure and operation mode, the dynamic linkage response between traffic signals and the psychological state of road users can be effectively realized, promoting the safe and efficient operation of the intelligent transportation system.

[0217] use Figure 14 The processing flow is explained.

[0218] Step 1:

[0219] The server acquires real-time data from cameras and emotion recognition sensors installed at the intersection. The input consists of a video stream from the cameras and emotion data from the emotion recognition sensors. The server decodes the video stream, temporarily stores each frame in a buffer, and synchronously writes the emotion sensor data to a database as structured raw data. The output consists of raw data files containing both frame image data and emotion data.

[0220] Step 2:

[0221] The server uses image processing software (such as OpenCV) and deep learning models (such as YOLOv5) to perform object detection on the captured video stream. The input is the frame image data from step 1. The server identifies vehicles and pedestrians in each frame and labels their coordinates (x, y), category (such as car, person), and speed, outputting a structured list of detection results.

[0222] Step 3:

[0223] The server uses an emotion recognition engine to analyze the raw emotion data. The input is the emotion data obtained in step 1. Data analysis includes facial expression feature extraction and behavior analysis. A specialized algorithm is used to determine the user's psychological state (such as anxiety, anger, etc.), and the output is a specified emotion label and confidence score for the user.

[0224] Step 4:

[0225] The server combines object detection results with sentiment analysis results to determine the current traffic volume and user psychological state. The inputs are the detection result list output from step 2 and the sentiment labels output from step 3. The server determines whether the logical conditions for signal change are met by counting vehicles and pedestrians and analyzing sentiment thresholds. The output is the decision result (e.g., "Signal can be changed" or "Maintain the status quo").

[0226] Step 5:

[0227] When the server determines that a signal change is needed, it sends a light-changing command to the signal control device via TCP / IP or MODBUS protocol. The input is the determination result from step 4. The server generates a command data packet, which includes the target of the change and the reason. After receiving the command, the signal control device performs a physical light-changing operation. The output is feedback information indicating that the light-changing action has been completed.

[0228] Step 6:

[0229] The server receives feedback from the signal control device and confirms that the signal change was successful. The input is the status feedback information from the signal control device. The server compares the actual signal status; if the change has been successful, it proceeds to the next step; otherwise, it records the error and retryes. The output is the final signal status (e.g., "turned green").

[0230] Step 7:

[0231] The server writes the signal status change and its reason to the log database and pushes the change information to the terminal via a wireless communication module (such as Bluetooth or Wi-Fi). The inputs are the final signal status from step 6 and the reason for the decision from step 4. Data processing includes generating a push message with a timestamp and reason. The output is the information displayed on the terminal and the database log entries.

[0232] Step 8:

[0233] After receiving a signal change message from the server, the terminal (e.g., smart glasses) displays the current signal status and the reason for the change on its screen in real time. The input is the information content pushed by the server. After parsing the message, the terminal performs information display operations, such as displaying "The signal has turned green ahead of schedule, reason: user anxiety detected." The output is the visual information perceived by the user.

[0234] Step 9:

[0235] Users see the prompts on the terminal and, based on the actual road conditions, take appropriate driving or pedestrian actions. The input is the signal status and reason for the change displayed on the terminal. Users make decisions such as proceeding or waiting based on the prompts, improving traffic efficiency and safety. The output is the user's actual behavioral response.

[0236] The specific processing unit 290 sends the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires sound representing user input regarding the result of the specific processing. The control unit 46A sends the sound data representing user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0237] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0238] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart device 14 or external devices, and the smart device 14 acquires or collects information required for processing from the data processing device 12 or external devices.

[0239] For example, the collection unit is implemented by the control unit 46A of the smart device 14 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart device 14 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the output device 40 of the smart device 14 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0240] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart device 14.

[0241] Second Implementation Method

[0242] Figure 3 An example of the configuration of the data processing system 210 according to the second embodiment is shown.

[0243] like Figure 3 As shown, the data processing system 210 includes a data processing device 12 and smart glasses 214. A server can be cited as an example of the data processing device 12.

[0244] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0245] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, and communication I / F 44 are also connected to the bus 52.

[0246] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0247] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to capture images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0248] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0249] Figure 4 This illustrates an example of the main functions of the data processing device 12 and the smart glasses 214. For example... Figure 4 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0250] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0251] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0252] In the smart glasses 214, the processor 46 performs reception and output processing. The memory 50 stores the reception and output program 60. The processor 46 reads the reception and output program 60 from the memory 50 and executes the read reception and output program 60 on the RAM 48. The reception and output processing is implemented by the processor 46 operating as a control unit 46A according to the reception and output program 60 executed on the RAM 48. Furthermore, the smart glasses 214 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290.

[0253] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart glasses 214. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0254] Example 1

[0255] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0256] Application Example 1

[0257] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0258] Example 2

[0259] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0260] Application Example 2

[0261] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0262] The specific processing unit 290 sends the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A outputs the result of the specific processing to the speaker 240. The microphone 238 acquires sound input representing the user's input regarding the result of the specific processing. The control unit 46A sends the sound data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0263] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0264] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or external devices, and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or external devices.

[0265] For example, the collection unit is implemented by the control unit 46A of the smart glasses 214 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart glasses 214 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the smart glasses 214 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0266] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart glasses 214.

[0267] Third Implementation Method

[0268] Figure 5 An example of the configuration of the data processing system 310 according to the third embodiment is shown.

[0269] like Figure 5 As shown, the data processing system 310 includes a data processing device 12 and a head-mounted terminal 314. A server can be cited as an example of the data processing device 12.

[0270] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0271] The head-mounted terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, display 343, and communication I / F 44 are also connected to the bus 52.

[0272] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0273] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to capture images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0274] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0275] Figure 6 This illustrates an example of the main functions of the data processing device 12 and the head-mounted terminal 314. For example... Figure 6 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0276] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0277] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0278] In the head-mounted terminal 314, the processor 46 performs the acceptance / output processing. The memory 50 stores the acceptance / output program 60. The processor 46 reads the acceptance / output program 60 from the memory 50 and executes the read acceptance / output program 60 on the RAM 48. The acceptance / output processing is implemented by the processor 46 operating as a control unit 46A according to the acceptance / output program 60 executed on the RAM 48.

[0279] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the head-mounted terminal 314. In the following description, the data processing device 12 will be referred to as the "server" and the head-mounted terminal 314 will be referred to as the "terminal".

[0280] Example 1

[0281] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0282] Application Example 1

[0283] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0284] Example 2

[0285] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0286] Application Example 2

[0287] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0288] The specific processing unit 290 sends the result of the specific processing to the head-mounted terminal 314. In the head-mounted terminal 314, the control unit 46A outputs the result of the specific processing to the speaker 240 and the display 343. The microphone 238 acquires sound input representing the user's input regarding the result of the specific processing. The control unit 46A sends the sound data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0289] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 includes prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0290] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the head-mounted terminal 314, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the head-mounted terminal 314. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the head-mounted terminal 314 or external devices, and the head-mounted terminal 314 acquires or collects information required for processing from the data processing device 12 or external devices.

[0291] For example, the collection unit is implemented by the control unit 46A of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the head-mounted terminal 314 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 to analyze the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12 to generate a menu using a generation AI. For example, the serving unit is implemented by the speaker 240 and display 343 of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12 to provide the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0292] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the head-mounted terminal 314.

[0293] Fourth Implementation Method

[0294] Figure 7 An example of the configuration of the data processing system 410 according to the fourth embodiment is shown.

[0295] like Figure 7 As shown, the data processing system 410 includes a data processing device 12 and a robot 414. A server can be cited as an example of the data processing device 12.

[0296] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0297] Robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, controlled object 443, and communication I / F 44 are also connected to the bus 52.

[0298] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0299] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to photograph the area around robot 414 (e.g., the field of view defined by a perspective equivalent to the field of vision of an average healthy person).

[0300] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0301] The controlled object 443 includes a display device, LEDs (light-emitting diodes) for the eyes, and motors for driving the arms, hands, and feet. The posture or movement of the robot 414 is controlled by controlling the motors in the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. In addition, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0302] Figure 8 This illustrates an example of the main functions of the data processing device 12 and the robot 414. For example... Figure 8 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0303] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0304] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0305] In robot 414, the processor 46 performs the acceptance and output processing. The memory 50 stores the acceptance and output program 60. The processor 46 reads the acceptance and output program 60 from the memory 50 and executes the read acceptance and output program 60 on RAM 48. The acceptance and output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance and output program 60 executed on RAM 48.

[0306] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the robot 414. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 will be referred to as the "terminal".

[0307] Example 1

[0308] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0309] Application Example 1

[0310] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0311] Example 2

[0312] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0313] Application Example 2

[0314] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0315] The specific processing unit 290 sends the result of the specific processing to the robot 414. In the robot 414, the control unit 46A outputs the result of the specific processing to the speaker 240 and the controlled object 443. The microphone 238 acquires sound input representing the result of the specific processing. The control unit 46A sends the sound data representing the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0316] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0317] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or external devices, and the robot 414 acquires or collects information required for processing from the data processing device 12 or external devices.

[0318] For example, the collection unit is implemented by the control unit 46A of the robot 414 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the robot 414 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the robot 414 and the control object 443 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0319] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the robot 414.

[0320] Furthermore, the emotion-specific model 59, acting as an emotion engine, can determine a user's emotion based on a specific mapping. Specifically, the emotion-specific model 59 can determine a user's emotion based on an emotion graph that serves as a specific mapping (see [reference]). Figure 9 The system determines the user's emotions. Furthermore, the emotion-specific model 59 can similarly determine the robot's emotions, and the specific processing unit 290 performs specific processing based on the robot's emotions.

[0321] Figure 9 This is a diagram representing an emotion map 400 that maps multiple emotions. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotion is. On the outer side of the concentric circles, emotions representing states or behaviors arising from mood are arranged. Emotions are concepts that include feelings and mental states. Emotions generated by reactions occurring in the brain are arranged roughly to the left of the concentric circles. Emotions derived from situational judgments are arranged roughly to the right of the concentric circles. Emotions generated by reactions occurring in the brain and derived from situational judgments are arranged roughly above and below the concentric circles. Furthermore, "pleasant" emotions are arranged above the concentric circles, and "unpleasant" emotions are arranged below them. Thus, in the emotion map 400, multiple emotions are mapped based on the structure that generates emotions, and emotions that are likely to occur simultaneously are mapped close to each other.

[0322] These emotions are distributed at the three o'clock position of the emotion map 400, typically fluctuating between peace and anxiety. In the right half of the emotion map 400, situational awareness dominates over internal sensation, thus resulting in an impression of calm.

[0323] The inner side of the emotion map 400 represents the inner state, while the outer side represents behavior. Therefore, the further outward you are from the emotion map 400, the more visible the emotion becomes (manifested in behavior).

[0324] Here, human emotions are based on various balances such as posture and blood sugar levels. When these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotions in robots, cars, motorcycles, etc., can also be created in the following way: based on various balances such as posture and remaining battery power, when these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotion maps can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a Brain Physiological Signal Analysis System for Voice Emotion Recognition and Emotion, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). In the left half of the emotion map, emotions belonging to the sensory-dominated region, called "response," are arranged. Furthermore, in the right half of the emotion map, emotions belonging to the situational cognition-dominated region, called "situation," are arranged.

[0325] In the emotion map, two types of emotions that promote learning are defined. One is a negative emotion on the situational side, in the middle or peripheral region of "repentance" or "reflection." This occurs when the robot experiences negative emotions such as "I don't want to experience this feeling again" or "I don't want to be blamed again." The other is a positive emotion on the response side, near the "desire" region. This occurs when there are positive feelings such as "wanting more" or "wanting to know more."

[0326] The emotion-specific model 59 inputs user input into a pre-trained neural network to obtain emotion values ​​representing each emotion shown in the emotion map 400, thereby determining the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network... Figure 10 As shown in the sentiment graph 900, it was trained in a way that sentiments that are configured close to each other have similar values. Figure 10 The text shows examples of emotions such as "peace of mind", "stability", and "reassurance" that have similar emotion values.

[0327] The above description focuses on the functions of the data processing device 12, but the system of this disclosure is not necessarily installed on a server. The system of this disclosure can also be installed as a general information processing system. This disclosure can also be installed, for example, as a software program running on a personal computer, an application running on a smartphone, etc. The method of this disclosure can also be provided to users in the form of SaaS (Software as a Service).

[0328] In the above embodiments, an example of a specific process being performed by a single computer 22 is given. However, the technology disclosed herein is not limited to this, and the specific process can also be distributed among multiple computers, including computer 22. For example, the data generation model 58 can be located on an external device of the data processing apparatus 12, where data is generated based on the input data.

[0329] In the above embodiments, examples of storing a specific processing program 56 in the memory 32 have been described, but the technology disclosed herein is not limited thereto. For example, the specific processing program 56 may also be stored in a portable computer-readable non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed into the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0330] Alternatively, a specific processing program 56 may be pre-stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 according to the requirements of the data processing device 12.

[0331] In addition, it is not necessary to store all the specific processing program 56 in the storage device such as the server connected to the data processing device 12 via the network 54 or in the memory 32; a portion of the specific processing program 56 may be stored in advance.

[0332] As hardware resources for performing specific processes, various processors, as shown below, can be used. For example, a CPU can be listed as a processor, which functions as a general-purpose processor that performs specific processes by executing software, i.e., a program. Furthermore, processors can be listed as special-purpose circuits such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application-Specific Integrated Circuits), which are processors with circuitry specifically designed to perform specific processes. Each processor has built-in or connected memory, and each processor executes specific processes using that memory.

[0333] The hardware resources for performing a specific process can consist of one of these various processors, or a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resources for performing a specific process can be a single processor.

[0334] As an example of a single processor, there are two approaches: First, a processor is composed of a combination of one or more CPUs and software, which functions as a hardware resource to perform a specific process; second, as represented by a SoC (System-on-a-chip), a processor is used to implement the functionality of the entire system, which includes multiple hardware resources for performing a specific process, using a single IC (Integrated Circuit) chip. In this way, the specific process is implemented by using one or more of the aforementioned processors as hardware resources.

[0335] Furthermore, the hardware architecture of these various processors, more specifically, can utilize circuits that combine semiconductor elements and other circuit components. Moreover, the specific process described above is just one example. Therefore, without departing from the main point, unnecessary steps can certainly be deleted, new steps added, or the processing order changed.

[0336] The descriptions and illustrations above are detailed explanations of a portion of the technology disclosed herein, and are merely one example of the technology disclosed herein. For example, the above descriptions of the structure, function, effect, and results are just one example of the structure, function, effect, and results of a portion of the technology disclosed herein. Therefore, without departing from the spirit of the technology disclosed herein, unnecessary parts may be deleted, new elements added, or replacements may be made to the descriptions and illustrations above. Furthermore, to avoid confusion and facilitate understanding of a portion of the technology disclosed herein, explanations of common technical knowledge that do not require special explanation under the premise of being able to implement the technology disclosed herein have been omitted from the descriptions and illustrations above.

[0337] All documents, patent applications and technical specifications set forth in this specification are incorporated herein by reference to the same extent that each document, patent application and technical specification is specifically and individually described therein and referenced by reference.

[0338] In addition, the following notes are provided in response to the above explanation.

[0339] Example 1

[0340] (Note 1)

[0341] An information processing system includes: a device for acquiring real-time image information of the area surrounding an intersection using an image information acquisition device; a device for temporarily storing the image information in a temporary storage device by the information processing device; a device for applying image recognition algorithms and machine learning models to the image information to detect moving objects by the information processing device; a device for recording the location information, direction of movement, and speed information of the detected moving objects in structured data form; a device for determining the traffic volume within a predetermined time unit based on the detection results and comparing the determination results with a set threshold; a device for sending a signal status change command to a traffic control device via an information communication device when the determined traffic volume is below the threshold; a device for re-acquiring the signal status of the traffic control device via an information communication device and confirming the content after the signal status change; and a device for recording the series of processing results and the reasons for the determination in an information recording and management system.

[0342] (Note 2)

[0343] The information processing system according to Appendix 1 further includes a processing device for acquiring signal status.

[0344] (Note 3)

[0345] According to the information processing system described in Appendix 1, the information recording and management system includes a processing device for recording the content of signal status changes and the reasons for the judgment.

[0346] Application Example 1

[0347] (Note 1)

[0348] An information processing system includes: a device for acquiring dynamic status data and user attribute data in real time from multiple information acquisition devices; a device for processing the dynamic status data through a data processing device to identify target objects and moving bodies; a device for calculating the amount of movement in a spatial region based on the identification results; a device for performing a threshold determination based on the user attribute data and the calculation results; a device for generating a control signal based on the threshold determination result and user attribute information and sending it to a control device; a device for acquiring the current state of the controlled object and confirming whether the control signal has been executed; a device for recording the execution of the control signal and the parameters of the determination process to a status recording device; a device for providing the control signal or status information to an external moving body or terminal device; and a device for enabling the external moving body or terminal device to optimize its movement path based on the information.

[0349] (Note 2)

[0350] The information processing system according to Appendix 1 further includes means for acquiring attribute data from the emotion recognition device and taking it into consideration when determining the threshold.

[0351] (Note 3)

[0352] According to the information processing system described in Appendix 1, the acquisition, identification, judgment, control, recording and distribution of the information are all performed automatically and periodically.

[0353] Example 2

[0354] (Note 1)

[0355] An information processing system includes: a device for acquiring various environmental information from an information acquisition device based on time information; a device for processing the acquired environmental information through an analysis device and identifying target objects using an identification device to extract their attribute information and location information; a device for analyzing the attribute information and action information of target individuals in the acquired environmental information to determine their psychological state information; a device for calculating traffic flow evaluation values ​​and psychological state evaluation values ​​based on the extracted target object attribute information, location information, and determined psychological state information; a device for outputting a signal to a signal control device to display change information when the calculated traffic flow evaluation value is lower than a set threshold or the psychological state evaluation value is in a specific state; and a device for comparing and confirming the changed signal display state with the acquired information.

[0356] (Note 2)

[0357] The information processing system according to Appendix 1 further includes a device for obtaining the current signal display status from the signal display device.

[0358] (Note 3)

[0359] The information processing system according to Appendix 1 further includes a device for saving information related to the change of signal display status, the cause, and the associated time information to an information recording device.

[0360] Application Example 2

[0361] (Note 1)

[0362] An information processing system includes: a device for acquiring information in real time from an acquisition device; a device for parsing the acquired information to detect mobile bodies and mobile body users; a device for inferring the psychological state of mobile body users based on the detection results; a device for determining the traffic volume of mobile bodies and mobile body users and the psychological state of mobile body users, and changing a controlled signal when predetermined conditions are met; a device for confirming the changed signal state; a device for displaying the changed signal state and the reason for the change on an information prompting device; and a device for recording the changed signal state and the reason for the change.

[0363] (Note 2)

[0364] The information processing system according to Appendix 1 further includes a device for acquiring the current status of the signal.

[0365] (Note 3)

[0366] The information processing system according to Appendix 1 further includes means for generating output prompt statements for the generative information processing system.

Claims

1. An information processing system, characterized in that, include: A device for acquiring real-time video of intersections; A device used to analyze acquired video and detect passing vehicles and pedestrians; A device for determining traffic volume based on detection results; Device for changing traffic lights when the traffic volume is determined to be below a predetermined threshold; A device used to confirm the status of a changed signal.

2. The information processing system according to claim 1, characterized in that, It further includes means for obtaining the current state of the traffic light.

3. The information processing system according to claim 1, characterized in that, It further includes means for recording the changed signal status in a log.

Images