system

Abstract

Provide a system. 【Solution means】 Data collection means for collecting data from a plurality of moving bodies, Analysis means for analyzing the situation of the moving body based on the data, Decision-making means for realizing cooperation between moving bodies based on the analysis result, Instruction distribution means for distributing the instructions generated by the decision-making means to each moving body, Execution means for each moving body to execute operations based on the instructions, Feedback means for feeding back the operation status and improving the operation, Information providing means for a user to receive the movement and safe route information of the moving body through a visual device, Warning generation means for generating warnings regarding the traveling direction and safety of the moving body, A system including the above.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] It is important to ensure the safety when an autonomous vehicle and an automated guided vehicle share a public road. In particular, in a situation where vehicles with different manufacturers and technical specifications are mixed, the risk of accidents and collisions may increase. Furthermore, in order for these vehicles to coexist safely with road users, especially pedestrians and human-powered vehicles, a mechanism is required to analyze the traffic situation in real time and generate and distribute appropriate instructions instantaneously. Against this background, there is a problem of providing an operation management system that can simultaneously achieve mutual cooperation and safety assurance, which could not be achieved by conventional systems.

Means for Solving the Problems

[0005] The present invention solves the aforementioned problems with a system comprising: data collection means for collecting data from multiple vehicles; analysis means for analyzing traffic conditions based on the collected data; decision-making means for realizing cooperative operation between vehicles based on the analysis results; instruction distribution means for distributing instructions based on the decision; execution means for each vehicle to perform an action based on the instructions; and feedback means for providing feedback on the operating status and improving operations. Furthermore, by controlling traffic signals based on priority using the instruction distribution means and providing an anomaly detection function that responds quickly to unexpected situations using the feedback means, safe vehicle operation on public roads is realized.

[0006] A "data collection method" is a means of acquiring real-time driving data such as location information and speed from multiple vehicles.

[0007] "Analysis methods" refer to means of analyzing traffic conditions and distances between vehicles based on collected data, in order to identify safety-related problems.

[0008] A "decision-making tool" is a means of determining vehicle priorities and routes based on the results obtained from analysis tools, and instructing appropriate actions.

[0009] The "instruction distribution means" is a means for distributing instructions generated by the decision-making means to each vehicle, thereby promoting the autonomous operation of the vehicles.

[0010] "Execution means" refers to the means by which each vehicle performs actions, including actual driving and stopping, in accordance with the instructions it receives.

[0011] A "feedback mechanism" is a means of monitoring the operating conditions and taking improvement measures to enhance the overall system performance based on the data obtained. [Brief explanation of the drawing]

[0012] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0013] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0016] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0017] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0018] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), and the like.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0020] [First Embodiment]

[0021] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0022] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0025] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0028] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0032] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0033] The present invention provides an operation management system that enables mutual cooperation between multiple vehicles using AI. This system mainly consists of data collection means, analysis means, decision-making means, instruction distribution means, execution means, and feedback means. This provides an environment in which autonomous vehicles and automated transport robots can safely coexist on public roads.

[0034] In this system, the server first collects driving data from each vehicle through sensors and communication devices. This includes the vehicle's position, speed, acceleration, and information about surrounding obstacles and other vehicles. Next, the server analyzes the collected data in real time on terminals to assess traffic conditions and potential risks.

[0035] Based on the analysis results, the server determines the priority order between vehicles and makes decisions regarding their routes and speed adjustments. The instructions generated by the server are then sent to each terminal, which distribute them to the vehicles. For example, at an intersection, the server determines the order in which each vehicle should pass and issues instructions to ensure safe passage based on that order.

[0036] The terminal controls the vehicle's operation based on received instructions. This control includes adjusting the vehicle's direction, speed, and stopping position. Furthermore, operational data is fed back to the server, and any abnormalities detected are promptly addressed. The feedback data is also used to improve the system's performance for future operations.

[0037] As a concrete example, when an autonomous bus travels through a congested urban area, it can coordinate with other automated transport robots and traffic signals to safely reach its destination along a designated route. This ensures the safety of all road users and creates a smooth flow of traffic.

[0038] Thus, the present invention provides an environment in which a variety of autonomous vehicles can operate safely on public roads, and in particular ensures traffic safety in situations where vehicles with different specifications and from different manufacturers are mixed together.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] The server collects data in real time from each vehicle traveling on public roads. This data includes the vehicle's location, speed, and information about surrounding obstacles. This data collection is performed through sensors and communication devices mounted on the vehicles.

[0042] Step 2:

[0043] The terminal uses data received from the server to analyze traffic conditions and assess the distance and speed between vehicles, as well as the potential collision risk at intersections. AI algorithms are used to improve the accuracy of the analysis results.

[0044] Step 3:

[0045] The server makes decisions based on the analysis results. This includes determining vehicle priorities and planning optimal routes and speed adjustments. In complex situations such as intersections, it determines priority and generates appropriate instructions.

[0046] Step 4:

[0047] The server generates specific instructions for each vehicle based on the decisions made and distributes them to the terminals. These instructions may include instructions to adjust speed, change direction, or stop temporarily.

[0048] Step 5:

[0049] The terminal relays the received instructions to the vehicle's control system, ensuring appropriate operation. Based on these instructions, the vehicle autonomously adjusts its course and speed, driving in harmony with other vehicles and road users.

[0050] Step 6:

[0051] The terminal monitors data during vehicle operation and reports any abnormalities or unexpected events to the server in real time. The server then takes appropriate action based on this information.

[0052] Step 7:

[0053] The server accumulates operational data, evaluates the overall system performance, and uses this data for future improvements. Through this feedback process, the system learns and improves its performance.

[0054] (Example 1)

[0055] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0056] In modern transportation systems involving mobile vehicles, there is a demand for improved safety and efficient traffic flow. However, in situations where vehicles with different specifications and from different manufacturers coexist, technical challenges exist in achieving harmony between them. Furthermore, there is a need for methods to quickly respond to unpredictable situations during operation and provide optimal transportation strategies in real time.

[0057] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0058] In this invention, the server includes information gathering means for collecting information from multiple moving objects, analysis means for analyzing the movement situation based on the information, and means for proposing a new operation strategy using a generative AI model. This improves the harmony between different moving objects and enables safe and efficient traffic management while immediately adapting to the situation during operation.

[0059] A "mobile object" is a means of transport designed for public or private use, and includes vehicles, machinery, and other similar devices.

[0060] "Information gathering means" refers to a device or system for receiving and accumulating various types of data from a moving object.

[0061] "Analysis means" refers to devices or software used to process collected data and evaluate movement patterns and traffic conditions.

[0062] A "decision-making mechanism" is a device or process for formulating a movement strategy based on the analysis results and determining the action plan for each moving object.

[0063] "Instruction and distribution means" refers to a device or communication method for accurately transmitting a decided action plan to each mobile entity.

[0064] "Implementation means" refers to a device or program for actually controlling the movement of a mobile object in accordance with the instructions that have been delivered.

[0065] "Adjustment means" refers to a device or system for feeding back operational information and optimizing the operation of a moving object.

[0066] A "generative AI model" is an algorithm or model that uses machine learning techniques to analyze data and generate new operational strategies or solutions.

[0067] An "anomaly detection method" is a process or system for quickly identifying and responding to unexpected situations or problems during operation.

[0068] This invention is an operation management system that enables mutual cooperation between multiple mobile entities by utilizing AI technology. This system includes means for information gathering, analysis, decision-making, instruction distribution, implementation, coordination, and operation strategy proposal using a generated AI model.

[0069] In implementing the system, the server first collects information in real time from each moving object through hardware such as GPS sensors and LiDAR. This allows the server to obtain information on the object's position, speed, acceleration, and surrounding obstacles. Next, the server uses powerful analytical software to analyze the collected information. This includes AI algorithms for modeling traffic conditions and assessing potential risks.

[0070] Based on the analysis results, the terminal determines the priority between each moving object and the optimal route. During the decision-making process, the generating AI model proposes a new operational strategy through prompt messages. For example, in a scenario where an electric bus in an urban area automatically avoids congestion and reaches its destination along a designated route, an example of a prompt message might be, "How can an autonomous bus operate smoothly in a congested urban area?"

[0071] The user plays a role in feeding back information obtained during operation to the server, and the system uses this feedback to improve its performance. This allows the system to be continuously optimized, enabling safe and efficient operation management even in situations where mobile vehicles with different specifications and manufacturers are mixed.

[0072] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0073] Step 1:

[0074] The server's role is to collect information from each mobile object in real time. Specifically, the server uses GPS sensors and LiDAR to acquire location, velocity, acceleration, and information about surrounding obstacles. The input is sensor data transmitted from the mobile object, which is received and stored in the server as integrated information. The output is a complete dataset for analysis.

[0075] Step 2:

[0076] The server analyzes the collected data. Using analysis software, it models traffic conditions and performs risk assessments. Specifically, it utilizes AI algorithms to predict potential hazards and congestion based on sensor data. The input is the dataset obtained in step 1, and the output is the evaluation results and risk assessment report regarding traffic conditions.

[0077] Step 3:

[0078] The terminal makes decisions based on the analysis results sent from the server. Specifically, it processes the prioritization of mobile objects and the optimal travel route. At this stage, the generating AI model may propose an operation strategy based on the prompt message. The input is the analysis results from step 2 and the prompt message from the generating AI model. The output is specific travel instructions for each mobile object.

[0079] Step 4:

[0080] The server distributes the determined instructions to each terminal. Specifically, it transmits instruction information to the mobile device using a communication protocol and accurately conveys its contents. The input is the instruction content determined in step 3. The output is the action plan transmitted to the terminal.

[0081] Step 5:

[0082] The terminal controls the mobile object based on the instructions it receives. Specifically, it sends signals to the vehicle's control system and performs appropriate operations. The input is the instructions received in step 4. The output is the specific operation performed by the mobile object, such as adjustments to direction or speed.

[0083] Step 6:

[0084] The user feeds data generated during operation back to the server. Specifically, this involves reporting abnormal situations to the server and prompting system adjustments. Inputs are operational data and abnormal information obtained during operation. Outputs are system improvement suggestions for the next operation.

[0085] (Application Example 1)

[0086] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0087] With the spread of autonomous driving technology, it is necessary to properly manage traffic situations between autonomous vehicles and pedestrians to enhance safety. However, conventional technologies lack the means to provide users with detailed real-time traffic information, and warnings for unexpected situations and sudden lane changes are particularly insufficient. This creates a challenge in that safety around autonomous vehicles may be compromised.

[0088] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0089] In this invention, the server includes data collection means for collecting data from multiple moving objects, analysis means for analyzing the status of the moving objects, and information provision means for users to receive information on the movement of the moving objects and safe route information through visual devices. This makes it possible to notify users of the movement of moving objects in real time and to warn them in advance of unexpected situations.

[0090] "Mobile devices" refer to vehicles and equipment equipped with autonomous driving technology that operate in a variety of environments.

[0091] "Data collection means" refers to devices and methods for acquiring driving conditions and surrounding information from a moving object.

[0092] "Analysis means" refers to methods and functions for evaluating the status of a moving object based on collected data and making necessary adjustments or decisions.

[0093] "Decision-making tools" refer to algorithms and systems used to determine priorities and routes between moving objects based on analysis results.

[0094] "Instruction distribution means" refers to a system or technology that transmits determined instructions to each mobile unit to prompt them to act.

[0095] "Execution means" refers to functions and technologies for controlling a moving object based on the instructions given.

[0096] "Feedback mechanisms" refer to methods and functions for continuously recording operational status and using that information to improve the system.

[0097] "Information provision means" refers to a system that notifies users in real time of the movement of a moving object and information on safe routes.

[0098] "Warning generation means" refers to a function that issues a warning to the user in situations where danger is anticipated.

[0099] The system for implementing this invention primarily operates through cooperation between three parties: a server, a terminal, and a user.

[0100] First, the server collects data from multiple mobile vehicles equipped with autonomous driving technology. The hardware used includes sensors and communication devices mounted on the vehicles. The data includes the location, speed, direction of travel, and information about the surrounding environment of the mobile vehicles. The server processes this data in real time and is responsible for analyzing the behavior of the mobile vehicles.

[0101] After analysis, the server determines priorities among the moving objects and sets appropriate routes and speeds. This decision-making process utilizes AI-powered algorithms. The analysis results are transmitted to terminals via a command distribution system.

[0102] The terminal receives these instructions and specifically controls the movement of each mobile object. Furthermore, it provides information about safe routes to the user's visual devices, such as smartphones or smart glasses. In doing so, it also issues warnings via voice or vibration as needed.

[0103] A feedback function is also important. The operating status of the mobile devices is sent to the server in real time, and if an anomaly is detected, immediate action is taken. The collected data is used to improve the overall performance of the system and prevent malfunctions.

[0104] As a concrete example, let's assume a user is walking through a crowded urban area. In this case, the smart glasses visually display the movement of the autonomous vehicle ahead and guide the user to the sidewalk where they can safely avoid it. Furthermore, if the direction of the moving object suddenly changes, an immediate voice warning is issued.

[0105] Examples of prompts for the generated AI model include: "Design an app that provides real-time safe route information to nearby pedestrians based on traffic data from autonomous vehicles," and "How can a smart device track an autonomous vehicle safely operating in an urban area during rainy weather?" From these prompts, the AI ​​model generates a solution appropriate to the specific situation the user faces.

[0106] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0107] Step 1:

[0108] The server collects data from multiple moving objects. Inputs include position, speed, direction of travel, and surrounding environment information from sensors mounted on these objects. The server receives and centrally stores this data. This process forms a base data set for detailed real-time traffic conditions.

[0109] Step 2:

[0110] The server performs analysis based on the collected data. The input is the traffic data obtained in Step 1. The server uses an AI algorithm to predict the trajectory of moving objects and assess collision risk, determining the priority of each object and the optimal route. This process generates information that enables rational decision-making.

[0111] Step 3:

[0112] The server generates instructions based on the analysis results and distributes them to the terminals. The input is the priority and route information of the mobile units determined in step 2. The server transmits these instructions to the control systems of each mobile unit through the instruction distribution system. This procedure provides specific guidance for the safe and efficient operation of the mobile units.

[0113] Step 4:

[0114] The terminal receives instructions from the server and actually controls the mobile object. The input is the instruction sent in step 3. The terminal adjusts the movement of the mobile object, changing the direction of travel, speed, and stopping position as needed. This control ensures safety and efficiency during actual operation.

[0115] Step 5:

[0116] The terminal provides information to the user's visual device. The input is the instruction information obtained in step 3. The terminal displays the movement of surrounding objects and safe route information on the user's smart device, and also issues warnings via voice and vibration as needed. This operation enables the provision of real-time safety information to the user.

[0117] Step 6:

[0118] The terminal provides feedback on the operational status at the site and transmits it to the server. The input is the latest operational data from the sensors of the moving vehicle. The terminal analyzes the data during operation and immediately reports any anomalies to the server. This feedback helps to improve the entire system and prevent accidents.

[0119] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0120] This invention is a cooperative operation management system that combines an emotion engine, and by linking the emotional states of multiple vehicles and users, it realizes more comfortable and safer travel. The system has a configuration that includes data collection means, analysis means, decision-making means, instruction distribution means, execution means, and feedback means, as well as an emotion engine.

[0121] (Data collection)

[0122] First, the server collects driving data from multiple vehicles. This data includes GPS location information, speed, road conditions, information about surrounding vehicles, and the user's vital signs (heart rate, facial expression, voice, etc.) obtained from sensors inside the vehicle.

[0123] (emotional analysis)

[0124] After collecting data, the device uses an emotion engine to analyze the user's emotional state based on their vital signs. For example, if it determines that the user is experiencing stress, it will suggest adjustments to operational support and environmental settings that correspond to that emotion.

[0125] (decision making)

[0126] The server combines analyzed emotions with driving data to make optimal decisions that allow the user to continue their journey comfortably and safely. In doing so, it takes into account surrounding traffic conditions and emotional state, and makes decisions to change the driving route or make temporary stops as needed.

[0127] (Instruction distribution)

[0128] The server generates instructions based on decisions and distributes them to each vehicle via terminals. These instructions include speed limits for vehicles, changes in direction, and automatic adjustments to environmental settings (such as music changes and lighting adjustments).

[0129] (Operation carried out)

[0130] The system controls the vehicle based on instructions received by the terminal, achieving operation that is adjusted to the user's emotional state. In particular, the goal is to create an operation that puts the user in a relaxed state.

[0131] (Feedback and Learning)

[0132] The server accumulates user emotion data and vehicle history data observed during operation, and uses this data through an emotion engine to help with future decision-making. This refines the entire system.

[0133] For example, if a user boards a vehicle while stressed after a meeting, the emotional engine will detect this and suggest relaxing lighting settings and classical music to provide a comfortable travel experience. In this way, the present invention aims to promote a more comfortable travel environment by considering not only the safety of travel but also the user's emotional state.

[0134] The following describes the processing flow.

[0135] Step 1:

[0136] The server collects real-time driving data and user vital signs from each vehicle. The data is acquired via sensors and includes GPS information, vehicle speed, and user heart rate and facial expression data.

[0137] Step 2:

[0138] The device uses an emotion engine to analyze the user's emotional state based on biometric data received from the server. Here, an AI algorithm is used to determine whether the user is relaxed or stressed.

[0139] Step 3:

[0140] The server evaluates the current operating status by combining the analyzed emotional state with driving data. In addition, it makes decisions based on the user's emotional state and plans to optimize the route and adjust the vehicle environment to improve comfort during the ride.

[0141] Step 4:

[0142] The server generates specific instructions for the vehicle based on the decision. These include speed limits, changes in direction, and adjustments to the operating environment (e.g., music selection, air conditioning settings, etc.).

[0143] Step 5:

[0144] The terminal distributes the generated instructions to each vehicle, and the vehicle's operation is controlled based on these instructions. This enables control that responds to the user's emotional state, providing a more comfortable riding experience.

[0145] Step 6:

[0146] The terminals and servers monitor the emotional state of users during operation and the results of environmental adjustments in real time, making immediate adjustments when necessary. This continuously improves user satisfaction.

[0147] Step 7:

[0148] After each operation, the server stores all operational data and user sentiment data, and uses AI to automatically learn areas for improvement for future operations, thereby enhancing the overall system functionality.

[0149] (Example 2)

[0150] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0151] In modern automotive technology, accurately understanding the emotional state of passengers and formulating a corresponding operational plan is difficult. This can lead to situations where passengers experience stress or where safety is not adequately ensured. Furthermore, it is difficult to respond quickly to unforeseen circumstances that occur during operation and to continuously improve the quality of service.

[0152] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0153] In this invention, the server includes information gathering means for collecting information from multiple mechanical devices, analysis means for analyzing the user's emotional state, and decision-making means for generating an optimal travel plan. This makes it possible to create a comfortable and safe travel plan that is tailored to the user's emotional state.

[0154] "Information gathering means" refers to a function that collects information in real time from multiple machines and devices and uses it in subsequent processing.

[0155] "Analysis means" refers to a function that identifies the emotional state of users from collected information and provides information necessary for optimizing operations through data analysis.

[0156] "Decision-making tool" refers to a function that generates the most efficient and comfortable travel plan for users based on analyzed information.

[0157] "Instruction distribution means" refers to a function that accurately and quickly transmits the generated operation plan to each machine and device, ensuring that the devices operate according to the plan.

[0158] "Execution means" refers to the function that controls the machine equipment to operate appropriately based on the operation plan received from the instruction distribution means.

[0159] "Feedback mechanisms" refer to functions that collect and evaluate operational results and user sentiment data, and utilize this information to continuously improve operational efficiency.

[0160] This invention is a system that utilizes an emotion analysis engine to understand the emotional state of users and, based on that understanding, achieve optimal vehicle operation. This system is designed to provide users with a comfortable and safe travel experience through data collection, analysis, decision-making, instruction distribution, operation execution, and feedback processing.

[0161] In this system, a server collects information from multiple mechanical devices (e.g., autonomous vehicles and smart devices). Information is collected using vehicle sensors (e.g., GPS, advanced driver assistance systems, cameras, microphones) to obtain location information, speed data, surrounding environment information, and the user's vital signs (heart rate, facial expressions, voice analysis, etc.).

[0162] Next, the device uses this data to activate its emotion analysis engine and analyze the user's emotional state. This analysis process uses software to process and analyze data, determining whether the user is stressed or relaxed. The results of the emotion analysis provide crucial input for improving the user experience.

[0163] The server uses analyzed emotional data and physical operational data to generate an optimal operational plan and distributes instructions to each vehicle. Network communication is used for instruction distribution, and the necessary actions are taken by the vehicles via terminals.

[0164] As a concrete example, consider a scenario where the entire system is in a vehicle after a stressful meeting. In this situation, the terminal activates an emotion analysis engine and recommends relaxing music and lighting settings based on the user's stress level.

[0165] Furthermore, the system utilizes a generative AI model to create prompt messages, improving the overall operational efficiency of the system. An example of a prompt message used is user feedback such as, "If the user is experiencing stress, please explain how to adjust the vehicle's environment."

[0166] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0167] Step 1:

[0168] The server collects information. Specifically, it receives sensor data from vehicles and smart devices. The input data includes GPS location information, speed, surrounding environment information, and the user's vital signs (e.g., heart rate, facial expression, voice). By aggregating this data, a basic dataset necessary for subsequent analysis is generated.

[0169] Step 2:

[0170] The device performs emotion analysis based on collected data. It uses data such as vital signs received from a server as input. An emotion engine analyzes the data to evaluate the user's emotional state. For example, it can determine stress levels from voice tone and interpret emotions by analyzing facial expressions. The output is an emotional state determination result, such as "relaxed" or "stressed."

[0171] Step 3:

[0172] The server integrates sentiment analysis results with physical operation data to make decisions. Input data includes sentiment analysis results and vehicle operation status data. An algorithm is used to create an optimal operation plan, and suggested adjustments include environmental settings (music, lighting) and changes to the driving route. The output is specific operation instructions.

[0173] Step 4:

[0174] The server distributes the generated operational instructions through the instruction distribution system. The input is the operational instructions generated in step 3. These instructions are sent to each terminal via the network, ensuring that each vehicle operates according to plan. The output is the operational protocol applied to each vehicle.

[0175] Step 5:

[0176] The terminal controls the vehicle based on the instructions it receives. The input is the operational instructions received from the server. By using actual feedback to adjust each system (e.g., navigation, infotainment) appropriately, a comfortable travel environment is provided to the user. The output is the result of the operation in accordance with the instructions.

[0177] Step 6:

[0178] The server aggregates operational data and performs feedback and learning. Inputs include user sentiment data and operational history collected during operation. This data is accumulated and used to train the sentiment engine for future decision-making. Outputs are improved operational algorithms and new learning models.

[0179] (Application Example 2)

[0180] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0181] Currently, most autonomous driving systems prioritize safety and efficiency above all else, with little consideration given to passengers' emotional state or comfort. This leads to problems such as operation in stressful environments and a disregard for passengers' emotional state. Therefore, there is a need for systems that can flexibly respond to diverse traffic conditions and provide comfortable operation while considering passengers' emotional state.

[0182] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0183] In this invention, the server includes information gathering means for collecting information from multiple mobile devices, analysis means for analyzing road conditions, decision-making means for realizing coordination between mobile devices, environment adjustment means for optimizing environmental settings according to emotional states, and feedback means for improving operations. This makes it possible to provide a comfortable and safe mobile environment that takes into account the emotional state of the user.

[0184] "Mobile devices" is a general term for means of transportation on land, sea, or air, and is a machine equipped with an autonomous driving function.

[0185] "Information gathering means" refers to means including sensors and devices for collecting data related to mobile devices and users.

[0186] "Analysis means" refers to a computer system that processes data obtained through information gathering means to analyze traffic conditions and user status.

[0187] A "decision-making tool" is a tool equipped with a processor or algorithm to determine appropriate actions or adjustments based on analysis results and to generate those instructions.

[0188] "Instruction distribution means" refers to communication infrastructure or systems for transmitting instructions generated by decision-making means to mobile devices.

[0189] "Execution means" refers to a mechanical or electronic device that receives instructions from an instruction distribution means, controls a mobile device, and performs an action.

[0190] "Environmental adjustment means" refers to devices that optimize the in-vehicle environment according to the user's emotional state, and are means of adjusting elements such as sound, lighting, and temperature.

[0191] "Feedback mechanisms" refer to procedures and devices used to improve the operating system based on data collected during operation.

[0192] The system of this invention analyzes the emotional state of users and generates corresponding operational instructions in order to improve the comfort and safety of mobile devices. Specifically, a server plays a crucial role in aggregating and analyzing information from mobile devices and providing appropriate instructions to each device, thereby maximizing the overall efficiency of travel.

[0193] The server collects location information, speed, road conditions, and user vital signs (heart rate, facial expressions, voice, etc.) data through sensors mounted on multiple mobile devices. This is implemented as a means of information gathering.

[0194] Next, the server uses analysis tools to process the collected data and analyze traffic conditions and the emotional state of users. Here, an emotion analysis engine (e.g., Emotion-SDK) is used to detect the user's stress level, etc.

[0195] Based on the analysis results, the decision-making system generates optimal instructions to ensure safe and comfortable travel. These instructions include adjustments to the route and in-vehicle environmental settings (music, lighting, etc.).

[0196] The server distributes the generated instructions to each mobile device via the instruction distribution means. The mobile devices receive these instructions and perform specific actions or environmental adjustments using the execution means.

[0197] Through feedback mechanisms, the system learns from data collected during operation and further optimizes future operations. This accumulated data is continuously analyzed and used to improve the overall accuracy of the system.

[0198] As a concrete example, consider a scenario where a user is experiencing increased stress as they head home after a full day of meetings. In this case, the server suggests relaxing music and gentle lighting settings. This allows the user to enjoy a pleasant travel experience.

[0199] An example of a prompt message might be: "Analyze the current user's emotional state and suggest a driving route and in-car environment that will reduce stress and improve comfort."

[0200] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0201] Step 1:

[0202] The server acquires location information, speed, road conditions, and user vital signs (heart rate, facial expressions, voice, etc.) from multiple mobile devices via information gathering means. This input data is collected in real time through sensors and communication networks and stored as an initial dataset.

[0203] Step 2:

[0204] The server processes the collected data using analytical tools to analyze traffic conditions and the emotional state of users. This analysis normalizes the data and uses an emotion analysis engine (e.g., Emotion-SDK) to determine stress levels. Input data includes vital signs and voice analysis results, and the server outputs the user's overall emotional state accordingly.

[0205] Step 3:

[0206] The server generates optimal operational instructions using decision-making mechanisms based on the analysis results. This process considers the analyzed emotional state and traffic information, and, if necessary, decides to change the route or adjust the in-vehicle environment settings. This process takes emotional state and traffic analysis results as input and provides specific instructions as output.

[0207] Step 4:

[0208] The server distributes the generated operation instructions to each mobile device through the instruction distribution means. In this distribution step, the generated instructions are transmitted to each mobile device in real time via the communication network. Instructions from the decision-making means are used as input, and the instructions are output as notifications to the mobile devices.

[0209] Step 5:

[0210] Each terminal controls the vehicle based on the instructions it receives and adjusts its operation using execution methods. Specifically, it adjusts speed, changes direction of travel, and changes settings for the in-vehicle environment (music, lighting). The input for this step is instructions from the server, and the output is a change in the state of the moving device.

[0211] Step 6:

[0212] The server monitors operational status through feedback mechanisms and continuously learns and improves the system based on the collected data. In this process, it analyzes errors that occur during execution and changes in sentiment data, gaining insights to improve future operations. Inputs are various data after execution, and outputs are updated analysis algorithms and feedback on improvement measures.

[0213] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0214] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0215] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0216] [Second Embodiment]

[0217] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0218] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0219] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0220] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0221] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0222] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0223] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0224] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0225] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0226] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0227] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0228] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0229] The present invention provides an operation management system that enables mutual cooperation between multiple vehicles using AI. This system mainly consists of data collection means, analysis means, decision-making means, instruction distribution means, execution means, and feedback means. This provides an environment in which autonomous vehicles and automated transport robots can safely coexist on public roads.

[0230] In this system, the server first collects driving data from each vehicle through sensors and communication devices. This includes the vehicle's position, speed, acceleration, and information about surrounding obstacles and other vehicles. Next, the server analyzes the collected data in real time on terminals to assess traffic conditions and potential risks.

[0231] Based on the analysis results, the server determines the priority order between vehicles and makes decisions regarding their routes and speed adjustments. The instructions generated by the server are then sent to each terminal, which distribute them to the vehicles. For example, at an intersection, the server determines the order in which each vehicle should pass and issues instructions to ensure safe passage based on that order.

[0232] The terminal controls the vehicle's operation based on received instructions. This control includes adjusting the vehicle's direction, speed, and stopping position. Furthermore, operational data is fed back to the server, and any abnormalities detected are promptly addressed. The feedback data is also used to improve the system's performance for future operations.

[0233] As a concrete example, when an autonomous bus travels through a congested urban area, it can coordinate with other automated transport robots and traffic signals to safely reach its destination along a designated route. This ensures the safety of all road users and creates a smooth flow of traffic.

[0234] Thus, the present invention provides an environment in which a variety of autonomous vehicles can operate safely on public roads, and in particular ensures traffic safety in situations where vehicles with different specifications and from different manufacturers are mixed together.

[0235] The following describes the processing flow.

[0236] Step 1:

[0237] The server collects data in real time from each vehicle traveling on public roads. This data includes the vehicle's location, speed, and information about surrounding obstacles. This data collection is performed through sensors and communication devices mounted on the vehicles.

[0238] Step 2:

[0239] The terminal uses data received from the server to analyze traffic conditions and assess the distance and speed between vehicles, as well as the potential collision risk at intersections. AI algorithms are used to improve the accuracy of the analysis results.

[0240] Step 3:

[0241] The server makes decisions based on the analysis results. This includes determining vehicle priorities and planning optimal routes and speed adjustments. In complex situations such as intersections, it determines priority and generates appropriate instructions.

[0242] Step 4:

[0243] The server generates specific instructions for each vehicle based on the decisions made and distributes them to the terminals. These instructions may include instructions to adjust speed, change direction, or stop temporarily.

[0244] Step 5:

[0245] The terminal relays the received instructions to the vehicle's control system, ensuring appropriate operation. Based on these instructions, the vehicle autonomously adjusts its course and speed, driving in harmony with other vehicles and road users.

[0246] Step 6:

[0247] The terminal monitors data during vehicle operation and reports any abnormalities or unexpected events to the server in real time. The server then takes appropriate action based on this information.

[0248] Step 7:

[0249] The server accumulates operational data, evaluates the overall system performance, and uses this data for future improvements. Through this feedback process, the system learns and improves its performance.

[0250] (Example 1)

[0251] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0252] In modern transportation systems involving mobile vehicles, there is a demand for improved safety and efficient traffic flow. However, in situations where vehicles with different specifications and from different manufacturers coexist, technical challenges exist in achieving harmony between them. Furthermore, there is a need for methods to quickly respond to unpredictable situations during operation and provide optimal transportation strategies in real time.

[0253] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0254] In this invention, the server includes information gathering means for collecting information from multiple moving objects, analysis means for analyzing the movement situation based on the information, and means for proposing a new operation strategy using a generative AI model. This improves the harmony between different moving objects and enables safe and efficient traffic management while immediately adapting to the situation during operation.

[0255] A "mobile object" is a means of transport designed for public or private use, and includes vehicles, machinery, and other similar devices.

[0256] "Information gathering means" refers to a device or system for receiving and accumulating various types of data from a moving object.

[0257] "Analysis means" refers to devices or software used to process collected data and evaluate movement patterns and traffic conditions.

[0258] A "decision-making mechanism" is a device or process for formulating a movement strategy based on the analysis results and determining the action plan for each moving object.

[0259] "Instruction and distribution means" refers to a device or communication method for accurately transmitting a decided action plan to each mobile entity.

[0260] "Implementation means" refers to a device or program for actually controlling the movement of a mobile object in accordance with the instructions that have been delivered.

[0261] "Adjustment means" refers to a device or system for feeding back operational information and optimizing the operation of a moving object.

[0262] A "generative AI model" is an algorithm or model that uses machine learning techniques to analyze data and generate new operational strategies or solutions.

[0263] An "anomaly detection method" is a process or system for quickly identifying and responding to unexpected situations or problems during operation.

[0264] This invention is an operation management system that enables mutual cooperation between multiple mobile entities by utilizing AI technology. This system includes means for information gathering, analysis, decision-making, instruction distribution, implementation, coordination, and operation strategy proposal using a generated AI model.

[0265] In implementing the system, the server first collects information in real time from each moving object through hardware such as GPS sensors and LiDAR. This allows the server to obtain information on the object's position, speed, acceleration, and surrounding obstacles. Next, the server uses powerful analytical software to analyze the collected information. This includes AI algorithms for modeling traffic conditions and assessing potential risks.

[0266] Based on the analysis results, the terminal determines the priority between each moving object and the optimal route. During the decision-making process, the generating AI model proposes a new operational strategy through prompt messages. For example, in a scenario where an electric bus in an urban area automatically avoids congestion and reaches its destination along a designated route, an example of a prompt message might be, "How can an autonomous bus operate smoothly in a congested urban area?"

[0267] The user plays a role in feeding back information obtained during operation to the server, and the system uses this feedback to improve its performance. This allows the system to be continuously optimized, enabling safe and efficient operation management even in situations where mobile vehicles with different specifications and manufacturers are mixed.

[0268] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0269] Step 1:

[0270] The server's role is to collect information from each mobile object in real time. Specifically, the server uses GPS sensors and LiDAR to acquire location, velocity, acceleration, and information about surrounding obstacles. The input is sensor data transmitted from the mobile object, which is received and stored in the server as integrated information. The output is a complete dataset for analysis.

[0271] Step 2:

[0272] The server analyzes the collected data. Using analysis software, it models traffic conditions and performs risk assessments. Specifically, it utilizes AI algorithms to predict potential hazards and congestion based on sensor data. The input is the dataset obtained in step 1, and the output is the evaluation results and risk assessment report regarding traffic conditions.

[0273] Step 3:

[0274] The terminal makes decisions based on the analysis results sent from the server. Specifically, it processes the prioritization of mobile objects and the optimal travel route. At this stage, the generating AI model may propose an operation strategy based on the prompt message. The input is the analysis results from step 2 and the prompt message from the generating AI model. The output is specific travel instructions for each mobile object.

[0275] Step 4:

[0276] The server distributes the determined instructions to each terminal. Specifically, it transmits instruction information to the mobile device using a communication protocol and accurately conveys its contents. The input is the instruction content determined in step 3. The output is the action plan transmitted to the terminal.

[0277] Step 5:

[0278] The terminal controls the mobile object based on the instructions it receives. Specifically, it sends signals to the vehicle's control system and performs appropriate operations. The input is the instructions received in step 4. The output is the specific operation performed by the mobile object, such as adjustments to direction or speed.

[0279] Step 6:

[0280] The user feeds data generated during operation back to the server. Specifically, this involves reporting abnormal situations to the server and prompting system adjustments. Inputs are operational data and abnormal information obtained during operation. Outputs are system improvement suggestions for the next operation.

[0281] (Application Example 1)

[0282] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0283] With the spread of autonomous driving technology, it is necessary to properly manage traffic situations between autonomous vehicles and pedestrians to enhance safety. However, conventional technologies lack the means to provide users with detailed real-time traffic information, and warnings for unexpected situations and sudden lane changes are particularly insufficient. This creates a challenge in that safety around autonomous vehicles may be compromised.

[0284] The specific processing by the specific processing unit 290 of the data processing apparatus 12 in Application Example 1 is realized by the following means.

[0285] In this invention, the server includes data collection means for collecting data from a plurality of moving objects, analysis means for analyzing the situation of the moving objects, and information providing means for enabling a user to receive movement and safe route information of the moving objects through a visual device. Thereby, it becomes possible to notify the user of the trend of the moving objects in real time and give a warning in advance for unexpected situations.

[0286] The "moving object" refers to a vehicle or device equipped with an automatic driving technology that operates in various environments.

[0287] The "data collection means" refers to a device or method for acquiring driving status and surrounding information from a moving object.

[0288] The "analysis means" refers to a method or function for evaluating the situation of a moving object based on the collected data and making necessary adjustments and decisions.

[0289] The "decision-making means" refers to an algorithm or system for determining the priority and route between moving objects based on the analysis results.

[0290] The "instruction distribution means" refers to a system or technology for transmitting the determined instruction to each moving object and prompting the operation.

[0291] The "execution means" refers to a function or technology for controlling a moving object based on the instructed content.

[0292] The "feedback means" refers to a method or function for continuously recording the operation status and using it for system improvement.

[0293] The "information providing means" refers to a system for notifying a user of the movement and safe route information of a moving object in real time.

[0294] "Warning generation means" refers to a function that issues a warning to the user in situations where danger is anticipated.

[0295] The system for implementing this invention primarily operates through cooperation between three parties: a server, a terminal, and a user.

[0296] First, the server collects data from multiple mobile vehicles equipped with autonomous driving technology. The hardware used includes sensors and communication devices mounted on the vehicles. The data includes the location, speed, direction of travel, and information about the surrounding environment of the mobile vehicles. The server processes this data in real time and is responsible for analyzing the behavior of the mobile vehicles.

[0297] After analysis, the server determines priorities among the moving objects and sets appropriate routes and speeds. This decision-making process utilizes AI-powered algorithms. The analysis results are transmitted to terminals via a command distribution system.

[0298] The terminal receives these instructions and specifically controls the movement of each mobile object. Furthermore, it provides information about safe routes to the user's visual devices, such as smartphones or smart glasses. In doing so, it also issues warnings via voice or vibration as needed.

[0299] A feedback function is also important. The operating status of the mobile devices is sent to the server in real time, and if an anomaly is detected, immediate action is taken. The collected data is used to improve the overall performance of the system and prevent malfunctions.

[0300] As a concrete example, let's assume a user is walking through a crowded urban area. In this case, the smart glasses visually display the movement of the autonomous vehicle ahead and guide the user to the sidewalk where they can safely avoid it. Furthermore, if the direction of the moving object suddenly changes, an immediate voice warning is issued.

[0301] Examples of prompt texts for generating an AI model include the following. Forms such as "Please devise an app that provides real-time route information safe for pedestrians in the vicinity based on traffic data of self-driving vehicles." and "What is the method of tracking self-driving vehicles operating safely with a smart device when moving in an urban area during rainy weather?" can be cited. From these prompts, the AI model generates solutions suitable for the specific situations faced by the user.

[0302] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0303] Step 1:

[0304] The server collects data from multiple moving objects. The input is the position, speed, direction of travel, and surrounding environmental information from sensors mounted on the moving objects. The server receives these data and stores them in a batch. By this operation, base data on the detailed traffic situation in real time is formed.

[0305] Step 2:

[0306] The server performs analysis based on the collected data. The input is the traffic data obtained in Step 1. The server uses an AI algorithm to predict the trajectory of the moving object and evaluate the collision risk, and determines the priority and optimal route of each moving object. Through this process, information for enabling rational decision-making is generated.

[0307] Step 3:

[0308] The server generates instructions based on the analysis results and distributes them to the terminals. The input is the priority and route information of the moving objects determined in Step 2. The server transmits these instructions to the control systems of each moving object through the instruction distribution system. By this procedure, specific guidelines for the safe and efficient operation of the moving objects are provided.

[0309] Step 4:

[0310] The terminal receives instructions from the server and actually controls the mobile object. The input is the instruction sent in step 3. The terminal adjusts the movement of the mobile object, changing the direction of travel, speed, and stopping position as needed. This control ensures safety and efficiency during actual operation.

[0311] Step 5:

[0312] The terminal provides information to the user's visual device. The input is the instruction information obtained in step 3. The terminal displays the movement of surrounding objects and safe route information on the user's smart device, and also issues warnings via voice and vibration as needed. This operation enables the provision of real-time safety information to the user.

[0313] Step 6:

[0314] The terminal provides feedback on the operational status at the site and transmits it to the server. The input is the latest operational data from the sensors of the moving vehicle. The terminal analyzes the data during operation and immediately reports any anomalies to the server. This feedback helps to improve the entire system and prevent accidents.

[0315] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0316] This invention is a cooperative operation management system that combines an emotion engine, and by linking the emotional states of multiple vehicles and users, it realizes more comfortable and safer travel. The system has a configuration that includes data collection means, analysis means, decision-making means, instruction distribution means, execution means, and feedback means, as well as an emotion engine.

[0317] (Data collection)

[0318] First, the server collects driving data from multiple vehicles. This data includes GPS location information, speed, road conditions, information about surrounding vehicles, and the user's vital signs (heart rate, facial expression, voice, etc.) obtained from sensors inside the vehicle.

[0319] (emotional analysis)

[0320] After collecting data, the device uses an emotion engine to analyze the user's emotional state based on their vital signs. For example, if it determines that the user is experiencing stress, it will suggest adjustments to operational support and environmental settings that correspond to that emotion.

[0321] (decision making)

[0322] The server combines analyzed emotions with driving data to make optimal decisions that allow the user to continue their journey comfortably and safely. In doing so, it takes into account surrounding traffic conditions and emotional state, and makes decisions to change the driving route or make temporary stops as needed.

[0323] (Instruction distribution)

[0324] The server generates instructions based on decisions and distributes them to each vehicle via terminals. These instructions include speed limits for vehicles, changes in direction, and automatic adjustments to environmental settings (such as music changes and lighting adjustments).

[0325] (Operation carried out)

[0326] The system controls the vehicle based on instructions received by the terminal, achieving operation that is adjusted to the user's emotional state. In particular, the goal is to create an operation that puts the user in a relaxed state.

[0327] (Feedback and Learning)

[0328] The server accumulates user emotion data and vehicle history data observed during operation, and uses this data through an emotion engine to help with future decision-making. This refines the entire system.

[0329] For example, if a user boards a vehicle while stressed after a meeting, the emotional engine will detect this and suggest relaxing lighting settings and classical music to provide a comfortable travel experience. In this way, the present invention aims to promote a more comfortable travel environment by considering not only the safety of travel but also the user's emotional state.

[0330] The following describes the processing flow.

[0331] Step 1:

[0332] The server collects real-time driving data and user vital signs from each vehicle. The data is acquired via sensors and includes GPS information, vehicle speed, and user heart rate and facial expression data.

[0333] Step 2:

[0334] The device uses an emotion engine to analyze the user's emotional state based on biometric data received from the server. Here, an AI algorithm is used to determine whether the user is relaxed or stressed.

[0335] Step 3:

[0336] The server evaluates the current operating status by combining the analyzed emotional state with driving data. In addition, it makes decisions based on the user's emotional state and plans to optimize the route and adjust the vehicle environment to improve comfort during the ride.

[0337] Step 4:

[0338] The server generates specific instructions for the vehicle based on the decision. These include speed limits, changes in direction, and adjustments to the operating environment (e.g., music selection, air conditioning settings, etc.).

[0339] Step 5:

[0340] The terminal distributes the generated instructions to each vehicle, and the vehicle's operation is controlled based on these instructions. This enables control that responds to the user's emotional state, providing a more comfortable riding experience.

[0341] Step 6:

[0342] The terminals and servers monitor the emotional state of users during operation and the results of environmental adjustments in real time, making immediate adjustments when necessary. This continuously improves user satisfaction.

[0343] Step 7:

[0344] After each operation, the server stores all operational data and user sentiment data, and uses AI to automatically learn areas for improvement for future operations, thereby enhancing the overall system functionality.

[0345] (Example 2)

[0346] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0347] In modern automotive technology, accurately understanding the emotional state of passengers and formulating a corresponding operational plan is difficult. This can lead to situations where passengers experience stress or where safety is not adequately ensured. Furthermore, it is difficult to respond quickly to unforeseen circumstances that occur during operation and to continuously improve the quality of service.

[0348] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0349] In this invention, the server includes information gathering means for collecting information from multiple mechanical devices, analysis means for analyzing the user's emotional state, and decision-making means for generating an optimal travel plan. This makes it possible to create a comfortable and safe travel plan that is tailored to the user's emotional state.

[0350] "Information gathering means" refers to a function that collects information in real time from multiple machines and devices and uses it in subsequent processing.

[0351] "Analysis means" refers to a function that identifies the emotional state of users from collected information and provides information necessary for optimizing operations through data analysis.

[0352] "Decision-making tool" refers to a function that generates the most efficient and comfortable travel plan for users based on analyzed information.

[0353] "Instruction distribution means" refers to a function that accurately and quickly transmits the generated operation plan to each machine and device, ensuring that the devices operate according to the plan.

[0354] "Execution means" refers to the function that controls the machine equipment to operate appropriately based on the operation plan received from the instruction distribution means.

[0355] "Feedback mechanisms" refer to functions that collect and evaluate operational results and user sentiment data, and utilize this information to continuously improve operational efficiency.

[0356] This invention is a system that utilizes an emotion analysis engine to understand the emotional state of users and, based on that understanding, achieve optimal vehicle operation. This system is designed to provide users with a comfortable and safe travel experience through data collection, analysis, decision-making, instruction distribution, operation execution, and feedback processing.

[0357] In this system, a server collects information from multiple mechanical devices (e.g., autonomous vehicles and smart devices). Information is collected using vehicle sensors (e.g., GPS, advanced driver assistance systems, cameras, microphones) to obtain location information, speed data, surrounding environment information, and the user's vital signs (heart rate, facial expressions, voice analysis, etc.).

[0358] Next, the device uses this data to activate its emotion analysis engine and analyze the user's emotional state. This analysis process uses software to process and analyze data, determining whether the user is stressed or relaxed. The results of the emotion analysis provide crucial input for improving the user experience.

[0359] The server uses analyzed emotional data and physical operational data to generate an optimal operational plan and distributes instructions to each vehicle. Network communication is used for instruction distribution, and the necessary actions are taken by the vehicles via terminals.

[0360] As a concrete example, consider a scenario where the entire system is in a vehicle after a stressful meeting. In this situation, the terminal activates an emotion analysis engine and recommends relaxing music and lighting settings based on the user's stress level.

[0361] Furthermore, the system utilizes a generative AI model to create prompt messages, improving the overall operational efficiency of the system. An example of a prompt message used is user feedback such as, "If the user is experiencing stress, please explain how to adjust the vehicle's environment."

[0362] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0363] Step 1:

[0364] The server collects information. Specifically, it receives sensor data from vehicles and smart devices. The input data includes GPS location information, speed, surrounding environment information, and the user's vital signs (e.g., heart rate, facial expression, voice). By aggregating this data, a basic dataset necessary for subsequent analysis is generated.

[0365] Step 2:

[0366] The device performs emotion analysis based on collected data. It uses data such as vital signs received from a server as input. An emotion engine analyzes the data to evaluate the user's emotional state. For example, it can determine stress levels from voice tone and interpret emotions by analyzing facial expressions. The output is an emotional state determination result, such as "relaxed" or "stressed."

[0367] Step 3:

[0368] The server integrates sentiment analysis results with physical operation data to make decisions. Input data includes sentiment analysis results and vehicle operation status data. An algorithm is used to create an optimal operation plan, and suggested adjustments include environmental settings (music, lighting) and changes to the driving route. The output is specific operation instructions.

[0369] Step 4:

[0370] The server distributes the generated operational instructions through the instruction distribution system. The input is the operational instructions generated in step 3. These instructions are sent to each terminal via the network, ensuring that each vehicle operates according to plan. The output is the operational protocol applied to each vehicle.

[0371] Step 5:

[0372] The terminal controls the vehicle based on the instructions it receives. The input is the operational instructions received from the server. By using actual feedback to adjust each system (e.g., navigation, infotainment) appropriately, a comfortable travel environment is provided to the user. The output is the result of the operation in accordance with the instructions.

[0373] Step 6:

[0374] The server aggregates operational data and performs feedback and learning. Inputs include user sentiment data and operational history collected during operation. This data is accumulated and used to train the sentiment engine for future decision-making. Outputs are improved operational algorithms and new learning models.

[0375] (Application Example 2)

[0376] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0377] Currently, most autonomous driving systems prioritize safety and efficiency above all else, with little consideration given to passengers' emotional state or comfort. This leads to problems such as operation in stressful environments and a disregard for passengers' emotional state. Therefore, there is a need for systems that can flexibly respond to diverse traffic conditions and provide comfortable operation while considering passengers' emotional state.

[0378] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0379] In this invention, the server includes information gathering means for collecting information from multiple mobile devices, analysis means for analyzing road conditions, decision-making means for realizing coordination between mobile devices, environment adjustment means for optimizing environmental settings according to emotional states, and feedback means for improving operations. This makes it possible to provide a comfortable and safe mobile environment that takes into account the emotional state of the user.

[0380] "Mobile devices" is a general term for means of transportation on land, sea, or air, and is a machine equipped with an autonomous driving function.

[0381] "Information gathering means" refers to means including sensors and devices for collecting data related to mobile devices and users.

[0382] "Analysis means" refers to a computer system that processes data obtained through information gathering means to analyze traffic conditions and user status.

[0383] A "decision-making tool" is a tool equipped with a processor or algorithm to determine appropriate actions or adjustments based on analysis results and to generate those instructions.

[0384] "Instruction distribution means" refers to communication infrastructure or systems for transmitting instructions generated by decision-making means to mobile devices.

[0385] "Execution means" refers to a mechanical or electronic device that receives instructions from an instruction distribution means, controls a mobile device, and performs an action.

[0386] "Environmental adjustment means" refers to devices that optimize the in-vehicle environment according to the user's emotional state, and are means of adjusting elements such as sound, lighting, and temperature.

[0387] "Feedback mechanisms" refer to procedures and devices used to improve the operating system based on data collected during operation.

[0388] The system of this invention analyzes the emotional state of users and generates corresponding operational instructions in order to improve the comfort and safety of mobile devices. Specifically, a server plays a crucial role in aggregating and analyzing information from mobile devices and providing appropriate instructions to each device, thereby maximizing the overall efficiency of travel.

[0389] The server collects location information, speed, road conditions, and user vital signs (heart rate, facial expressions, voice, etc.) data through sensors mounted on multiple mobile devices. This is implemented as a means of information gathering.

[0390] Next, the server uses analysis tools to process the collected data and analyze traffic conditions and the emotional state of users. Here, an emotion analysis engine (e.g., Emotion-SDK) is used to detect the user's stress level, etc.

[0391] Based on the analysis results, the decision-making system generates optimal instructions to ensure safe and comfortable travel. These instructions include adjustments to the route and in-vehicle environmental settings (music, lighting, etc.).

[0392] The server distributes the generated instructions to each mobile device via the instruction distribution means. The mobile devices receive these instructions and perform specific actions or environmental adjustments using the execution means.

[0393] Through feedback mechanisms, the system learns from data collected during operation and further optimizes future operations. This accumulated data is continuously analyzed and used to improve the overall accuracy of the system.

[0394] As a concrete example, consider a scenario where a user is experiencing increased stress as they head home after a full day of meetings. In this case, the server suggests relaxing music and gentle lighting settings. This allows the user to enjoy a pleasant travel experience.

[0395] An example of a prompt message might be: "Analyze the current user's emotional state and suggest a driving route and in-car environment that will reduce stress and improve comfort."

[0396] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0397] Step 1:

[0398] The server acquires location information, speed, road conditions, and user vital signs (heart rate, facial expressions, voice, etc.) from multiple mobile devices via information gathering means. This input data is collected in real time through sensors and communication networks and stored as an initial dataset.

[0399] Step 2:

[0400] The server processes the collected data using analytical tools to analyze traffic conditions and the emotional state of users. This analysis normalizes the data and uses an emotion analysis engine (e.g., Emotion-SDK) to determine stress levels. Input data includes vital signs and voice analysis results, and the server outputs the user's overall emotional state accordingly.

[0401] Step 3:

[0402] The server generates optimal operational instructions using decision-making mechanisms based on the analysis results. This process considers the analyzed emotional state and traffic information, and, if necessary, decides to change the route or adjust the in-vehicle environment settings. This process takes emotional state and traffic analysis results as input and provides specific instructions as output.

[0403] Step 4:

[0404] The server distributes the generated operation instructions to each mobile device through the instruction distribution means. In this distribution step, the generated instructions are transmitted to each mobile device in real time via the communication network. Instructions from the decision-making means are used as input, and the instructions are output as notifications to the mobile devices.

[0405] Step 5:

[0406] Each terminal controls the vehicle based on the instructions it receives and adjusts its operation using execution methods. Specifically, it adjusts speed, changes direction of travel, and changes settings for the in-vehicle environment (music, lighting). The input for this step is instructions from the server, and the output is a change in the state of the moving device.

[0407] Step 6:

[0408] The server monitors operational status through feedback mechanisms and continuously learns and improves the system based on the collected data. In this process, it analyzes errors that occur during execution and changes in sentiment data, gaining insights to improve future operations. Inputs are various data after execution, and outputs are updated analysis algorithms and feedback on improvement measures.

[0409] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0410] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0411] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0412] [Third Embodiment]

[0413] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0414] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0415] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0416] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0417] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0418] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0419] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0420] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0421] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0422] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0423] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0424] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0425] The present invention provides an operation management system that enables mutual cooperation between multiple vehicles using AI. This system mainly consists of data collection means, analysis means, decision-making means, instruction distribution means, execution means, and feedback means. This provides an environment in which autonomous vehicles and automated transport robots can safely coexist on public roads.

[0426] In this system, the server first collects driving data from each vehicle through sensors and communication devices. This includes the vehicle's position, speed, acceleration, and information about surrounding obstacles and other vehicles. Next, the server analyzes the collected data in real time on terminals to assess traffic conditions and potential risks.

[0427] Based on the analysis results, the server determines the priority order between vehicles and makes decisions regarding their routes and speed adjustments. The instructions generated by the server are then sent to each terminal, which distribute them to the vehicles. For example, at an intersection, the server determines the order in which each vehicle should pass and issues instructions to ensure safe passage based on that order.

[0428] The terminal controls the vehicle's operation based on received instructions. This control includes adjusting the vehicle's direction, speed, and stopping position. Furthermore, operational data is fed back to the server, and any abnormalities detected are promptly addressed. The feedback data is also used to improve the system's performance for future operations.

[0429] As a concrete example, when an autonomous bus travels through a congested urban area, it can coordinate with other automated transport robots and traffic signals to safely reach its destination along a designated route. This ensures the safety of all road users and creates a smooth flow of traffic.

[0430] Thus, the present invention provides an environment in which a variety of autonomous vehicles can operate safely on public roads, and in particular ensures traffic safety in situations where vehicles with different specifications and from different manufacturers are mixed together.

[0431] The following describes the processing flow.

[0432] Step 1:

[0433] The server collects data in real time from each vehicle traveling on public roads. This data includes the vehicle's location, speed, and information about surrounding obstacles. This data collection is performed through sensors and communication devices mounted on the vehicles.

[0434] Step 2:

[0435] The terminal uses data received from the server to analyze traffic conditions and assess the distance and speed between vehicles, as well as the potential collision risk at intersections. AI algorithms are used to improve the accuracy of the analysis results.

[0436] Step 3:

[0437] The server makes decisions based on the analysis results. This includes determining vehicle priorities and planning optimal routes and speed adjustments. In complex situations such as intersections, it determines priority and generates appropriate instructions.

[0438] Step 4:

[0439] The server generates specific instructions for each vehicle based on the decisions made and distributes them to the terminals. These instructions may include instructions to adjust speed, change direction, or stop temporarily.

[0440] Step 5:

[0441] The terminal relays the received instructions to the vehicle's control system, ensuring appropriate operation. Based on these instructions, the vehicle autonomously adjusts its course and speed, driving in harmony with other vehicles and road users.

[0442] Step 6:

[0443] The terminal monitors data during vehicle operation and reports any abnormalities or unexpected events to the server in real time. The server then takes appropriate action based on this information.

[0444] Step 7:

[0445] The server accumulates operational data, evaluates the overall system performance, and uses this data for future improvements. Through this feedback process, the system learns and improves its performance.

[0446] (Example 1)

[0447] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0448] In modern transportation systems involving mobile vehicles, there is a demand for improved safety and efficient traffic flow. However, in situations where vehicles with different specifications and from different manufacturers coexist, technical challenges exist in achieving harmony between them. Furthermore, there is a need for methods to quickly respond to unpredictable situations during operation and provide optimal transportation strategies in real time.

[0449] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0450] In this invention, the server includes information gathering means for collecting information from multiple moving objects, analysis means for analyzing the movement situation based on the information, and means for proposing a new operation strategy using a generative AI model. This improves the harmony between different moving objects and enables safe and efficient traffic management while immediately adapting to the situation during operation.

[0451] A "mobile object" is a means of transport designed for public or private use, and includes vehicles, machinery, and other similar devices.

[0452] "Information gathering means" refers to a device or system for receiving and accumulating various types of data from a moving object.

[0453] "Analysis means" refers to devices or software used to process collected data and evaluate movement patterns and traffic conditions.

[0454] A "decision-making mechanism" is a device or process for formulating a movement strategy based on the analysis results and determining the action plan for each moving object.

[0455] "Instruction and distribution means" refers to a device or communication method for accurately transmitting a decided action plan to each mobile entity.

[0456] "Implementation means" refers to a device or program for actually controlling the movement of a mobile object in accordance with the instructions that have been delivered.

[0457] "Adjustment means" refers to a device or system for feeding back operational information and optimizing the operation of a moving object.

[0458] A "generative AI model" is an algorithm or model that uses machine learning techniques to analyze data and generate new operational strategies or solutions.

[0459] An "anomaly detection method" is a process or system for quickly identifying and responding to unexpected situations or problems during operation.

[0460] This invention is an operation management system that enables mutual cooperation between multiple mobile entities by utilizing AI technology. This system includes means for information gathering, analysis, decision-making, instruction distribution, implementation, coordination, and operation strategy proposal using a generated AI model.

[0461] In implementing the system, the server first collects information in real time from each moving object through hardware such as GPS sensors and LiDAR. This allows the server to obtain information on the object's position, speed, acceleration, and surrounding obstacles. Next, the server uses powerful analytical software to analyze the collected information. This includes AI algorithms for modeling traffic conditions and assessing potential risks.

[0462] Based on the analysis results, the terminal determines the priority between each moving object and the optimal route. During the decision-making process, the generating AI model proposes a new operational strategy through prompt messages. For example, in a scenario where an electric bus in an urban area automatically avoids congestion and reaches its destination along a designated route, an example of a prompt message might be, "How can an autonomous bus operate smoothly in a congested urban area?"

[0463] The user plays a role in feeding back information obtained during operation to the server, and the system uses this feedback to improve its performance. This allows the system to be continuously optimized, enabling safe and efficient operation management even in situations where mobile vehicles with different specifications and manufacturers are mixed.

[0464] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0465] Step 1:

[0466] The server's role is to collect information from each mobile object in real time. Specifically, the server uses GPS sensors and LiDAR to acquire location, velocity, acceleration, and information about surrounding obstacles. The input is sensor data transmitted from the mobile object, which is received and stored in the server as integrated information. The output is a complete dataset for analysis.

[0467] Step 2:

[0468] The server analyzes the collected data. Using analysis software, it models traffic conditions and performs risk assessments. Specifically, it utilizes AI algorithms to predict potential hazards and congestion based on sensor data. The input is the dataset obtained in step 1, and the output is the evaluation results and risk assessment report regarding traffic conditions.

[0469] Step 3:

[0470] The terminal makes decisions based on the analysis results sent from the server. Specifically, it processes the prioritization of mobile objects and the optimal travel route. At this stage, the generating AI model may propose an operation strategy based on the prompt message. The input is the analysis results from step 2 and the prompt message from the generating AI model. The output is specific travel instructions for each mobile object.

[0471] Step 4:

[0472] The server distributes the determined instructions to each terminal. Specifically, it transmits instruction information to the mobile device using a communication protocol and accurately conveys its contents. The input is the instruction content determined in step 3. The output is the action plan transmitted to the terminal.

[0473] Step 5:

[0474] The terminal controls the mobile object based on the instructions it receives. Specifically, it sends signals to the vehicle's control system and performs appropriate operations. The input is the instructions received in step 4. The output is the specific operation performed by the mobile object, such as adjustments to direction or speed.

[0475] Step 6:

[0476] The user feeds data generated during operation back to the server. Specifically, this involves reporting abnormal situations to the server and prompting system adjustments. Inputs are operational data and abnormal information obtained during operation. Outputs are system improvement suggestions for the next operation.

[0477] (Application Example 1)

[0478] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0479] With the spread of autonomous driving technology, it is necessary to properly manage traffic situations between autonomous vehicles and pedestrians to enhance safety. However, conventional technologies lack the means to provide users with detailed real-time traffic information, and warnings for unexpected situations and sudden lane changes are particularly insufficient. This creates a challenge in that safety around autonomous vehicles may be compromised.

[0480] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0481] In this invention, the server includes data collection means for collecting data from multiple moving objects, analysis means for analyzing the status of the moving objects, and information provision means for users to receive information on the movement of the moving objects and safe route information through visual devices. This makes it possible to notify users of the movement of moving objects in real time and to warn them in advance of unexpected situations.

[0482] "Mobile devices" refer to vehicles and equipment equipped with autonomous driving technology that operate in a variety of environments.

[0483] "Data collection means" refers to devices and methods for acquiring driving conditions and surrounding information from a moving object.

[0484] "Analysis means" refers to methods and functions for evaluating the status of a moving object based on collected data and making necessary adjustments or decisions.

[0485] "Decision-making tools" refer to algorithms and systems used to determine priorities and routes between moving objects based on analysis results.

[0486] "Instruction distribution means" refers to a system or technology that transmits determined instructions to each mobile unit to prompt them to act.

[0487] "Execution means" refers to functions and technologies for controlling a moving object based on the instructions given.

[0488] "Feedback mechanisms" refer to methods and functions for continuously recording operational status and using that information to improve the system.

[0489] "Information provision means" refers to a system that notifies users in real time of the movement of a moving object and information on safe routes.

[0490] "Warning generation means" refers to a function that issues a warning to the user in situations where danger is anticipated.

[0491] The system for implementing this invention primarily operates through cooperation between three parties: a server, a terminal, and a user.

[0492] First, the server collects data from multiple mobile vehicles equipped with autonomous driving technology. The hardware used includes sensors and communication devices mounted on the vehicles. The data includes the location, speed, direction of travel, and information about the surrounding environment of the mobile vehicles. The server processes this data in real time and is responsible for analyzing the behavior of the mobile vehicles.

[0493] After analysis, the server determines priorities among the moving objects and sets appropriate routes and speeds. This decision-making process utilizes AI-powered algorithms. The analysis results are transmitted to terminals via a command distribution system.

[0494] The terminal receives these instructions and specifically controls the movement of each mobile object. Furthermore, it provides information about safe routes to the user's visual devices, such as smartphones or smart glasses. In doing so, it also issues warnings via voice or vibration as needed.

[0495] A feedback function is also important. The operating status of the mobile devices is sent to the server in real time, and if an anomaly is detected, immediate action is taken. The collected data is used to improve the overall performance of the system and prevent malfunctions.

[0496] As a concrete example, let's assume a user is walking through a crowded urban area. In this case, the smart glasses visually display the movement of the autonomous vehicle ahead and guide the user to the sidewalk where they can safely avoid it. Furthermore, if the direction of the moving object suddenly changes, an immediate voice warning is issued.

[0497] Examples of prompts for the generated AI model include: "Design an app that provides real-time safe route information to nearby pedestrians based on traffic data from autonomous vehicles," and "How can a smart device track an autonomous vehicle safely operating in an urban area during rainy weather?" From these prompts, the AI ​​model generates a solution appropriate to the specific situation the user faces.

[0498] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0499] Step 1:

[0500] The server collects data from multiple moving objects. Inputs include position, speed, direction of travel, and surrounding environment information from sensors mounted on these objects. The server receives and centrally stores this data. This process forms a base data set for detailed real-time traffic conditions.

[0501] Step 2:

[0502] The server performs analysis based on the collected data. The input is the traffic data obtained in Step 1. The server uses an AI algorithm to predict the trajectory of moving objects and assess collision risk, determining the priority of each object and the optimal route. This process generates information that enables rational decision-making.

[0503] Step 3:

[0504] The server generates instructions based on the analysis results and distributes them to the terminals. The input is the priority and route information of the mobile units determined in step 2. The server transmits these instructions to the control systems of each mobile unit through the instruction distribution system. This procedure provides specific guidance for the safe and efficient operation of the mobile units.

[0505] Step 4:

[0506] The terminal receives instructions from the server and actually controls the mobile object. The input is the instruction sent in step 3. The terminal adjusts the movement of the mobile object, changing the direction of travel, speed, and stopping position as needed. This control ensures safety and efficiency during actual operation.

[0507] Step 5:

[0508] The terminal provides information to the user's visual device. The input is the instruction information obtained in step 3. The terminal displays the movement of surrounding objects and safe route information on the user's smart device, and also issues warnings via voice and vibration as needed. This operation enables the provision of real-time safety information to the user.

[0509] Step 6:

[0510] The terminal provides feedback on the operational status at the site and transmits it to the server. The input is the latest operational data from the sensors of the moving vehicle. The terminal analyzes the data during operation and immediately reports any anomalies to the server. This feedback helps to improve the entire system and prevent accidents.

[0511] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0512] This invention is a cooperative operation management system that combines an emotion engine, and by linking the emotional states of multiple vehicles and users, it realizes more comfortable and safer travel. The system has a configuration that includes data collection means, analysis means, decision-making means, instruction distribution means, execution means, and feedback means, as well as an emotion engine.

[0513] (Data collection)

[0514] First, the server collects driving data from multiple vehicles. This data includes GPS location information, speed, road conditions, information about surrounding vehicles, and the user's vital signs (heart rate, facial expression, voice, etc.) obtained from sensors inside the vehicle.

[0515] (emotional analysis)

[0516] After collecting data, the device uses an emotion engine to analyze the user's emotional state based on their vital signs. For example, if it determines that the user is experiencing stress, it will suggest adjustments to operational support and environmental settings that correspond to that emotion.

[0517] (decision making)

[0518] The server combines analyzed emotions with driving data to make optimal decisions that allow the user to continue their journey comfortably and safely. In doing so, it takes into account surrounding traffic conditions and emotional state, and makes decisions to change the driving route or make temporary stops as needed.

[0519] (Instruction distribution)

[0520] The server generates instructions based on decisions and distributes them to each vehicle via terminals. These instructions include speed limits for vehicles, changes in direction, and automatic adjustments to environmental settings (such as music changes and lighting adjustments).

[0521] (Operation carried out)

[0522] The system controls the vehicle based on instructions received by the terminal, achieving operation that is adjusted to the user's emotional state. In particular, the goal is to create an operation that puts the user in a relaxed state.

[0523] (Feedback and Learning)

[0524] The server accumulates user emotion data and vehicle history data observed during operation, and uses this data through an emotion engine to help with future decision-making. This refines the entire system.

[0525] For example, if a user boards a vehicle while stressed after a meeting, the emotional engine will detect this and suggest relaxing lighting settings and classical music to provide a comfortable travel experience. In this way, the present invention aims to promote a more comfortable travel environment by considering not only the safety of travel but also the user's emotional state.

[0526] The following describes the processing flow.

[0527] Step 1:

[0528] The server collects real-time driving data and user vital signs from each vehicle. The data is acquired via sensors and includes GPS information, vehicle speed, and user heart rate and facial expression data.

[0529] Step 2:

[0530] The device uses an emotion engine to analyze the user's emotional state based on biometric data received from the server. Here, an AI algorithm is used to determine whether the user is relaxed or stressed.

[0531] Step 3:

[0532] The server evaluates the current operating status by combining the analyzed emotional state with driving data. In addition, it makes decisions based on the user's emotional state and plans to optimize the route and adjust the vehicle environment to improve comfort during the ride.

[0533] Step 4:

[0534] The server generates specific instructions for the vehicle based on the decision. These include speed limits, changes in direction, and adjustments to the operating environment (e.g., music selection, air conditioning settings, etc.).

[0535] Step 5:

[0536] The terminal distributes the generated instructions to each vehicle, and the vehicle's operation is controlled based on these instructions. This enables control that responds to the user's emotional state, providing a more comfortable riding experience.

[0537] Step 6:

[0538] The terminals and servers monitor the emotional state of users during operation and the results of environmental adjustments in real time, making immediate adjustments when necessary. This continuously improves user satisfaction.

[0539] Step 7:

[0540] After each operation, the server stores all operational data and user sentiment data, and uses AI to automatically learn areas for improvement for future operations, thereby enhancing the overall system functionality.

[0541] (Example 2)

[0542] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0543] In modern automotive technology, accurately understanding the emotional state of passengers and formulating a corresponding operational plan is difficult. This can lead to situations where passengers experience stress or where safety is not adequately ensured. Furthermore, it is difficult to respond quickly to unforeseen circumstances that occur during operation and to continuously improve the quality of service.

[0544] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0545] In this invention, the server includes information gathering means for collecting information from multiple mechanical devices, analysis means for analyzing the user's emotional state, and decision-making means for generating an optimal travel plan. This makes it possible to create a comfortable and safe travel plan that is tailored to the user's emotional state.

[0546] "Information gathering means" refers to a function that collects information in real time from multiple machines and devices and uses it in subsequent processing.

[0547] "Analysis means" refers to a function that identifies the emotional state of users from collected information and provides information necessary for optimizing operations through data analysis.

[0548] "Decision-making tool" refers to a function that generates the most efficient and comfortable travel plan for users based on analyzed information.

[0549] "Instruction distribution means" refers to a function that accurately and quickly transmits the generated operation plan to each machine and device, ensuring that the devices operate according to the plan.

[0550] "Execution means" refers to the function that controls the machine equipment to operate appropriately based on the operation plan received from the instruction distribution means.

[0551] "Feedback mechanisms" refer to functions that collect and evaluate operational results and user sentiment data, and utilize this information to continuously improve operational efficiency.

[0552] This invention is a system that utilizes an emotion analysis engine to understand the emotional state of users and, based on that understanding, achieve optimal vehicle operation. This system is designed to provide users with a comfortable and safe travel experience through data collection, analysis, decision-making, instruction distribution, operation execution, and feedback processing.

[0553] In this system, a server collects information from multiple mechanical devices (e.g., autonomous vehicles and smart devices). Information is collected using vehicle sensors (e.g., GPS, advanced driver assistance systems, cameras, microphones) to obtain location information, speed data, surrounding environment information, and the user's vital signs (heart rate, facial expressions, voice analysis, etc.).

[0554] Next, the device uses this data to activate its emotion analysis engine and analyze the user's emotional state. This analysis process uses software to process and analyze data, determining whether the user is stressed or relaxed. The results of the emotion analysis provide crucial input for improving the user experience.

[0555] The server uses analyzed emotional data and physical operational data to generate an optimal operational plan and distributes instructions to each vehicle. Network communication is used for instruction distribution, and the necessary actions are taken by the vehicles via terminals.

[0556] As a concrete example, consider a scenario where the entire system is in a vehicle after a stressful meeting. In this situation, the terminal activates an emotion analysis engine and recommends relaxing music and lighting settings based on the user's stress level.

[0557] Furthermore, the system utilizes a generative AI model to create prompt messages, improving the overall operational efficiency of the system. An example of a prompt message used is user feedback such as, "If the user is experiencing stress, please explain how to adjust the vehicle's environment."

[0558] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0559] Step 1:

[0560] The server collects information. Specifically, it receives sensor data from vehicles and smart devices. The input data includes GPS location information, speed, surrounding environment information, and the user's vital signs (e.g., heart rate, facial expression, voice). By aggregating this data, a basic dataset necessary for subsequent analysis is generated.

[0561] Step 2:

[0562] The device performs emotion analysis based on collected data. It uses data such as vital signs received from a server as input. An emotion engine analyzes the data to evaluate the user's emotional state. For example, it can determine stress levels from voice tone and interpret emotions by analyzing facial expressions. The output is an emotional state determination result, such as "relaxed" or "stressed."

[0563] Step 3:

[0564] The server integrates sentiment analysis results with physical operation data to make decisions. Input data includes sentiment analysis results and vehicle operation status data. An algorithm is used to create an optimal operation plan, and suggested adjustments include environmental settings (music, lighting) and changes to the driving route. The output is specific operation instructions.

[0565] Step 4:

[0566] The server distributes the generated operational instructions through the instruction distribution system. The input is the operational instructions generated in step 3. These instructions are sent to each terminal via the network, ensuring that each vehicle operates according to plan. The output is the operational protocol applied to each vehicle.

[0567] Step 5:

[0568] The terminal controls the vehicle based on the instructions it receives. The input is the operational instructions received from the server. By using actual feedback to adjust each system (e.g., navigation, infotainment) appropriately, a comfortable travel environment is provided to the user. The output is the result of the operation in accordance with the instructions.

[0569] Step 6:

[0570] The server aggregates operational data and performs feedback and learning. Inputs include user sentiment data and operational history collected during operation. This data is accumulated and used to train the sentiment engine for future decision-making. Outputs are improved operational algorithms and new learning models.

[0571] (Application Example 2)

[0572] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0573] Currently, most autonomous driving systems prioritize safety and efficiency above all else, with little consideration given to passengers' emotional state or comfort. This leads to problems such as operation in stressful environments and a disregard for passengers' emotional state. Therefore, there is a need for systems that can flexibly respond to diverse traffic conditions and provide comfortable operation while considering passengers' emotional state.

[0574] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0575] In this invention, the server includes information gathering means for collecting information from multiple mobile devices, analysis means for analyzing road conditions, decision-making means for realizing coordination between mobile devices, environment adjustment means for optimizing environmental settings according to emotional states, and feedback means for improving operations. This makes it possible to provide a comfortable and safe mobile environment that takes into account the emotional state of the user.

[0576] "Mobile devices" is a general term for means of transportation on land, sea, or air, and is a machine equipped with an autonomous driving function.

[0577] "Information gathering means" refers to means including sensors and devices for collecting data related to mobile devices and users.

[0578] "Analysis means" refers to a computer system that processes data obtained through information gathering means to analyze traffic conditions and user status.

[0579] A "decision-making tool" is a tool equipped with a processor or algorithm to determine appropriate actions or adjustments based on analysis results and to generate those instructions.

[0580] "Instruction distribution means" refers to communication infrastructure or systems for transmitting instructions generated by decision-making means to mobile devices.

[0581] "Execution means" refers to a mechanical or electronic device that receives instructions from an instruction distribution means, controls a mobile device, and performs an action.

[0582] "Environmental adjustment means" refers to devices that optimize the in-vehicle environment according to the user's emotional state, and are means of adjusting elements such as sound, lighting, and temperature.

[0583] "Feedback mechanisms" refer to procedures and devices used to improve the operating system based on data collected during operation.

[0584] The system of this invention analyzes the emotional state of users and generates corresponding operational instructions in order to improve the comfort and safety of mobile devices. Specifically, a server plays a crucial role in aggregating and analyzing information from mobile devices and providing appropriate instructions to each device, thereby maximizing the overall efficiency of travel.

[0585] The server collects location information, speed, road conditions, and user vital signs (heart rate, facial expressions, voice, etc.) data through sensors mounted on multiple mobile devices. This is implemented as a means of information gathering.

[0586] Next, the server uses analysis tools to process the collected data and analyze traffic conditions and the emotional state of users. Here, an emotion analysis engine (e.g., Emotion-SDK) is used to detect the user's stress level, etc.

[0587] Based on the analysis results, the decision-making system generates optimal instructions to ensure safe and comfortable travel. These instructions include adjustments to the route and in-vehicle environmental settings (music, lighting, etc.).

[0588] The server distributes the generated instructions to each mobile device via the instruction distribution means. The mobile devices receive these instructions and perform specific actions or environmental adjustments using the execution means.

[0589] Through feedback mechanisms, the system learns from data collected during operation and further optimizes future operations. This accumulated data is continuously analyzed and used to improve the overall accuracy of the system.

[0590] As a concrete example, consider a scenario where a user is experiencing increased stress as they head home after a full day of meetings. In this case, the server suggests relaxing music and gentle lighting settings. This allows the user to enjoy a pleasant travel experience.

[0591] An example of a prompt message might be: "Analyze the current user's emotional state and suggest a driving route and in-car environment that will reduce stress and improve comfort."

[0592] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0593] Step 1:

[0594] The server acquires location information, speed, road conditions, and user vital signs (heart rate, facial expressions, voice, etc.) from multiple mobile devices via information gathering means. This input data is collected in real time through sensors and communication networks and stored as an initial dataset.

[0595] Step 2:

[0596] The server processes the collected data using analytical tools to analyze traffic conditions and the emotional state of users. This analysis normalizes the data and uses an emotion analysis engine (e.g., Emotion-SDK) to determine stress levels. Input data includes vital signs and voice analysis results, and the server outputs the user's overall emotional state accordingly.

[0597] Step 3:

[0598] The server generates optimal operational instructions using decision-making mechanisms based on the analysis results. This process considers the analyzed emotional state and traffic information, and, if necessary, decides to change the route or adjust the in-vehicle environment settings. This process takes emotional state and traffic analysis results as input and provides specific instructions as output.

[0599] Step 4:

[0600] The server distributes the generated operation instructions to each mobile device through the instruction distribution means. In this distribution step, the generated instructions are transmitted to each mobile device in real time via the communication network. Instructions from the decision-making means are used as input, and the instructions are output as notifications to the mobile devices.

[0601] Step 5:

[0602] Each terminal controls the vehicle based on the instructions it receives and adjusts its operation using execution methods. Specifically, it adjusts speed, changes direction of travel, and changes settings for the in-vehicle environment (music, lighting). The input for this step is instructions from the server, and the output is a change in the state of the moving device.

[0603] Step 6:

[0604] The server monitors operational status through feedback mechanisms and continuously learns and improves the system based on the collected data. In this process, it analyzes errors that occur during execution and changes in sentiment data, gaining insights to improve future operations. Inputs are various data after execution, and outputs are updated analysis algorithms and feedback on improvement measures.

[0605] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0606] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0607] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0608] [Fourth Embodiment]

[0609] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0610] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0611] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0612] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0613] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0614] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0615] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0616] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0617] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0618] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0619] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0620] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0621] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0622] The present invention provides an operation management system that enables mutual cooperation between multiple vehicles using AI. This system mainly consists of data collection means, analysis means, decision-making means, instruction distribution means, execution means, and feedback means. This provides an environment in which autonomous vehicles and automated transport robots can safely coexist on public roads.

[0623] In this system, the server first collects driving data from each vehicle through sensors and communication devices. This includes the vehicle's position, speed, acceleration, and information about surrounding obstacles and other vehicles. Next, the server analyzes the collected data in real time on terminals to assess traffic conditions and potential risks.

[0624] Based on the analysis results, the server determines the priority order between vehicles and makes decisions regarding their routes and speed adjustments. The instructions generated by the server are then sent to each terminal, which distribute them to the vehicles. For example, at an intersection, the server determines the order in which each vehicle should pass and issues instructions to ensure safe passage based on that order.

[0625] The terminal controls the vehicle's operation based on received instructions. This control includes adjusting the vehicle's direction, speed, and stopping position. Furthermore, operational data is fed back to the server, and any abnormalities detected are promptly addressed. The feedback data is also used to improve the system's performance for future operations.

[0626] As a concrete example, when an autonomous bus travels through a congested urban area, it can coordinate with other automated transport robots and traffic signals to safely reach its destination along a designated route. This ensures the safety of all road users and creates a smooth flow of traffic.

[0627] Thus, the present invention provides an environment in which a variety of autonomous vehicles can operate safely on public roads, and in particular ensures traffic safety in situations where vehicles with different specifications and from different manufacturers are mixed together.

[0628] The following describes the processing flow.

[0629] Step 1:

[0630] The server collects data in real time from each vehicle traveling on public roads. This data includes the vehicle's location, speed, and information about surrounding obstacles. This data collection is performed through sensors and communication devices mounted on the vehicles.

[0631] Step 2:

[0632] The terminal uses data received from the server to analyze traffic conditions and assess the distance and speed between vehicles, as well as the potential collision risk at intersections. AI algorithms are used to improve the accuracy of the analysis results.

[0633] Step 3:

[0634] The server makes decisions based on the analysis results. This includes determining vehicle priorities and planning optimal routes and speed adjustments. In complex situations such as intersections, it determines priority and generates appropriate instructions.

[0635] Step 4:

[0636] The server generates specific instructions for each vehicle based on the decisions made and distributes them to the terminals. These instructions may include instructions to adjust speed, change direction, or stop temporarily.

[0637] Step 5:

[0638] The terminal relays the received instructions to the vehicle's control system, ensuring appropriate operation. Based on these instructions, the vehicle autonomously adjusts its course and speed, driving in harmony with other vehicles and road users.

[0639] Step 6:

[0640] The terminal monitors data during vehicle operation and reports any abnormalities or unexpected events to the server in real time. The server then takes appropriate action based on this information.

[0641] Step 7:

[0642] The server accumulates operational data, evaluates the overall system performance, and uses this data for future improvements. Through this feedback process, the system learns and improves its performance.

[0643] (Example 1)

[0644] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0645] In modern transportation systems involving mobile vehicles, there is a demand for improved safety and efficient traffic flow. However, in situations where vehicles with different specifications and from different manufacturers coexist, technical challenges exist in achieving harmony between them. Furthermore, there is a need for methods to quickly respond to unpredictable situations during operation and provide optimal transportation strategies in real time.

[0646] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0647] In this invention, the server includes information gathering means for collecting information from multiple moving objects, analysis means for analyzing the movement situation based on the information, and means for proposing a new operation strategy using a generative AI model. This improves the harmony between different moving objects and enables safe and efficient traffic management while immediately adapting to the situation during operation.

[0648] A "mobile object" is a means of transport designed for public or private use, and includes vehicles, machinery, and other similar devices.

[0649] "Information gathering means" refers to a device or system for receiving and accumulating various types of data from a moving object.

[0650] "Analysis means" refers to devices or software used to process collected data and evaluate movement patterns and traffic conditions.

[0651] A "decision-making mechanism" is a device or process for formulating a movement strategy based on the analysis results and determining the action plan for each moving object.

[0652] "Instruction and distribution means" refers to a device or communication method for accurately transmitting a decided action plan to each mobile entity.

[0653] "Implementation means" refers to a device or program for actually controlling the movement of a mobile object in accordance with the instructions that have been delivered.

[0654] "Adjustment means" refers to a device or system for feeding back operational information and optimizing the operation of a moving object.

[0655] A "generative AI model" is an algorithm or model that uses machine learning techniques to analyze data and generate new operational strategies or solutions.

[0656] An "anomaly detection method" is a process or system for quickly identifying and responding to unexpected situations or problems during operation.

[0657] This invention is an operation management system that enables mutual cooperation between multiple mobile entities by utilizing AI technology. This system includes means for information gathering, analysis, decision-making, instruction distribution, implementation, coordination, and operation strategy proposal using a generated AI model.

[0658] In implementing the system, the server first collects information in real time from each moving object through hardware such as GPS sensors and LiDAR. This allows the server to obtain information on the object's position, speed, acceleration, and surrounding obstacles. Next, the server uses powerful analytical software to analyze the collected information. This includes AI algorithms for modeling traffic conditions and assessing potential risks.

[0659] Based on the analysis results, the terminal determines the priority between each moving object and the optimal route. During the decision-making process, the generating AI model proposes a new operational strategy through prompt messages. For example, in a scenario where an electric bus in an urban area automatically avoids congestion and reaches its destination along a designated route, an example of a prompt message might be, "How can an autonomous bus operate smoothly in a congested urban area?"

[0660] The user plays a role in feeding back information obtained during operation to the server, and the system uses this feedback to improve its performance. This allows the system to be continuously optimized, enabling safe and efficient operation management even in situations where mobile vehicles with different specifications and manufacturers are mixed.

[0661] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0662] Step 1:

[0663] The server's role is to collect information from each mobile object in real time. Specifically, the server uses GPS sensors and LiDAR to acquire location, velocity, acceleration, and information about surrounding obstacles. The input is sensor data transmitted from the mobile object, which is received and stored in the server as integrated information. The output is a complete dataset for analysis.

[0664] Step 2:

[0665] The server analyzes the collected data. Using analysis software, it models traffic conditions and performs risk assessments. Specifically, it utilizes AI algorithms to predict potential hazards and congestion based on sensor data. The input is the dataset obtained in step 1, and the output is the evaluation results and risk assessment report regarding traffic conditions.

[0666] Step 3:

[0667] The terminal makes decisions based on the analysis results sent from the server. Specifically, it processes the prioritization of mobile objects and the optimal travel route. At this stage, the generating AI model may propose an operation strategy based on the prompt message. The input is the analysis results from step 2 and the prompt message from the generating AI model. The output is specific travel instructions for each mobile object.

[0668] Step 4:

[0669] The server distributes the determined instructions to each terminal. Specifically, it transmits instruction information to the mobile device using a communication protocol and accurately conveys its contents. The input is the instruction content determined in step 3. The output is the action plan transmitted to the terminal.

[0670] Step 5:

[0671] The terminal controls the mobile object based on the instructions it receives. Specifically, it sends signals to the vehicle's control system and performs appropriate operations. The input is the instructions received in step 4. The output is the specific operation performed by the mobile object, such as adjustments to direction or speed.

[0672] Step 6:

[0673] The user feeds data generated during operation back to the server. Specifically, this involves reporting abnormal situations to the server and prompting system adjustments. Inputs are operational data and abnormal information obtained during operation. Outputs are system improvement suggestions for the next operation.

[0674] (Application Example 1)

[0675] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0676] With the spread of autonomous driving technology, it is necessary to properly manage traffic situations between autonomous vehicles and pedestrians to enhance safety. However, conventional technologies lack the means to provide users with detailed real-time traffic information, and warnings for unexpected situations and sudden lane changes are particularly insufficient. This creates a challenge in that safety around autonomous vehicles may be compromised.

[0677] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0678] In this invention, the server includes data collection means for collecting data from multiple moving objects, analysis means for analyzing the status of the moving objects, and information provision means for users to receive information on the movement of the moving objects and safe route information through visual devices. This makes it possible to notify users of the movement of moving objects in real time and to warn them in advance of unexpected situations.

[0679] "Mobile devices" refer to vehicles and equipment equipped with autonomous driving technology that operate in a variety of environments.

[0680] "Data collection means" refers to devices and methods for acquiring driving conditions and surrounding information from a moving object.

[0681] "Analysis means" refers to methods and functions for evaluating the status of a moving object based on collected data and making necessary adjustments or decisions.

[0682] "Decision-making tools" refer to algorithms and systems used to determine priorities and routes between moving objects based on analysis results.

[0683] "Instruction distribution means" refers to a system or technology that transmits determined instructions to each mobile unit to prompt them to act.

[0684] "Execution means" refers to functions and technologies for controlling a moving object based on the instructions given.

[0685] "Feedback mechanisms" refer to methods and functions for continuously recording operational status and using that information to improve the system.

[0686] "Information provision means" refers to a system that notifies users in real time of the movement of a moving object and information on safe routes.

[0687] "Warning generation means" refers to a function that issues a warning to the user in situations where danger is anticipated.

[0688] The system for implementing this invention primarily operates through cooperation between three parties: a server, a terminal, and a user.

[0689] First, the server collects data from multiple mobile vehicles equipped with autonomous driving technology. The hardware used includes sensors and communication devices mounted on the vehicles. The data includes the location, speed, direction of travel, and information about the surrounding environment of the mobile vehicles. The server processes this data in real time and is responsible for analyzing the behavior of the mobile vehicles.

[0690] After analysis, the server determines priorities among the moving objects and sets appropriate routes and speeds. This decision-making process utilizes AI-powered algorithms. The analysis results are transmitted to terminals via a command distribution system.

[0691] The terminal receives these instructions and specifically controls the movement of each mobile object. Furthermore, it provides information about safe routes to the user's visual devices, such as smartphones or smart glasses. In doing so, it also issues warnings via voice or vibration as needed.

[0692] A feedback function is also important. The operating status of the mobile devices is sent to the server in real time, and if an anomaly is detected, immediate action is taken. The collected data is used to improve the overall performance of the system and prevent malfunctions.

[0693] As a concrete example, let's assume a user is walking through a crowded urban area. In this case, the smart glasses visually display the movement of the autonomous vehicle ahead and guide the user to the sidewalk where they can safely avoid it. Furthermore, if the direction of the moving object suddenly changes, an immediate voice warning is issued.

[0694] Examples of prompts for the generated AI model include: "Design an app that provides real-time safe route information to nearby pedestrians based on traffic data from autonomous vehicles," and "How can a smart device track an autonomous vehicle safely operating in an urban area during rainy weather?" From these prompts, the AI ​​model generates a solution appropriate to the specific situation the user faces.

[0695] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0696] Step 1:

[0697] The server collects data from multiple moving objects. Inputs include position, speed, direction of travel, and surrounding environment information from sensors mounted on these objects. The server receives and centrally stores this data. This process forms a base data set for detailed real-time traffic conditions.

[0698] Step 2:

[0699] The server performs analysis based on the collected data. The input is the traffic data obtained in Step 1. The server uses an AI algorithm to predict the trajectory of moving objects and assess collision risk, determining the priority of each object and the optimal route. This process generates information that enables rational decision-making.

[0700] Step 3:

[0701] The server generates instructions based on the analysis results and distributes them to the terminals. The input is the priority and route information of the mobile units determined in step 2. The server transmits these instructions to the control systems of each mobile unit through the instruction distribution system. This procedure provides specific guidance for the safe and efficient operation of the mobile units.

[0702] Step 4:

[0703] The terminal receives instructions from the server and actually controls the mobile object. The input is the instruction sent in step 3. The terminal adjusts the movement of the mobile object, changing the direction of travel, speed, and stopping position as needed. This control ensures safety and efficiency during actual operation.

[0704] Step 5:

[0705] The terminal provides information to the user's visual device. The input is the instruction information obtained in step 3. The terminal displays the movement of surrounding objects and safe route information on the user's smart device, and also issues warnings via voice and vibration as needed. This operation enables the provision of real-time safety information to the user.

[0706] Step 6:

[0707] The terminal provides feedback on the operational status at the site and transmits it to the server. The input is the latest operational data from the sensors of the moving vehicle. The terminal analyzes the data during operation and immediately reports any anomalies to the server. This feedback helps to improve the entire system and prevent accidents.

[0708] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0709] This invention is a cooperative operation management system that combines an emotion engine, and by linking the emotional states of multiple vehicles and users, it realizes more comfortable and safer travel. The system has a configuration that includes data collection means, analysis means, decision-making means, instruction distribution means, execution means, and feedback means, as well as an emotion engine.

[0710] (Data collection)

[0711] First, the server collects driving data from multiple vehicles. This data includes GPS location information, speed, road conditions, information about surrounding vehicles, and the user's vital signs (heart rate, facial expression, voice, etc.) obtained from sensors inside the vehicle.

[0712] (emotional analysis)

[0713] After collecting data, the device uses an emotion engine to analyze the user's emotional state based on their vital signs. For example, if it determines that the user is experiencing stress, it will suggest adjustments to operational support and environmental settings that correspond to that emotion.

[0714] (decision making)

[0715] The server combines analyzed emotions with driving data to make optimal decisions that allow the user to continue their journey comfortably and safely. In doing so, it takes into account surrounding traffic conditions and emotional state, and makes decisions to change the driving route or make temporary stops as needed.

[0716] (Instruction distribution)

[0717] The server generates instructions based on decisions and distributes them to each vehicle via terminals. These instructions include speed limits for vehicles, changes in direction, and automatic adjustments to environmental settings (such as music changes and lighting adjustments).

[0718] (Operation carried out)

[0719] The system controls the vehicle based on instructions received by the terminal, achieving operation that is adjusted to the user's emotional state. In particular, the goal is to create an operation that puts the user in a relaxed state.

[0720] (Feedback and Learning)

[0721] The server accumulates user emotion data and vehicle history data observed during operation, and uses this data through an emotion engine to help with future decision-making. This refines the entire system.

[0722] For example, if a user boards a vehicle while stressed after a meeting, the emotional engine will detect this and suggest relaxing lighting settings and classical music to provide a comfortable travel experience. In this way, the present invention aims to promote a more comfortable travel environment by considering not only the safety of travel but also the user's emotional state.

[0723] The following describes the processing flow.

[0724] Step 1:

[0725] The server collects real-time driving data and user vital signs from each vehicle. The data is acquired via sensors and includes GPS information, vehicle speed, and user heart rate and facial expression data.

[0726] Step 2:

[0727] The device uses an emotion engine to analyze the user's emotional state based on biometric data received from the server. Here, an AI algorithm is used to determine whether the user is relaxed or stressed.

[0728] Step 3:

[0729] The server evaluates the current operating status by combining the analyzed emotional state with driving data. In addition, it makes decisions based on the user's emotional state and plans to optimize the route and adjust the vehicle environment to improve comfort during the ride.

[0730] Step 4:

[0731] The server generates specific instructions for the vehicle based on the decision. These include speed limits, changes in direction, and adjustments to the operating environment (e.g., music selection, air conditioning settings, etc.).

[0732] Step 5:

[0733] The terminal distributes the generated instructions to each vehicle, and the vehicle's operation is controlled based on these instructions. This enables control that responds to the user's emotional state, providing a more comfortable riding experience.

[0734] Step 6:

[0735] The terminals and servers monitor the emotional state of users during operation and the results of environmental adjustments in real time, making immediate adjustments when necessary. This continuously improves user satisfaction.

[0736] Step 7:

[0737] After each operation, the server stores all operational data and user sentiment data, and uses AI to automatically learn areas for improvement for future operations, thereby enhancing the overall system functionality.

[0738] (Example 2)

[0739] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0740] In modern automotive technology, accurately understanding the emotional state of passengers and formulating a corresponding operational plan is difficult. This can lead to situations where passengers experience stress or where safety is not adequately ensured. Furthermore, it is difficult to respond quickly to unforeseen circumstances that occur during operation and to continuously improve the quality of service.

[0741] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0742] In this invention, the server includes information gathering means for collecting information from multiple mechanical devices, analysis means for analyzing the user's emotional state, and decision-making means for generating an optimal travel plan. This makes it possible to create a comfortable and safe travel plan that is tailored to the user's emotional state.

[0743] "Information gathering means" refers to a function that collects information in real time from multiple machines and devices and uses it in subsequent processing.

[0744] "Analysis means" refers to a function that identifies the emotional state of users from collected information and provides information necessary for optimizing operations through data analysis.

[0745] "Decision-making tool" refers to a function that generates the most efficient and comfortable travel plan for users based on analyzed information.

[0746] "Instruction distribution means" refers to a function that accurately and quickly transmits the generated operation plan to each machine and device, ensuring that the devices operate according to the plan.

[0747] "Execution means" refers to the function that controls the machine equipment to operate appropriately based on the operation plan received from the instruction distribution means.

[0748] "Feedback mechanisms" refer to functions that collect and evaluate operational results and user sentiment data, and utilize this information to continuously improve operational efficiency.

[0749] This invention is a system that utilizes an emotion analysis engine to understand the emotional state of users and, based on that understanding, achieve optimal vehicle operation. This system is designed to provide users with a comfortable and safe travel experience through data collection, analysis, decision-making, instruction distribution, operation execution, and feedback processing.

[0750] In this system, a server collects information from multiple mechanical devices (e.g., autonomous vehicles and smart devices). Information is collected using vehicle sensors (e.g., GPS, advanced driver assistance systems, cameras, microphones) to obtain location information, speed data, surrounding environment information, and the user's vital signs (heart rate, facial expressions, voice analysis, etc.).

[0751] Next, the device uses this data to activate its emotion analysis engine and analyze the user's emotional state. This analysis process uses software to process and analyze data, determining whether the user is stressed or relaxed. The results of the emotion analysis provide crucial input for improving the user experience.

[0752] The server uses analyzed emotional data and physical operational data to generate an optimal operational plan and distributes instructions to each vehicle. Network communication is used for instruction distribution, and the necessary actions are taken by the vehicles via terminals.

[0753] As a concrete example, consider a scenario where the entire system is in a vehicle after a stressful meeting. In this situation, the terminal activates an emotion analysis engine and recommends relaxing music and lighting settings based on the user's stress level.

[0754] Furthermore, the system utilizes a generative AI model to create prompt messages, improving the overall operational efficiency of the system. An example of a prompt message used is user feedback such as, "If the user is experiencing stress, please explain how to adjust the vehicle's environment."

[0755] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0756] Step 1:

[0757] The server collects information. Specifically, it receives sensor data from vehicles and smart devices. The input data includes GPS location information, speed, surrounding environment information, and the user's vital signs (e.g., heart rate, facial expression, voice). By aggregating this data, a basic dataset necessary for subsequent analysis is generated.

[0758] Step 2:

[0759] The device performs emotion analysis based on collected data. It uses data such as vital signs received from a server as input. An emotion engine analyzes the data to evaluate the user's emotional state. For example, it can determine stress levels from voice tone and interpret emotions by analyzing facial expressions. The output is an emotional state determination result, such as "relaxed" or "stressed."

[0760] Step 3:

[0761] The server integrates sentiment analysis results with physical operation data to make decisions. Input data includes sentiment analysis results and vehicle operation status data. An algorithm is used to create an optimal operation plan, and suggested adjustments include environmental settings (music, lighting) and changes to the driving route. The output is specific operation instructions.

[0762] Step 4:

[0763] The server distributes the generated operational instructions through the instruction distribution system. The input is the operational instructions generated in step 3. These instructions are sent to each terminal via the network, ensuring that each vehicle operates according to plan. The output is the operational protocol applied to each vehicle.

[0764] Step 5:

[0765] The terminal controls the vehicle based on the instructions it receives. The input is the operational instructions received from the server. By using actual feedback to adjust each system (e.g., navigation, infotainment) appropriately, a comfortable travel environment is provided to the user. The output is the result of the operation in accordance with the instructions.

[0766] Step 6:

[0767] The server aggregates operational data and performs feedback and learning. Inputs include user sentiment data and operational history collected during operation. This data is accumulated and used to train the sentiment engine for future decision-making. Outputs are improved operational algorithms and new learning models.

[0768] (Application Example 2)

[0769] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0770] Currently, most autonomous driving systems prioritize safety and efficiency above all else, with little consideration given to passengers' emotional state or comfort. This leads to problems such as operation in stressful environments and a disregard for passengers' emotional state. Therefore, there is a need for systems that can flexibly respond to diverse traffic conditions and provide comfortable operation while considering passengers' emotional state.

[0771] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0772] In this invention, the server includes information gathering means for collecting information from multiple mobile devices, analysis means for analyzing road conditions, decision-making means for realizing coordination between mobile devices, environment adjustment means for optimizing environmental settings according to emotional states, and feedback means for improving operations. This makes it possible to provide a comfortable and safe mobile environment that takes into account the emotional state of the user.

[0773] "Mobile devices" is a general term for means of transportation on land, sea, or air, and is a machine equipped with an autonomous driving function.

[0774] "Information gathering means" refers to means including sensors and devices for collecting data related to mobile devices and users.

[0775] "Analysis means" refers to a computer system that processes data obtained through information gathering means to analyze traffic conditions and user status.

[0776] A "decision-making tool" is a tool equipped with a processor or algorithm to determine appropriate actions or adjustments based on analysis results and to generate those instructions.

[0777] "Instruction distribution means" refers to communication infrastructure or systems for transmitting instructions generated by decision-making means to mobile devices.

[0778] "Execution means" refers to a mechanical or electronic device that receives instructions from an instruction distribution means, controls a mobile device, and performs an action.

[0779] "Environmental adjustment means" refers to devices that optimize the in-vehicle environment according to the user's emotional state, and are means of adjusting elements such as sound, lighting, and temperature.

[0780] "Feedback mechanisms" refer to procedures and devices used to improve the operating system based on data collected during operation.

[0781] The system of this invention analyzes the emotional state of users and generates corresponding operational instructions in order to improve the comfort and safety of mobile devices. Specifically, a server plays a crucial role in aggregating and analyzing information from mobile devices and providing appropriate instructions to each device, thereby maximizing the overall efficiency of travel.

[0782] The server collects location information, speed, road conditions, and user vital signs (heart rate, facial expressions, voice, etc.) data through sensors mounted on multiple mobile devices. This is implemented as a means of information gathering.

[0783] Next, the server uses analysis tools to process the collected data and analyze traffic conditions and the emotional state of users. Here, an emotion analysis engine (e.g., Emotion-SDK) is used to detect the user's stress level, etc.

[0784] Based on the analysis results, the decision-making system generates optimal instructions to ensure safe and comfortable travel. These instructions include adjustments to the route and in-vehicle environmental settings (music, lighting, etc.).

[0785] The server distributes the generated instructions to each mobile device via the instruction distribution means. The mobile devices receive these instructions and perform specific actions or environmental adjustments using the execution means.

[0786] Through feedback mechanisms, the system learns from data collected during operation and further optimizes future operations. This accumulated data is continuously analyzed and used to improve the overall accuracy of the system.

[0787] As a concrete example, consider a scenario where a user is experiencing increased stress as they head home after a full day of meetings. In this case, the server suggests relaxing music and gentle lighting settings. This allows the user to enjoy a pleasant travel experience.

[0788] An example of a prompt message might be: "Analyze the current user's emotional state and suggest a driving route and in-car environment that will reduce stress and improve comfort."

[0789] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0790] Step 1:

[0791] The server acquires location information, speed, road conditions, and user vital signs (heart rate, facial expressions, voice, etc.) from multiple mobile devices via information gathering means. This input data is collected in real time through sensors and communication networks and stored as an initial dataset.

[0792] Step 2:

[0793] The server processes the collected data using analytical tools to analyze traffic conditions and the emotional state of users. This analysis normalizes the data and uses an emotion analysis engine (e.g., Emotion-SDK) to determine stress levels. Input data includes vital signs and voice analysis results, and the server outputs the user's overall emotional state accordingly.

[0794] Step 3:

[0795] The server generates optimal operational instructions using decision-making mechanisms based on the analysis results. This process considers the analyzed emotional state and traffic information, and, if necessary, decides to change the route or adjust the in-vehicle environment settings. This process takes emotional state and traffic analysis results as input and provides specific instructions as output.

[0796] Step 4:

[0797] The server distributes the generated operation instructions to each mobile device through the instruction distribution means. In this distribution step, the generated instructions are transmitted to each mobile device in real time via the communication network. Instructions from the decision-making means are used as input, and the instructions are output as notifications to the mobile devices.

[0798] Step 5:

[0799] Each terminal controls the vehicle based on the instructions it receives and adjusts its operation using execution methods. Specifically, it adjusts speed, changes direction of travel, and changes settings for the in-vehicle environment (music, lighting). The input for this step is instructions from the server, and the output is a change in the state of the moving device.

[0800] Step 6:

[0801] The server monitors operational status through feedback mechanisms and continuously learns and improves the system based on the collected data. In this process, it analyzes errors that occur during execution and changes in sentiment data, gaining insights to improve future operations. Inputs are various data after execution, and outputs are updated analysis algorithms and feedback on improvement measures.

[0802] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0803] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0804] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0805] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0806] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0807] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0808] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0809] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0810] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0811] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0812] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0813] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0814] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0815] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0816] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0817] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0818] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0819] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0820] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0821] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0822] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0823] The following is further disclosed regarding the embodiments described above.

[0824] (Claim 1)

[0825] A data collection method for collecting data from multiple vehicles,

[0826] An analysis means for analyzing traffic conditions based on the aforementioned data,

[0827] Based on the aforementioned analysis results, a decision-making means for achieving cooperation between vehicles,

[0828] Instruction distribution means for distributing instructions generated by the decision-making means to each vehicle,

[0829] Each of the aforementioned vehicles includes an execution means for performing an action based on the instructions,

[0830] A feedback mechanism for receiving feedback on operational status and improving operations,

[0831] A system that includes this.

[0832] (Claim 2)

[0833] The system according to claim 1, wherein the instruction distribution means comprises means for performing traffic signal control based on priority.

[0834] (Claim 3)

[0835] The system according to claim 1, wherein the feedback means comprises an anomaly detection means for responding to unexpected situations during operation.

[0836] "Example 1"

[0837] (Claim 1)

[0838] Information gathering means for collecting information from multiple moving objects,

[0839] An analysis means for analyzing movement status based on the aforementioned information,

[0840] Based on the above analysis results, a decision-making means for achieving harmony between mobile bodies,

[0841] Instruction distribution means for distributing instructions generated by the aforementioned determination means to each mobile unit,

[0842] Each of the aforementioned moving bodies includes an implementation means for performing an operation based on the instruction,

[0843] A means of adjustment to improve operations by providing reverse feedback on operational status,

[0844] A means of proposing new operational strategies using generative AI models,

[0845] A system that includes this.

[0846] (Claim 2)

[0847] The system according to claim 1, wherein the instruction distribution means comprises means for performing movement signal control based on priority.

[0848] (Claim 3)

[0849] The system according to claim 1, wherein the adjustment means comprises an anomaly detection means for responding to unexpected situations during operation.

[0850] "Application Example 1"

[0851] (Claim 1)

[0852] A data collection means for collecting data from multiple mobile objects,

[0853] An analysis means for analyzing the status of a moving object based on the aforementioned data,

[0854] Based on the aforementioned analysis results, a decision-making means for realizing cooperation between mobile bodies,

[0855] Instruction distribution means for distributing instructions generated by the decision-making means to each mobile body,

[0856] Each of the aforementioned moving bodies includes an execution means for performing an action based on the instructions,

[0857] A feedback mechanism for receiving feedback on operational status and improving operations,

[0858] A means of providing information to allow users to receive information on the movement of moving objects and safe routes through visual devices,

[0859] Warning generation means for generating warnings regarding the direction of movement and safety of a moving object,

[0860] A system that includes this.

[0861] (Claim 2)

[0862] The system according to claim 1, wherein the instruction distribution means comprises means for performing traffic control based on priority.

[0863] (Claim 3)

[0864] The system according to claim 1, wherein the feedback means comprises an anomaly detection means for responding to unexpected situations during operation.

[0865] "Example 2 of combining an emotion engine"

[0866] (Claim 1)

[0867] Information gathering means for collecting information from multiple mechanical devices,

[0868] An analytical means for analyzing the user's emotional state based on the aforementioned information,

[0869] A decision-making means for generating an optimal travel plan for the user based on the aforementioned analysis results and driving information,

[0870] Instruction distribution means for distributing the movement plan generated by the decision-making means to each machine device,

[0871] Each of the aforementioned mechanical devices includes an execution means for performing an operation based on the instructions,

[0872] A feedback mechanism for receiving feedback on operational status and conducting learning that takes into account emotional states,

[0873] A system that includes this.

[0874] (Claim 2)

[0875] The system according to claim 1, comprising means for controlling traffic signals and setting environmental parameters based on priority.

[0876] (Claim 3)

[0877] The system according to claim 1, comprising an anomaly detection means for responding to unexpected situations during operation, and a means for improving user safety through emotion analysis.

[0878] "Application example 2 when combining with an emotional engine"

[0879] (Claim 1)

[0880] Information gathering means for collecting information from multiple mobile devices,

[0881] An analysis means for analyzing road conditions based on the aforementioned information,

[0882] Based on the aforementioned analysis results, a decision-making means for realizing coordination between mobile devices,

[0883] Instruction distribution means for distributing instructions generated by the decision-making means to each mobile device,

[0884] Each of the aforementioned mobile devices includes an execution means for performing an operation based on the instruction,

[0885] An environmental adjustment means for analyzing the emotional state of the user and optimizing the environmental settings according to that state,

[0886] A feedback mechanism for receiving feedback on operational status and improving operations,

[0887] A system that includes this.

[0888] (Claim 2)

[0889] The system according to claim 1, wherein the instruction distribution means comprises means for performing road signal control based on priority.

[0890] (Claim 3)

[0891] The system according to claim 1, wherein the feedback means comprises fault detection means for responding to unexpected situations during operation. [Explanation of symbols]

[0892] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A data collection means for collecting data from multiple mobile objects, An analysis means for analyzing the status of a moving object based on the aforementioned data, Based on the aforementioned analysis results, a decision-making means for realizing cooperation between mobile bodies, Instruction distribution means for distributing instructions generated by the decision-making means to each mobile body, Each of the aforementioned moving bodies includes an execution means for performing an action based on the instructions, A feedback mechanism for receiving feedback on operational status and improving operations, A means of providing information to allow users to receive information on the movement of moving objects and safe routes through visual devices, Warning generation means for generating warnings regarding the direction of movement and safety of a moving object, A system that includes this.

2. The system according to claim 1, wherein the instruction distribution means comprises means for performing traffic control based on priority.

3. The system according to claim 1, wherein the feedback means comprises an anomaly detection means for responding to unexpected situations during operation.

Images