System

Abstract

A system is provided.SOLUTION: A vehicle system having an automated driving function, comprising: a central management unit configured to collect operation information and calculate an operation route and a speed; an operation control unit configured to receive an instruction from the central management unit and control an operation of a vehicle; a monitoring unit configured to monitor a surrounding situation during driving and detect an abnormality; an abnormality notification unit configured to notify the central management unit when an abnormality is detected; and a vehicle on which a person who responds when an emergency occurs is on board.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Current autonomous driving technology has difficulty responding flexibly and quickly in emergency situations. As a result, there is a high possibility that safety will be compromised. In particular, Japan's railway system requires a rapid response in emergencies due to its high frequency and high density of service. Furthermore, the difficulty of securing drivers has led to a reduction in the number of trains in regional areas. It is an urgent task to resolve these issues. [Means for solving the problem]

[0005] The present invention provides a vehicle system with an automatic driving function. This system includes a central management means for collecting operation information and calculating operation routes and speeds, an operation control means for receiving instructions from the central management means and controlling the operation of the vehicle, and a monitoring means for monitoring the surrounding conditions and detecting abnormalities while the vehicle is in operation. Furthermore, the system includes an abnormality notification means for notifying the central management means when an abnormality is detected, and a vehicle equipped with personnel to respond in the event of an emergency, thereby improving emergency response capabilities.

[0006] Furthermore, the central control means includes a means for issuing an emergency stop command when an abnormality occurs and for notifying personnel of emergency response procedures, enabling prompt and appropriate responses even in emergencies. Furthermore, the operation control means includes a means for receiving new operation commands from the central control means and resuming operation, enabling smooth resumption of operation after the emergency is resolved.

[0007] "Autonomous driving function" refers to the function that enables a vehicle to operate autonomously without human operation.

[0008] A "vehicle system" is an overall mechanism in which multiple parts and means work together to operate a vehicle.

[0009] "Operation information" refers to all information related to operation, such as vehicle position, speed, destination, intermediate points, route, weather information, and track conditions.

[0010] The "central control means" is a central component that collects and analyzes operational information to calculate operational routes and speeds, and issues instructions necessary for operation.

[0011] The "operation control means" is a component that receives instructions from the central control means and controls the actual operation of the vehicle.

[0012] "Monitoring means" refers to devices such as sensors and cameras that continuously monitor the surrounding environment and conditions while the vehicle is in operation.

[0013] The "abnormality notification means" is a means for immediately notifying the central management means of information when an abnormality is detected by the monitoring means.

[0014] An "emergency" is an extraordinary situation that makes it difficult to continue normal operations, such as an obstruction on the tracks or an unexpected mechanical failure.

[0015] "Instructions" refer to specific operation routes and speed settings, emergency stop orders, etc. given by the central management means to the operation control means.

[0016] "Personnel" refers to the people on board the train who respond to emergencies, usually railway staff such as conductors. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention relates to a vehicle system with an automatic driving function, and in particular to a system that enables safe and efficient operation even in an emergency. Hereinafter, an embodiment of the present invention will be described in detail.

[0039] Overall system configuration

[0040] This system mainly consists of a central management means, operation control means, monitoring means, abnormality notification means, and vehicles carrying emergency response personnel (usually conductors). The central management means functions as a server, collecting and analyzing operation information and generating instructions. The operation control means functions as a train terminal, operating the vehicle based on instructions from the server. The monitoring means and abnormality notification means are composed of devices such as cameras and sensors built into the terminals.

[0041] Program processing

[0042] Server Processing

[0043] 1. Collection of operational information:

[0044] The server collects train operation information, such as the train's current location, speed, destination, and route information, from a database, as well as real-time weather and track condition data, and analyzes this information.

[0045] 2. Route and speed calculation:

[0046] The server uses AI algorithms to calculate the optimal route and speed based on the collected operational information, enabling efficient and safe operation.

[0047] 3. Generate and send instructions:

[0048] Based on the calculation results, the server generates and sends specific instructions on the driving route and speed to the driving control means (terminal).

[0049] Terminal (train) processing

[0050] 1. Receiving instructions:

[0051] The terminal receives the driving route and speed instructions sent from the server.

[0052] 2. Navigation control:

[0053] Based on the received instructions, the terminal controls the operation of the train through the automated driving system, specifically, driving at a specified speed along a specified route.

[0054] 3. Surrounding Area Monitoring:

[0055] The terminals are equipped with cameras and sensors as monitoring tools, and continuously monitor the surroundings while in operation, sending information to a server in real time.

[0056] 4. Anomaly detection and notification:

[0057] If the monitoring means detects an abnormality (for example, an obstacle on the tracks or an abnormal speed limit), the information is immediately sent to the server via the abnormality notification means.

[0058] User (conductor) role

[0059] 1. Monitoring normal operations:

[0060] During normal operation, the user (conductor) monitors the situation inside the train and checks for any problems. He or she patrols the train regularly to ensure safety.

[0061] 2. Emergency Response:

[0062] When the server notifies the conductor of an emergency, the conductor will act according to the appropriate emergency response procedures, such as explaining the situation to passengers, guiding them to an evacuation, and checking for and removing obstacles if possible.

[0063] 3. Reporting and Feedback:

[0064] After the emergency situation has been resolved, the conductor will report the situation in detail to the server, including details of the emergency response and current safety confirmation.

[0065] Specific examples

[0066] For example, consider the process that is performed when a train detects an obstacle on the tracks while in operation.

[0067] 1. Obstacle detection:

[0068] The device's camera detects obstacles on the tracks and sends that information to a server.

[0069] 2. Emergency stop instruction:

[0070] The server analyzes the obstacle information and immediately sends an emergency stop instruction to the terminal if it determines it is necessary.

[0071] 3. Performing an emergency stop:

[0072] The terminal receives the emergency stop command and activates the emergency brake to stop the train.

[0073] 4. Conductor's response:

[0074] The conductor will guide passengers and guide them to safety based on instructions from the server. In addition, if possible, the conductor will check for and remove obstacles.

[0075] 5. Service resumes:

[0076] After the obstacle is removed and safety is confirmed, the server calculates a new route and speed and sends it to the device, which receives the new instructions and resumes operation.

[0077] In this way, the vehicle system of the present invention can operate safely and efficiently both during normal operation and in emergency situations.

[0078] The processing flow will be explained below.

[0079] Server Processing

[0080] Step 1:

[0081] The server retrieves information about the train's current location, speed, destination, stops, and route from a database, as well as real-time weather and track condition data.

[0082] Step 2:

[0083] The server uses AI algorithms to calculate the optimal route and speed based on the collected operational information, records the calculation results, and generates instructions.

[0084] Step 3:

[0085] The server sends the generated instructions to the operation control means (terminal), which include detailed operation routes, speeds, and emergency response procedures.

[0086] Step 4:

[0087] Even after the vehicle has started moving, the server continues to receive and analyze data on the surrounding situation sent from the device in real time, and sends new operating instructions to the device as needed.

[0088] Terminal (train) processing

[0089] Step 1:

[0090] The terminal receives the route and speed instructions sent from the server, checks the received information, and sets it up in the autonomous driving system.

[0091] Step 2:

[0092] The device's autonomous driving system operates the train based on the received route and speed instructions, and the train travels along the specified route at the specified speed.

[0093] Step 3:

[0094] The terminals are equipped with cameras and sensors as monitoring tools, and continuously monitor the surroundings while in operation. This information is sent to a server in real time.

[0095] Step 4:

[0096] If the monitoring means detects an abnormality (for example, an obstacle on the tracks or an abnormal speed limit), the information is immediately sent to the server via the abnormality notification means.

[0097] Step 5:

[0098] When the terminal receives new instructions from the server, it will take emergency measures such as applying the emergency brakes, and will wait until it receives an instruction to resume operation.

[0099] User (conductor) role

[0100] Step 1:

[0101] The user (conductor) monitors the situation inside the train during normal operation and checks for any particular problems. He or she patrols the train regularly to ensure safety.

[0102] Step 2:

[0103] When the server notifies the conductor of an emergency, the conductor acts according to the server's instructions and follows the appropriate emergency response procedures, such as providing explanations to passengers and guiding them to safety.

[0104] Step 3:

[0105] The user (conductor) follows the instructions generated by the server and uses in-car announcements to communicate operational information and emergency response measures to passengers.

[0106] Step 4:

[0107] After the emergency situation has been resolved, the conductor will report the situation in detail to the server, including the details of the emergency response and confirmation of the current safety situation.

[0108] Step 5:

[0109] The conductor confirms that new operating instructions have been sent from the server, informs passengers that operation has resumed, and makes another round to ensure passengers are able to travel safely.

[0110] In this way, the server, terminal, and user (conductor) work together to ensure that the entire process, from normal operation to the occurrence of an emergency and then resumption of operation, proceeds safely and efficiently.

[0111] Example 1

[0112] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0113] To ensure safe and efficient operation in a vehicle system with autonomous driving functions, it is necessary to quickly and accurately collect operational information, optimize routes, monitor surrounding conditions, detect abnormalities, and respond to emergencies. However, in current systems, these functions are performed separately, resulting in problems such as information delays and inefficient processing. Furthermore, if an emergency response is not implemented immediately, it could lead to a serious accident. It is necessary to solve these issues and provide a system in which all functions are integrated.

[0114] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0115] In this invention, the server includes means for collecting operation information and calculating operation routes and speeds, means for receiving instructions from a central control means and controlling vehicle operation, means for monitoring the surrounding conditions while driving and detecting abnormalities, means for notifying the central control means when an abnormality is detected, a vehicle with personnel on board who will respond in the event of an emergency, means for receiving instructions from the operation control means, executing an emergency stop, and monitoring the surrounding conditions, and communication means for transmitting information to the central control means in real time. This enables real-time collection and analysis of operation information, optimization of operation, rapid detection and response to abnormalities, and seamless information transmission.

[0116] The "central management means" is a device or system that collects operation information, calculates operation routes and speeds based on that information, and sends instructions to the operation control means and abnormality notification means.

[0117] The "operation control means" is a device or system that receives instructions from the central control means and controls the operation of the vehicle based on those instructions.

[0118] "Monitoring means" refers to devices or systems such as sensors or cameras that monitor the surrounding conditions while driving and detect any abnormalities that may occur.

[0119] The "abnormality notification means" is a device or system for notifying the central management means of information when the monitoring means detects an abnormality.

[0120] "Emergency response measures" refer to systems and personnel that can respond quickly and appropriately when an emergency occurs.

[0121] "Communication means" refers to a device or system for transmitting information from the operation control means and monitoring means to the central management means.

[0122] "Operation information" refers to real-time information including data such as the train's current location, speed, destination, and route points, as well as weather data and track condition data.

[0123] An "operation route" is the path a train takes to reach its destination, which is optimized by a central control means.

[0124] An "automatic driving system" is a system that automatically controls train operation based on instructions from an operation control means.

[0125] An "emergency brake" is a braking device that immediately stops a train in an emergency.

[0126] The present invention relates to a vehicle system with an automatic driving function, which enables safe and efficient operation even in emergency situations. The system comprises the following components:

[0127] Overall system configuration

[0128] This system consists of a central management means, operation control means, monitoring means, abnormality notification means, and vehicles on board with personnel to respond to emergencies. The central management means functions as a server, collecting and analyzing operation information and generating instructions. The operation control means functions as a train terminal, operating the vehicle based on instructions from the server. The monitoring means and abnormality notification means are made up of devices such as cameras and sensors built into the terminals. Emergency response means include having personnel on board to respond to emergencies (usually a conductor).

[0129] Program processing

[0130] Server Processing

[0131] Collection of operation information

[0132] The server collects train operation information such as the train's current location, speed, destination, and route points from a database (e.g., PostgreSQL, MongoDB), and also obtains real-time weather data from a weather API (e.g., OpenWeatherMap API) and track condition data from a track monitoring system API, and analyzes this information.

[0133] Route and speed calculation

[0134] The server uses AI algorithms to calculate the optimal route and speed based on the collected operational information. Specifically, machine learning libraries (e.g., TensorFlow, PyTorch) are used to enable efficient and safe operation.

[0135] Generate and send instructions

[0136] Based on the calculation results, the server generates specific operating route and speed instructions for the operation control means (terminal) and sends them via a communication protocol (e.g., HTTP, MQTT).

[0137] Terminal (train) processing

[0138] Receiving instructions

[0139] The terminal receives the route and speed instructions sent from the server using edge computing devices (e.g., Raspberry Pi, Arduino) and uses Wi-Fi as the communication method.

[0140] Operation control

[0141] Based on the received instructions, the terminal controls the operation of the train through an automated driving system (e.g., ROS), specifically, driving the train at a specified speed along a specified route.

[0142] Surroundings monitoring

[0143] The device is equipped with a camera (e.g., Logitech C920) and a LIDAR sensor as a monitoring tool, and continuously monitors the surroundings while in operation. This information is sent to a server in real time.

[0144] Anomaly detection and notification

[0145] If the monitoring means detects an abnormality (e.g., an obstacle on the track or an abnormal speed limit), it immediately sends the information to the server through the abnormality notification means using image analysis software (e.g., OpenCV, TensorFlow Object Detection API).

[0146] User (conductor) role

[0147] Monitoring normal operations

[0148] During normal operation, the user (conductor) monitors the situation inside the train and checks for any problems. He or she patrols the train regularly to ensure safety.

[0149] Emergency response

[0150] When the server notifies the user of an emergency, the user (conductor) provides explanations to passengers, guides them to evacuate, and checks for and removes obstacles depending on the situation.

[0151] Reporting and Feedback

[0152] After the emergency situation has been resolved, the user (conductor) reports the details of the emergency response and the current safety status to the server. Based on this report, the next steps are taken promptly.

[0153] Specific examples

[0154] For example, if a train is in motion and encounters an obstacle on the tracks, it might do the following:

[0155] 1. Obstacle detection

[0156] The device's camera detects obstacles on the tracks and sends the information to the server. OpenCV is used for image analysis.

[0157] 2. Emergency stop instruction

[0158] The server analyzes the obstacle information and immediately sends an emergency stop command to the terminal.

[0159] 3. Performing an emergency shutdown

[0160] The terminal receives the emergency stop command and activates the emergency brake to stop the train.

[0161] 4. Conductor's Response

[0162] The conductor will guide passengers and guide them to safety based on instructions from the server, and will also check for and remove obstacles.

[0163] 5. Service resumes

[0164] After the obstacle is removed and safety is confirmed, the server calculates a new route and speed and sends it to the device, which receives the new instructions and resumes operation.

[0165] Example prompts for generative AI models

[0166] "Based on the following automated driving vehicle system, please explain in detail the safe operation process in an emergency. Please include the entire procedure from detecting an obstacle, making an emergency stop, responding to passengers, and resuming operation."

[0167] By feeding this prompt into a generative AI model, a detailed explanation of how a similar system would respond to an emergency can be obtained.

[0168] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0169] Step 1: Collecting traffic information

[0170] The server collects train operation information such as the train's current location, speed, destination, and route information from a database (e.g., PostgreSQL, MongoDB). It also retrieves real-time weather data from a weather API (e.g., OpenWeatherMap API) and track condition data from a track monitoring system API.

[0171] Input: Operation information in the database, weather data from the weather API, track condition data from the track monitoring system API

[0172] Output: Operation information, weather data, track condition data

[0173] Specific operation: The server executes a database query to extract the necessary operation information, and calls the weather API and track monitoring system API to obtain real-time data.

[0174] Step 2: Calculate route and speed

[0175] Based on the collected operational information, the server uses AI algorithms (e.g., TensorFlow, PyTorch) to calculate the optimal route and speed.

[0176] Input: Operation information, weather data, and track condition data collected in Step 1

[0177] Output: Optimal route and speed

[0178] Specific operation: The server runs an AI algorithm, analyzes operation information, weather data, and track condition data, and calculates the optimal operating route and speed.

[0179] Step 3: Generate and send instructions

[0180] Based on the calculation results, the server generates and sends specific operating route and speed instructions to the operation control means (terminal) via a communication protocol (e.g., HTTP, MQTT).

[0181] Input: Optimal route and speed calculated in step 2

[0182] Output: Route instructions, speed instructions

[0183] Specific operation: The server generates route and speed instructions and uses a communication protocol to send them to the terminal.

[0184] Step 4: Receiving instructions

[0185] The terminal receives route and speed instructions sent from the server, and uses edge computing devices (e.g., Raspberry Pi, Arduino) as hardware.

[0186] Input: Route and speed instructions from the server

[0187] Output: Received route and speed instructions (data stored in the device)

[0188] Specific operation: The device receives data sent from the server via Wi-Fi.

[0189] Step 5: Controlling the operation

[0190] Based on the received instructions, the terminal controls the train operation through the autonomous driving system (e.g., ROS).

[0191] Input: Route and speed instructions received in step 4

[0192] Output: Specific operation control (speed adjustment, route selection)

[0193] Specific operation: The terminal activates the autonomous driving system and controls the train's operation according to the specified speed and route.

[0194] Step 6: Monitor your surroundings

[0195] The device is equipped with a camera (e.g., Logitech C920) and a LIDAR sensor to continuously monitor the surroundings while driving, and this information is sent to a server in real time.

[0196] Input: Video data from cameras and LIDAR sensors, sensor data

[0197] Output: Surrounding situation data (sent to server)

[0198] Specific operation: The device uses the camera and sensors to capture the surrounding situation and sends the data to the server.

[0199] Step 7: Anomaly detection and notification

[0200] When the device detects an abnormality using a camera or sensor, it analyzes it using image analysis software (e.g., OpenCV), and if an abnormality is confirmed, it immediately sends the information to the server via HTTP POST.

[0201] Input: Surroundings data collected in step 6

[0202] Output: Anomaly detection data (notification to server)

[0203] Specific operation: The device analyzes the data using image analysis software, and if an abnormality is detected, it notifies the server.

[0204] Step 8: Monitor normal operations

[0205] During normal operation, the user (conductor) monitors the situation inside the train and checks for any problems. He or she patrols the train regularly to ensure safety.

[0206] Input: In-car situation

[0207] Output: Safety confirmation report

[0208] Specific actions: The conductor will patrol the train regularly to ensure the safety of passengers.

[0209] Step 9: Emergency response

[0210] When the server notifies the user of an emergency, the user (conductor) provides explanations to passengers and guides them to evacuate. The user also checks for and removes obstacles depending on the situation.

[0211] Input: Emergency notification from the server, actual emergency situation

[0212] Output: Passenger response, evacuation guidance, obstacle removal report

[0213] Specific actions: The conductor will make an announcement to passengers and take appropriate action and provide evacuation guidance.

[0214] Step 10: Reporting and Feedback

[0215] After the emergency situation has been resolved, the user (conductor) reports the details of the emergency response and the current safety status to the server.

[0216] Input: Emergency response history, current safety confirmation

[0217] Output: Detailed report

[0218] Specific operation: The conductor records the series of steps and results of the emergency response and sends them to the server.

[0219] (Application example 1)

[0220] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0221] While conventional autonomous vehicle systems can provide safe and efficient operation, they have the problem of being unable to take immediate and appropriate action when an emergency occurs. In particular, there are cases where personnel on board the vehicle are unable to respond quickly, and it takes time to ensure the safety of passengers. In addition, operation information and abnormality notifications are only sent to the operation manager, resulting in a lack of information for on-site response.

[0222] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0223] In this invention, the server includes a central management means for collecting operation information and calculating operation routes and speeds, an operation control means for receiving instructions from the central management means and controlling the operation of the vehicle, a monitoring means for monitoring the surrounding conditions while driving and detecting abnormalities, an abnormality notification means for notifying the central management means when an abnormality is detected, and a means for sending a notification to a smart device using the information when an emergency occurs. This enables a quick and appropriate response when an emergency occurs, thereby improving the safety of passengers and vehicles.

[0224] "Operation information" is a general term for detailed data such as the vehicle's current location, speed, destination, route points, weather data, and track conditions.

[0225] The "central management means" is a device or system that functions as a server, collects and analyzes operation information, and generates instructions.

[0226] The "operation control means" is a device or system that receives instructions from the central control means and controls the operation of the vehicle.

[0227] "Monitoring means" refers to a device or system that includes devices such as cameras and sensors for monitoring the surrounding conditions during operation and detecting abnormalities.

[0228] The "abnormality notification means" is a device or system that, when an abnormality is detected, promptly notifies the central management means of that information.

[0229] An "emergency situation" refers to an unexpected accident, malfunction, or abnormal event that occurs while a vehicle is in operation.

[0230] A "means for sending notifications to smart devices" is a device or system for quickly sending corresponding information to devices such as smartphones and tablets when an emergency occurs.

[0231] "Personnel" refers to personnel in the vehicle who are intended to respond in the event of an emergency.

[0232] This invention relates to a vehicle system with an automated driving function, which enables safe and efficient operation even in emergency situations. The system includes a central management means for collecting operation information and calculating operation routes and speeds, an operation control means for receiving instructions from the central management means and controlling the operation of the vehicle, a monitoring means for monitoring the surrounding conditions while driving and detecting abnormalities, an abnormality notification means for notifying the central management means when an abnormality is detected, a vehicle carrying personnel who will respond in the event of an emergency, and a means for using the information to send a notification to a smart device when an emergency occurs.

[0233] System program description

[0234] The program of this system is configured as follows:

[0235] 1. Collection of operation information

[0236] The server retrieves operational information from the database, such as vehicle location, speed, destination, and route information, as well as real-time weather and track condition data. This information is continuously updated and sent to a central control unit.

[0237] 2. Route and speed calculation

[0238] The server uses AI algorithms to calculate optimal routes and speeds based on collected traffic information, using standard server hardware and software to run the AI ​​models.

[0239] 3. Instruction Generation and Transmission

[0240] Based on the calculation results, the server proposes specific driving routes and speeds and sends them to the central management means, which then receives the instructions and controls the driving of the vehicles.

[0241] 4. Surrounding situation monitoring and abnormality detection

[0242] The device's onboard cameras and sensors monitor the surroundings while driving, and if an abnormality is detected, the information is sent to a server in real time. This monitoring method is implemented using high-performance image recognition software and sensors.

[0243] 5. Abnormality notification and emergency response

[0244] When an emergency occurs, the server analyzes the information and issues appropriate instructions for response. The server then sends an emergency notification to smart devices, prompting occupants and other responders to act quickly. This functionality is achieved using a notification service such as Firebase Cloud Messaging (FCM).

[0245] Example of a system

[0246] For example, the following is an example of a prompt that may be issued when an obstacle is discovered on the tracks during operation:

[0247] Example of a text prompt:

[0248] "Information about autonomous vehicles is obtained from a central server, and operational status is monitored in real time. If an abnormality occurs, an emergency notification is sent to the user's smartphone, instructing them on the appropriate response."

[0249] This system will enable a swift and appropriate response in the event of an emergency, enhancing the safety of passengers and trains. Conductors and train operators will be able to respond immediately to emergencies, provide guidance to passengers, and guide them to safety.

[0250] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0251] Step 1:

[0252] The server obtains operation information such as the vehicle's current location, speed, destination, and route points from the database. It also obtains real-time weather data and track condition data and integrates this information. The input is data from various sensors and the database, and the output is integrated operation information. This allows the server to have the most up-to-date operation information.

[0253] Step 2:

[0254] The server uses an AI algorithm based on the collected traffic information to calculate the optimal route and speed. The input is the traffic information obtained in the previous step, and the output is the optimal route and speed instructions. Specifically, the AI ​​model analyzes real-time data and calculates an efficient and safe route.

[0255] Step 3:

[0256] The server generates specific driving route and speed instructions based on the calculation results and sends the instructions to the central control means, where the input is the optimized driving route and speed instructions and the output is the actual instruction information, so that the central control means receives the latest driving instructions.

[0257] Step 4:

[0258] The terminal receives the driving route and speed instructions sent from the central control means and controls the driving of the vehicle through the automatic driving system. The input is the driving instructions from the central control means and the output is the actual driving status. In concrete terms, the terminal controls the speed of the vehicle and drives it along the specified route.

[0259] Step 5:

[0260] The device uses its on-board camera and sensors to monitor the surroundings while driving and detect any abnormalities. The input is real-time data from the camera and sensors, and the output is the results of any abnormalities detected. Specifically, image analysis software analyzes the video and detects obstacles and abnormal speeds.

[0261] Step 6:

[0262] When an abnormality is detected, the terminal notifies the server in real time. The input is the abnormality detection data, and the output is an abnormality notification to the central management means. This allows the server to immediately recognize the abnormality.

[0263] Step 7:

[0264] The server analyzes the anomaly information and generates appropriate response instructions. The input is the anomaly notification data, and the output is emergency response instructions. Depending on the anomaly that has occurred, the server generates instructions such as an emergency shutdown.

[0265] Step 8:

[0266] The server sends emergency response instructions to smart devices and notifies crew members and other relevant parties. The input is the emergency response instructions, and the output is a notification message. Specifically, the server sends an emergency notification to smartphones and tablets via Firebase Cloud Messaging (FCM).

[0267] Step 9:

[0268] The user (crew member / conductor) implements emergency response based on the notification from the server. The input is the emergency response instructions displayed on the smart device, and the output is the actual response action. Specific actions include explaining the situation to passengers and guiding them to evacuate.

[0269] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0270] The present invention further improves the safety and efficiency of driving by combining an emotion engine with a vehicle system having an automatic driving function. Hereinafter, embodiments of the present invention will be described in detail.

[0271] Overall system configuration

[0272] The system includes the following major components:

[0273] Central management means (server): Manages and analyzes operation information and generates instructions.

[0274] Operation control means (terminal): Receives instructions from the central control means and controls the operation of the vehicle.

[0275] Monitoring Measures: Continuously monitor the surrounding environment and conditions while the vehicle is in operation.

[0276] Abnormality notification means: Notifies the central management means when an abnormality is detected.

[0277] Emotion engine: Recognizes the emotional state of personnel (conductors) and reflects it in management and response.

[0278] Personnel (Conductor): Operates within the vehicle to respond to emergencies.

[0279] Program processing

[0280] Server Processing

[0281] 1. Collection of operational information:

[0282] The server retrieves information about the train's current location, speed, destination, stops, and route from a database, as well as real-time weather and track condition data.

[0283] 2. Route and speed calculation:

[0284] The server uses AI algorithms to calculate the optimal route and speed based on the collected operational information, records the calculation results, and generates instructions.

[0285] 3. Generate and send instructions:

[0286] The server sends the generated instructions to the operation control means (terminal), which include detailed operation routes, speeds, and emergency response procedures.

[0287] 4. Real-time analysis:

[0288] Even after the vehicle has started moving, the server continues to receive and analyze data on the surrounding situation sent from the device in real time, and sends new operating instructions to the device as needed.

[0289] 5. Emotion recognition and response:

[0290] The server receives and analyzes the conductor's emotional data sent from the emotion engine, and responds accordingly based on the conductor's mental state and stress level.

[0291] Terminal (train) processing

[0292] 1. Receiving and setting instructions:

[0293] The terminal receives the route and speed instructions sent from the server, checks the received information, and sets it up in the autonomous driving system.

[0294] 2. Navigation control:

[0295] The device's autonomous driving system operates the train based on the received route and speed instructions, and the train travels along the specified route at the specified speed.

[0296] 3. Surrounding Area Monitoring:

[0297] The terminals are equipped with cameras and sensors as monitoring tools, and continuously monitor the surroundings while in operation. This information is sent to a server in real time.

[0298] 4. Anomaly detection and notification:

[0299] If the monitoring means detects an abnormality (for example, an obstacle on the tracks or an abnormal speed limit), the information is immediately sent to the server via the abnormality notification means.

[0300] 5. Collecting Emotional Data:

[0301] The emotion engine continuously monitors the conductor's emotional state and transmits the data to the server.

[0302] User (conductor) role

[0303] 1. Monitoring normal operations:

[0304] During normal operation, the user (conductor) monitors the situation inside the train and checks for any particular problems. He or she also patrols the train regularly to ensure safety.

[0305] 2. Emergency Response:

[0306] When an emergency situation is notified by the server, the conductor will act according to the appropriate emergency response procedures, such as explaining the situation to passengers and guiding them to an evacuation.

[0307] 3. Status reports and feedback:

[0308] After the emergency situation has been resolved, the conductor will report the situation in detail to the server, including the details of the emergency response and confirmation of the current safety situation.

[0309] 4. Emotional management:

[0310] Based on feedback from the emotion engine, the conductor can take measures according to his or her emotional state, such as taking a break if stress levels are high.

[0311] Specific examples

[0312] For example, consider the process that is performed when a train detects an obstacle on the tracks while in operation.

[0313] 1. Obstacle detection:

[0314] The device's camera detects obstacles on the tracks and sends that information to a server.

[0315] 2. Emergency stop instruction:

[0316] The server analyzes the obstacle information and immediately sends an emergency stop command to the terminal if it determines it is necessary. At the same time, it notifies the conductor of the emergency response procedure.

[0317] 3. Performing an emergency stop:

[0318] The terminal receives the emergency stop command and activates the emergency brake to stop the train.

[0319] 4. Conductor's response:

[0320] The conductor will guide passengers and guide them to safety based on instructions from the server. In addition, if possible, the conductor will check for and remove obstacles.

[0321] 5. Emotional data feedback:

[0322] When an emergency occurs, the emotion engine monitors the conductor's emotional state and sends the data to a server, which analyzes the data and provides a response guide based on the conductor's mental state.

[0323] 6. Service resumes:

[0324] After the obstacle is removed and safety is confirmed, the server calculates a new route and speed and sends it to the device, which receives the new instructions and resumes operation.

[0325] In this way, by combining the emotion engine, the vehicle system of the present invention can operate more safely and efficiently, taking into account the mental health of the conductor, from normal operation to emergency situations and even when operation resumes.

[0326] The processing flow will be explained below.

[0327] Server Processing

[0328] Step 1:

[0329] The server retrieves information about the train's current location, speed, destination, stops, and route from a database, as well as real-time weather and track condition data.

[0330] Step 2:

[0331] The server uses AI algorithms to calculate the optimal route and speed based on the collected operational information, records the calculation results, and generates instructions.

[0332] Step 3:

[0333] The server sends the generated instructions to the operation control means (terminal), which include detailed operation routes, speeds, and emergency response procedures.

[0334] Step 4:

[0335] Even after the vehicle has started moving, the server continues to receive and analyze data on the surrounding situation sent from the device in real time, and sends new operating instructions to the device as needed.

[0336] Step 5:

[0337] The server receives and analyzes the conductor's emotional data sent from the emotion engine, and based on the results, provides a response guide according to the conductor's mental state and stress level.

[0338] Terminal (train) processing

[0339] Step 1:

[0340] The terminal receives the route and speed instructions sent from the server, checks the received information, and sets it up in the autonomous driving system.

[0341] Step 2:

[0342] The device's autonomous driving system operates the train based on the received route and speed instructions, and the train travels along the specified route at the specified speed.

[0343] Step 3:

[0344] The terminals are equipped with cameras and sensors as monitoring tools, and continuously monitor the surroundings while in operation. This information is sent to a server in real time.

[0345] Step 4:

[0346] If the monitoring means detects an abnormality (for example, an obstacle on the tracks or an abnormal speed limit), the information is immediately sent to the server via the abnormality notification means.

[0347] Step 5:

[0348] The terminal continuously monitors the conductor's emotional data through the emotion engine and sends that data to the server in real time.

[0349] User (conductor) role

[0350] Step 1:

[0351] During normal operation, the user (conductor) monitors the situation inside the train and checks for any particular problems. He or she also patrols the train regularly to ensure safety.

[0352] Step 2:

[0353] When the server notifies the conductor of an emergency, the conductor acts according to the server's instructions and follows the appropriate emergency response procedures, such as providing explanations to passengers and guiding them to safety.

[0354] Step 3:

[0355] Based on feedback from the emotion engine, the conductor can take measures according to his or her emotional state, such as taking a break if stress levels are high.

[0356] Step 4:

[0357] After the emergency situation has been resolved, the conductor will report the situation in detail to the server, including the details of the emergency response and confirmation of the current safety situation.

[0358] Step 5:

[0359] The conductor confirms that new operating instructions have been sent from the server, informs passengers that operation has resumed, and makes another round to ensure passengers are able to travel safely.

[0360] Specific examples

[0361] For example, consider the process that is performed when a train detects an obstacle on the tracks while in operation.

[0362] Step 1:

[0363] The device's camera detects obstacles on the tracks and sends that information to a server.

[0364] Step 2:

[0365] The server analyzes the obstacle information and immediately sends an emergency stop command to the terminal if it determines it is necessary. At the same time, it notifies the conductor of the emergency response procedure.

[0366] Step 3:

[0367] The terminal receives the emergency stop command and activates the emergency brake to stop the train.

[0368] Step 4:

[0369] The conductor will guide passengers and guide them to safety based on instructions from the server. In addition, if possible, the conductor will check for and remove obstacles.

[0370] Step 5:

[0371] When an emergency occurs, the emotion engine monitors the conductor's emotional state and sends the data to a server, which analyzes the data and provides a response guide based on the conductor's mental state.

[0372] Step 6:

[0373] After the obstacle is removed and safety is confirmed, the server calculates a new route and speed and sends it to the device, which receives the new instructions and resumes operation.

[0374] Example 2

[0375] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0376] While conventional autonomous vehicle systems have certain functions for collecting operational information, calculating operational routes, and detecting abnormalities, they are not yet able to respond quickly and effectively to emergencies or sudden abnormalities.In addition, they lack the functionality to consider the mental state and stress level of the conductor as human factors, which has led to issues with the safety and efficiency of operations.

[0377] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for collecting operation information and calculating operation routes and speeds, a means for analyzing the surrounding situation in real time and generating new operation instructions, and an emotion recognition means for recognizing the emotional state of passengers on board and taking appropriate action. This makes it possible to ensure the safety of operation while improving the efficiency of operation by taking human factors into consideration.

[0378] "Operation information" is data related to the vehicle's current location, speed, destination, intermediate stops, and operating route.

[0379] A "central control means" is a system that collects operational information, calculates operational routes and speeds, and generates instructions.

[0380] The "operation control means" is a device that receives instructions from the central control means and automatically controls the operation of the vehicle.

[0381] "Monitoring means" refers to devices such as cameras and sensors that continuously monitor the surrounding conditions while the vehicle is in operation and detect abnormalities.

[0382] The "abnormality notification means" is a system that notifies the central management means of abnormalities detected by the monitoring means.

[0383] An "emergency situation" refers to an unexpected malfunction, accident, or abnormal situation that occurs during operation.

[0384] The "emotion recognition means" is a system that monitors the emotional state of onboard personnel in real time and analyzes the data.

[0385] "Personnel" refers to the people on board who will respond in the event of an emergency.

[0386] "New driving instructions" are updated driving routes and speed instructions generated based on the results of real-time analysis.

[0387] A "conductor" is a person on board a train who is responsible for normal operation and responding to emergencies.

[0388] The present invention further improves the safety and efficiency of driving by combining emotion recognition means with a vehicle system having an automatic driving function. Hereinafter, embodiments of the present invention will be described in detail.

[0389] Overall system configuration

[0390] The system includes the following main components:

[0391] Central control means (server): The server collects driving information, calculates driving routes and speeds, and analyzes the surrounding situation in real time during driving to generate new driving instructions.

[0392] Operation control means (terminal): The terminal receives instructions from the server and automatically controls the operation of the vehicle.

[0393] Monitoring means: Includes cameras and sensors to continuously monitor the surroundings while the vehicle is in operation and detect any abnormalities.

[0394] Abnormality notification means: Notifies the server of any abnormalities detected by the monitoring means.

[0395] Emotion recognition means: The emotion engine recognizes and analyzes the emotional state of the conductor (personnel), allowing it to respond according to the conductor's mental state and stress level.

[0396] Personnel (conductor): Responsible for responding to emergencies in the vehicle.

[0397] Specific server processing

[0398] The server manages and analyzes operation information in the following steps:

[0399] 1. Collection of operational information:

[0400] The server retrieves information on the train's current location, speed, destination, intermediate stops, and route from the database, as well as real-time weather data and track condition data, and manages this information as comprehensive operational information.

[0401] 2. Route and speed calculation:

[0402] Based on the collected operational information, the server uses the AI ​​algorithm "Route Optimizer" to calculate the optimal route and speed, thereby supporting safe and efficient operation.

[0403] 3. Generate and send instructions:

[0404] The server then sends the generated driving route and speed instructions to the terminal, which include detailed driving route, speed, and emergency response procedures.

[0405] 4. Real-time analysis:

[0406] While the vehicle is in operation, the server receives and analyzes real-time data on the surrounding environment sent from the device, and sends new operating instructions to the device as needed to ensure safe operation.

[0407] 5. Emotion recognition and response:

[0408] The server analyzes the conductor's emotional data sent from the emotion engine "EmotionAnalyzer," and generates and provides the conductor with a response guide based on the conductor's mental state and stress level.

[0409] Specific processing of the terminal (train)

[0410] The terminal automatically controls operation through the following steps:

[0411] 1. Receiving and setting instructions:

[0412] The terminal receives route and speed instructions sent from the server and sets them in the autonomous driving system.

[0413] 2. Navigation control:

[0414] The automated driving system operates the vehicle based on the received route and speed, and follows the specified route at the specified speed.

[0415] 3. Surrounding Area Monitoring:

[0416] Cameras and sensors are used to continuously monitor the surrounding conditions while the vehicle is in operation, and the data is sent to a server.

[0417] 4. Anomaly detection and notification:

[0418] If the monitoring means detects an abnormality, it immediately transmits the information to the server.

[0419] 5. Collecting Emotional Data:

[0420] The emotion engine continuously monitors the conductor's emotional state and transmits the data to the server.

[0421] User (conductor) role

[0422] 1. Monitoring normal operations:

[0423] During normal operation, the conductor monitors the situation inside the train and checks for any particular problems. He also patrols the train regularly to ensure safety.

[0424] 2. Emergency Response:

[0425] When an emergency situation is notified by the server, the conductor will act according to the appropriate emergency response procedures, such as explaining the situation to passengers and guiding them to an evacuation.

[0426] 3. Status reports and feedback:

[0427] After the emergency situation has been resolved, the conductor will report the situation in detail to the server, including the details of the emergency response and confirmation of the current safety situation.

[0428] 4. Emotional management:

[0429] The conductor takes measures based on his or her own emotional state based on feedback from the emotion engine. For example, if stress levels are high, the conductor may take a break.

[0430] Specific examples

[0431] For example, if a train detects an obstacle on the tracks while in operation, the process is as follows.

[0432] 1. Obstacle detection:

[0433] The device's camera detects obstacles on the tracks and sends that information to a server.

[0434] 2. Emergency stop instruction:

[0435] The server analyzes the obstacle information and, if necessary, immediately sends an emergency stop instruction to the terminal, while simultaneously informing the conductor of the emergency response procedure.

[0436] 3. Performing an emergency stop:

[0437] The terminal receives an emergency stop command and activates the emergency brake, bringing the train to a halt.

[0438] 4. Conductor's response:

[0439] The conductor will provide passenger guidance and evacuation guidance according to instructions from the server, and will also check for and remove obstacles if possible.

[0440] 5. Emotional data feedback:

[0441] The emotion engine monitors the conductor's emotional state in the event of an emergency and sends the data to a server, which analyzes the data and provides a response guide based on the conductor's mental state.

[0442] 6. Service resumes:

[0443] After the obstacle is removed and safety is confirmed, the server calculates a new route and speed and sends it to the device, which receives the new instructions and resumes operation.

[0444] Prompt Sentence Examples

[0445] To use a generative AI model to respond to specific situations, we use prompts like the following:

[0446] Example prompt 1: "If a train is in motion and discovers an obstacle on the tracks, explain how you should respond based on your roles as the server, the terminal, and the conductor."

[0447] Example prompt 2: "Please explain in detail how the emotion engine monitors the conductor's emotional state and notifies the server in the event of an emergency."

[0448] These prompts allow the generative AI model to learn system-wide responses and response procedures based on specific scenarios.

[0449] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0450] Server Processing

[0451] Step 1:

[0452] Collection of operation information

[0453] Input: Database, real-time weather data, track condition data

[0454] Data processing / calculation: The server retrieves and organizes information on the train's current location, speed, destination, intermediate stops, and route from the database. It also retrieves and integrates real-time weather information and track conditions from the weather data API.

[0455] Output: Complete set of traffic information

[0456] What it does: The server periodically retrieves data from the database and API and stores it as a train information set, including the train's current location, speed, destination, temperature, precipitation, wind speed, etc.

[0457] Step 2:

[0458] Route and speed calculation

[0459] Input: Service information set

[0460] Data processing / calculation: Based on the operation information, the server uses the AI ​​algorithm "Route Optimizer" to calculate the optimal route and speed, taking into account traffic and weather conditions to derive a safe and efficient route.

[0461] Output: Optimized route and speed instructions

[0462] Specific operation: The server inputs operation information into "RouteOptimizer" and generates a file containing the resulting route and speed instructions, including the route to the next station, speed limits, and emergency stop points.

[0463] Step 3:

[0464] Generate and send instructions

[0465] Input: Route and speed instructions

[0466] Data processing / calculation: The server formats the instructions and sends them to the operation control means (terminal). Safety checks are performed and the contents of the instructions are confirmed.

[0467] Output: Instructions sent to the terminal

[0468] What happens: The server sends instructions to the device via email or a digital communications protocol, and receives a confirmation message to confirm that they were received in the correct format.

[0469] Step 4:

[0470] Real-time analytics

[0471] Input: Ambient data from the device

[0472] Data processing / calculation: The server analyzes the surrounding situation data received from the device in real time, determines whether there are any abnormalities, and generates new driving instructions as necessary.

[0473] Output: New operating instructions

[0474] Specific operation: The server analyzes camera footage and sensor data sent from moving trains, and if it detects an obstacle or abnormality, it generates new operating instructions and immediately sends them to the terminal.

[0475] Step 5:

[0476] Emotion recognition and response

[0477] Input: Emotion data from the emotion engine

[0478] Data processing / calculation: The server analyzes the conductor's emotional data sent from the emotion engine "EmotionAnalyzer." It evaluates the conductor's stress level and emotional state and generates a response guide.

[0479] Output: Conductor's response guide

[0480] Specific actions: Based on the emotional data, the server generates a response guide that includes stress reduction techniques and recommendations for taking breaks, and notifies the conductor. This guide includes specific advice, such as "take deep breaths and stretch to relax."

[0481] Terminal (train) processing

[0482] Step 1:

[0483] Receiving and Setting Instructions

[0484] Input: Route and speed instructions sent from the server

[0485] Data processing / calculation: The device checks the received route and speed instructions and sets them in the autonomous driving system. It also checks the settings and verifies that the vehicle can be driven safely.

[0486] Output: Route and speed set for the autonomous driving system

[0487] Specific operation: The device analyzes the received instructions and sends the appropriate configuration commands to the autonomous driving system. Once the configuration is complete, it sends a confirmation message to the server.

[0488] Step 2:

[0489] Operation control

[0490] Input: Set route and speed

[0491] Data processing / calculation: The autonomous driving system controls the vehicle based on the route and speed instructions it receives, adjusting speed and stopping positions to ensure safe and efficient driving.

[0492] Output: Actual driving data (current location, speed, etc.)

[0493] Specific operation: The device uses an autonomous driving system to drive along a designated route at a set speed. During operation, the device constantly monitors its speedometer and location information and makes adjustments as necessary.

[0494] Step 3:

[0495] Surroundings monitoring

[0496] Input: Camera and sensor data

[0497] Data processing / calculation: Analyzes data sent from cameras and sensors, which are used as monitoring tools, and monitors the surrounding situation. If an abnormality is detected, the data is sent to the server.

[0498] Output: Surroundings data

[0499] Specific operation: High-resolution cameras and various sensors installed on the terminal acquire data in real time, and perform image analysis and pattern recognition to determine whether there are any abnormalities. This data is then sent to the server.

[0500] Step 4:

[0501] Anomaly detection and notification

[0502] Input: Camera and sensor data

[0503] Data processing / calculation: If the monitoring means detects an abnormality, the information is immediately sent to the server via the abnormality notification means.

[0504] Output: Error notification data

[0505] Specific operation: If an abnormality is detected, the device will send the information to the server via the abnormality notification means. For example, if an obstacle is detected on the tracks, the device will send the video and data to the server.

[0506] Step 5:

[0507] Collecting Emotional Data

[0508] Input: Data from the emotion engine "EmotionAnalyzer"

[0509] Data processing / calculation: The emotion engine continuously monitors the conductor's emotional state and sends the data to the server.

[0510] Output: Conductor's emotion data

[0511] Specific operation: The emotion engine monitors the conductor's facial expressions, voice, heart rate, etc. in real time, analyzes this data, and processes it into emotion data. The analyzed data is sent to a server, and a response is made based on the conductor's emotional state.

[0512] User (conductor) role

[0513] Step 1:

[0514] Monitoring normal operations

[0515] Input: Passenger and vehicle status data

[0516] Data processing / calculation: Conductors monitor the situation inside the train during normal operation and check for any particular problems. They also make regular rounds to check safety.

[0517] Output: Safety report data

[0518] Specific actions: The conductor checks the train announcements and security camera footage to see if there are any problems. Depending on the situation, he or she will make announcements to passengers and ensure their safety.

[0519] Step 2:

[0520] Emergency response

[0521] Input: Emergency notification data from the server

[0522] Data Processing / Calculation: When an emergency situation is notified by the server, the conductor will act according to the appropriate emergency response procedures.

[0523] Output: Emergency response report data

[0524] Specific actions: In the event of an emergency, the conductor will explain the situation to passengers and guide them to evacuate. For example, an announcement will be made on the train saying, "This is an emergency, please disembark."

[0525] Step 3:

[0526] Status reports and feedback

[0527] Input: Post-emergency data

[0528] Data processing / calculation: After the emergency situation has been resolved, the conductor will report the situation in detail to the server.

[0529] Output: Emergency report details

[0530] Specific operation: The conductor records the details of the emergency response and the results, and reports them to the server. For example, he / she describes the success of evacuation guidance and the results of safety confirmation.

[0531] Step 4:

[0532] emotional management

[0533] Input: Feedback from the emotion engine

[0534] Data processing / calculation: The conductor takes measures based on his / her own emotional state based on feedback from the emotion engine.

[0535] Output: Improved emotional state data

[0536] Specific Actions: Conductors receive feedback from the Emotion Engine and implement stress management and relaxation techniques, such as deep breathing and relaxation exercises during designated breaks.

[0537] The above is the specific processing flow of the program of this system and the specific operations performed at each processing step.

[0538] (Application example 2)

[0539] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0540] In order to improve the safety and efficiency of autonomous driving vehicle systems, it is necessary not only to monitor the surrounding environment and situation, but also to grasp the emotional state of the personnel on board in real time and respond accordingly. However, conventional autonomous driving systems lack the ability to monitor emotional states, and lack means to reduce the mental burden on conductors and drivers in emergencies. Therefore, comprehensive improvements to operational safety and comfort are a challenge.

[0541] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting operation information and calculating operation routes and speeds, means for receiving instructions from the central control means and controlling the operation of the vehicle, means for monitoring the surrounding conditions while driving and detecting abnormalities, and means for monitoring the emotional state of the vehicle in real time and transmitting the emotional data to the central control means. This makes it possible to improve the safety and comfort of operation by adjusting operation and reducing stress according to the emotional state of the conductor and driver.

[0542] "Operation information" is data related to the vehicle's current location, speed, destination, intermediate stops, and operating route.

[0543] "Central control means" refers to the system that collects, analyzes, and generates instructions regarding traffic information.

[0544] The "operation control means" is a system that receives instructions from the central control means and controls the operation of the vehicle.

[0545] The "monitoring means" is a system that continuously monitors the surrounding environment and conditions while driving and detects abnormalities.

[0546] The "abnormality notification means" is a system that notifies the central management means of information when an abnormality is detected.

[0547] The "emotion recognition means" is a system that monitors the emotional state inside the vehicle in real time and transmits the data to a central control means.

[0548] "Vehicle" refers to a land transportation means that has an automatic driving function and has personnel on board to respond in the event of an abnormality or emergency during operation.

[0549] "Route" refers to the route chosen for a vehicle to travel to its destination.

[0550] An "emergency situation" refers to an unexpected abnormality or malfunction that occurs during operation and requires immediate response.

[0551] The "operation adjustment means" is a system that adjusts the route and speed of the vehicle depending on the vehicle's emotional state and surrounding circumstances.

[0552] A specific system configuration and its operation will be described below as an embodiment of the invention.

[0553] Overall system configuration

[0554] The system includes the following major components:

[0555] 1. Central management means (server): Collects operation information, analyzes it, and generates instructions. The server calculates the route and speed, and generates and sends operation instructions.

[0556] 2. Operation control means (terminal): A terminal that receives instructions from the central control means and controls the operation of vehicles. It operates vehicles based on the instructions for route and speed.

[0557] 3. Monitoring means: Using cameras and sensors, the surrounding environment and conditions are monitored during operation. If an abnormality is detected, it is notified to the central control means via a terminal.

[0558] 4. Abnormality notification means: Notifies the central control means when an abnormality is detected. This means immediately sends information to the central control means when an abnormality occurs.

[0559] 5. Emotion recognition means: The emotional state of personnel on board (conductors and drivers) is monitored in real time using cameras and emotion recognition software, and the data is sent to a central management means.

[0560] 6. Vehicles: These vehicles have autonomous driving capabilities and are manned by personnel who can respond to emergencies.

[0561] Server Processing

[0562] The server performs the following process:

[0563] 1. Operational information collection: The server collects information on the vehicle's current location, speed, destination, intermediate stops, and route, as well as real-time weather and track condition data.

[0564] 2. Calculation of route and speed: Based on the collected operation information, an AI algorithm is used to calculate the optimal route and speed.

[0565] 3. Generation and transmission of instructions: Generate instructions based on the calculation results and send them to the operation control means.

[0566] 4. Real-time analysis: Analyzes surrounding situation data and emotional data sent from the operation control means and sends new instructions based on the results.

[0567] 5. Emotion recognition and response: Data from the emotion recognition system is received and analyzed, and operation adjustments and stress reduction measures are implemented based on the driver's stress level.

[0568] Terminal handling

[0569] The terminal performs the following process:

[0570] 1. Receiving and setting instructions: Receive driving instructions from the server and set them in the autonomous driving system.

[0571] 2. Operation control: Operate the vehicle based on the received route and speed instructions.

[0572] 3. Surroundings monitoring: Cameras and sensors are used to monitor the surroundings while driving.

[0573] 4. Abnormality notification: If an abnormality is detected, the information is immediately sent to a central control means.

[0574] Emotion Recognition System Processing

[0575] 1. Real-time emotion monitoring: Using cameras and emotion recognition software, the facial expressions of the vehicle's occupants are analyzed to determine their emotional state.

[0576] 2. Sending emotion data to the server: The recognized emotion data is sent to the server and analyzed in real time.

[0577] 3. Operational adjustment: If the emotional state is judged to be stressful, the route or speed of the driver will be adjusted.

[0578] 4. Stress reduction measures: Reduce stress levels for drivers and passengers by playing stress-reducing music and adjusting lighting.

[0579] Specific examples

[0580] For example, if a camera inside the vehicle detects that the driver is fatigued, the information is immediately sent to a server. The server uses the data to adjust the route and speed to ensure the driver can continue driving safely. If the server determines that the driver is feeling stressed, it will play relaxing music and adjust the lighting inside the vehicle to reduce stress.

[0581] Using generative AI models

[0582] Example prompt: "The driver is stressed, please play some relaxing music."

[0583] In this way, the system of the present invention combines emotion recognition to comprehensively improve driving safety and comfort.

[0584] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0585] Step 1:

[0586] Collection of operation information

[0587] The server retrieves information about the vehicle's current location, speed, destination, intermediate stops, and route from a database, as well as real-time weather and track condition data.

[0588] Input: Vehicle location information, speed information, weather data, track condition data

[0589] Data processing: Organize and store collected operation information in an analyzable format

[0590] Output: Formatted traffic information data

[0591] Step 2:

[0592] Route and speed calculation

[0593] Based on the operation information collected by the server, an AI algorithm is used to calculate the optimal operation route and speed.

[0594] Input: Formatted traffic information data

[0595] Data calculation: Optimization of route and speed is performed using AI algorithms

[0596] Output: Calculated optimal route and speed instructions

[0597] Step 3:

[0598] Generate and send instructions

[0599] The server transmits the generated travel route and speed instructions to the travel control means.

[0600] Input: Optimal route and speed instructions

[0601] Data processing: Converting data into a format for transmission

[0602] Output: Data sent to the operation control means

[0603] Step 4:

[0604] Receiving and Setting Instructions

[0605] The terminal receives driving instructions sent from the server and sets them in the autonomous driving system.

[0606] Input: Data sent to the operation control means

[0607] Data processing: Converting received instructions into the setting format of the autonomous driving system

[0608] Output: Configured autonomous driving system

[0609] Step 5:

[0610] Operation control

[0611] The device's autonomous driving system operates the vehicle based on the received route and speed instructions.

[0612] Input: Configured autonomous driving system

[0613] Specific behavior: The vehicle travels along a specified route at a specified speed.

[0614] Output: Vehicles in operation

[0615] Step 6:

[0616] Surrounding Area Monitoring

[0617] The device uses cameras and sensors to monitor the surroundings while in operation.

[0618] Input: Surrounding situation data (video and sensor information)

[0619] Data processing: Real-time situation analysis

[0620] Output: Parsed situation data

[0621] Step 7:

[0622] Abnormality notification

[0623] If the monitoring means detects an abnormality, it immediately transmits the information to the central control means.

[0624] Input: Parsed situation data

[0625] Data processing: Analysis to see if there are any abnormalities

[0626] Output: Error notification data

[0627] Step 8:

[0628] Collecting and transmitting emotional data

[0629] An emotion recognition means monitors the emotional state of the personnel in the vehicle and transmits the data to a server.

[0630] Input: Camera footage, emotion recognition software analysis results

[0631] Data Computation: Analysis with Emotion Recognition Algorithms

[0632] Output: Parsed emotion data

[0633] Step 9:

[0634] Emotion recognition and response

[0635] The server receives and analyzes the data from the emotion recognition means, and based on the results, adjusts operation and takes measures to reduce stress according to the driver's stress level.

[0636] Input: Parsed emotion data

[0637] Data processing: assessing stress states and generating response instructions

[0638] Output: Operation adjustment instructions, stress reduction measures

[0639] Step 10:

[0640] Operation adjustments

[0641] The terminal receives operation adjustment instructions from the server and adjusts the operation route and speed.

[0642] Input: Operation adjustment instructions

[0643] Specific behavior: Resetting the route and speed

[0644] Output: Operating state after adjustment

[0645] Step 11:

[0646] Stress reduction measures

[0647] The device will implement stress-reducing measures (such as playing relaxing music or adjusting the lighting).

[0648] Input: Stress reduction measures (specific instructions)

[0649] Specific actions: Change the in-car environment (play music, adjust lighting, etc.)

[0650] Output: Improved in-car environment

[0651] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0652] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0653] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0654] [Second embodiment]

[0655] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0656] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0657] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0658] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0659] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0660] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0661] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0662] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0663] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0664] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0665] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0666] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0667] The present invention relates to a vehicle system with an automatic driving function, and in particular to a system that enables safe and efficient operation even in an emergency. Hereinafter, an embodiment of the present invention will be described in detail.

[0668] Overall system configuration

[0669] This system mainly consists of a central management means, operation control means, monitoring means, abnormality notification means, and vehicles carrying emergency response personnel (usually conductors). The central management means functions as a server, collecting and analyzing operation information and generating instructions. The operation control means functions as a train terminal, operating the vehicle based on instructions from the server. The monitoring means and abnormality notification means are composed of devices such as cameras and sensors built into the terminals.

[0670] Program processing

[0671] Server Processing

[0672] 1. Collection of operational information:

[0673] The server collects train operation information, such as the train's current location, speed, destination, and route information, from a database, as well as real-time weather and track condition data, and analyzes this information.

[0674] 2. Route and speed calculation:

[0675] The server uses AI algorithms to calculate the optimal route and speed based on the collected operational information, enabling efficient and safe operation.

[0676] 3. Generate and send instructions:

[0677] Based on the calculation results, the server generates and sends specific instructions on the driving route and speed to the driving control means (terminal).

[0678] Terminal (train) processing

[0679] 1. Receiving instructions:

[0680] The terminal receives the driving route and speed instructions sent from the server.

[0681] 2. Navigation control:

[0682] Based on the received instructions, the terminal controls the operation of the train through the automated driving system, specifically, driving at a specified speed along a specified route.

[0683] 3. Surrounding Area Monitoring:

[0684] The terminals are equipped with cameras and sensors as monitoring tools, and continuously monitor the surroundings while in operation, sending information to a server in real time.

[0685] 4. Anomaly detection and notification:

[0686] If the monitoring means detects an abnormality (for example, an obstacle on the tracks or an abnormal speed limit), the information is immediately sent to the server via the abnormality notification means.

[0687] User (conductor) role

[0688] 1. Monitoring normal operations:

[0689] During normal operation, the user (conductor) monitors the situation inside the train and checks for any problems. He or she patrols the train regularly to ensure safety.

[0690] 2. Emergency Response:

[0691] When the server notifies the conductor of an emergency, the conductor will act according to the appropriate emergency response procedures, such as explaining the situation to passengers, guiding them to an evacuation, and checking for and removing obstacles if possible.

[0692] 3. Reporting and Feedback:

[0693] After the emergency situation has been resolved, the conductor will report the situation in detail to the server, including details of the emergency response and current safety confirmation.

[0694] Specific examples

[0695] For example, consider the process that is performed when a train detects an obstacle on the tracks while in operation.

[0696] 1. Obstacle detection:

[0697] The device's camera detects obstacles on the tracks and sends that information to a server.

[0698] 2. Emergency stop instruction:

[0699] The server analyzes the obstacle information and immediately sends an emergency stop instruction to the terminal if it determines it is necessary.

[0700] 3. Perform an emergency stop:

[0701] The terminal receives the emergency stop command and activates the emergency brake to stop the train.

[0702] 4. Conductor's response:

[0703] The conductor will guide passengers and guide them to safety based on instructions from the server. In addition, if possible, the conductor will check for and remove obstacles.

[0704] 5. Service resumes:

[0705] After the obstacle is removed and safety is confirmed, the server calculates a new route and speed and sends it to the device, which receives the new instructions and resumes operation.

[0706] In this way, the vehicle system of the present invention can operate safely and efficiently both during normal operation and in emergency situations.

[0707] The processing flow will be explained below.

[0708] Server Processing

[0709] Step 1:

[0710] The server retrieves information about the train's current location, speed, destination, stops, and route from a database, as well as real-time weather and track condition data.

[0711] Step 2:

[0712] The server uses AI algorithms to calculate the optimal route and speed based on the collected operational information, records the calculation results, and generates instructions.

[0713] Step 3:

[0714] The server sends the generated instructions to the operation control means (terminal), which include detailed operation routes, speeds, and emergency response procedures.

[0715] Step 4:

[0716] Even after the vehicle has started moving, the server continues to receive and analyze data on the surrounding situation sent from the device in real time, and sends new operating instructions to the device as needed.

[0717] Terminal (train) processing

[0718] Step 1:

[0719] The terminal receives the route and speed instructions sent from the server, checks the received information, and sets it up in the autonomous driving system.

[0720] Step 2:

[0721] The device's autonomous driving system operates the train based on the received route and speed instructions, and the train travels along the specified route at the specified speed.

[0722] Step 3:

[0723] The terminals are equipped with cameras and sensors as monitoring tools, and continuously monitor the surroundings while in operation. This information is sent to a server in real time.

[0724] Step 4:

[0725] If the monitoring means detects an abnormality (for example, an obstacle on the tracks or an abnormal speed limit), the information is immediately sent to the server via the abnormality notification means.

[0726] Step 5:

[0727] When the terminal receives new instructions from the server, it will take emergency measures such as applying the emergency brakes, and will wait until it receives an instruction to resume operation.

[0728] User (conductor) role

[0729] Step 1:

[0730] The user (conductor) monitors the situation inside the train during normal operation and checks for any particular problems. He or she patrols the train regularly to ensure safety.

[0731] Step 2:

[0732] When the server notifies the conductor of an emergency, the conductor acts according to the server's instructions and follows the appropriate emergency response procedures, such as providing explanations to passengers and guiding them to safety.

[0733] Step 3:

[0734] The user (conductor) follows the instructions generated by the server and uses in-car announcements to communicate operational information and emergency response measures to passengers.

[0735] Step 4:

[0736] After the emergency situation has been resolved, the conductor will report the situation in detail to the server, including the details of the emergency response and confirmation of the current safety situation.

[0737] Step 5:

[0738] The conductor confirms that new operating instructions have been sent from the server, informs passengers that operation has resumed, and makes another round to ensure passengers are able to travel safely.

[0739] In this way, the server, terminal, and user (conductor) work together to ensure that the entire process, from normal operation to the occurrence of an emergency and then resumption of operation, proceeds safely and efficiently.

[0740] Example 1

[0741] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0742] To ensure safe and efficient operation in a vehicle system with autonomous driving functions, it is necessary to quickly and accurately collect operational information, optimize routes, monitor surrounding conditions, detect abnormalities, and respond to emergencies. However, in current systems, these functions are performed separately, resulting in problems such as information delays and inefficient processing. Furthermore, if an emergency response is not implemented immediately, it could lead to a serious accident. It is necessary to solve these issues and provide a system in which all functions are integrated.

[0743] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0744] In this invention, the server includes means for collecting operation information and calculating operation routes and speeds, means for receiving instructions from a central control means and controlling vehicle operation, means for monitoring the surrounding conditions while driving and detecting abnormalities, means for notifying the central control means when an abnormality is detected, a vehicle with personnel on board who will respond in the event of an emergency, means for receiving instructions from the operation control means, executing an emergency stop, and monitoring the surrounding conditions, and communication means for transmitting information to the central control means in real time. This enables real-time collection and analysis of operation information, optimization of operation, rapid detection and response to abnormalities, and seamless information transmission.

[0745] The "central management means" is a device or system that collects operation information, calculates operation routes and speeds based on that information, and sends instructions to the operation control means and abnormality notification means.

[0746] The "operation control means" is a device or system that receives instructions from the central control means and controls the operation of the vehicle based on those instructions.

[0747] "Monitoring means" refers to devices or systems such as sensors or cameras that monitor the surrounding conditions while driving and detect any abnormalities that may occur.

[0748] The "abnormality notification means" is a device or system for notifying the central management means of information when the monitoring means detects an abnormality.

[0749] "Emergency response measures" refer to systems and personnel that can respond quickly and appropriately when an emergency occurs.

[0750] "Communication means" refers to a device or system for transmitting information from the operation control means and monitoring means to the central management means.

[0751] "Operation information" refers to real-time information including data such as the train's current location, speed, destination, and route points, as well as weather data and track condition data.

[0752] An "operation route" is the path a train takes to reach its destination, which is optimized by a central control means.

[0753] An "automatic driving system" is a system that automatically controls train operation based on instructions from an operation control means.

[0754] An "emergency brake" is a braking device that immediately stops a train in an emergency.

[0755] The present invention relates to a vehicle system with an automatic driving function, which enables safe and efficient operation even in emergency situations. The system comprises the following components:

[0756] Overall system configuration

[0757] This system consists of a central management means, operation control means, monitoring means, abnormality notification means, and vehicles on board with personnel to respond to emergencies. The central management means functions as a server, collecting and analyzing operation information and generating instructions. The operation control means functions as a train terminal, operating the vehicle based on instructions from the server. The monitoring means and abnormality notification means are made up of devices such as cameras and sensors built into the terminals. Emergency response means include having personnel on board to respond to emergencies (usually a conductor).

[0758] Program processing

[0759] Server Processing

[0760] Collection of operation information

[0761] The server collects train operation information such as the train's current location, speed, destination, and route points from a database (e.g., PostgreSQL, MongoDB), and also obtains real-time weather data from a weather API (e.g., OpenWeatherMap API) and track condition data from a track monitoring system API, and analyzes this information.

[0762] Route and speed calculation

[0763] The server uses AI algorithms to calculate the optimal route and speed based on the collected operational information. Specifically, machine learning libraries (e.g., TensorFlow, PyTorch) are used to enable efficient and safe operation.

[0764] Generate and send instructions

[0765] Based on the calculation results, the server generates specific operating route and speed instructions for the operation control means (terminal) and sends them via a communication protocol (e.g., HTTP, MQTT).

[0766] Terminal (train) processing

[0767] Receiving instructions

[0768] The terminal receives the route and speed instructions sent from the server using edge computing devices (e.g., Raspberry Pi, Arduino) and uses Wi-Fi as the communication method.

[0769] Operation control

[0770] Based on the received instructions, the terminal controls the operation of the train through an automated driving system (e.g., ROS), specifically, driving the train at a specified speed along a specified route.

[0771] Surroundings monitoring

[0772] The device is equipped with a camera (e.g., Logitech C920) and a LIDAR sensor as a monitoring tool, and continuously monitors the surroundings while in operation. This information is sent to a server in real time.

[0773] Anomaly detection and notification

[0774] If the monitoring means detects an abnormality (e.g., an obstacle on the track or an abnormal speed limit), it immediately sends the information to the server through the abnormality notification means using image analysis software (e.g., OpenCV, TensorFlow Object Detection API).

[0775] User (conductor) role

[0776] Monitoring normal operations

[0777] During normal operation, the user (conductor) monitors the situation inside the train and checks for any problems. He or she patrols the train regularly to ensure safety.

[0778] Emergency response

[0779] When the server notifies the user of an emergency, the user (conductor) provides explanations to passengers, guides them to evacuate, and checks for and removes obstacles depending on the situation.

[0780] Reporting and Feedback

[0781] After the emergency situation has been resolved, the user (conductor) reports the details of the emergency response and the current safety status to the server. Based on this report, the next steps are taken promptly.

[0782] Specific examples

[0783] For example, if a train is in motion and encounters an obstacle on the tracks, it might do the following:

[0784] 1. Obstacle detection

[0785] The device's camera detects obstacles on the tracks and sends the information to the server. OpenCV is used for image analysis.

[0786] 2. Emergency stop instruction

[0787] The server analyzes the obstacle information and immediately sends an emergency stop command to the terminal.

[0788] 3. Performing an emergency shutdown

[0789] The terminal receives the emergency stop command and activates the emergency brake to stop the train.

[0790] 4. Conductor's Response

[0791] The conductor will guide passengers and guide them to safety based on instructions from the server, and will also check for and remove obstacles.

[0792] 5. Service resumes

[0793] After the obstacle is removed and safety is confirmed, the server calculates a new route and speed and sends it to the device, which receives the new instructions and resumes operation.

[0794] Example prompts for generative AI models

[0795] "Based on the following automated driving vehicle system, please explain in detail the safe operation process in an emergency. Please include the entire procedure from detecting an obstacle, making an emergency stop, responding to passengers, and resuming operation."

[0796] By feeding this prompt into a generative AI model, a detailed explanation of how a similar system would respond to an emergency can be obtained.

[0797] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0798] Step 1: Collecting traffic information

[0799] The server collects train operation information such as the train's current location, speed, destination, and route information from a database (e.g., PostgreSQL, MongoDB). It also retrieves real-time weather data from a weather API (e.g., OpenWeatherMap API) and track condition data from a track monitoring system API.

[0800] Input: Operation information in the database, weather data from the weather API, track condition data from the track monitoring system API

[0801] Output: Operation information, weather data, track condition data

[0802] Specific operation: The server executes a database query to extract the necessary operation information, and calls the weather API and track monitoring system API to obtain real-time data.

[0803] Step 2: Calculate route and speed

[0804] Based on the collected operational information, the server uses AI algorithms (e.g., TensorFlow, PyTorch) to calculate the optimal route and speed.

[0805] Input: Operation information, weather data, and track condition data collected in Step 1

[0806] Output: Optimal route and speed

[0807] Specific operation: The server runs an AI algorithm, analyzes operation information, weather data, and track condition data, and calculates the optimal operating route and speed.

[0808] Step 3: Generate and send instructions

[0809] Based on the calculation results, the server generates and sends specific operating route and speed instructions to the operation control means (terminal) via a communication protocol (e.g., HTTP, MQTT).

[0810] Input: Optimal route and speed calculated in step 2

[0811] Output: Route instructions, speed instructions

[0812] Specific operation: The server generates route and speed instructions and uses a communication protocol to send them to the terminal.

[0813] Step 4: Receiving instructions

[0814] The terminal receives route and speed instructions sent from the server, and uses edge computing devices (e.g., Raspberry Pi, Arduino) as hardware.

[0815] Input: Route and speed instructions from the server

[0816] Output: Received route and speed instructions (data stored in the device)

[0817] Specific operation: The device receives data sent from the server via Wi-Fi.

[0818] Step 5: Controlling the operation

[0819] Based on the received instructions, the terminal controls the train operation through the autonomous driving system (e.g., ROS).

[0820] Input: Route and speed instructions received in step 4

[0821] Output: Specific operation control (speed adjustment, route selection)

[0822] Specific operation: The terminal activates the autonomous driving system and controls the train's operation according to the specified speed and route.

[0823] Step 6: Monitor your surroundings

[0824] The device is equipped with a camera (e.g., Logitech C920) and a LIDAR sensor to continuously monitor the surroundings while driving, and this information is sent to a server in real time.

[0825] Input: Video data from cameras and LIDAR sensors, sensor data

[0826] Output: Surrounding situation data (sent to server)

[0827] Specific operation: The device uses the camera and sensors to capture the surrounding situation and sends the data to the server.

[0828] Step 7: Anomaly detection and notification

[0829] When the device detects an abnormality using a camera or sensor, it analyzes it using image analysis software (e.g., OpenCV), and if an abnormality is confirmed, it immediately sends the information to the server via HTTP POST.

[0830] Input: Surroundings data collected in step 6

[0831] Output: Anomaly detection data (notification to server)

[0832] Specific operation: The device analyzes the data using image analysis software, and if an abnormality is detected, it notifies the server.

[0833] Step 8: Monitor normal operations

[0834] During normal operation, the user (conductor) monitors the situation inside the train and checks for any problems. He or she patrols the train regularly to ensure safety.

[0835] Input: In-car situation

[0836] Output: Safety confirmation report

[0837] Specific actions: The conductor will patrol the train regularly to ensure the safety of passengers.

[0838] Step 9: Emergency response

[0839] When the server notifies the user of an emergency, the user (conductor) provides explanations to passengers and guides them to evacuate. The user also checks for and removes obstacles depending on the situation.

[0840] Input: Emergency notification from the server, actual emergency situation

[0841] Output: Passenger response, evacuation guidance, obstacle removal report

[0842] Specific actions: The conductor will make an announcement to passengers and take appropriate action and provide evacuation guidance.

[0843] Step 10: Reporting and Feedback

[0844] After the emergency situation has been resolved, the user (conductor) reports the details of the emergency response and the current safety status to the server.

[0845] Input: Emergency response history, current safety confirmation

[0846] Output: Detailed report

[0847] Specific operation: The conductor records the series of steps and results of the emergency response and sends them to the server.

[0848] (Application example 1)

[0849] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0850] While conventional autonomous vehicle systems can provide safe and efficient operation, they have the problem of being unable to take immediate and appropriate action when an emergency occurs. In particular, there are cases where personnel on board the vehicle are unable to respond quickly, and it takes time to ensure the safety of passengers. In addition, operation information and abnormality notifications are only sent to the operation manager, resulting in a lack of information for on-site response.

[0851] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0852] In this invention, the server includes a central management means for collecting operation information and calculating operation routes and speeds, an operation control means for receiving instructions from the central management means and controlling the operation of the vehicle, a monitoring means for monitoring the surrounding conditions while driving and detecting abnormalities, an abnormality notification means for notifying the central management means when an abnormality is detected, and a means for sending a notification to a smart device using the information when an emergency occurs. This enables a quick and appropriate response when an emergency occurs, thereby improving the safety of passengers and vehicles.

[0853] "Operation information" is a general term for detailed data such as the vehicle's current location, speed, destination, route points, weather data, and track conditions.

[0854] The "central management means" is a device or system that functions as a server, collects and analyzes operation information, and generates instructions.

[0855] The "operation control means" is a device or system that receives instructions from the central control means and controls the operation of the vehicle.

[0856] "Monitoring means" refers to a device or system that includes devices such as cameras and sensors for monitoring the surrounding conditions during operation and detecting abnormalities.

[0857] The "abnormality notification means" is a device or system that, when an abnormality is detected, promptly notifies the central management means of that information.

[0858] An "emergency situation" refers to an unexpected accident, malfunction, or abnormal event that occurs while a vehicle is in operation.

[0859] A "means for sending notifications to smart devices" is a device or system for quickly sending corresponding information to devices such as smartphones and tablets when an emergency occurs.

[0860] "Personnel" refers to personnel in the vehicle who are intended to respond in the event of an emergency.

[0861] This invention relates to a vehicle system with an automated driving function, which enables safe and efficient operation even in emergency situations. The system includes a central management means for collecting operation information and calculating operation routes and speeds, an operation control means for receiving instructions from the central management means and controlling the operation of the vehicle, a monitoring means for monitoring the surrounding conditions while driving and detecting abnormalities, an abnormality notification means for notifying the central management means when an abnormality is detected, a vehicle carrying personnel who will respond in the event of an emergency, and a means for using the information to send a notification to a smart device when an emergency occurs.

[0862] System program description

[0863] The program of this system is configured as follows:

[0864] 1. Collection of operation information

[0865] The server retrieves operational information from the database, such as vehicle location, speed, destination, and route information, as well as real-time weather and track condition data. This information is continuously updated and sent to a central control unit.

[0866] 2. Route and speed calculation

[0867] The server uses AI algorithms to calculate optimal routes and speeds based on collected traffic information, using standard server hardware and software to run the AI ​​models.

[0868] 3. Instruction Generation and Transmission

[0869] Based on the calculation results, the server proposes specific driving routes and speeds and sends them to the central management means, which then receives the instructions and controls the driving of the vehicles.

[0870] 4. Surrounding situation monitoring and abnormality detection

[0871] The device's onboard cameras and sensors monitor the surroundings while driving, and if an abnormality is detected, the information is sent to a server in real time. This monitoring method is implemented using high-performance image recognition software and sensors.

[0872] 5. Abnormality notification and emergency response

[0873] When an emergency occurs, the server analyzes the information and issues appropriate instructions for response. The server then sends an emergency notification to smart devices, prompting occupants and other responders to act quickly. This functionality is achieved using a notification service such as Firebase Cloud Messaging (FCM).

[0874] Example of a system

[0875] For example, the following is an example of a prompt that may be issued when an obstacle is discovered on the tracks during operation:

[0876] Example of a text prompt:

[0877] "Information about autonomous vehicles is obtained from a central server, and operational status is monitored in real time. If an abnormality occurs, an emergency notification is sent to the user's smartphone, instructing them on the appropriate response."

[0878] This system will enable a swift and appropriate response in the event of an emergency, enhancing the safety of passengers and trains. Conductors and train operators will be able to respond immediately to emergencies, provide guidance to passengers, and guide them to safety.

[0879] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0880] Step 1:

[0881] The server obtains operation information such as the vehicle's current location, speed, destination, and route points from the database. It also obtains real-time weather data and track condition data and integrates this information. The input is data from various sensors and the database, and the output is integrated operation information. This allows the server to have the most up-to-date operation information.

[0882] Step 2:

[0883] The server uses an AI algorithm based on the collected traffic information to calculate the optimal route and speed. The input is the traffic information obtained in the previous step, and the output is the optimal route and speed instructions. Specifically, the AI ​​model analyzes real-time data and calculates an efficient and safe route.

[0884] Step 3:

[0885] The server generates specific driving route and speed instructions based on the calculation results and sends the instructions to the central control means, where the input is the optimized driving route and speed instructions and the output is the actual instruction information, so that the central control means receives the latest driving instructions.

[0886] Step 4:

[0887] The terminal receives the driving route and speed instructions sent from the central control means and controls the driving of the vehicle through the automatic driving system. The input is the driving instructions from the central control means and the output is the actual driving status. In concrete terms, the terminal controls the speed of the vehicle and drives it along the specified route.

[0888] Step 5:

[0889] The device uses its on-board camera and sensors to monitor the surroundings while driving and detect any abnormalities. The input is real-time data from the camera and sensors, and the output is the results of any abnormalities detected. Specifically, image analysis software analyzes the video and detects obstacles and abnormal speeds.

[0890] Step 6:

[0891] When an abnormality is detected, the terminal notifies the server in real time. The input is the abnormality detection data, and the output is an abnormality notification to the central management means. This allows the server to immediately recognize the abnormality.

[0892] Step 7:

[0893] The server analyzes the anomaly information and generates appropriate response instructions. The input is the anomaly notification data, and the output is emergency response instructions. Depending on the anomaly that has occurred, the server generates instructions such as an emergency shutdown.

[0894] Step 8:

[0895] The server sends emergency response instructions to smart devices and notifies crew members and other relevant parties. The input is the emergency response instructions, and the output is a notification message. Specifically, the server sends an emergency notification to smartphones and tablets via Firebase Cloud Messaging (FCM).

[0896] Step 9:

[0897] The user (crew member / conductor) implements emergency response based on the notification from the server. The input is the emergency response instructions displayed on the smart device, and the output is the actual response action. Specific actions include explaining the situation to passengers and guiding them to evacuate.

[0898] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0899] The present invention further improves the safety and efficiency of driving by combining an emotion engine with a vehicle system having an automatic driving function. Hereinafter, embodiments of the present invention will be described in detail.

[0900] Overall system configuration

[0901] The system includes the following major components:

[0902] Central management means (server): Manages and analyzes operation information and generates instructions.

[0903] Operation control means (terminal): Receives instructions from the central control means and controls the operation of the vehicle.

[0904] Monitoring Measures: Continuously monitor the surrounding environment and conditions while the vehicle is in operation.

[0905] Abnormality notification means: Notifies the central management means when an abnormality is detected.

[0906] Emotion engine: Recognizes the emotional state of personnel (conductors) and reflects it in management and response.

[0907] Personnel (Conductor): Operates within the vehicle to respond to emergencies.

[0908] Program processing

[0909] Server Processing

[0910] 1. Collection of operational information:

[0911] The server retrieves information about the train's current location, speed, destination, stops, and route from a database, as well as real-time weather and track condition data.

[0912] 2. Route and speed calculation:

[0913] The server uses AI algorithms to calculate the optimal route and speed based on the collected operational information, records the calculation results, and generates instructions.

[0914] 3. Generate and send instructions:

[0915] The server sends the generated instructions to the operation control means (terminal), which include detailed operation routes, speeds, and emergency response procedures.

[0916] 4. Real-time analysis:

[0917] Even after the vehicle has started moving, the server continues to receive and analyze data on the surrounding situation sent from the device in real time, and sends new operating instructions to the device as needed.

[0918] 5. Emotion recognition and response:

[0919] The server receives and analyzes the conductor's emotional data sent from the emotion engine, and responds accordingly based on the conductor's mental state and stress level.

[0920] Terminal (train) processing

[0921] 1. Receiving and setting instructions:

[0922] The terminal receives the route and speed instructions sent from the server, checks the received information, and sets it up in the autonomous driving system.

[0923] 2. Navigation control:

[0924] The device's autonomous driving system operates the train based on the received route and speed instructions, and the train travels along the specified route at the specified speed.

[0925] 3. Surrounding Area Monitoring:

[0926] The terminals are equipped with cameras and sensors as monitoring tools, and continuously monitor the surroundings while in operation. This information is sent to a server in real time.

[0927] 4. Anomaly detection and notification:

[0928] If the monitoring means detects an abnormality (for example, an obstacle on the tracks or an abnormal speed limit), the information is immediately sent to the server via the abnormality notification means.

[0929] 5. Collecting Emotional Data:

[0930] The emotion engine continuously monitors the conductor's emotional state and transmits the data to the server.

[0931] User (conductor) role

[0932] 1. Monitoring normal operations:

[0933] During normal operation, the user (conductor) monitors the situation inside the train and checks for any particular problems. He or she also patrols the train regularly to ensure safety.

[0934] 2. Emergency Response:

[0935] When an emergency situation is notified by the server, the conductor will act according to the appropriate emergency response procedures, such as explaining the situation to passengers and guiding them to an evacuation.

[0936] 3. Status reports and feedback:

[0937] After the emergency situation has been resolved, the conductor will report the situation in detail to the server, including the details of the emergency response and confirmation of the current safety situation.

[0938] 4. Emotional management:

[0939] Based on feedback from the emotion engine, the conductor can take measures according to his or her emotional state, such as taking a break if stress levels are high.

[0940] Specific examples

[0941] For example, consider the process that is performed when a train detects an obstacle on the tracks while in operation.

[0942] 1. Obstacle detection:

[0943] The device's camera detects obstacles on the tracks and sends that information to a server.

[0944] 2. Emergency stop instruction:

[0945] The server analyzes the obstacle information and immediately sends an emergency stop command to the terminal if it determines it is necessary. At the same time, it notifies the conductor of the emergency response procedure.

[0946] 3. Performing an emergency stop:

[0947] The terminal receives the emergency stop command and activates the emergency brake to stop the train.

[0948] 4. Conductor's response:

[0949] The conductor will guide passengers and guide them to safety based on instructions from the server. In addition, if possible, the conductor will check for and remove obstacles.

[0950] 5. Emotional data feedback:

[0951] When an emergency occurs, the emotion engine monitors the conductor's emotional state and sends the data to a server, which analyzes the data and provides a response guide based on the conductor's mental state.

[0952] 6. Service resumes:

[0953] After the obstacle is removed and safety is confirmed, the server calculates a new route and speed and sends it to the device, which receives the new instructions and resumes operation.

[0954] In this way, by combining the emotion engine, the vehicle system of the present invention can operate more safely and efficiently, taking into account the mental health of the conductor, from normal operation to emergency situations and even when operation resumes.

[0955] The processing flow will be explained below.

[0956] Server Processing

[0957] Step 1:

[0958] The server retrieves information about the train's current location, speed, destination, stops, and route from a database, as well as real-time weather and track condition data.

[0959] Step 2:

[0960] The server uses AI algorithms to calculate the optimal route and speed based on the collected operational information, records the calculation results, and generates instructions.

[0961] Step 3:

[0962] The server sends the generated instructions to the operation control means (terminal), which include detailed operation routes, speeds, and emergency response procedures.

[0963] Step 4:

[0964] Even after the vehicle has started moving, the server continues to receive and analyze data on the surrounding situation sent from the device in real time, and sends new operating instructions to the device as needed.

[0965] Step 5:

[0966] The server receives and analyzes the conductor's emotional data sent from the emotion engine, and based on the results, provides a response guide according to the conductor's mental state and stress level.

[0967] Terminal (train) processing

[0968] Step 1:

[0969] The terminal receives the route and speed instructions sent from the server, checks the received information, and sets it up in the autonomous driving system.

[0970] Step 2:

[0971] The device's autonomous driving system operates the train based on the received route and speed instructions, and the train travels along the specified route at the specified speed.

[0972] Step 3:

[0973] The terminals are equipped with cameras and sensors as monitoring tools, and continuously monitor the surroundings while in operation. This information is sent to a server in real time.

[0974] Step 4:

[0975] If the monitoring means detects an abnormality (for example, an obstacle on the tracks or an abnormal speed limit), the information is immediately sent to the server via the abnormality notification means.

[0976] Step 5:

[0977] The terminal continuously monitors the conductor's emotional data through the emotion engine and sends that data to the server in real time.

[0978] User (conductor) role

[0979] Step 1:

[0980] During normal operation, the user (conductor) monitors the situation inside the train and checks for any particular problems. He or she also patrols the train regularly to ensure safety.

[0981] Step 2:

[0982] When the server notifies the conductor of an emergency, the conductor acts according to the server's instructions and follows the appropriate emergency response procedures, such as providing explanations to passengers and guiding them to safety.

[0983] Step 3:

[0984] Based on feedback from the emotion engine, the conductor can take measures according to his or her emotional state, such as taking a break if stress levels are high.

[0985] Step 4:

[0986] After the emergency situation has been resolved, the conductor will report the situation in detail to the server, including the details of the emergency response and confirmation of the current safety situation.

[0987] Step 5:

[0988] The conductor confirms that new operating instructions have been sent from the server, informs passengers that operation has resumed, and makes another round to ensure passengers are able to travel safely.

[0989] Specific examples

[0990] For example, consider the process that is performed when a train detects an obstacle on the tracks while in operation.

[0991] Step 1:

[0992] The device's camera detects obstacles on the tracks and sends that information to a server.

[0993] Step 2:

[0994] The server analyzes the obstacle information and immediately sends an emergency stop command to the terminal if it determines it is necessary. At the same time, it notifies the conductor of the emergency response procedure.

[0995] Step 3:

[0996] The terminal receives the emergency stop command and activates the emergency brake to stop the train.

[0997] Step 4:

[0998] The conductor will guide passengers and guide them to safety based on instructions from the server. In addition, if possible, the conductor will check for and remove obstacles.

[0999] Step 5:

[1000] When an emergency occurs, the emotion engine monitors the conductor's emotional state and sends the data to a server, which analyzes the data and provides a response guide based on the conductor's mental state.

[1001] Step 6:

[1002] After the obstacle is removed and safety is confirmed, the server calculates a new route and speed and sends it to the device, which receives the new instructions and resumes operation.

[1003] Example 2

[1004] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1005] While conventional autonomous vehicle systems have certain functions for collecting operational information, calculating operational routes, and detecting abnormalities, they are not yet able to respond quickly and effectively to emergencies or sudden abnormalities.In addition, they lack the functionality to consider the mental state and stress level of the conductor as human factors, which has led to issues with the safety and efficiency of operations.

[1006] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for collecting operation information and calculating operation routes and speeds, a means for analyzing the surrounding situation in real time and generating new operation instructions, and an emotion recognition means for recognizing the emotional state of passengers on board and taking appropriate action. This makes it possible to ensure the safety of operation while improving the efficiency of operation by taking human factors into consideration.

[1007] "Operation information" is data related to the vehicle's current location, speed, destination, intermediate stops, and operating route.

[1008] A "central control means" is a system that collects operational information, calculates operational routes and speeds, and generates instructions.

[1009] The "operation control means" is a device that receives instructions from the central control means and automatically controls the operation of the vehicle.

[1010] "Monitoring means" refers to devices such as cameras and sensors that continuously monitor the surrounding conditions while the vehicle is in operation and detect abnormalities.

[1011] The "abnormality notification means" is a system that notifies the central management means of abnormalities detected by the monitoring means.

[1012] An "emergency situation" refers to an unexpected malfunction, accident, or abnormal situation that occurs during operation.

[1013] The "emotion recognition means" is a system that monitors the emotional state of onboard personnel in real time and analyzes the data.

[1014] "Personnel" refers to the people on board who will respond in the event of an emergency.

[1015] "New driving instructions" are updated driving routes and speed instructions generated based on the results of real-time analysis.

[1016] A "conductor" is a person on board a train who is responsible for normal operation and responding to emergencies.

[1017] The present invention further improves the safety and efficiency of driving by combining emotion recognition means with a vehicle system having an automatic driving function. Hereinafter, embodiments of the present invention will be described in detail.

[1018] Overall system configuration

[1019] The system includes the following main components:

[1020] Central control means (server): The server collects driving information, calculates driving routes and speeds, and analyzes the surrounding situation in real time during driving to generate new driving instructions.

[1021] Operation control means (terminal): The terminal receives instructions from the server and automatically controls the operation of the vehicle.

[1022] Monitoring means: Includes cameras and sensors to continuously monitor the surroundings while the vehicle is in operation and detect any abnormalities.

[1023] Abnormality notification means: Notifies the server of any abnormalities detected by the monitoring means.

[1024] Emotion recognition means: The emotion engine recognizes and analyzes the emotional state of the conductor (personnel), allowing it to respond according to the conductor's mental state and stress level.

[1025] Personnel (conductor): Responsible for responding to emergencies in the vehicle.

[1026] Specific server processing

[1027] The server manages and analyzes operation information in the following steps:

[1028] 1. Collection of operational information:

[1029] The server retrieves information on the train's current location, speed, destination, intermediate stops, and route from the database, as well as real-time weather data and track condition data, and manages this information as comprehensive operational information.

[1030] 2. Route and speed calculation:

[1031] Based on the collected operational information, the server uses the AI ​​algorithm "Route Optimizer" to calculate the optimal route and speed, thereby supporting safe and efficient operation.

[1032] 3. Generate and send instructions:

[1033] The server then sends the generated driving route and speed instructions to the terminal, which include detailed driving route, speed, and emergency response procedures.

[1034] 4. Real-time analysis:

[1035] While the vehicle is in operation, the server receives and analyzes real-time data on the surrounding environment sent from the device, and sends new operating instructions to the device as needed to ensure safe operation.

[1036] 5. Emotion recognition and response:

[1037] The server analyzes the conductor's emotional data sent from the emotion engine "EmotionAnalyzer," and generates and provides the conductor with a response guide based on the conductor's mental state and stress level.

[1038] Specific processing of the terminal (train)

[1039] The terminal automatically controls operation through the following steps:

[1040] 1. Receiving and setting instructions:

[1041] The terminal receives route and speed instructions sent from the server and sets them in the autonomous driving system.

[1042] 2. Navigation control:

[1043] The automated driving system operates the vehicle based on the received route and speed, and follows the specified route at the specified speed.

[1044] 3. Surrounding Area Monitoring:

[1045] Cameras and sensors are used to continuously monitor the surrounding conditions while the vehicle is in operation, and the data is sent to a server.

[1046] 4. Anomaly detection and notification:

[1047] If the monitoring means detects an abnormality, it immediately transmits the information to the server.

[1048] 5. Collecting Emotional Data:

[1049] The emotion engine continuously monitors the conductor's emotional state and transmits the data to the server.

[1050] User (conductor) role

[1051] 1. Monitoring normal operations:

[1052] During normal operation, the conductor monitors the situation inside the train and checks for any particular problems. He also patrols the train regularly to ensure safety.

[1053] 2. Emergency Response:

[1054] When an emergency situation is notified by the server, the conductor will act according to the appropriate emergency response procedures, such as explaining the situation to passengers and guiding them to an evacuation.

[1055] 3. Status reports and feedback:

[1056] After the emergency situation has been resolved, the conductor will report the situation in detail to the server, including the details of the emergency response and confirmation of the current safety situation.

[1057] 4. Emotional management:

[1058] The conductor takes measures based on his or her own emotional state based on feedback from the emotion engine. For example, if stress levels are high, the conductor may take a break.

[1059] Specific examples

[1060] For example, if a train detects an obstacle on the tracks while in operation, the process is as follows.

[1061] 1. Obstacle detection:

[1062] The device's camera detects obstacles on the tracks and sends that information to a server.

[1063] 2. Emergency stop instruction:

[1064] The server analyzes the obstacle information and, if necessary, immediately sends an emergency stop instruction to the terminal, while simultaneously informing the conductor of the emergency response procedure.

[1065] 3. Performing an emergency stop:

[1066] The terminal receives an emergency stop command and activates the emergency brake, bringing the train to a halt.

[1067] 4. Conductor's response:

[1068] The conductor will provide passenger guidance and evacuation guidance according to instructions from the server, and will also check for and remove obstacles if possible.

[1069] 5. Emotional data feedback:

[1070] The emotion engine monitors the conductor's emotional state in the event of an emergency and sends the data to a server, which analyzes the data and provides a response guide based on the conductor's mental state.

[1071] 6. Service resumes:

[1072] After the obstacle is removed and safety is confirmed, the server calculates a new route and speed and sends it to the device, which receives the new instructions and resumes operation.

[1073] Prompt Sentence Examples

[1074] To use a generative AI model to respond to specific situations, we use prompts like the following:

[1075] Example prompt 1: "If a train is in motion and discovers an obstacle on the tracks, explain how you should respond based on your roles as the server, the terminal, and the conductor."

[1076] Example prompt 2: "Please explain in detail how the emotion engine monitors the conductor's emotional state and notifies the server in the event of an emergency."

[1077] These prompts allow the generative AI model to learn system-wide responses and response procedures based on specific scenarios.

[1078] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1079] Server Processing

[1080] Step 1:

[1081] Collection of operation information

[1082] Input: Database, real-time weather data, track condition data

[1083] Data processing / calculation: The server retrieves and organizes information on the train's current location, speed, destination, intermediate stops, and route from the database. It also retrieves and integrates real-time weather information and track conditions from the weather data API.

[1084] Output: Complete set of traffic information

[1085] What it does: The server periodically retrieves data from the database and API and stores it as a train information set, including the train's current location, speed, destination, temperature, precipitation, wind speed, etc.

[1086] Step 2:

[1087] Route and speed calculation

[1088] Input: Service information set

[1089] Data processing / calculation: Based on the operation information, the server uses the AI ​​algorithm "Route Optimizer" to calculate the optimal route and speed, taking into account traffic and weather conditions to derive a safe and efficient route.

[1090] Output: Optimized route and speed instructions

[1091] Specific operation: The server inputs operation information into "RouteOptimizer" and generates a file containing the resulting route and speed instructions, including the route to the next station, speed limits, and emergency stop points.

[1092] Step 3:

[1093] Generate and send instructions

[1094] Input: Route and speed instructions

[1095] Data processing / calculation: The server formats the instructions and sends them to the operation control means (terminal). Safety checks are performed and the contents of the instructions are confirmed.

[1096] Output: Instructions sent to the terminal

[1097] What happens: The server sends instructions to the device via email or a digital communications protocol, and receives a confirmation message to confirm that they were received in the correct format.

[1098] Step 4:

[1099] Real-time analytics

[1100] Input: Ambient data from the device

[1101] Data processing / calculation: The server analyzes the surrounding situation data received from the device in real time, determines whether there are any abnormalities, and generates new driving instructions as necessary.

[1102] Output: New operating instructions

[1103] Specific operation: The server analyzes camera footage and sensor data sent from moving trains, and if it detects an obstacle or abnormality, it generates new operating instructions and immediately sends them to the terminal.

[1104] Step 5:

[1105] Emotion recognition and response

[1106] Input: Emotion data from the emotion engine

[1107] Data processing / calculation: The server analyzes the conductor's emotional data sent from the emotion engine "EmotionAnalyzer." It evaluates the conductor's stress level and emotional state and generates a response guide.

[1108] Output: Conductor's response guide

[1109] Specific actions: Based on the emotional data, the server generates a response guide that includes stress reduction techniques and recommendations for taking breaks, and notifies the conductor. This guide includes specific advice, such as "take deep breaths and stretch to relax."

[1110] Terminal (train) processing

[1111] Step 1:

[1112] Receiving and Setting Instructions

[1113] Input: Route and speed instructions sent from the server

[1114] Data processing / calculation: The device checks the received route and speed instructions and sets them in the autonomous driving system. It also checks the settings and verifies that the vehicle can be driven safely.

[1115] Output: Route and speed set for the autonomous driving system

[1116] Specific operation: The device analyzes the received instructions and sends the appropriate configuration commands to the autonomous driving system. Once the configuration is complete, it sends a confirmation message to the server.

[1117] Step 2:

[1118] Operation control

[1119] Input: Set route and speed

[1120] Data processing / calculation: The autonomous driving system controls the vehicle based on the route and speed instructions it receives, adjusting speed and stopping positions to ensure safe and efficient driving.

[1121] Output: Actual driving data (current location, speed, etc.)

[1122] Specific operation: The device uses an autonomous driving system to drive along a designated route at a set speed. During operation, the device constantly monitors its speedometer and location information and makes adjustments as necessary.

[1123] Step 3:

[1124] Surroundings monitoring

[1125] Input: Camera and sensor data

[1126] Data processing / calculation: Analyzes data sent from cameras and sensors, which are used as monitoring tools, and monitors the surrounding situation. If an abnormality is detected, the data is sent to the server.

[1127] Output: Surroundings data

[1128] Specific operation: High-resolution cameras and various sensors installed on the terminal acquire data in real time, and perform image analysis and pattern recognition to determine whether there are any abnormalities. This data is then sent to the server.

[1129] Step 4:

[1130] Anomaly detection and notification

[1131] Input: Camera and sensor data

[1132] Data processing / calculation: If the monitoring means detects an abnormality, the information is immediately sent to the server via the abnormality notification means.

[1133] Output: Error notification data

[1134] Specific operation: If an abnormality is detected, the device will send the information to the server via the abnormality notification means. For example, if an obstacle is detected on the tracks, the device will send the video and data to the server.

[1135] Step 5:

[1136] Collecting Emotional Data

[1137] Input: Data from the emotion engine "EmotionAnalyzer"

[1138] Data processing / calculation: The emotion engine continuously monitors the conductor's emotional state and sends the data to the server.

[1139] Output: Conductor's emotion data

[1140] Specific operation: The emotion engine monitors the conductor's facial expressions, voice, heart rate, etc. in real time, analyzes this data, and processes it into emotion data. The analyzed data is sent to a server, and a response is made based on the conductor's emotional state.

[1141] User (conductor) role

[1142] Step 1:

[1143] Monitoring normal operations

[1144] Input: Passenger and vehicle status data

[1145] Data processing / calculation: Conductors monitor the situation inside the train during normal operation and check for any particular problems. They also make regular rounds to check safety.

[1146] Output: Safety report data

[1147] Specific actions: The conductor checks the train announcements and security camera footage to see if there are any problems. Depending on the situation, he or she will make announcements to passengers and ensure their safety.

[1148] Step 2:

[1149] Emergency response

[1150] Input: Emergency notification data from the server

[1151] Data Processing / Calculation: When an emergency situation is notified by the server, the conductor will act according to the appropriate emergency response procedures.

[1152] Output: Emergency response report data

[1153] Specific actions: In the event of an emergency, the conductor will explain the situation to passengers and guide them to evacuate. For example, an announcement will be made on the train saying, "This is an emergency, please disembark."

[1154] Step 3:

[1155] Status reports and feedback

[1156] Input: Post-emergency data

[1157] Data processing / calculation: After the emergency situation has been resolved, the conductor will report the situation in detail to the server.

[1158] Output: Emergency report details

[1159] Specific operation: The conductor records the details of the emergency response and the results, and reports them to the server. For example, he / she describes the success of evacuation guidance and the results of safety confirmation.

[1160] Step 4:

[1161] emotional management

[1162] Input: Feedback from the emotion engine

[1163] Data processing / calculation: The conductor takes measures based on his / her own emotional state based on feedback from the emotion engine.

[1164] Output: Improved emotional state data

[1165] Specific Actions: Conductors receive feedback from the Emotion Engine and implement stress management and relaxation techniques, such as deep breathing and relaxation exercises during designated breaks.

[1166] The above is the specific processing flow of the program of this system and the specific operations performed at each processing step.

[1167] (Application example 2)

[1168] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1169] In order to improve the safety and efficiency of autonomous driving vehicle systems, it is necessary not only to monitor the surrounding environment and situation, but also to grasp the emotional state of the personnel on board in real time and respond accordingly. However, conventional autonomous driving systems lack the ability to monitor emotional states, and lack means to reduce the mental burden on conductors and drivers in emergencies. Therefore, comprehensive improvements to operational safety and comfort are a challenge.

[1170] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting operation information and calculating operation routes and speeds, means for receiving instructions from the central control means and controlling the operation of the vehicle, means for monitoring the surrounding conditions while driving and detecting abnormalities, and means for monitoring the emotional state of the vehicle in real time and transmitting the emotional data to the central control means. This makes it possible to improve the safety and comfort of operation by adjusting operation and reducing stress according to the emotional state of the conductor and driver.

[1171] "Operation information" is data related to the vehicle's current location, speed, destination, intermediate stops, and operating route.

[1172] "Central control means" refers to the system that collects, analyzes, and generates instructions regarding traffic information.

[1173] The "operation control means" is a system that receives instructions from the central control means and controls the operation of the vehicle.

[1174] The "monitoring means" is a system that continuously monitors the surrounding environment and conditions while driving and detects abnormalities.

[1175] The "abnormality notification means" is a system that notifies the central management means of information when an abnormality is detected.

[1176] The "emotion recognition means" is a system that monitors the emotional state inside the vehicle in real time and transmits the data to a central control means.

[1177] "Vehicle" refers to a land transportation means that has an automatic driving function and has personnel on board to respond in the event of an abnormality or emergency during operation.

[1178] "Route" refers to the route chosen for a vehicle to travel to its destination.

[1179] An "emergency situation" refers to an unexpected abnormality or malfunction that occurs during operation and requires immediate response.

[1180] The "operation adjustment means" is a system that adjusts the route and speed of the vehicle depending on the vehicle's emotional state and surrounding circumstances.

[1181] A specific system configuration and its operation will be described below as an embodiment of the invention.

[1182] Overall system configuration

[1183] The system includes the following major components:

[1184] 1. Central management means (server): Collects operation information, analyzes it, and generates instructions. The server calculates the route and speed, and generates and sends operation instructions.

[1185] 2. Operation control means (terminal): A terminal that receives instructions from the central control means and controls the operation of vehicles. It operates vehicles based on the instructions for route and speed.

[1186] 3. Monitoring means: Using cameras and sensors, the surrounding environment and conditions are monitored during operation. If an abnormality is detected, it is notified to the central control means via a terminal.

[1187] 4. Abnormality notification means: Notifies the central control means when an abnormality is detected. This means immediately sends information to the central control means when an abnormality occurs.

[1188] 5. Emotion recognition means: The emotional state of personnel on board (conductors and drivers) is monitored in real time using cameras and emotion recognition software, and the data is sent to a central management means.

[1189] 6. Vehicles: These vehicles have autonomous driving capabilities and are manned by personnel who can respond to emergencies.

[1190] Server Processing

[1191] The server performs the following process:

[1192] 1. Operational information collection: The server collects information on the vehicle's current location, speed, destination, intermediate stops, and route, as well as real-time weather and track condition data.

[1193] 2. Calculation of route and speed: Based on the collected operation information, an AI algorithm is used to calculate the optimal route and speed.

[1194] 3. Generation and transmission of instructions: Generate instructions based on the calculation results and send them to the operation control means.

[1195] 4. Real-time analysis: Analyzes surrounding situation data and emotional data sent from the operation control means and sends new instructions based on the results.

[1196] 5. Emotion recognition and response: Data from the emotion recognition system is received and analyzed, and operation adjustments and stress reduction measures are implemented based on the driver's stress level.

[1197] Terminal handling

[1198] The terminal performs the following process:

[1199] 1. Receiving and setting instructions: Receive driving instructions from the server and set them in the autonomous driving system.

[1200] 2. Operation control: Operate the vehicle based on the received route and speed instructions.

[1201] 3. Surroundings monitoring: Cameras and sensors are used to monitor the surroundings while driving.

[1202] 4. Abnormality notification: If an abnormality is detected, the information is immediately sent to a central control means.

[1203] Emotion Recognition System Processing

[1204] 1. Real-time emotion monitoring: Using cameras and emotion recognition software, the facial expressions of the vehicle's occupants are analyzed to determine their emotional state.

[1205] 2. Sending emotion data to the server: The recognized emotion data is sent to the server and analyzed in real time.

[1206] 3. Operational adjustment: If the emotional state is judged to be stressful, the route or speed of the driver will be adjusted.

[1207] 4. Stress reduction measures: Reduce stress levels for drivers and passengers by playing stress-reducing music and adjusting lighting.

[1208] Specific examples

[1209] For example, if a camera inside the vehicle detects that the driver is fatigued, the information is immediately sent to a server. The server uses the data to adjust the route and speed to ensure the driver can continue driving safely. If the server determines that the driver is feeling stressed, it will play relaxing music and adjust the lighting inside the vehicle to reduce stress.

[1210] Using generative AI models

[1211] Example prompt: "The driver is stressed, please play some relaxing music."

[1212] In this way, the system of the present invention combines emotion recognition to comprehensively improve driving safety and comfort.

[1213] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1214] Step 1:

[1215] Collection of operation information

[1216] The server retrieves information about the vehicle's current location, speed, destination, intermediate stops, and route from a database, as well as real-time weather and track condition data.

[1217] Input: Vehicle location information, speed information, weather data, track condition data

[1218] Data processing: Organize and store collected operation information in an analyzable format

[1219] Output: Formatted traffic information data

[1220] Step 2:

[1221] Route and speed calculation

[1222] Based on the operation information collected by the server, an AI algorithm is used to calculate the optimal operation route and speed.

[1223] Input: Formatted traffic information data

[1224] Data calculation: Optimization of route and speed is performed using AI algorithms

[1225] Output: Calculated optimal route and speed instructions

[1226] Step 3:

[1227] Generate and send instructions

[1228] The server transmits the generated travel route and speed instructions to the travel control means.

[1229] Input: Optimal route and speed instructions

[1230] Data processing: Converting data into a format for transmission

[1231] Output: Data sent to the operation control means

[1232] Step 4:

[1233] Receiving and Setting Instructions

[1234] The terminal receives driving instructions sent from the server and sets them in the autonomous driving system.

[1235] Input: Data sent to the operation control means

[1236] Data processing: Converting received instructions into the setting format of the autonomous driving system

[1237] Output: Configured autonomous driving system

[1238] Step 5:

[1239] Operation control

[1240] The device's autonomous driving system operates the vehicle based on the received route and speed instructions.

[1241] Input: Configured autonomous driving system

[1242] Specific behavior: The vehicle travels along a specified route at a specified speed.

[1243] Output: Vehicles in operation

[1244] Step 6:

[1245] Surrounding Area Monitoring

[1246] The device uses cameras and sensors to monitor the surroundings while in operation.

[1247] Input: Surrounding situation data (video and sensor information)

[1248] Data processing: Real-time situation analysis

[1249] Output: Parsed situation data

[1250] Step 7:

[1251] Abnormality notification

[1252] If the monitoring means detects an abnormality, it immediately transmits the information to the central control means.

[1253] Input: Parsed situation data

[1254] Data processing: Analysis to see if there are any abnormalities

[1255] Output: Error notification data

[1256] Step 8:

[1257] Collecting and transmitting emotional data

[1258] An emotion recognition means monitors the emotional state of the personnel in the vehicle and transmits the data to a server.

[1259] Input: Camera footage, emotion recognition software analysis results

[1260] Data Computation: Analysis with Emotion Recognition Algorithms

[1261] Output: Parsed emotion data

[1262] Step 9:

[1263] Emotion recognition and response

[1264] The server receives and analyzes the data from the emotion recognition means, and based on the results, adjusts operation and takes measures to reduce stress according to the driver's stress level.

[1265] Input: Parsed emotion data

[1266] Data processing: assessing stress states and generating response instructions

[1267] Output: Operation adjustment instructions, stress reduction measures

[1268] Step 10:

[1269] Operation adjustments

[1270] The terminal receives operation adjustment instructions from the server and adjusts the operation route and speed.

[1271] Input: Operation adjustment instructions

[1272] Specific behavior: Resetting the route and speed

[1273] Output: Operating state after adjustment

[1274] Step 11:

[1275] Stress reduction measures

[1276] The device will implement stress-reducing measures (such as playing relaxing music or adjusting the lighting).

[1277] Input: Stress reduction measures (specific instructions)

[1278] Specific actions: Change the in-car environment (play music, adjust lighting, etc.)

[1279] Output: Improved in-car environment

[1280] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1281] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1282] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1283] [Third embodiment]

[1284] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1285] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1286] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1287] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1288] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1289] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1290] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1291] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1292] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1293] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1294] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1295] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1296] The present invention relates to a vehicle system with an automatic driving function, and in particular to a system that enables safe and efficient operation even in an emergency. Hereinafter, an embodiment of the present invention will be described in detail.

[1297] Overall system configuration

[1298] This system mainly consists of a central management means, operation control means, monitoring means, abnormality notification means, and vehicles carrying emergency response personnel (usually conductors). The central management means functions as a server, collecting and analyzing operation information and generating instructions. The operation control means functions as a train terminal, operating the vehicle based on instructions from the server. The monitoring means and abnormality notification means are composed of devices such as cameras and sensors built into the terminals.

[1299] Program processing

[1300] Server Processing

[1301] 1. Collection of operational information:

[1302] The server collects train operation information, such as the train's current location, speed, destination, and route information, from a database, as well as real-time weather and track condition data, and analyzes this information.

[1303] 2. Route and speed calculation:

[1304] The server uses AI algorithms to calculate the optimal route and speed based on the collected operational information, enabling efficient and safe operation.

[1305] 3. Generate and send instructions:

[1306] Based on the calculation results, the server generates and sends specific instructions on the driving route and speed to the driving control means (terminal).

[1307] Terminal (train) processing

[1308] 1. Receiving instructions:

[1309] The terminal receives the driving route and speed instructions sent from the server.

[1310] 2. Navigation control:

[1311] Based on the received instructions, the terminal controls the operation of the train through the automated driving system, specifically, driving at a specified speed along a specified route.

[1312] 3. Surrounding Area Monitoring:

[1313] The terminals are equipped with cameras and sensors as monitoring tools, and continuously monitor the surroundings while in operation, sending information to a server in real time.

[1314] 4. Anomaly detection and notification:

[1315] If the monitoring means detects an abnormality (for example, an obstacle on the tracks or an abnormal speed limit), the information is immediately sent to the server via the abnormality notification means.

[1316] User (conductor) role

[1317] 1. Monitoring normal operations:

[1318] During normal operation, the user (conductor) monitors the situation inside the train and checks for any problems. He or she patrols the train regularly to ensure safety.

[1319] 2. Emergency Response:

[1320] When the server notifies the conductor of an emergency, the conductor will act according to the appropriate emergency response procedures, such as explaining the situation to passengers, guiding them to an evacuation, and checking for and removing obstacles if possible.

[1321] 3. Reporting and Feedback:

[1322] After the emergency situation has been resolved, the conductor will report the situation in detail to the server, including details of the emergency response and current safety confirmation.

[1323] Specific examples

[1324] For example, consider the process that is performed when a train detects an obstacle on the tracks while in operation.

[1325] 1. Obstacle detection:

[1326] The device's camera detects obstacles on the tracks and sends that information to a server.

[1327] 2. Emergency stop instruction:

[1328] The server analyzes the obstacle information and immediately sends an emergency stop instruction to the terminal if it determines it is necessary.

[1329] 3. Perform an emergency stop:

[1330] The terminal receives the emergency stop command and activates the emergency brake to stop the train.

[1331] 4. Conductor's response:

[1332] The conductor will guide passengers and guide them to safety based on instructions from the server. In addition, if possible, the conductor will check for and remove obstacles.

[1333] 5. Service resumes:

[1334] After the obstacle is removed and safety is confirmed, the server calculates a new route and speed and sends it to the device, which receives the new instructions and resumes operation.

[1335] In this way, the vehicle system of the present invention can operate safely and efficiently both during normal operation and in emergency situations.

[1336] The processing flow will be explained below.

[1337] Server Processing

[1338] Step 1:

[1339] The server retrieves information about the train's current location, speed, destination, stops, and route from a database, as well as real-time weather and track condition data.

[1340] Step 2:

[1341] The server uses AI algorithms to calculate the optimal route and speed based on the collected operational information, records the calculation results, and generates instructions.

[1342] Step 3:

[1343] The server sends the generated instructions to the operation control means (terminal), which include detailed operation routes, speeds, and emergency response procedures.

[1344] Step 4:

[1345] Even after the vehicle has started moving, the server continues to receive and analyze data on the surrounding situation sent from the device in real time, and sends new operating instructions to the device as needed.

[1346] Terminal (train) processing

[1347] Step 1:

[1348] The terminal receives the route and speed instructions sent from the server, checks the received information, and sets it up in the autonomous driving system.

[1349] Step 2:

[1350] The device's autonomous driving system operates the train based on the received route and speed instructions, and the train travels along the specified route at the specified speed.

[1351] Step 3:

[1352] The terminals are equipped with cameras and sensors as monitoring tools, and continuously monitor the surroundings while in operation. This information is sent to a server in real time.

[1353] Step 4:

[1354] If the monitoring means detects an abnormality (for example, an obstacle on the tracks or an abnormal speed limit), the information is immediately sent to the server via the abnormality notification means.

[1355] Step 5:

[1356] When the terminal receives new instructions from the server, it will take emergency measures such as applying the emergency brakes, and will wait until it receives an instruction to resume operation.

[1357] User (conductor) role

[1358] Step 1:

[1359] The user (conductor) monitors the situation inside the train during normal operation and checks for any particular problems. He or she patrols the train regularly to ensure safety.

[1360] Step 2:

[1361] When the server notifies the conductor of an emergency, the conductor acts according to the server's instructions and follows the appropriate emergency response procedures, such as providing explanations to passengers and guiding them to safety.

[1362] Step 3:

[1363] The user (conductor) follows the instructions generated by the server and uses in-car announcements to communicate operational information and emergency response measures to passengers.

[1364] Step 4:

[1365] After the emergency situation has been resolved, the conductor will report the situation in detail to the server, including the details of the emergency response and confirmation of the current safety situation.

[1366] Step 5:

[1367] The conductor confirms that new operating instructions have been sent from the server, informs passengers that operation has resumed, and makes another round to ensure passengers are able to travel safely.

[1368] In this way, the server, terminal, and user (conductor) work together to ensure that the entire process, from normal operation to the occurrence of an emergency and then resumption of operation, proceeds safely and efficiently.

[1369] Example 1

[1370] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1371] To ensure safe and efficient operation in a vehicle system with autonomous driving functions, it is necessary to quickly and accurately collect operational information, optimize routes, monitor surrounding conditions, detect abnormalities, and respond to emergencies. However, in current systems, these functions are performed separately, resulting in problems such as information delays and inefficient processing. Furthermore, if an emergency response is not implemented immediately, it could lead to a serious accident. It is necessary to solve these issues and provide a system in which all functions are integrated.

[1372] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1373] In this invention, the server includes means for collecting operation information and calculating operation routes and speeds, means for receiving instructions from a central control means and controlling vehicle operation, means for monitoring the surrounding conditions while driving and detecting abnormalities, means for notifying the central control means when an abnormality is detected, a vehicle with personnel on board who will respond in the event of an emergency, means for receiving instructions from the operation control means, executing an emergency stop, and monitoring the surrounding conditions, and communication means for transmitting information to the central control means in real time. This enables real-time collection and analysis of operation information, optimization of operation, rapid detection and response to abnormalities, and seamless information transmission.

[1374] The "central management means" is a device or system that collects operation information, calculates operation routes and speeds based on that information, and sends instructions to the operation control means and abnormality notification means.

[1375] The "operation control means" is a device or system that receives instructions from the central control means and controls the operation of the vehicle based on those instructions.

[1376] "Monitoring means" refers to devices or systems such as sensors or cameras that monitor the surrounding conditions while driving and detect any abnormalities that may occur.

[1377] The "abnormality notification means" is a device or system for notifying the central management means of information when the monitoring means detects an abnormality.

[1378] "Emergency response measures" refer to systems and personnel that can respond quickly and appropriately when an emergency occurs.

[1379] "Communication means" refers to a device or system for transmitting information from the operation control means and monitoring means to the central management means.

[1380] "Operation information" refers to real-time information including data such as the train's current location, speed, destination, and route points, as well as weather data and track condition data.

[1381] An "operation route" is the path a train takes to reach its destination, which is optimized by a central control means.

[1382] An "automatic driving system" is a system that automatically controls train operation based on instructions from an operation control means.

[1383] An "emergency brake" is a braking device that immediately stops a train in an emergency.

[1384] The present invention relates to a vehicle system with an automatic driving function, which enables safe and efficient operation even in emergency situations. The system comprises the following components:

[1385] Overall system configuration

[1386] This system consists of a central management means, operation control means, monitoring means, abnormality notification means, and vehicles on board with personnel to respond to emergencies. The central management means functions as a server, collecting and analyzing operation information and generating instructions. The operation control means functions as a train terminal, operating the vehicle based on instructions from the server. The monitoring means and abnormality notification means are made up of devices such as cameras and sensors built into the terminals. Emergency response means include having personnel on board to respond to emergencies (usually a conductor).

[1387] Program processing

[1388] Server Processing

[1389] Collection of operation information

[1390] The server collects train operation information such as the train's current location, speed, destination, and route points from a database (e.g., PostgreSQL, MongoDB), and also obtains real-time weather data from a weather API (e.g., OpenWeatherMap API) and track condition data from a track monitoring system API, and analyzes this information.

[1391] Route and speed calculation

[1392] The server uses AI algorithms to calculate the optimal route and speed based on the collected operational information. Specifically, machine learning libraries (e.g., TensorFlow, PyTorch) are used to enable efficient and safe operation.

[1393] Generate and send instructions

[1394] Based on the calculation results, the server generates specific operating route and speed instructions for the operation control means (terminal) and sends them via a communication protocol (e.g., HTTP, MQTT).

[1395] Terminal (train) processing

[1396] Receiving instructions

[1397] The terminal receives the route and speed instructions sent from the server using edge computing devices (e.g., Raspberry Pi, Arduino) and uses Wi-Fi as the communication method.

[1398] Operation control

[1399] Based on the received instructions, the terminal controls the operation of the train through an automated driving system (e.g., ROS), specifically, driving the train at a specified speed along a specified route.

[1400] Surroundings monitoring

[1401] The device is equipped with a camera (e.g., Logitech C920) and a LIDAR sensor as a monitoring tool, and continuously monitors the surroundings while in operation. This information is sent to a server in real time.

[1402] Anomaly detection and notification

[1403] If the monitoring means detects an abnormality (e.g., an obstacle on the track or an abnormal speed limit), it immediately sends the information to the server through the abnormality notification means using image analysis software (e.g., OpenCV, TensorFlow Object Detection API).

[1404] User (conductor) role

[1405] Monitoring normal operations

[1406] During normal operation, the user (conductor) monitors the situation inside the train and checks for any problems. He or she patrols the train regularly to ensure safety.

[1407] Emergency response

[1408] When the server notifies the user of an emergency, the user (conductor) provides explanations to passengers, guides them to evacuate, and checks for and removes obstacles depending on the situation.

[1409] Reporting and Feedback

[1410] After the emergency situation has been resolved, the user (conductor) reports the details of the emergency response and the current safety status to the server. Based on this report, the next steps are taken promptly.

[1411] Specific examples

[1412] For example, if a train is in motion and encounters an obstacle on the tracks, it might do the following:

[1413] 1. Obstacle detection

[1414] The device's camera detects obstacles on the tracks and sends the information to the server. OpenCV is used for image analysis.

[1415] 2. Emergency stop instruction

[1416] The server analyzes the obstacle information and immediately sends an emergency stop command to the terminal.

[1417] 3. Performing an emergency shutdown

[1418] The terminal receives the emergency stop command and activates the emergency brake to stop the train.

[1419] 4. Conductor's Response

[1420] The conductor will guide passengers and guide them to safety based on instructions from the server, and will also check for and remove obstacles.

[1421] 5. Service resumes

[1422] After the obstacle is removed and safety is confirmed, the server calculates a new route and speed and sends it to the device, which receives the new instructions and resumes operation.

[1423] Example prompts for generative AI models

[1424] "Based on the following automated driving vehicle system, please explain in detail the safe operation process in an emergency. Please include the entire procedure from detecting an obstacle, making an emergency stop, responding to passengers, and resuming operation."

[1425] By feeding this prompt into a generative AI model, a detailed explanation of how a similar system would respond to an emergency can be obtained.

[1426] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1427] Step 1: Collecting traffic information

[1428] The server collects train operation information such as the train's current location, speed, destination, and route information from a database (e.g., PostgreSQL, MongoDB). It also retrieves real-time weather data from a weather API (e.g., OpenWeatherMap API) and track condition data from a track monitoring system API.

[1429] Input: Operation information in the database, weather data from the weather API, track condition data from the track monitoring system API

[1430] Output: Operation information, weather data, track condition data

[1431] Specific operation: The server executes a database query to extract the necessary operation information, and calls the weather API and track monitoring system API to obtain real-time data.

[1432] Step 2: Calculate route and speed

[1433] Based on the collected operational information, the server uses AI algorithms (e.g., TensorFlow, PyTorch) to calculate the optimal route and speed.

[1434] Input: Operation information, weather data, and track condition data collected in Step 1

[1435] Output: Optimal route and speed

[1436] Specific operation: The server runs an AI algorithm, analyzes operation information, weather data, and track condition data, and calculates the optimal operating route and speed.

[1437] Step 3: Generate and send instructions

[1438] Based on the calculation results, the server generates and sends specific operating route and speed instructions to the operation control means (terminal) via a communication protocol (e.g., HTTP, MQTT).

[1439] Input: Optimal route and speed calculated in step 2

[1440] Output: Route instructions, speed instructions

[1441] Specific operation: The server generates route and speed instructions and uses a communication protocol to send them to the terminal.

[1442] Step 4: Receiving instructions

[1443] The terminal receives route and speed instructions sent from the server, and uses edge computing devices (e.g., Raspberry Pi, Arduino) as hardware.

[1444] Input: Route and speed instructions from the server

[1445] Output: Received route and speed instructions (data stored in the device)

[1446] Specific operation: The device receives data sent from the server via Wi-Fi.

[1447] Step 5: Controlling the operation

[1448] Based on the received instructions, the terminal controls the train operation through the autonomous driving system (e.g., ROS).

[1449] Input: Route and speed instructions received in step 4

[1450] Output: Specific operation control (speed adjustment, route selection)

[1451] Specific operation: The terminal activates the autonomous driving system and controls the train's operation according to the specified speed and route.

[1452] Step 6: Monitor your surroundings

[1453] The device is equipped with a camera (e.g., Logitech C920) and a LIDAR sensor to continuously monitor the surroundings while driving, and this information is sent to a server in real time.

[1454] Input: Video data from cameras and LIDAR sensors, sensor data

[1455] Output: Surrounding situation data (sent to server)

[1456] Specific operation: The device uses the camera and sensors to capture the surrounding situation and sends the data to the server.

[1457] Step 7: Anomaly detection and notification

[1458] When the device detects an abnormality using a camera or sensor, it analyzes it using image analysis software (e.g., OpenCV), and if an abnormality is confirmed, it immediately sends the information to the server via HTTP POST.

[1459] Input: Surroundings data collected in step 6

[1460] Output: Anomaly detection data (notification to server)

[1461] Specific operation: The device analyzes the data using image analysis software, and if an abnormality is detected, it notifies the server.

[1462] Step 8: Monitor normal operations

[1463] During normal operation, the user (conductor) monitors the situation inside the train and checks for any problems. He or she patrols the train regularly to ensure safety.

[1464] Input: In-car situation

[1465] Output: Safety confirmation report

[1466] Specific actions: The conductor will patrol the train regularly to ensure the safety of passengers.

[1467] Step 9: Emergency response

[1468] When the server notifies the user of an emergency, the user (conductor) provides explanations to passengers and guides them to evacuate. The user also checks for and removes obstacles depending on the situation.

[1469] Input: Emergency notification from the server, actual emergency situation

[1470] Output: Passenger response, evacuation guidance, obstacle removal report

[1471] Specific actions: The conductor will make an announcement to passengers and take appropriate action and provide evacuation guidance.

[1472] Step 10: Reporting and Feedback

[1473] After the emergency situation has been resolved, the user (conductor) reports the details of the emergency response and the current safety status to the server.

[1474] Input: Emergency response history, current safety confirmation

[1475] Output: Detailed report

[1476] Specific operation: The conductor records the series of steps and results of the emergency response and sends them to the server.

[1477] (Application example 1)

[1478] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1479] While conventional autonomous vehicle systems can provide safe and efficient operation, they have the problem of being unable to take immediate and appropriate action when an emergency occurs. In particular, there are cases where personnel on board the vehicle are unable to respond quickly, and it takes time to ensure the safety of passengers. In addition, operation information and abnormality notifications are only sent to the operation manager, resulting in a lack of information for on-site response.

[1480] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1481] In this invention, the server includes a central management means for collecting operation information and calculating operation routes and speeds, an operation control means for receiving instructions from the central management means and controlling the operation of the vehicle, a monitoring means for monitoring the surrounding conditions while driving and detecting abnormalities, an abnormality notification means for notifying the central management means when an abnormality is detected, and a means for sending a notification to a smart device using the information when an emergency occurs. This enables a quick and appropriate response when an emergency occurs, thereby improving the safety of passengers and vehicles.

[1482] "Operation information" is a general term for detailed data such as the vehicle's current location, speed, destination, route points, weather data, and track conditions.

[1483] The "central management means" is a device or system that functions as a server, collects and analyzes operation information, and generates instructions.

[1484] The "operation control means" is a device or system that receives instructions from the central control means and controls the operation of the vehicle.

[1485] "Monitoring means" refers to a device or system that includes devices such as cameras and sensors for monitoring the surrounding conditions during operation and detecting abnormalities.

[1486] The "abnormality notification means" is a device or system that, when an abnormality is detected, promptly notifies the central management means of that information.

[1487] An "emergency situation" refers to an unexpected accident, malfunction, or abnormal event that occurs while a vehicle is in operation.

[1488] A "means for sending notifications to smart devices" is a device or system for quickly sending corresponding information to devices such as smartphones and tablets when an emergency occurs.

[1489] "Personnel" refers to personnel in the vehicle who are intended to respond in the event of an emergency.

[1490] This invention relates to a vehicle system with an automated driving function, which enables safe and efficient operation even in emergency situations. The system includes a central management means for collecting operation information and calculating operation routes and speeds, an operation control means for receiving instructions from the central management means and controlling the operation of the vehicle, a monitoring means for monitoring the surrounding conditions while driving and detecting abnormalities, an abnormality notification means for notifying the central management means when an abnormality is detected, a vehicle carrying personnel who will respond in the event of an emergency, and a means for using the information to send a notification to a smart device when an emergency occurs.

[1491] System program description

[1492] The program of this system is configured as follows:

[1493] 1. Collection of operation information

[1494] The server retrieves operational information from the database, such as vehicle location, speed, destination, and route information, as well as real-time weather and track condition data. This information is continuously updated and sent to a central control unit.

[1495] 2. Route and speed calculation

[1496] The server uses AI algorithms to calculate optimal routes and speeds based on collected traffic information, using standard server hardware and software to run the AI ​​models.

[1497] 3. Instruction Generation and Transmission

[1498] Based on the calculation results, the server proposes specific driving routes and speeds and sends them to the central management means, which then receives the instructions and controls the driving of the vehicles.

[1499] 4. Surrounding situation monitoring and abnormality detection

[1500] The device's onboard cameras and sensors monitor the surroundings while driving, and if an abnormality is detected, the information is sent to a server in real time. This monitoring method is implemented using high-performance image recognition software and sensors.

[1501] 5. Abnormality notification and emergency response

[1502] When an emergency occurs, the server analyzes the information and issues appropriate instructions for response. The server then sends an emergency notification to smart devices, prompting occupants and other responders to act quickly. This functionality is achieved using a notification service such as Firebase Cloud Messaging (FCM).

[1503] Example of a system

[1504] For example, the following is an example of a prompt that may be issued when an obstacle is discovered on the tracks during operation:

[1505] Example of a text prompt:

[1506] "It obtains information about autonomous vehicles from a central server and monitors their operating status in real time. If an abnormality occurs, it sends an emergency notification to the user's smartphone, instructing them on the appropriate response."

[1507] This system will enable a swift and appropriate response in the event of an emergency, enhancing the safety of passengers and trains. Conductors and train operators will be able to respond immediately to emergencies, provide guidance to passengers, and guide them to safety.

[1508] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1509] Step 1:

[1510] The server obtains operation information such as the vehicle's current location, speed, destination, and route points from the database. It also obtains real-time weather data and track condition data and integrates this information. The input is data from various sensors and the database, and the output is integrated operation information. This allows the server to have the most up-to-date operation information.

[1511] Step 2:

[1512] The server uses an AI algorithm based on the collected traffic information to calculate the optimal route and speed. The input is the traffic information obtained in the previous step, and the output is the optimal route and speed instructions. Specifically, the AI ​​model analyzes real-time data and calculates an efficient and safe route.

[1513] Step 3:

[1514] The server generates specific driving route and speed instructions based on the calculation results and sends the instructions to the central control means, where the input is the optimized driving route and speed instructions and the output is the actual instruction information, so that the central control means receives the latest driving instructions.

[1515] Step 4:

[1516] The terminal receives the driving route and speed instructions sent from the central control means and controls the driving of the vehicle through the automatic driving system. The input is the driving instructions from the central control means and the output is the actual driving status. In concrete terms, the terminal controls the speed of the vehicle and drives it along the specified route.

[1517] Step 5:

[1518] The device uses its on-board camera and sensors to monitor the surroundings while driving and detect any abnormalities. The input is real-time data from the camera and sensors, and the output is the results of any abnormalities detected. Specifically, image analysis software analyzes the video and detects obstacles and abnormal speeds.

[1519] Step 6:

[1520] When an abnormality is detected, the terminal notifies the server in real time. The input is the abnormality detection data, and the output is an abnormality notification to the central management means. This allows the server to immediately recognize the abnormality.

[1521] Step 7:

[1522] The server analyzes the anomaly information and generates appropriate response instructions. The input is the anomaly notification data, and the output is emergency response instructions. Depending on the anomaly that has occurred, the server generates instructions such as an emergency shutdown.

[1523] Step 8:

[1524] The server sends emergency response instructions to smart devices and notifies crew members and other relevant parties. The input is the emergency response instructions, and the output is a notification message. Specifically, the server sends an emergency notification to smartphones and tablets via Firebase Cloud Messaging (FCM).

[1525] Step 9:

[1526] The user (crew member / conductor) implements emergency response based on the notification from the server. The input is the emergency response instructions displayed on the smart device, and the output is the actual response action. Specific actions include explaining the situation to passengers and guiding them to evacuate.

[1527] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1528] The present invention further improves the safety and efficiency of driving by combining an emotion engine with a vehicle system having an automatic driving function. Hereinafter, embodiments of the present invention will be described in detail.

[1529] Overall system configuration

[1530] The system includes the following major components:

[1531] Central management means (server): Manages and analyzes operation information and generates instructions.

[1532] Operation control means (terminal): Receives instructions from the central control means and controls the operation of the vehicle.

[1533] Monitoring Measures: Continuously monitor the surrounding environment and conditions while the vehicle is in operation.

[1534] Abnormality notification means: Notifies the central management means when an abnormality is detected.

[1535] Emotion engine: Recognizes the emotional state of personnel (conductors) and reflects it in management and response.

[1536] Personnel (Conductor): Operates within the vehicle to respond to emergencies.

[1537] Program processing

[1538] Server Processing

[1539] 1. Collection of operational information:

[1540] The server retrieves information about the train's current location, speed, destination, stops, and route from a database, as well as real-time weather and track condition data.

[1541] 2. Route and speed calculation:

[1542] The server uses AI algorithms to calculate the optimal route and speed based on the collected operational information, records the calculation results, and generates instructions.

[1543] 3. Generate and send instructions:

[1544] The server sends the generated instructions to the operation control means (terminal), which include detailed operation routes, speeds, and emergency response procedures.

[1545] 4. Real-time analysis:

[1546] Even after the vehicle has started moving, the server continues to receive and analyze data on the surrounding situation sent from the device in real time, and sends new operating instructions to the device as needed.

[1547] 5. Emotion recognition and response:

[1548] The server receives and analyzes the conductor's emotional data sent from the emotion engine, and responds accordingly based on the conductor's mental state and stress level.

[1549] Terminal (train) processing

[1550] 1. Receiving and setting instructions:

[1551] The terminal receives the route and speed instructions sent from the server, checks the received information, and sets it up in the autonomous driving system.

[1552] 2. Navigation control:

[1553] The device's autonomous driving system operates the train based on the received route and speed instructions, and the train travels along the specified route at the specified speed.

[1554] 3. Surrounding Area Monitoring:

[1555] The terminals are equipped with cameras and sensors as monitoring tools, and continuously monitor the surroundings while in operation. This information is sent to a server in real time.

[1556] 4. Anomaly detection and notification:

[1557] If the monitoring means detects an abnormality (for example, an obstacle on the tracks or an abnormal speed limit), the information is immediately sent to the server via the abnormality notification means.

[1558] 5. Collecting Emotional Data:

[1559] The emotion engine continuously monitors the conductor's emotional state and transmits the data to the server.

[1560] User (conductor) role

[1561] 1. Monitoring normal operations:

[1562] During normal operation, the user (conductor) monitors the situation inside the train and checks for any particular problems. He or she also patrols the train regularly to ensure safety.

[1563] 2. Emergency Response:

[1564] When an emergency situation is notified by the server, the conductor will act according to the appropriate emergency response procedures, such as explaining the situation to passengers and guiding them to an evacuation.

[1565] 3. Status reports and feedback:

[1566] After the emergency situation has been resolved, the conductor will report the situation in detail to the server, including the details of the emergency response and confirmation of the current safety situation.

[1567] 4. Emotional management:

[1568] Based on feedback from the emotion engine, the conductor can take measures according to his or her emotional state, such as taking a break if stress levels are high.

[1569] Specific examples

[1570] For example, consider the process that is performed when a train detects an obstacle on the tracks while in operation.

[1571] 1. Obstacle detection:

[1572] The device's camera detects obstacles on the tracks and sends that information to a server.

[1573] 2. Emergency stop instruction:

[1574] The server analyzes the obstacle information and immediately sends an emergency stop command to the terminal if it determines it is necessary. At the same time, it notifies the conductor of the emergency response procedure.

[1575] 3. Performing an emergency stop:

[1576] The terminal receives the emergency stop command and activates the emergency brake to stop the train.

[1577] 4. Conductor's response:

[1578] The conductor will guide passengers and guide them to safety based on instructions from the server. In addition, if possible, the conductor will check for and remove obstacles.

[1579] 5. Emotional data feedback:

[1580] When an emergency occurs, the emotion engine monitors the conductor's emotional state and sends the data to a server, which analyzes the data and provides a response guide based on the conductor's mental state.

[1581] 6. Resumption of service:

[1582] After the obstacle is removed and safety is confirmed, the server calculates a new route and speed and sends it to the device, which receives the new instructions and resumes operation.

[1583] In this way, by combining the emotion engine, the vehicle system of the present invention can operate more safely and efficiently, taking into account the mental health of the conductor, from normal operation to emergency situations and even when operation resumes.

[1584] The processing flow will be explained below.

[1585] Server Processing

[1586] Step 1:

[1587] The server retrieves information about the train's current location, speed, destination, stops, and route from a database, as well as real-time weather and track condition data.

[1588] Step 2:

[1589] The server uses AI algorithms to calculate the optimal route and speed based on the collected operational information, records the calculation results, and generates instructions.

[1590] Step 3:

[1591] The server sends the generated instructions to the operation control means (terminal), which include detailed operation routes, speeds, and emergency response procedures.

[1592] Step 4:

[1593] Even after the vehicle has started moving, the server continues to receive and analyze data on the surrounding situation sent from the device in real time, and sends new operating instructions to the device as needed.

[1594] Step 5:

[1595] The server receives and analyzes the conductor's emotional data sent from the emotion engine, and based on the results, provides a response guide according to the conductor's mental state and stress level.

[1596] Terminal (train) processing

[1597] Step 1:

[1598] The terminal receives the route and speed instructions sent from the server, checks the received information, and sets it up in the autonomous driving system.

[1599] Step 2:

[1600] The device's autonomous driving system operates the train based on the received route and speed instructions, and the train travels along the specified route at the specified speed.

[1601] Step 3:

[1602] The terminals are equipped with cameras and sensors as monitoring tools, and continuously monitor the surroundings while in operation. This information is sent to a server in real time.

[1603] Step 4:

[1604] If the monitoring means detects an abnormality (for example, an obstacle on the tracks or an abnormal speed limit), the information is immediately sent to the server via the abnormality notification means.

[1605] Step 5:

[1606] The terminal continuously monitors the conductor's emotional data through the emotion engine and sends that data to the server in real time.

[1607] User (conductor) role

[1608] Step 1:

[1609] During normal operation, the user (conductor) monitors the situation inside the train and checks for any particular problems. He or she also patrols the train regularly to ensure safety.

[1610] Step 2:

[1611] When the server notifies the conductor of an emergency, the conductor acts according to the server's instructions and follows the appropriate emergency response procedures, such as providing explanations to passengers and guiding them to safety.

[1612] Step 3:

[1613] Based on feedback from the emotion engine, the conductor can take measures according to his or her emotional state, such as taking a break if stress levels are high.

[1614] Step 4:

[1615] After the emergency situation has been resolved, the conductor will report the situation in detail to the server, including the details of the emergency response and confirmation of the current safety situation.

[1616] Step 5:

[1617] The conductor confirms that new operating instructions have been sent from the server, informs passengers that operation has resumed, and makes another round to ensure passengers are able to travel safely.

[1618] Specific examples

[1619] For example, consider the process that is performed when a train detects an obstacle on the tracks while in operation.

[1620] Step 1:

[1621] The device's camera detects obstacles on the tracks and sends that information to a server.

[1622] Step 2:

[1623] The server analyzes the obstacle information and immediately sends an emergency stop command to the terminal if it determines it is necessary. At the same time, it notifies the conductor of the emergency response procedure.

[1624] Step 3:

[1625] The terminal receives the emergency stop command and activates the emergency brake to stop the train.

[1626] Step 4:

[1627] The conductor will guide passengers and guide them to safety based on instructions from the server. In addition, if possible, the conductor will check for and remove obstacles.

[1628] Step 5:

[1629] When an emergency occurs, the emotion engine monitors the conductor's emotional state and sends the data to a server, which analyzes the data and provides a response guide based on the conductor's mental state.

[1630] Step 6:

[1631] After the obstacle is removed and safety is confirmed, the server calculates a new route and speed and sends it to the device, which receives the new instructions and resumes operation.

[1632] Example 2

[1633] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1634] While conventional autonomous vehicle systems have certain functions for collecting operational information, calculating operational routes, and detecting abnormalities, they are not yet able to respond quickly and effectively to emergencies or sudden abnormalities.In addition, they lack the functionality to consider the mental state and stress level of the conductor as human factors, which has led to issues with the safety and efficiency of operations.

[1635] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for collecting operation information and calculating operation routes and speeds, a means for analyzing the surrounding situation in real time and generating new operation instructions, and an emotion recognition means for recognizing the emotional state of passengers on board and taking appropriate action. This makes it possible to ensure the safety of operation while improving the efficiency of operation by taking human factors into consideration.

[1636] "Operation information" is data related to the vehicle's current location, speed, destination, intermediate stops, and operating route.

[1637] A "central control means" is a system that collects operational information, calculates operational routes and speeds, and generates instructions.

[1638] The "operation control means" is a device that receives instructions from the central control means and automatically controls the operation of the vehicle.

[1639] "Monitoring means" refers to devices such as cameras and sensors that continuously monitor the surrounding conditions while the vehicle is in operation and detect abnormalities.

[1640] The "abnormality notification means" is a system that notifies the central management means of abnormalities detected by the monitoring means.

[1641] An "emergency situation" refers to an unexpected malfunction, accident, or abnormal situation that occurs during operation.

[1642] The "emotion recognition means" is a system that monitors the emotional state of onboard personnel in real time and analyzes the data.

[1643] "Personnel" refers to the people on board who will respond in the event of an emergency.

[1644] "New driving instructions" are updated driving routes and speed instructions generated based on the results of real-time analysis.

[1645] A "conductor" is a person on board a train who is responsible for normal operation and responding to emergencies.

[1646] The present invention further improves the safety and efficiency of driving by combining emotion recognition means with a vehicle system equipped with an automatic driving function. Hereinafter, embodiments of the present invention will be described in detail.

[1647] Overall system configuration

[1648] The system includes the following main components:

[1649] Central control means (server): The server collects driving information, calculates driving routes and speeds, and analyzes the surrounding situation in real time during driving to generate new driving instructions.

[1650] Operation control means (terminal): The terminal receives instructions from the server and automatically controls the operation of the vehicle.

[1651] Monitoring means: Includes cameras and sensors to continuously monitor the surroundings while the vehicle is in operation and detect any abnormalities.

[1652] Abnormality notification means: Notifies the server of any abnormalities detected by the monitoring means.

[1653] Emotion recognition means: The emotion engine recognizes and analyzes the emotional state of the conductor (personnel), allowing it to respond according to the conductor's mental state and stress level.

[1654] Personnel (conductor): Responsible for responding to emergencies in the vehicle.

[1655] Specific server processing

[1656] The server manages and analyzes operation information in the following steps:

[1657] 1. Collection of operational information:

[1658] The server retrieves information on the train's current location, speed, destination, intermediate stops, and route from the database, as well as real-time weather data and track condition data, and manages this information as comprehensive operational information.

[1659] 2. Route and speed calculation:

[1660] The server uses the collected operational information to calculate the optimal route and speed using the AI ​​algorithm "Route Optimizer," thereby supporting safe and efficient operation.

[1661] 3. Generate and send instructions:

[1662] The server then sends the generated driving route and speed instructions to the terminal, which include detailed driving route, speed, and emergency response procedures.

[1663] 4. Real-time analysis:

[1664] While the vehicle is in operation, the server receives and analyzes real-time data on the surrounding environment sent from the device, and sends new operating instructions to the device as needed to ensure safe operation.

[1665] 5. Emotion recognition and response:

[1666] The server analyzes the conductor's emotional data sent from the emotion engine "EmotionAnalyzer," and generates and provides the conductor with a response guide based on the conductor's mental state and stress level.

[1667] Specific processing of the terminal (train)

[1668] The terminal automatically controls operation through the following steps:

[1669] 1. Receiving and setting instructions:

[1670] The terminal receives route and speed instructions sent from the server and sets them in the autonomous driving system.

[1671] 2. Navigation control:

[1672] The automated driving system operates the vehicle based on the received route and speed, and follows the specified route at the specified speed.

[1673] 3. Surrounding Area Monitoring:

[1674] Cameras and sensors are used to continuously monitor the surrounding conditions while the vehicle is in operation, and the data is sent to a server.

[1675] 4. Anomaly detection and notification:

[1676] If the monitoring means detects an abnormality, it immediately transmits the information to the server.

[1677] 5. Collecting Emotional Data:

[1678] The emotion engine continuously monitors the conductor's emotional state and transmits the data to the server.

[1679] User (conductor) role

[1680] 1. Monitoring normal operations:

[1681] During normal operation, the conductor monitors the situation inside the train and checks for any particular problems. He also patrols the train regularly to ensure safety.

[1682] 2. Emergency Response:

[1683] When an emergency situation is notified by the server, the conductor will act according to the appropriate emergency response procedures, such as explaining the situation to passengers and guiding them to an evacuation.

[1684] 3. Status reports and feedback:

[1685] After the emergency situation has been resolved, the conductor will report the situation in detail to the server, including the details of the emergency response and confirmation of the current safety situation.

[1686] 4. Emotional management:

[1687] The conductor takes measures based on his or her own emotional state based on feedback from the emotion engine. For example, if stress levels are high, the conductor may take a break.

[1688] Specific examples

[1689] For example, if a train detects an obstacle on the tracks while in operation, the process is as follows.

[1690] 1. Obstacle detection:

[1691] The device's camera detects obstacles on the tracks and sends that information to a server.

[1692] 2. Emergency stop instruction:

[1693] The server analyzes the obstacle information and immediately sends an emergency stop command to the terminal if necessary, while also notifying the conductor of the emergency response procedure.

[1694] 3. Perform an emergency stop:

[1695] The terminal receives an emergency stop command and activates the emergency brake, bringing the train to a halt.

[1696] 4. Conductor's response:

[1697] The conductor will provide passenger guidance and evacuation guidance according to instructions from the server, and will also check for and remove obstacles if possible.

[1698] 5. Emotional data feedback:

[1699] The emotion engine monitors the conductor's emotional state in the event of an emergency and sends the data to a server, which analyzes the data and provides a response guide based on the conductor's mental state.

[1700] 6. Resumption of service:

[1701] After the obstacle is removed and safety is confirmed, the server calculates a new route and speed and sends it to the device, which receives the new instructions and resumes operation.

[1702] Prompt Sentence Examples

[1703] To use a generative AI model to respond to specific situations, we use prompts like the following:

[1704] Example prompt 1: "If a train is in motion and discovers an obstacle on the tracks, explain how you should respond based on your roles as the server, the terminal, and the conductor."

[1705] Example prompt 2: "Please explain in detail how the emotion engine monitors the conductor's emotional state and notifies the server in the event of an emergency."

[1706] These prompts allow the generative AI model to learn system-wide responses and response procedures based on specific scenarios.

[1707] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1708] Server Processing

[1709] Step 1:

[1710] Collection of operation information

[1711] Input: Database, real-time weather data, track condition data

[1712] Data processing / calculation: The server retrieves and organizes information on the train's current location, speed, destination, intermediate stops, and route from the database. It also retrieves and integrates real-time weather information and track conditions from the weather data API.

[1713] Output: Complete set of traffic information

[1714] What it does: The server periodically retrieves data from the database and API and stores it as a train information set, including the train's current location, speed, destination, temperature, precipitation, wind speed, etc.

[1715] Step 2:

[1716] Route and speed calculation

[1717] Input: Service information set

[1718] Data processing / calculation: Based on the operation information, the server uses the AI ​​algorithm "Route Optimizer" to calculate the optimal route and speed, taking into account traffic and weather conditions to derive a safe and efficient route.

[1719] Output: Optimized route and speed instructions

[1720] Specific operation: The server inputs operation information into "RouteOptimizer" and generates a file containing the resulting route and speed instructions, including the route to the next station, speed limits, and emergency stop points.

[1721] Step 3:

[1722] Generate and send instructions

[1723] Input: Route and speed instructions

[1724] Data processing / calculation: The server formats the instructions and sends them to the operation control means (terminal). Safety checks are performed and the contents of the instructions are confirmed.

[1725] Output: Instructions sent to the terminal

[1726] What happens: The server sends instructions to the device via email or a digital communications protocol, and receives a confirmation message to confirm that it was received in the correct format.

[1727] Step 4:

[1728] Real-time analytics

[1729] Input: Ambient data from the device

[1730] Data processing / calculation: The server analyzes the surrounding situation data received from the device in real time, determines whether there are any abnormalities, and generates new driving instructions as necessary.

[1731] Output: New operating instructions

[1732] Specific operation: The server analyzes camera footage and sensor data sent from moving trains, and if it detects an obstacle or abnormality, it generates new operating instructions and immediately sends them to the terminal.

[1733] Step 5:

[1734] Emotion recognition and response

[1735] Input: Emotion data from the emotion engine

[1736] Data processing / calculation: The server analyzes the conductor's emotional data sent from the emotion engine "EmotionAnalyzer." It evaluates the conductor's stress level and emotional state and generates a response guide.

[1737] Output: Conductor's response guide

[1738] Specific actions: Based on the emotional data, the server generates a response guide that includes stress reduction techniques and recommendations for taking breaks, and notifies the conductor. This guide includes specific advice, such as "take deep breaths and stretch to relax."

[1739] Terminal (train) processing

[1740] Step 1:

[1741] Receiving and Setting Instructions

[1742] Input: Route and speed instructions sent from the server

[1743] Data processing / calculation: The device checks the received route and speed instructions and sets them in the autonomous driving system. It also checks the settings and verifies that the vehicle can be driven safely.

[1744] Output: Route and speed set for the autonomous driving system

[1745] Specific operation: The device analyzes the received instructions and sends the appropriate configuration commands to the autonomous driving system. Once the configuration is complete, it sends a confirmation message to the server.

[1746] Step 2:

[1747] Operation control

[1748] Input: Set route and speed

[1749] Data processing / calculation: The autonomous driving system controls the vehicle based on the route and speed instructions it receives, adjusting speed and stopping positions to ensure safe and efficient driving.

[1750] Output: Actual driving data (current location, speed, etc.)

[1751] Specific operation: The device uses an autonomous driving system to drive along a designated route at a set speed, constantly monitoring its speedometer and location while driving and making adjustments as necessary.

[1752] Step 3:

[1753] Surroundings monitoring

[1754] Input: Camera and sensor data

[1755] Data processing / calculation: Analyzes data sent from cameras and sensors, which are used as monitoring tools, and monitors the surrounding situation. If an abnormality is detected, the data is sent to the server.

[1756] Output: Surroundings data

[1757] Specific operation: High-resolution cameras and various sensors installed on the terminal acquire data in real time, and perform image analysis and pattern recognition to determine whether there are any abnormalities. This data is then sent to the server.

[1758] Step 4:

[1759] Anomaly detection and notification

[1760] Input: Camera and sensor data

[1761] Data processing / calculation: If the monitoring means detects an abnormality, the information is immediately sent to the server via the abnormality notification means.

[1762] Output: Error notification data

[1763] Specific operation: If an abnormality is detected, the device will send the information to the server via the abnormality notification means. For example, if an obstacle is detected on the tracks, the device will send the video and data to the server.

[1764] Step 5:

[1765] Collecting Emotional Data

[1766] Input: Data from the emotion engine "EmotionAnalyzer"

[1767] Data processing / calculation: The emotion engine continuously monitors the conductor's emotional state and sends the data to the server.

[1768] Output: Conductor's emotion data

[1769] Specific operation: The emotion engine monitors the conductor's facial expressions, voice, heart rate, etc. in real time, analyzes this data, and processes it into emotion data. The analyzed data is sent to a server, and a response is made based on the conductor's emotional state.

[1770] User (conductor) role

[1771] Step 1:

[1772] Monitoring normal operations

[1773] Input: Passenger and vehicle status data

[1774] Data processing / calculation: Conductors monitor the situation inside the train during normal operation and check for any particular problems. They also make regular rounds to check safety.

[1775] Output: Safety report data

[1776] Specific actions: The conductor checks the train announcements and security camera footage to see if there are any problems. Depending on the situation, he or she will make announcements to passengers and ensure their safety.

[1777] Step 2:

[1778] Emergency response

[1779] Input: Emergency notification data from the server

[1780] Data Processing / Calculation: When an emergency situation is notified by the server, the conductor will act according to the appropriate emergency response procedures.

[1781] Output: Emergency response report data

[1782] Specific actions: In the event of an emergency, the conductor will explain the situation to passengers and guide them to evacuate. For example, an announcement will be made on the train saying, "This is an emergency, please disembark."

[1783] Step 3:

[1784] Status reports and feedback

[1785] Input: Post-emergency data

[1786] Data processing / calculation: After the emergency situation has been resolved, the conductor will report the situation in detail to the server.

[1787] Output: Detailed emergency report data

[1788] Specific operation: The conductor records the details of the emergency response and the results, and reports them to the server. For example, he / she describes the success of evacuation guidance and the results of safety confirmation.

[1789] Step 4:

[1790] emotional management

[1791] Input: Feedback from the emotion engine

[1792] Data processing / calculation: The conductor takes measures based on his / her own emotional state based on feedback from the emotion engine.

[1793] Output: Improved emotional state data

[1794] Specific Actions: Conductors receive feedback from the Emotion Engine and implement stress management and relaxation techniques, such as deep breathing and relaxation exercises during designated breaks.

[1795] The above is the specific processing flow of the program of this system and the specific operations performed at each processing step.

[1796] (Application example 2)

[1797] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1798] In order to improve the safety and efficiency of autonomous driving vehicle systems, it is necessary not only to monitor the surrounding environment and situation, but also to grasp the emotional state of the personnel on board in real time and respond accordingly. However, conventional autonomous driving systems lack the ability to monitor emotional states, and lack means to reduce the mental burden on conductors and drivers in emergencies. Therefore, comprehensive improvements to operational safety and comfort are a challenge.

[1799] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting operation information and calculating operation routes and speeds, means for receiving instructions from the central control means and controlling the operation of the vehicle, means for monitoring the surrounding conditions while driving and detecting abnormalities, and means for monitoring the emotional state of the vehicle in real time and transmitting the emotional data to the central control means. This makes it possible to improve the safety and comfort of operation by adjusting operation and reducing stress according to the emotional state of the conductor and driver.

[1800] "Operation information" is data related to the vehicle's current location, speed, destination, intermediate stops, and operating route.

[1801] "Central control means" refers to the system that collects, analyzes, and generates instructions regarding traffic information.

[1802] The "operation control means" is a system that receives instructions from the central control means and controls the operation of the vehicle.

[1803] The "monitoring means" is a system that continuously monitors the surrounding environment and conditions while driving and detects abnormalities.

[1804] The "abnormality notification means" is a system that notifies the central management means of information when an abnormality is detected.

[1805] The "emotion recognition means" is a system that monitors the emotional state inside the vehicle in real time and transmits the data to a central control means.

[1806] "Vehicle" refers to a land transportation means that has an automatic driving function and has personnel on board to respond in the event of an abnormality or emergency during operation.

[1807] "Route" refers to the route chosen for a vehicle to travel to its destination.

[1808] An "emergency situation" refers to an unexpected abnormality or malfunction that occurs during operation and requires immediate response.

[1809] The "operation adjustment means" is a system that adjusts the route and speed of the vehicle depending on the vehicle's emotional state and surrounding circumstances.

[1810] A specific system configuration and its operation will be described below as an embodiment of the invention.

[1811] Overall system configuration

[1812] The system includes the following major components:

[1813] 1. Central management means (server): Collects operation information, analyzes it, and generates instructions. The server calculates the route and speed, and generates and sends operation instructions.

[1814] 2. Operation control means (terminal): A terminal that receives instructions from the central control means and controls the operation of vehicles. It operates vehicles based on the instructions for route and speed.

[1815] 3. Monitoring means: Using cameras and sensors, the surrounding environment and conditions are monitored during operation. If an abnormality is detected, it is notified to the central control means via a terminal.

[1816] 4. Abnormality notification means: Notifies the central control means when an abnormality is detected. This means immediately sends information to the central control means when an abnormality occurs.

[1817] 5. Emotion recognition means: The emotional state of personnel on board (conductors and drivers) is monitored in real time using cameras and emotion recognition software, and the data is sent to a central management means.

[1818] 6. Vehicles: These vehicles have autonomous driving capabilities and are manned by personnel who can respond to emergencies.

[1819] Server Processing

[1820] The server performs the following process:

[1821] 1. Operational information collection: The server collects information on the vehicle's current location, speed, destination, intermediate stops, and route, as well as real-time weather and track condition data.

[1822] 2. Calculation of route and speed: Based on the collected operation information, an AI algorithm is used to calculate the optimal route and speed.

[1823] 3. Generation and transmission of instructions: Generate instructions based on the calculation results and send them to the operation control means.

[1824] 4. Real-time analysis: Analyzes surrounding situation data and emotional data sent from the operation control means and sends new instructions based on the results.

[1825] 5. Emotion recognition and response: Data from the emotion recognition system is received and analyzed, and operation adjustments and stress reduction measures are implemented based on the driver's stress level.

[1826] Terminal handling

[1827] The terminal performs the following process:

[1828] 1. Receiving and setting instructions: Receive driving instructions from the server and set them in the autonomous driving system.

[1829] 2. Operation control: Operate the vehicle based on the received route and speed instructions.

[1830] 3. Surroundings monitoring: Cameras and sensors are used to monitor the surroundings while driving.

[1831] 4. Abnormality notification: If an abnormality is detected, the information is immediately sent to a central control means.

[1832] Emotion Recognition System Processing

[1833] 1. Real-time emotion monitoring: Using cameras and emotion recognition software, the facial expressions of the vehicle's occupants are analyzed to determine their emotional state.

[1834] 2. Sending emotion data to the server: The recognized emotion data is sent to the server and analyzed in real time.

[1835] 3. Operational adjustment: If the emotional state is judged to be stressful, the route or speed of the vehicle will be adjusted.

[1836] 4. Stress reduction measures: Reduce stress levels for drivers and passengers by playing stress-reducing music and adjusting lighting.

[1837] Specific examples

[1838] For example, if a camera inside the vehicle detects that the driver is fatigued, the information is immediately sent to a server. The server uses the data to adjust the route and speed to ensure the driver can continue driving safely. If the server determines that the driver is feeling stressed, it will play relaxing music and adjust the lighting inside the vehicle to reduce stress.

[1839] Using generative AI models

[1840] Example prompt: "The driver is stressed, please play some relaxing music."

[1841] In this way, the system of the present invention combines emotion recognition to comprehensively improve driving safety and comfort.

[1842] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1843] Step 1:

[1844] Collection of operation information

[1845] The server retrieves information about the vehicle's current location, speed, destination, stops, and route from a database, as well as real-time weather and track condition data.

[1846] Input: vehicle location information, speed information, weather data, track condition data

[1847] Data processing: Organize and store collected operation information and make it into an analyzable format

[1848] Output: Formatted traffic information data

[1849] Step 2:

[1850] Route and speed calculation

[1851] Based on the operation information collected by the server, an AI algorithm is used to calculate the optimal operation route and speed.

[1852] Input: Formatted traffic information data

[1853] Data calculation: Optimization of route and speed is performed using AI algorithms

[1854] Output: Calculated optimal route and speed instructions

[1855] Step 3:

[1856] Generate and send instructions

[1857] The server transmits the generated travel route and speed instructions to the travel control means.

[1858] Input: Optimal route and speed instructions

[1859] Data processing: Converting data into a format for transmission

[1860] Output: Data sent to the operation control means

[1861] Step 4:

[1862] Receiving and Setting Instructions

[1863] The terminal receives driving instructions sent from the server and sets them in the autonomous driving system.

[1864] Input: Data sent to the operation control means

[1865] Data processing: Converting received instructions into the setting format for the autonomous driving system

[1866] Output: Configured autonomous driving system

[1867] Step 5:

[1868] Operation control

[1869] The device's autonomous driving system operates the vehicle based on the received route and speed instructions.

[1870] Input: Configured autonomous driving system

[1871] Specific behavior: The vehicle travels a specified route at a specified speed.

[1872] Output: Vehicles in operation

[1873] Step 6:

[1874] Surrounding Area Monitoring

[1875] The device uses cameras and sensors to monitor the surroundings while in operation.

[1876] Input: Surrounding situation data (video and sensor information)

[1877] Data processing: Real-time situation analysis

[1878] Output: Parsed situation data

[1879] Step 7:

[1880] Abnormality notification

[1881] If the monitoring means detects an abnormality, it immediately transmits the information to the central control means.

[1882] Input: Parsed situation data

[1883] Data processing: Analysis to see if there are any abnormalities

[1884] Output: Error notification data

[1885] Step 8:

[1886] Collecting and transmitting emotional data

[1887] An emotion recognition means monitors the emotional state of the personnel in the vehicle and transmits the data to a server.

[1888] Input: Camera footage, emotion recognition software analysis results

[1889] Data Computation: Analysis with Emotion Recognition Algorithms

[1890] Output: Parsed emotion data

[1891] Step 9:

[1892] Emotion recognition and response

[1893] The server receives and analyzes the data from the emotion recognition means, and based on the results, adjusts operation and takes measures to reduce stress according to the driver's stress level.

[1894] Input: Parsed emotion data

[1895] Data processing: assessing stress states and generating response instructions

[1896] Output: Operation adjustment instructions, stress reduction measures

[1897] Step 10:

[1898] Operation adjustments

[1899] The terminal receives operation adjustment instructions from the server and adjusts the operation route and speed.

[1900] Input: Operation adjustment instructions

[1901] Specific behavior: Resetting the route and speed

[1902] Output: Operating state after adjustment

[1903] Step 11:

[1904] Stress reduction measures

[1905] The device will implement stress-reducing measures (such as playing relaxing music or adjusting the lighting).

[1906] Input: Stress reduction measures (specific instructions)

[1907] Specific actions: Change the in-car environment (play music, adjust lighting, etc.)

[1908] Output: Improved in-car environment

[1909] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1910] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1911] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1912] [Fourth embodiment]

[1913] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1914] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1915] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1916] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1917] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1918] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1919] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1920] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1921] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1922] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1923] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1924] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1925] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1926] The present invention relates to a vehicle system with an automatic driving function, and in particular to a system that enables safe and efficient operation even in an emergency. Hereinafter, an embodiment of the present invention will be described in detail.

[1927] Overall system configuration

[1928] This system mainly consists of a central management means, operation control means, monitoring means, abnormality notification means, and vehicles carrying emergency response personnel (usually conductors). The central management means functions as a server, collecting and analyzing operation information and generating instructions. The operation control means functions as a train terminal, operating the vehicle based on instructions from the server. The monitoring means and abnormality notification means are composed of devices such as cameras and sensors built into the terminals.

[1929] Program processing

[1930] Server Processing

[1931] 1. Collection of operational information:

[1932] The server collects train operation information, such as the train's current location, speed, destination, and route information, from a database, as well as real-time weather and track condition data, and analyzes this information.

[1933] 2. Route and speed calculation:

[1934] The server uses AI algorithms to calculate the optimal route and speed based on the collected operational information, enabling efficient and safe operation.

[1935] 3. Generate and send instructions:

[1936] Based on the calculation results, the server generates and sends specific instructions on the driving route and speed to the driving control means (terminal).

[1937] Terminal (train) processing

[1938] 1. Receiving instructions:

[1939] The terminal receives the driving route and speed instructions sent from the server.

[1940] 2. Navigation control:

[1941] Based on the received instructions, the terminal controls the operation of the train through the automated driving system, specifically, driving at a specified speed along a specified route.

[1942] 3. Surrounding Area Monitoring:

[1943] The terminals are equipped with cameras and sensors as monitoring tools, and continuously monitor the surroundings while in operation, sending information to a server in real time.

[1944] 4. Anomaly detection and notification:

[1945] If the monitoring means detects an abnormality (for example, an obstacle on the tracks or an abnormal speed limit), the information is immediately sent to the server via the abnormality notification means.

[1946] User (conductor) role

[1947] 1. Monitoring normal operations:

[1948] During normal operation, the user (conductor) monitors the situation inside the train and checks for any problems. He or she patrols the train regularly to ensure safety.

[1949] 2. Emergency Response:

[1950] When the server notifies the conductor of an emergency, the conductor will act according to the appropriate emergency response procedures, such as explaining the situation to passengers, guiding them to an evacuation, and checking for and removing obstacles if possible.

[1951] 3. Reporting and Feedback:

[1952] After the emergency situation has been resolved, the conductor will report the situation in detail to the server, including details of the emergency response and current safety confirmation.

[1953] Specific examples

[1954] For example, consider the process that is performed when a train detects an obstacle on the tracks while in operation.

[1955] 1. Obstacle detection:

[1956] The device's camera detects obstacles on the tracks and sends that information to a server.

[1957] 2. Emergency stop instruction:

[1958] The server analyzes the obstacle information and immediately sends an emergency stop instruction to the terminal if it determines it is necessary.

[1959] 3. Perform an emergency stop:

[1960] The terminal receives the emergency stop command and activates the emergency brake to stop the train.

[1961] 4. Conductor's response:

[1962] The conductor will guide passengers and guide them to safety based on instructions from the server. In addition, if possible, the conductor will check for and remove obstacles.

[1963] 5. Service resumes:

[1964] After the obstacle is removed and safety is confirmed, the server calculates a new route and speed and sends it to the device, which receives the new instructions and resumes operation.

[1965] In this way, the vehicle system of the present invention can operate safely and efficiently both during normal operation and in emergency situations.

[1966] The processing flow will be explained below.

[1967] Server Processing

[1968] Step 1:

[1969] The server retrieves information about the train's current location, speed, destination, stops, and route from a database, as well as real-time weather and track condition data.

[1970] Step 2:

[1971] The server uses AI algorithms to calculate the optimal route and speed based on the collected operational information, records the calculation results, and generates instructions.

[1972] Step 3:

[1973] The server sends the generated instructions to the operation control means (terminal), which include detailed operation routes, speeds, and emergency response procedures.

[1974] Step 4:

[1975] Even after the vehicle has started moving, the server continues to receive and analyze data on the surrounding situation sent from the device in real time, and sends new operating instructions to the device as needed.

[1976] Terminal (train) processing

[1977] Step 1:

[1978] The terminal receives the route and speed instructions sent from the server, checks the received information, and sets it up in the autonomous driving system.

[1979] Step 2:

[1980] The device's autonomous driving system operates the train based on the received route and speed instructions, and the train travels along the specified route at the specified speed.

[1981] Step 3:

[1982] The terminals are equipped with cameras and sensors as monitoring tools, and continuously monitor the surroundings while in operation. This information is sent to a server in real time.

[1983] Step 4:

[1984] If the monitoring means detects an abnormality (for example, an obstacle on the tracks or an abnormal speed limit), the information is immediately sent to the server via the abnormality notification means.

[1985] Step 5:

[1986] When the terminal receives new instructions from the server, it will take emergency measures such as applying the emergency brakes, and will wait until it receives an instruction to resume operation.

[1987] User (conductor) role

[1988] Step 1:

[1989] The user (conductor) monitors the situation inside the train during normal operation and checks for any particular problems. He or she patrols the train regularly to ensure safety.

[1990] Step 2:

[1991] When the server notifies the conductor of an emergency, the conductor acts according to the server's instructions and follows the appropriate emergency response procedures, such as providing explanations to passengers and guiding them to safety.

[1992] Step 3:

[1993] The user (conductor) follows the instructions generated by the server and uses in-car announcements to communicate operational information and emergency response measures to passengers.

[1994] Step 4:

[1995] After the emergency situation has been resolved, the conductor will report the situation in detail to the server, including the details of the emergency response and confirmation of the current safety situation.

[1996] Step 5:

[1997] The conductor confirms that new operating instructions have been sent from the server, informs passengers that operation has resumed, and makes another round to ensure passengers are able to travel safely.

[1998] In this way, the server, terminal, and user (conductor) work together to ensure that the entire process, from normal operation to the occurrence of an emergency and then resumption of operation, proceeds safely and efficiently.

[1999] Example 1

[2000] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2001] To ensure safe and efficient operation in a vehicle system with autonomous driving functions, it is necessary to quickly and accurately collect operational information, optimize routes, monitor surrounding conditions, detect abnormalities, and respond to emergencies. However, in current systems, these functions are performed separately, resulting in problems such as information delays and inefficient processing. Furthermore, if an emergency response is not implemented immediately, it could lead to a serious accident. It is necessary to solve these issues and provide a system in which all functions are integrated.

[2002] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2003] In this invention, the server includes means for collecting operation information and calculating operation routes and speeds, means for receiving instructions from a central control means and controlling vehicle operation, means for monitoring the surrounding conditions while driving and detecting abnormalities, means for notifying the central control means when an abnormality is detected, a vehicle with personnel on board who will respond in the event of an emergency, means for receiving instructions from the operation control means, executing an emergency stop, and monitoring the surrounding conditions, and communication means for transmitting information to the central control means in real time. This enables real-time collection and analysis of operation information, optimization of operation, rapid detection and response to abnormalities, and seamless information transmission.

[2004] The "central management means" is a device or system that collects operation information, calculates operation routes and speeds based on that information, and sends instructions to the operation control means and abnormality notification means.

[2005] The "operation control means" is a device or system that receives instructions from the central control means and controls the operation of the vehicle based on those instructions.

[2006] "Monitoring means" refers to devices or systems such as sensors or cameras that monitor the surrounding conditions while driving and detect any abnormalities that may occur.

[2007] The "abnormality notification means" is a device or system for notifying the central management means of information when the monitoring means detects an abnormality.

[2008] "Emergency response measures" refer to systems and personnel that can respond quickly and appropriately when an emergency occurs.

[2009] "Communication means" refers to a device or system for transmitting information from the operation control means and monitoring means to the central management means.

[2010] "Operation information" refers to real-time information including data such as the train's current location, speed, destination, and route points, as well as weather data and track condition data.

[2011] An "operation route" is the path a train takes to reach its destination, which is optimized by a central control means.

[2012] An "automatic driving system" is a system that automatically controls train operation based on instructions from an operation control means.

[2013] An "emergency brake" is a braking device that immediately stops a train in an emergency.

[2014] The present invention relates to a vehicle system with an automatic driving function, which enables safe and efficient operation even in emergency situations. The system comprises the following components:

[2015] Overall system configuration

[2016] This system consists of a central management means, operation control means, monitoring means, abnormality notification means, and vehicles on board with personnel to respond to emergencies. The central management means functions as a server, collecting and analyzing operation information and generating instructions. The operation control means functions as a train terminal, operating the vehicle based on instructions from the server. The monitoring means and abnormality notification means are made up of devices such as cameras and sensors built into the terminals. Emergency response means include having personnel on board to respond to emergencies (usually a conductor).

[2017] Program processing

[2018] Server Processing

[2019] Collection of operation information

[2020] The server collects train operation information such as the train's current location, speed, destination, and route points from a database (e.g., PostgreSQL, MongoDB), and also obtains real-time weather data from a weather API (e.g., OpenWeatherMap API) and track condition data from a track monitoring system API, and analyzes this information.

[2021] Route and speed calculation

[2022] The server uses AI algorithms to calculate the optimal route and speed based on the collected operational information. Specifically, machine learning libraries (e.g., TensorFlow, PyTorch) are used to enable efficient and safe operation.

[2023] Generate and send instructions

[2024] Based on the calculation results, the server generates specific operating route and speed instructions for the operation control means (terminal) and sends them via a communication protocol (e.g., HTTP, MQTT).

[2025] Terminal (train) processing

[2026] Receiving instructions

[2027] The terminal receives the route and speed instructions sent from the server using edge computing devices (e.g., Raspberry Pi, Arduino) and uses Wi-Fi as the communication method.

[2028] Operation control

[2029] Based on the received instructions, the terminal controls the operation of the train through an automated driving system (e.g., ROS), specifically, driving the train at a specified speed along a specified route.

[2030] Surroundings monitoring

[2031] The device is equipped with a camera (e.g., Logitech C920) and a LIDAR sensor as a monitoring tool, and continuously monitors the surroundings while in operation. This information is sent to a server in real time.

[2032] Anomaly detection and notification

[2033] If the monitoring means detects an abnormality (e.g., an obstacle on the track or an abnormal speed limit), it immediately sends the information to the server through the abnormality notification means using image analysis software (e.g., OpenCV, TensorFlow Object Detection API).

[2034] User (conductor) role

[2035] Monitoring normal operations

[2036] During normal operation, the user (conductor) monitors the situation inside the train and checks for any problems. He or she patrols the train regularly to ensure safety.

[2037] Emergency response

[2038] When the server notifies the user of an emergency, the user (conductor) provides explanations to passengers, guides them to evacuate, and checks for and removes obstacles depending on the situation.

[2039] Reporting and Feedback

[2040] After the emergency situation has been resolved, the user (conductor) reports the details of the emergency response and the current safety status to the server. Based on this report, the next steps are taken promptly.

[2041] Specific examples

[2042] For example, if a train is in motion and encounters an obstacle on the tracks, it might do the following:

[2043] 1. Obstacle detection

[2044] The device's camera detects obstacles on the tracks and sends the information to the server. OpenCV is used for image analysis.

[2045] 2. Emergency stop instruction

[2046] The server analyzes the obstacle information and immediately sends an emergency stop command to the terminal.

[2047] 3. Performing an emergency shutdown

[2048] The terminal receives the emergency stop command and activates the emergency brake to stop the train.

[2049] 4. Conductor's Response

[2050] The conductor will guide passengers and guide them to safety based on instructions from the server, and will also check for and remove obstacles.

[2051] 5. Service resumes

[2052] After the obstacle is removed and safety is confirmed, the server calculates a new route and speed and sends it to the device, which receives the new instructions and resumes operation.

[2053] Example prompts for generative AI models

[2054] "Based on the following automated driving vehicle system, please explain in detail the safe operation process in an emergency. Please include the entire procedure from detecting an obstacle, making an emergency stop, responding to passengers, and resuming operation."

[2055] By feeding this prompt into a generative AI model, a detailed explanation of how a similar system would respond to an emergency can be obtained.

[2056] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2057] Step 1: Collecting traffic information

[2058] The server collects train operation information such as the train's current location, speed, destination, and route information from a database (e.g., PostgreSQL, MongoDB). It also retrieves real-time weather data from a weather API (e.g., OpenWeatherMap API) and track condition data from a track monitoring system API.

[2059] Input: Operation information in the database, weather data from the weather API, track condition data from the track monitoring system API

[2060] Output: Operation information, weather data, track condition data

[2061] Specific operation: The server executes a database query to extract the necessary operation information, and calls the weather API and track monitoring system API to obtain real-time data.

[2062] Step 2: Calculate route and speed

[2063] Based on the collected operational information, the server uses AI algorithms (e.g., TensorFlow, PyTorch) to calculate the optimal route and speed.

[2064] Input: Operation information, weather data, and track condition data collected in Step 1

[2065] Output: Optimal route and speed

[2066] Specific operation: The server runs an AI algorithm, analyzes operation information, weather data, and track condition data, and calculates the optimal operating route and speed.

[2067] Step 3: Generate and send instructions

[2068] Based on the calculation results, the server generates and sends specific operating route and speed instructions to the operation control means (terminal) via a communication protocol (e.g., HTTP, MQTT).

[2069] Input: Optimal route and speed calculated in step 2

[2070] Output: Route instructions, speed instructions

[2071] Specific operation: The server generates route and speed instructions and uses a communication protocol to send them to the terminal.

[2072] Step 4: Receiving instructions

[2073] The terminal receives route and speed instructions sent from the server, and uses edge computing devices (e.g., Raspberry Pi, Arduino) as hardware.

[2074] Input: Route and speed instructions from the server

[2075] Output: Received route and speed instructions (data stored in the device)

[2076] Specific operation: The device receives data sent from the server via Wi-Fi.

[2077] Step 5: Controlling the operation

[2078] Based on the received instructions, the terminal controls the train operation through the autonomous driving system (e.g., ROS).

[2079] Input: Route and speed instructions received in step 4

[2080] Output: Specific operation control (speed adjustment, route selection)

[2081] Specific operation: The terminal activates the autonomous driving system and controls the train's operation according to the specified speed and route.

[2082] Step 6: Monitor your surroundings

[2083] The device is equipped with a camera (e.g., Logitech C920) and a LIDAR sensor to continuously monitor the surroundings while driving, and this information is sent to a server in real time.

[2084] Input: Video data from cameras and LIDAR sensors, sensor data

[2085] Output: Surrounding situation data (sent to server)

[2086] Specific operation: The device uses the camera and sensors to capture the surrounding situation and sends the data to the server.

[2087] Step 7: Anomaly detection and notification

[2088] When a device detects an abnormality using a camera or sensor, it analyzes it using image analysis software (e.g., OpenCV), and if an abnormality is confirmed, it immediately sends the information to the server via HTTP POST.

[2089] Input: Surroundings data collected in step 6

[2090] Output: Anomaly detection data (notification to server)

[2091] Specific operation: The device analyzes the data using image analysis software, and if an abnormality is detected, it notifies the server.

[2092] Step 8: Monitor normal operations

[2093] During normal operation, the user (conductor) monitors the situation inside the train and checks for any problems. He or she patrols the train regularly to ensure safety.

[2094] Input: In-car situation

[2095] Output: Safety confirmation report

[2096] Specific actions: The conductor will patrol the train regularly to ensure the safety of passengers.

[2097] Step 9: Emergency response

[2098] When the server notifies the user of an emergency, the user (conductor) provides explanations to passengers and guides them to evacuate. The user also checks for and removes obstacles depending on the situation.

[2099] Input: Emergency notification from the server, actual emergency situation

[2100] Output: Passenger response, evacuation guidance, obstacle removal report

[2101] Specific actions: The conductor will make an announcement to passengers and take appropriate action and provide evacuation guidance.

[2102] Step 10: Reporting and Feedback

[2103] After the emergency situation has been resolved, the user (conductor) reports the details of the emergency response and the current safety status to the server.

[2104] Input: Emergency response history, current safety confirmation

[2105] Output: Detailed report

[2106] Specific operation: The conductor records the series of steps and results of the emergency response and sends them to the server.

[2107] (Application example 1)

[2108] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2109] While conventional autonomous vehicle systems can provide safe and efficient operation, they have the problem of being unable to take immediate and appropriate action when an emergency occurs. In particular, there are cases where personnel on board the vehicle are unable to respond quickly, and it takes time to ensure the safety of passengers. In addition, operation information and abnormality notifications are only sent to the operation manager, resulting in a lack of information for on-site response.

[2110] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2111] In this invention, the server includes a central management means for collecting operation information and calculating operation routes and speeds, an operation control means for receiving instructions from the central management means and controlling the operation of the vehicle, a monitoring means for monitoring the surrounding conditions while driving and detecting abnormalities, an abnormality notification means for notifying the central management means when an abnormality is detected, and a means for sending a notification to a smart device using the information when an emergency occurs. This enables a quick and appropriate response when an emergency occurs, thereby improving the safety of passengers and vehicles.

[2112] "Operation information" is a general term for detailed data such as the vehicle's current location, speed, destination, route points, weather data, and track conditions.

[2113] The "central management means" is a device or system that functions as a server, collects and analyzes operation information, and generates instructions.

[2114] The "operation control means" is a device or system that receives instructions from the central control means and controls the operation of the vehicle.

[2115] "Monitoring means" refers to a device or system that includes devices such as cameras and sensors for monitoring the surrounding conditions during operation and detecting abnormalities.

[2116] The "abnormality notification means" is a device or system that, when an abnormality is detected, promptly notifies the central management means of that information.

[2117] An "emergency situation" refers to an unexpected accident, malfunction, or abnormal event that occurs while a vehicle is in operation.

[2118] A "means for sending notifications to smart devices" is a device or system for quickly sending corresponding information to devices such as smartphones and tablets when an emergency occurs.

[2119] "Personnel" refers to personnel in the vehicle who are intended to respond in the event of an emergency.

[2120] This invention relates to a vehicle system with an automated driving function, which enables safe and efficient operation even in emergency situations. The system includes a central management means for collecting operation information and calculating operation routes and speeds, an operation control means for receiving instructions from the central management means and controlling the operation of the vehicle, a monitoring means for monitoring the surroundings while driving and detecting abnormalities, an abnormality notification means for notifying the central management means when an abnormality is detected, a vehicle carrying personnel who will respond in the event of an emergency, and a means for using the information to send a notification to a smart device when an emergency occurs.

[2121] System program description

[2122] The program of this system is configured as follows:

[2123] 1. Collection of operation information

[2124] The server retrieves operational information from the database, such as vehicle location, speed, destination, and route information, as well as real-time weather and track condition data. This information is continuously updated and sent to a central control unit.

[2125] 2. Route and speed calculation

[2126] The server uses AI algorithms to calculate optimal routes and speeds based on collected traffic information, using standard server hardware and software to run the AI ​​models.

[2127] 3. Instruction Generation and Transmission

[2128] Based on the calculation results, the server proposes specific driving routes and speeds and sends them to the central management means, which then receives the instructions and controls the driving of the vehicles.

[2129] 4. Surrounding situation monitoring and abnormality detection

[2130] The device's onboard cameras and sensors monitor the surroundings while driving, and if an abnormality is detected, the information is sent to a server in real time. This monitoring method is implemented using high-performance image recognition software and sensors.

[2131] 5. Abnormality notification and emergency response

[2132] When an emergency occurs, the server analyzes the information and issues appropriate instructions for response. The server then sends an emergency notification to smart devices, prompting occupants and other responders to act quickly. This functionality is achieved using a notification service such as Firebase Cloud Messaging (FCM).

[2133] Example of a system

[2134] For example, the following is an example of a prompt that may be issued when an obstacle is discovered on the tracks during operation:

[2135] Example of a text prompt:

[2136] "It obtains information about autonomous vehicles from a central server and monitors their operating status in real time. If an abnormality occurs, it sends an emergency notification to the user's smartphone, instructing them on the appropriate response."

[2137] This system will enable a swift and appropriate response in the event of an emergency, enhancing the safety of passengers and trains. Conductors and train operators will be able to respond immediately to emergencies, provide guidance to passengers, and guide them to safety.

[2138] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2139] Step 1:

[2140] The server obtains operation information such as the vehicle's current location, speed, destination, and route points from the database. It also obtains real-time weather data and track condition data and integrates this information. The input is data from various sensors and the database, and the output is integrated operation information. This allows the server to have the most up-to-date operation information.

[2141] Step 2:

[2142] The server uses an AI algorithm based on the collected traffic information to calculate the optimal route and speed. The input is the traffic information obtained in the previous step, and the output is the optimal route and speed instructions. Specifically, the AI ​​model analyzes real-time data and calculates an efficient and safe route.

[2143] Step 3:

[2144] The server generates specific driving route and speed instructions based on the calculation results and sends the instructions to the central control means, where the input is the optimized driving route and speed instructions and the output is the actual instruction information, so that the central control means receives the latest driving instructions.

[2145] Step 4:

[2146] The terminal receives the driving route and speed instructions sent from the central control means and controls the driving of the vehicle through the automatic driving system. The input is the driving instructions from the central control means and the output is the actual driving status. In concrete terms, the terminal controls the speed of the vehicle and drives it along the specified route.

[2147] Step 5:

[2148] The device uses its on-board camera and sensors to monitor the surroundings while driving and detect any abnormalities. The input is real-time data from the camera and sensors, and the output is the results of any abnormalities detected. Specifically, image analysis software analyzes the video and detects obstacles and abnormal speeds.

[2149] Step 6:

[2150] When an abnormality is detected, the terminal notifies the server in real time. The input is the abnormality detection data, and the output is an abnormality notification to the central management means. This allows the server to immediately recognize the abnormality.

[2151] Step 7:

[2152] The server analyzes the anomaly information and generates appropriate response instructions. The input is the anomaly notification data, and the output is emergency response instructions. Depending on the anomaly that has occurred, the server generates instructions such as an emergency shutdown.

[2153] Step 8:

[2154] The server sends emergency response instructions to smart devices and notifies crew members and other relevant parties. The input is the emergency response instructions, and the output is a notification message. Specifically, the server sends an emergency notification to smartphones and tablets via Firebase Cloud Messaging (FCM).

[2155] Step 9:

[2156] The user (crew member / conductor) implements emergency response based on the notification from the server. The input is the emergency response instructions displayed on the smart device, and the output is the actual response action. Specific actions include explaining the situation to passengers and guiding them to evacuate.

[2157] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2158] The present invention further improves the safety and efficiency of driving by combining an emotion engine with a vehicle system having an automatic driving function. Hereinafter, embodiments of the present invention will be described in detail.

[2159] Overall system configuration

[2160] The system includes the following major components:

[2161] Central management means (server): Manages and analyzes operation information and generates instructions.

[2162] Operation control means (terminal): Receives instructions from the central control means and controls the operation of the vehicle.

[2163] Monitoring Measures: Continuously monitor the surrounding environment and conditions while the vehicle is in operation.

[2164] Abnormality notification means: Notifies the central management means when an abnormality is detected.

[2165] Emotion engine: Recognizes the emotional state of personnel (conductors) and reflects it in management and response.

[2166] Personnel (Conductor): Operates within the vehicle to respond to emergencies.

[2167] Program processing

[2168] Server Processing

[2169] 1. Collection of operational information:

[2170] The server retrieves information about the train's current location, speed, destination, stops, and route from a database, as well as real-time weather and track condition data.

[2171] 2. Route and speed calculation:

[2172] The server uses AI algorithms to calculate the optimal route and speed based on the collected operational information, records the calculation results, and generates instructions.

[2173] 3. Generate and send instructions:

[2174] The server sends the generated instructions to the operation control means (terminal), which include detailed operation routes, speeds, and emergency response procedures.

[2175] 4. Real-time analysis:

[2176] Even after the vehicle has started moving, the server continues to receive and analyze data on the surrounding situation sent from the device in real time, and sends new operating instructions to the device as needed.

[2177] 5. Emotion recognition and response:

[2178] The server receives and analyzes the conductor's emotional data sent from the emotion engine, and responds accordingly based on the conductor's mental state and stress level.

[2179] Terminal (train) processing

[2180] 1. Receiving and setting instructions:

[2181] The terminal receives the route and speed instructions sent from the server, checks the received information, and sets it up in the autonomous driving system.

[2182] 2. Navigation control:

[2183] The device's autonomous driving system operates the train based on the received route and speed instructions, and the train travels along the specified route at the specified speed.

[2184] 3. Surrounding Area Monitoring:

[2185] The terminals are equipped with cameras and sensors as monitoring tools, and continuously monitor the surroundings while in operation. This information is sent to a server in real time.

[2186] 4. Anomaly detection and notification:

[2187] If the monitoring means detects an abnormality (for example, an obstacle on the tracks or an abnormal speed limit), the information is immediately sent to the server via the abnormality notification means.

[2188] 5. Collecting Emotional Data:

[2189] The emotion engine continuously monitors the conductor's emotional state and transmits the data to the server.

[2190] User (conductor) role

[2191] 1. Monitoring normal operations:

[2192] During normal operation, the user (conductor) monitors the situation inside the train and checks for any particular problems. He or she also patrols the train regularly to ensure safety.

[2193] 2. Emergency Response:

[2194] When an emergency situation is notified by the server, the conductor will act according to the appropriate emergency response procedures, such as explaining the situation to passengers and guiding them to an evacuation.

[2195] 3. Status reports and feedback:

[2196] After the emergency situation has been resolved, the conductor will report the situation in detail to the server, including the details of the emergency response and confirmation of the current safety situation.

[2197] 4. Emotional management:

[2198] Based on feedback from the emotion engine, the conductor can take measures according to his or her emotional state, such as taking a break if stress levels are high.

[2199] Specific examples

[2200] For example, consider the process that is performed when a train detects an obstacle on the tracks while in operation.

[2201] 1. Obstacle detection:

[2202] The device's camera detects obstacles on the tracks and sends that information to a server.

[2203] 2. Emergency stop instruction:

[2204] The server analyzes the obstacle information and immediately sends an emergency stop command to the terminal if it determines it is necessary. At the same time, it notifies the conductor of the emergency response proce...

Claims

1. A vehicle system with an automatic driving function, a central control means for collecting operation information and calculating operation routes and speeds; an operation control means for receiving instructions from the central control means and controlling the operation of the vehicle; a monitoring means for monitoring the surroundings during operation and detecting abnormalities; an abnormality notification means for notifying the central management means when an abnormality is detected; A vehicle carrying personnel who will respond in the event of an emergency, A system including:

2. 10. The system of claim 1, A system in which a central control means issues an emergency shutdown command when an abnormality occurs and notifies personnel of emergency response procedures.

3. 10. The system of claim 1, A system in which the operation control means includes means for receiving new operation instructions from the central management means and resuming operation.

Images