System

Abstract

A system is provided.SOLUTION: A system, comprising: means for collecting video recorded by a recording recorder; means for transmitting the collected video data to a server; means for analyzing the transmitted video data and detecting a situation of a traffic accident or road damage; means for notifying a relevant institution of detected abnormality information; and means for analyzing a traffic volume and determining a priority of road repair.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional road management systems have difficulty quickly and accurately grasping the status of traffic accidents and road damage, which can delay efficient responses and repairs. Furthermore, dashcam footage is only used individually, and there is a lack of a mechanism for comprehensively analyzing overall abnormalities and traffic volume. To solve these problems and improve public safety, a system is needed that effectively utilizes dashcam footage to detect and notify abnormalities in real time. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for collecting video recorded by a drive recorder, a means for transmitting the collected video data to a server, a means for analyzing the transmitted video data to detect traffic accidents and road damage, a means for notifying relevant authorities of detected abnormalities, and a means for analyzing traffic volume and determining the priority of road repairs. This system allows for quick detection of road abnormalities using the video from the drive recorder, allowing for prompt implementation of necessary measures. Furthermore, by acquiring location information along with the video and transmitting it to the server, the location of the abnormality can be accurately identified. The notification includes the location, type, and severity of the abnormality, as well as an estimated repair time, enabling relevant authorities to respond quickly.

[0006] A "dashcam" is a device installed in a vehicle that records video and audio while driving.

[0007] "Video data" refers to the video information recorded by a drive recorder, and is digital data that visualizes traffic conditions and road conditions.

[0008] A "server" is a computer system that receives, stores, and analyzes video data and GPS data over a network.

[0009] "Analysis" is the process of automatically extracting specific information, such as traffic accidents and road damage, from the collected video data.

[0010] "Anomaly information" is detailed data on abnormalities such as traffic accidents and road damage detected through analysis.

[0011] "Related organizations" refers to government agencies, administrative agencies, or private companies involved in road management and traffic safety.

[0012] "Traffic volume" is data that indicates the number and frequency of vehicles passing through a particular road section.

[0013] "Location information" is digital data that indicates a specific location using GPS or other means, and is used in combination with video data to identify abnormalities.

[0014] "Repair priority" is a criterion for determining which road sections should be given priority for repair based on the analysis results.

[0015] "Notification" refers to a message or alert that notifies relevant authorities of abnormal information.

[0016] "Estimated repair time" is data that estimates the time required for repair work to be performed on a detected abnormality. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The system of the present invention collects video footage recorded by a drive recorder and analyzes it on a server to quickly and efficiently detect traffic accidents and road damage. This system notifies relevant authorities of abnormality information, analyzes traffic volume, and prioritizes road repairs. Specific program processing and its implementation are explained in natural language below.

[0039] Video collection and transmission by dashcam

[0040] The device (dashcam) records high-resolution video while the vehicle is driving. For example, if the vehicle is driving throughout the day, it will record road conditions and traffic conditions in full HD resolution. The device also collects GPS data along with the video, linking the location information to the video. This data is sent to a server in real time via Wi-Fi or a mobile network.

[0041] Data reception and analysis by the server

[0042] The server receives the video data and GPS information sent from the device. The received data is stored in a database to ensure security and redundancy. Analysis software on the server then analyzes the video data to identify traffic accident situations and road damage. The analysis is performed using machine learning models to detect anomalies in each frame of the video. For example, if an anomaly such as a large hole, crack, or traffic accident scene is detected, its location and time are identified.

[0043] Notification and Arrangements

[0044] Based on the analysis results, the server notifies relevant authorities of any abnormalities in real time. The notification includes detailed information such as the location, type, severity, and estimated repair time of the abnormality. For example, if a large hole is detected on a specific road, detailed information about it will be sent to the road management bureau via email or SMS.

[0045] Traffic analysis and repair prioritization

[0046] The server analyzes traffic volume based on the video data to understand road usage. The analyzed traffic volume data is used to determine repair priorities. For example, repairs to defects on major roads with very heavy traffic may be planned as a priority, while roads with less traffic may be postponed.

[0047] User interface support

[0048] Users can check notifications from the system through the interface. For example, after a road maintenance officer receives a notification, he or she can check the detailed situation on the system interface and enter the necessary instructions, allowing a repair team to quickly head to the site and begin repair work.

[0049] Specific examples

[0050] For example, one morning, while a vehicle is driving down a major road, the dashcam detects a large pothole ahead and records the video. The video and GPS data are immediately sent to a server. The server receives the data, and analysis software detects the large pothole. The analysis results identify its location and severity, and notify the road management authority. The road management authority then issues repair orders through the system and arranges for the necessary resources, allowing repairs to be carried out quickly.

[0051] Such systems can improve public safety by quickly detecting and responding to road anomalies.

[0052] The processing flow will be explained below.

[0053] Step 1:

[0054] The device records video while the vehicle is in motion.

[0055] The dashcam captures high-resolution footage of the vehicle while driving and saves each frame.

[0056] The GPS sensor records your current location in real time and tags it in the footage.

[0057] Step 2:

[0058] The device sends the collected video and location information to the server.

[0059] The device uploads data to the server at regular intervals via Wi-Fi or mobile network.

[0060] Confirm the data transmission and record the successful transmission in the log.

[0061] Step 3:

[0062] The server stores the received video data and location information.

[0063] The server receives the data and stores it in a database.

[0064] Video files are assigned timestamps and location information and are centrally managed.

[0065] Step 4:

[0066] The server analyzes the stored video data.

[0067] Computer vision technology and machine learning models are used to detect anomalies in the video (e.g., potholes, cracks, accident scenes, etc.).

[0068] The type and severity of the abnormality is evaluated based on the detection results.

[0069] Step 5:

[0070] The server notifies the relevant authorities of the abnormality information.

[0071] Detailed information about the anomaly (location, type, severity, estimated repair time) is sent to relevant authorities via an automated notification system.

[0072] Notification methods include email, SMS, and in-system messaging.

[0073] Step 6:

[0074] The server analyzes traffic volume from video data and determines repair priorities.

[0075] The number of cars in the video and their frequency of passage are measured to estimate traffic volume.

[0076] Traffic volume data is compiled and repair priorities for abnormal areas are determined.

[0077] Step 7:

[0078] The user checks the notification through the interface and takes action.

[0079] Road management personnel check the notification and view detailed information on the system interface.

[0080] Necessary repair instructions and resource arrangements are made via the interface.

[0081] By using the above processing steps, the system of the present invention can quickly detect road abnormalities and take appropriate measures.

[0082] Example 1

[0083] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0084] Conventional systems for detecting traffic accidents and road damage required manual confirmation and reporting, making it difficult to respond quickly. Furthermore, delays in reporting and responding to abnormal information often threatened public safety. Traffic volume analysis and prioritization of road repairs were also inappropriate, preventing efficient repair work. The present invention aims to solve these problems.

[0085] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0086] In this invention, the server includes a means for storing video footage and GPS data recorded by the drive recorder in a database in real time, a means for detecting road damage and abnormalities using a machine learning model, a means for notifying relevant authorities of detected abnormalities, and a means for analyzing traffic volume and determining the priority of road repairs. This enables rapid and efficient detection of road condition abnormalities and real-time notification to relevant authorities, enabling prompt and appropriate response. Furthermore, determining repair priorities based on traffic volume data enables efficient road repairs.

[0087] A "dashcam" is a device that records video and audio while a vehicle is in motion.

[0088] An "external domain" refers to a remote data storage location, such as a server or cloud storage connected via a network.

[0089] "Video data" refers to video information recorded by a drive recorder.

[0090] "GPS Data" means location information collected using the Global Positioning System.

[0091] The "database" is a system for systematically storing and managing received video data and GPS data.

[0092] A "machine learning model" is an algorithm or system that learns from large amounts of data and performs pattern recognition and prediction.

[0093] "Abnormal information" refers to information that deviates from normal conditions, such as traffic accidents or road damage.

[0094] "Related agencies" refer to organizations and institutions that respond to traffic accidents and road damage, such as road management bureaus and traffic police.

[0095] "Traffic analysis" is the process of evaluating road usage and traffic flow through data analysis.

[0096] "Road repair priorities" are criteria for determining which areas should be given priority when carrying out road repairs or maintenance.

[0097] Embodiments of the present invention will be described in detail below.

[0098] The system of the present invention collects video data and GPS data recorded by a drive recorder and transmits them to a server. The device (drive recorder) records video in high resolution and simultaneously collects GPS data. The device transmits this data to the server in real time via Wi-Fi or a mobile network.

[0099] Specifically, when the vehicle starts moving, the drive recorder automatically starts recording video. The device records high-resolution video and GPS data while the vehicle is moving, and associates it with a timestamp and location information. This data is temporarily stored inside the device and periodically sent to the server. Communication errors are checked by receiving a transmission confirmation message from the server.

[0100] The server receives the video data and GPS data sent from the device and stores it in a database. When storing the data, the data is backed up in multiple storage devices to ensure safety and redundancy. The received data is analyzed using machine learning models by the server's analysis software. During the analysis, the video data is analyzed frame by frame to detect traffic accidents and road damage. For example, if an abnormality such as a large hole, crack, or traffic accident site is detected, the location and time of the abnormality are identified.

[0101] Based on the analysis results, the server notifies the relevant authorities of any detected anomalies. The notification includes information such as details of the anomaly, its location, type, severity, and estimated repair time. This information is sent to the relevant authorities (e.g., road management bureau) via email or SMS. In addition, the analyzed traffic volume data is used to determine the priority of repairs. Anomalies on major roads with very heavy traffic volume are given priority for repair planning, while roads with less traffic volume may be postponed.

[0102] The user (e.g., road maintenance personnel) checks the system's notification through the system's interface. After receiving the notification, the user can check the detailed situation in the interface and enter the necessary instructions, allowing repair teams to quickly head to the site and begin repair work.

[0103] Specific examples

[0104] For example, one morning, a vehicle traveling down a major road might detect a large pothole ahead and record video footage of it on its dashcam. The video and GPS data are immediately sent to a server, where analysis software detects the presence of the pothole. The analysis results identify its location and severity, and the road management authority is notified. The road management authority then issues repair orders through the system and arranges for the necessary resources to be used, ensuring prompt repairs.

[0105] Such a system can improve public safety by quickly detecting road anomalies and responding quickly.

[0106] Prompt Sentence Examples

[0107] "Please explain in detail, step by step, how you analyze footage recorded by a dashcam to detect road anomalies. Also, please explain in detail the notification and response process when an anomaly is detected."

[0108] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0109] Step 1:

[0110] The device collects video and GPS data while the vehicle is moving. It receives video of the vehicle moving and its current location as input, and outputs high-resolution video and GPS data. Specifically, the device automatically starts recording when the vehicle's engine starts, recording video and GPS data in full HD. This data is temporarily stored in the device's internal memory.

[0111] Step 2:

[0112] The device sends the collected video data and GPS data to the server. It takes the collected video data and GPS data as input and outputs the data to the server. Specifically, the device periodically uploads data packets to the server using Wi-Fi or a mobile network. It also checks for communication errors by receiving a confirmation message that transmission is complete.

[0113] Step 3:

[0114] The server receives the video data and GPS data sent from the device. It receives the data packets sent as input and outputs them to be safely stored in a database. Specifically, the server monitors the receiving port in real time and promptly stores the received data in the database. It also copies the data to multiple backup storages to ensure redundancy.

[0115] Step 4:

[0116] The server analyzes the received video data and GPS data. Using the stored video data and GPS data as input, it obtains an output that detects traffic accidents and road damage. Specifically, the analysis software on the server uses a machine learning model to detect anomalies in each video frame. For example, large holes, cracks, and traffic accident scenes on the road are analyzed. As a result of the analysis, the location and severity of the detected anomalies are identified.

[0117] Step 5:

[0118] Based on the analysis results, the server notifies relevant organizations of detected anomalies. It receives the analysis result data as input and obtains an output that sends detailed information to relevant organizations. Specifically, the server creates a notification message that includes the location, type, severity, estimated repair time, etc. of the anomaly, and sends it via email or SMS to relevant organizations such as road management bureaus. In addition, images or videos of the anomaly may be attached to the notification email if necessary.

[0119] Step 6:

[0120] The server analyzes traffic volume based on video data and determines repair priorities. It takes in all video data and road condition data as input and outputs a list of repair priorities. Specifically, the server calculates traffic volume for specific road sections from the video data and ranks the repair priority of each road based on the traffic volume information. The more major roads with heavy traffic have defects, the higher the priority for repair plans.

[0121] Step 7:

[0122] The user checks notifications from the system through an interface and inputs the necessary instructions. The input is a confirmation of the notification message, and the system outputs the instructions and resource arrangement information required for repair response. Specifically, the user (road management officer) logs into the system and checks the details of the notification (location, type of anomaly, severity, etc.). After that, the system registers and sends the arrangement of a repair team and detailed instructions for the repair content. This allows the repair team to respond quickly.

[0123] (Application example 1)

[0124] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0125] In modern road traffic, early detection and rapid response to traffic accidents and road damage are important. However, conventional traffic monitoring systems require dedicated infrastructure, which is costly and difficult to operate efficiently. It is also difficult to prioritize road repairs appropriately in response to fluctuations in traffic volume. Furthermore, with the spread of autonomous vehicles, there is a need for real-time understanding of the road conditions these vehicles face and support for safe operation.

[0126] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0127] In this invention, the server includes means for collecting video recorded by the drive recorder, means for transmitting the collected video data to the server, means for analyzing the transmitted video data and detecting traffic accidents and road damage, means for notifying relevant organizations of detected abnormality information, means for analyzing traffic volume and determining the priority of road repairs, and means for collecting and transmitting video and location information during driving in real time via an application installed in the autonomous vehicle, feeding back abnormality detection results to the server, and proposing safe avoidance routes to the autonomous vehicle. This not only enables early detection of traffic accidents and road damage and rapid response, but also supports the safe operation of the autonomous vehicle.

[0128] A "drive recorder" is a device that records video while a vehicle is driving.

[0129] A "server" is a computing system that receives, stores, analyzes, and processes data over a network.

[0130] An "autonomous vehicle" is a vehicle that can drive autonomously using artificial intelligence and sensors.

[0131] "Video data" refers to digital data of visual information recorded by a drive recorder.

[0132] "GPS Data" means location information obtained from the Global Positioning System (GPS) to indicate a precise location on Earth.

[0133] "Analysis Software" means the programs used to interpret collected data and extract specific information on demand.

[0134] "Abnormal information" is data on events that deviate from normal conditions, specifically traffic accidents and road damage.

[0135] "Related organizations" refers to government agencies and companies in charge of road management and traffic management.

[0136] "Traffic analysis" is the process of analyzing the flow of vehicles and traffic on a particular road to identify patterns and trends.

[0137] "Repair priority" is a criterion for determining the order of repair work for road sections where abnormalities have been detected.

[0138] A "user interface" is an interactive platform that allows users to view displayed information and perform necessary operations.

[0139] A "safe avoidance route" is an alternative route for a detected anomaly, including instructions for an autonomous vehicle to safely reach its destination.

[0140] The system of the present invention is mainly composed of a drive recorder, a GPS module, a server, and analysis software that are installed in a vehicle, including an autonomous driving vehicle.

[0141] Data collection and transmission by the device (autonomous vehicle):

[0142] The autonomous vehicle terminal is equipped with a dashcam and a GPS module. The dashcam records high-resolution video while the vehicle is driving, and the GPS module obtains precise location information. The collected video data and GPS data are transmitted to a server in real time. To achieve this, the terminal is connected to the Internet via Wi-Fi or a mobile network.

[0143] Data reception and analysis by the server:

[0144] The server receives the video data and GPS information sent from the device and stores it in a database for security and redundancy. Next, analysis software on the server uses machine learning models to analyze the video data and detect traffic accidents and road damage. This analysis uses software such as OpenCV and TensorFlow. If an abnormality is detected, the location information and details are notified to the relevant authorities.

[0145] Real-time safety assessment and route suggestions:

[0146] Based on the analysis results, the server proposes safe avoidance routes to autonomous vehicles in real time, enabling vehicles in operation to avoid abnormal areas and operate safely. The route proposal algorithm is implemented using Google Maps API, etc.

[0147] Notification and Arrangements:

[0148] Notification of abnormalities is made via communication methods such as email and SMS. Notifications include the location, type, severity, and estimated repair time of the abnormality, allowing relevant organizations to take prompt action based on this information.

[0149] User Interface:

[0150] Personnel from relevant agencies can check notifications from the system through a dedicated user interface, which allows them to check the detailed situation, input necessary instructions, and take prompt action such as dispatching a repair team.

[0151] Specific examples

[0152] For example, one morning, while an autonomous vehicle is driving down a major road, its dashcam detects a large pothole ahead and immediately sends the video and GPS data to a server. The server receives the data, and analysis software detects the large pothole. The analysis identifies its location and severity, and notifies the road management authority. The road management authority then issues repair orders through the system and arranges for the necessary resources, ensuring prompt repairs.

[0153] Example prompts for generative AI models

[0154] "Design an application that automates the response when a dashcam detects road damage ahead while an autonomous vehicle is driving. Please provide specific code examples and the processing flow for building a system that sends video data and GPS data to a server in real time and notifies the relevant authorities if an abnormality is detected."

[0155] In this way, by describing the mode for carrying out the invention in detail, the components of the system, the operation procedures, and specific examples will become clear.

[0156] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0157] Step 1:

[0158] The device activates the dashcam while the vehicle is moving, and collects video data and GPS data synchronously. The input is the vehicle's driving status and current location, and the output is high-resolution video data and location information. Video frames are stored in a buffer in real time, and GPS data is simultaneously linked.

[0159] Step 2:

[0160] The device transmits the collected video data and GPS data to the server in real time. The input is the video data and GPS data acquired in step 1, and the output is the status of successful transmission to the server. This process involves encoding the data using Wi-Fi or mobile network and sending it to the server via the HTTP protocol.

[0161] Step 3:

[0162] The server stores the received video data and GPS data in a database. The input is the data sent from the device, and the output is the status of successful storage in the database. The server performs data consistency checks to ensure redundancy and performs highly reliable storage processing.

[0163] Step 4:

[0164] The analysis software on the server analyzes the received data and detects anomalies such as traffic accidents and road damage. The input is the saved video data and GPS data, and the output is the identification of anomalies and detailed information about them. Here, a machine learning model is used to determine whether there are any anomalies in each video frame and match them with location information.

[0165] Step 5:

[0166] The server notifies the relevant organizations of the anomaly information. The input is the detailed information about the anomaly obtained in step 4, and the output is the status of sending the notification to the relevant organizations. In this process, detailed information including the location, type, severity, and estimated repair time of the anomaly is notified using email and SMS sending functions.

[0167] Step 6:

[0168] The server analyzes traffic volume and determines the priority of road repairs. The input is collected traffic data and anomaly information, and the output is a list of repair priorities. The server analyzes traffic volume fluctuations and calculates repair priorities based on the frequency of use of anomaly points.

[0169] Step 7:

[0170] The device proposes a safe avoidance route to the autonomous vehicle based on feedback from the server. The input is the abnormality information and the avoidance route proposal obtained from the server, and the output is the vehicle's new driving route. The device uses the navigation system to calculate a safe route in real time and update the driving plan.

[0171] This allows the above-mentioned processing steps to work together to create a system that enables safe and efficient operation of autonomous vehicles.

[0172] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0173] The system of the present invention collects video footage recorded by a drive recorder and analyzes it on a server to quickly and efficiently detect traffic accidents and road damage. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, it is possible to respond flexibly according to the user's condition. Below, specific program processing and its implementation are explained in natural language.

[0174] Video collection and transmission by dashcam

[0175] The device (dashcam) records high-resolution video while the vehicle is driving. For example, if the vehicle is driving in an urban area, it records road conditions and traffic conditions in full HD resolution. The device also collects GPS data along with the video, linking the location information to the video. This data is sent to a server in real time via Wi-Fi or a mobile network.

[0176] Data reception and analysis by the server

[0177] The server receives the video data and GPS information sent from the device. The received data is stored in a database to ensure security and redundancy. The analysis software on the server then analyzes the video data to identify traffic accident situations and road damage. The analysis is performed using a machine learning model to detect anomalies in each frame of the video. For example, if an anomaly such as a large hole or crack in the road or the scene of a traffic accident is detected, the location and time of the anomaly are identified.

[0178] Notification and Arrangements

[0179] Based on the analysis results, the server notifies relevant authorities of any abnormalities in real time. The notification includes detailed information such as the location, type, severity, and estimated repair time of the abnormality. For example, if a large hole is detected on a specific road, detailed information about it will be sent to the road management bureau via email or SMS.

[0180] Traffic analysis and repair prioritization

[0181] The server analyzes traffic volume based on the video data to understand road usage. The analyzed traffic volume data is used to determine repair priorities. For example, repairs to defects on major roads with very heavy traffic may be planned as a priority, while roads with less traffic may be postponed.

[0182] User response using an emotion engine

[0183] The server uses an emotion engine to analyze the user's emotional state. The emotion engine recognizes the user's emotions from video, audio, and the user's operation patterns, and can detect, for example, stress or impatience. This allows for flexible responses depending on the user's emotional state when reporting an abnormality. For example, if the user is feeling extremely stressed, the content of the notification and the priority of the response can be adjusted.

[0184] Specific examples

[0185] For example, one morning, while a vehicle is driving down a major street in the city, the dashcam detects a large pothole ahead and records the video. The video and GPS data are immediately sent to the server. The server receives the data, and analysis software detects the large pothole. The analysis results identify the location and severity of the pothole, and notify the road management bureau. The road management bureau then issues repair orders through the system and arranges for the necessary resources, ensuring that repairs are carried out quickly. At the same time, an emotion engine is used to analyze the user's emotional state when they report the abnormality, and further measures can be taken if the abnormality is particularly urgent. This type of system makes it possible to quickly detect road abnormalities and implement appropriate responses.

[0186] The processing flow will be explained below.

[0187] Step 1:

[0188] The device records video while the vehicle is in motion.

[0189] The dashcam captures high-resolution footage of the vehicle while driving and saves each frame.

[0190] The GPS sensor records your current location in real time and tags it in the footage.

[0191] Step 2:

[0192] The device sends the collected video and location information to the server.

[0193] The device uploads data to the server at regular intervals via Wi-Fi or mobile network.

[0194] Confirm the data transmission and record the successful transmission in the log.

[0195] Step 3:

[0196] The server stores the received video data and location information.

[0197] The server receives the data and stores it in a database.

[0198] Video files are assigned timestamps and location information and are centrally managed.

[0199] Step 4:

[0200] The server analyzes the stored video data.

[0201] Computer vision technology and machine learning models are used to detect anomalies in the video (e.g., potholes, cracks, accident scenes, etc.).

[0202] The type and severity of the abnormality is evaluated based on the detection results.

[0203] Step 5:

[0204] The server analyzes the user's emotional state.

[0205] It uses an emotion engine that recognizes user emotions from video and audio data and operation patterns.

[0206] Analyze and evaluate the user's emotional state (e.g., stress, impatience) in real time.

[0207] Step 6:

[0208] The server notifies the relevant authorities of the abnormality information.

[0209] Detailed information about the anomaly (location, type, severity, estimated repair time) is sent to relevant authorities via an automated notification system.

[0210] Notification methods include email, SMS, and in-system messaging.

[0211] Step 7:

[0212] The server reflects the emotion data in traffic volume analysis and repair priority determination.

[0213] The system takes into account the user's emotional state and adjusts the priority of repairs if the urgency is high.

[0214] The final decision is made by combining the priorities calculated based on traffic volume with the analysis results of the emotion engine.

[0215] Step 8:

[0216] The user checks the notification through the interface and takes action.

[0217] Road management personnel check the notification and view detailed information on the system interface.

[0218] Necessary repair instructions and resource arrangements are made via the interface.

[0219] By using the above processing steps, the system of the present invention can quickly detect road abnormalities and take appropriate measures. Furthermore, by taking the user's emotional state into consideration, more flexible and appropriate responses are possible.

[0220] Example 2

[0221] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0222] Traffic accidents and road damage must be detected quickly and accurately, but conventional methods often delay appropriate analysis and notification. Furthermore, responses that ignore the user's emotional state can increase stress and frustration. The purpose of this invention is to solve these problems by efficiently and quickly detecting anomalies, notifying relevant authorities, and enabling flexible responses based on the user's emotional state.

[0223] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing the transmitted video data and detecting the situation of a traffic accident or road damage, a means for notifying the relevant authorities of the detected abnormality information, a means for analyzing the emotional state of the user, and a means for providing a flexible response to the user based on the analysis result. This enables quick detection and notification of a traffic accident or road damage, and also enables a flexible response according to the emotional state of the user.

[0224] A "dashcam" is a device that records images of the front and rear of a vehicle while it is in motion.

[0225] "Video data" refers to video information collected by a drive recorder.

[0226] A "server" is a computer system that receives, analyzes, and stores data over a network and works in conjunction with other systems and devices.

[0227] "Transmitting means" refers to the mechanism by which collected data is sent to other systems via wireless or wired communication.

[0228] "Analysis means" refers to devices and technologies, including software and algorithms, that process collected data and detect abnormalities such as traffic accidents and road damage.

[0229] "Traffic accident" means an accident caused by a collision between vehicles or contact between a vehicle and a person or object.

[0230] "Road damage" refers to conditions that impede vehicle movement, such as cracks or holes in the road surface.

[0231] "Notification means" refers to the mechanism for communicating analysis results to relevant agencies by means of email, SMS, etc.

[0232] "Relevant organizations" refer to organizations that should respond to detected abnormalities, such as road management bureaus and police.

[0233] "Traffic volume analysis means" refers to technology that analyzes the number, speed, flow, etc. of vehicles based on video data.

[0234] "Means for determining repair priority" refers to a mechanism for determining the order and priority of road repairs based on analysis results and traffic volume data.

[0235] "Emotional state analysis means" refers to technology that analyzes the user's tone of voice, operation patterns, etc. to determine the user's emotions.

[0236] "Flexible response means" refers to techniques and processes for adjusting notification content and changing response priorities according to the user's emotional state.

[0237] The system of this invention collects video footage recorded by a dashcam and analyzes it on a server to quickly and efficiently detect traffic accidents and road damage.The system also incorporates an emotion engine that recognizes the user's emotional state and enables flexible responses.

[0238] Video collection and transmission by dashcam

[0239] The device (dashcam) records high-resolution video while the vehicle is driving. For example, when the vehicle is driving in an urban area, the dashcam records video of the road ahead in full HD resolution. This can be done using a commercially available high-performance camera. A GPS device also operates to obtain location information, collecting location data in real time. This data (video data and GPS data) is processed in real time within the device and sent to a server using Wi-Fi or a mobile network.

[0240] Data reception and storage by the server

[0241] The server receives the video data and GPS information sent from the device. The server is equipped with a high-performance data receiving module (e.g., an API server using Python and Flask) and receives data packets arriving via the Internet. It then temporarily stores this data in storage and stores it in a database (e.g., MySQL or PostgreSQL) in a highly redundant format.

[0242] Data analysis by server

[0243] The server analyzes the stored video data and location information. Machine learning models (e.g., deep learning models using TensorFlow or PyTorch) are used for video analysis, with anomaly detection algorithms being particularly important. The video data is processed frame by frame to detect anomalies (e.g., large holes or debris from traffic accidents) within each frame. Any anomalies detected by the analysis software are recorded in a database along with their location information.

[0244] Server notification of abnormal information

[0245] The server notifies the relevant authorities of the anomaly information based on the analysis results. This notification process is automated, and a message containing information such as the location, type, severity, and estimated repair time of the anomaly is generated based on the analysis results. The message is sent to the relevant authorities (e.g., road management authorities or police) via email or SMS. This can be done, for example, using the SMTP protocol or an SMS gateway.

[0246] Traffic analysis and repair priority determination by server

[0247] The server analyzes traffic volume from the video data. During the analysis process, the number of vehicles, speed, flow, etc. are analyzed and recorded in a traffic volume database. Based on these results, an algorithm is run to determine the priority of repair work. For example, the data analysis is performed using the Python Pandas library, and a high priority is assigned to abnormalities on major roads based on a priority algorithm.

[0248] Server-based user emotion analysis and response

[0249] The server uses an emotion engine to analyze the user's emotional state. The emotion engine uses a generative AI model (for example, a large-scale language model such as GPT-3) to process data such as the user's voice and operation patterns to recognize the user's emotional state. For example, it can detect stress or impatience from the user's typing speed and tone of voice when operating the device. Based on the results of this emotion analysis, the server can respond flexibly to the user. For example, for a user who feels stressed, it can simplify the notification content and provide a high-priority response.

[0250] Specific examples

[0251] For example, one morning, while a vehicle is driving down a major street in the city, the dashcam detects a large pothole ahead and records the video. The video and GPS data are immediately sent to the server. The server receives the data, and analysis software detects the large pothole. The analysis results identify the location and severity of the pothole, and notify the road management bureau. The road management bureau then issues repair orders through the system and arranges for the necessary resources, ensuring prompt repairs. At the same time, an emotion engine is used to analyze the user's emotional state when reporting the abnormality, and further measures can be taken if the abnormality is particularly urgent. This type of system makes it possible to quickly detect road abnormalities and implement appropriate responses.

[0252] Prompt Sentence Examples

[0253] "A large hole has been discovered on a major street in the city. We have provided detailed location information and urgent repairs are required."

[0254] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0255] Step 1:

[0256] The device (dashcam) records high-resolution video while the vehicle is driving. The dashcam is equipped with a high-performance camera that captures full HD resolution video of the road ahead, while a GPS device simultaneously collects location data in real time. The input is the vehicle's current driving video and location information, and the output is the video data and GPS data temporarily stored in the dashcam.

[0257] Step 2:

[0258] The device transmits the collected video data and GPS data to a server in real time via Wi-Fi or mobile networks. The input is the data stored in the dashcam, and the output is the video data and location information transferred to the server.

[0259] Step 3:

[0260] The server receives the video data and GPS information sent from the device. It receives the data packets via the Internet through a receiving API server (using Flask, for example) and temporarily stores them in storage. The input is the sent video data and location information, and the output is the raw data stored in temporary storage on the server.

[0261] Step 4:

[0262] The server stores the received data in a database for redundancy, for example, using MySQL or PostgreSQL. The input is video data and GPS information stored in the server's temporary storage, and the output is data stored in the database ready for analysis.

[0263] Step 5:

[0264] The server analyzes the stored video data. It uses a machine learning model (for example, a model using TensorFlow) to process the video data frame by frame to detect anomalies. The input is the video data read from the database, and the output contains anomaly detection information (for example, the occurrence of a large hole or a traffic accident) as the analysis result.

[0265] Step 6:

[0266] The server records the detected anomaly information in a database. The analyzed anomaly information and its location data are stored in the database. The input is the anomaly detection information, and the output is detailed anomaly information recorded in the database.

[0267] Step 7:

[0268] The server notifies relevant organizations based on the detected anomaly information. A notification message containing the location, type, severity, estimated repair time, etc. of the anomaly is generated and sent via email or SMS to the configured email address or phone number. The input is the anomaly information stored in the database, and the output is the notification message sent to the relevant organizations.

[0269] Step 8:

[0270] The server analyzes traffic volume from the video data. It uses Python's Pandas library to extract the number of vehicles, their speeds, and other data, and records them in a traffic volume database. The input is raw video data, and the output is analyzed traffic volume data.

[0271] Step 9:

[0272] The server determines the priority of road repairs based on traffic volume data. It applies a priority algorithm to assign higher priority to defects on major roads with heavy traffic. The input is traffic volume data, and the output is a repair priority list.

[0273] Step 10:

[0274] The server uses an emotion engine to analyze the user's emotional state. It uses a generative AI model (e.g., GPT-3) to analyze the user's tone of voice and operation patterns. The input is the user's voice data and operation patterns, and the output is the analyzed emotional state.

[0275] Step 11:

[0276] The server responds flexibly to users based on their emotional state. If a user is feeling stressed, it simplifies notifications and provides high-priority responses. The input is the emotion analysis results, and the output is tailored notifications and support responses.

[0277] (Application example 2)

[0278] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0279] Conventional systems for detecting traffic accidents and road damage have had the problem of being difficult to respond quickly and efficiently. Furthermore, while it is important for autonomous vehicles to detect road conditions and traffic abnormalities in real time and take appropriate measures, they often lack the ability to respond flexibly while taking into account the emotional state of the user. Furthermore, it is also necessary to accurately notify the location of abnormalities and determine repair priorities based on traffic volume.

[0280] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0281] In this invention, the server includes means for collecting video recorded by the drive recorder, means for transmitting the collected video data to the server, means for analyzing the transmitted video data and detecting traffic accidents and road damage, means for notifying the relevant authorities of the detected abnormality information, means for analyzing traffic volume and determining the priority of road repairs, means for analyzing the emotional state of the user, and means for adjusting the response according to the emotional state. This makes it possible to not only detect road conditions and traffic abnormalities in real time and notify the relevant authorities, but also to respond flexibly according to the emotional state of the user.

[0282] A "dashcam" is a device that records video footage of the front and rear of a vehicle, and is used to analyze traffic accidents and monitor road conditions.

[0283] A "server" is a computer system that stores and processes data via a network, and is responsible for functions such as analyzing video data and notifying users of abnormal information.

[0284] "Video data" refers to video information recorded by a drive recorder, and is data used to analyze traffic accidents and road damage.

[0285] "Traffic accidents" refer to accidents that occur when vehicles collide with each other or when a vehicle comes into contact with a pedestrian or object.

[0286] "Road damage" refers to cracks, holes, or other damage to the road surface that may impede vehicle movement.

[0287] "Relevant authorities" refers to public institutions responsible for responding to traffic accidents and road damage, such as road management authorities and traffic police.

[0288] "Traffic volume" refers to the number of vehicles passing through a particular road within a certain period of time, and is an indicator of road usage.

[0289] The "emotional state of the user" refers to the mental state of the driver and passengers while driving, and includes emotions such as stress and impatience.

[0290] "Flexible response" refers to the ability to change the way you respond depending on the situation or conditions, and to adjust the content of notifications and responses based on the user's emotional state.

[0291] "Location information" is data relating to geographical location obtained using technology such as GPS, and is transmitted to the server together with the video data.

[0292] "Analysis" refers to the process of examining collected data to find meaning and identify anomalies or specific situations.

[0293] "Abnormal information" is data relating to unusual situations such as traffic accidents and road damage, and is notified to relevant authorities.

[0294] "Priority" indicates the order or importance of how things should be handled, and is a criterion for determining the order of repairs based on traffic volume and road conditions.

[0295] MODE FOR CARRYING OUT THE INVENTION

[0296] System Overview

[0297] The system that realizes this invention collects video footage using a drive recorder installed in an autonomous vehicle, transmits the footage to a cloud server, and analyzes it to detect traffic accidents and road damage. It can also analyze the user's emotional state and provide flexible responses accordingly. Each component of the system and its function are described in detail below.

[0298] Device (Dashcam)

[0299] The dashcam records high-resolution footage of the front and rear of the vehicle. The footage is captured in Full HD resolution, providing crystal-clear video data. The dashcam also has a built-in GPS module, which simultaneously collects location information. This data is sent in real time to a cloud server via Wi-Fi or mobile network.

[0300] Data processing and analysis by the server

[0301] The server receives the video data and location information sent from the device. The received data is stored in a database to ensure security and redundancy. The server uses advanced analytics software to analyze the video data and identify traffic accidents and road damage. This analysis utilizes machine learning models to detect anomalies on a frame-by-frame basis.

[0302] For example, if a major road damage or traffic accident is detected in the video, the specific location and time of the incident are identified. When an anomaly is detected, the server immediately notifies the relevant authorities. The notification includes details such as the anomaly's location, type, severity, and estimated repair time.

[0303] User response using an emotion engine

[0304] The server uses an emotion engine to analyze the user's emotional state. This emotion engine recognizes emotions such as stress and impatience from video, audio, and the user's operation patterns. This makes it possible to provide a flexible response according to the user's emotional state when they report an abnormality.

[0305] For example, if it detects that a user is experiencing high levels of stress, it can adjust the content of notifications and the priority of responses, thereby reducing the psychological burden on the user.

[0306] Traffic analysis and repair prioritization

[0307] The server analyzes traffic volume based on the received video data. The analyzed traffic volume data is used to understand road usage. This data is used to determine the priority of road repairs. For example, if there is an abnormality on a major road with very heavy traffic, repairs will be planned as a priority.

[0308] Specific examples

[0309] As a concrete example, consider a case where a vehicle detects a large pothole while driving on a major city street. At this time, the dashcam records the video and location information and immediately sends it to a cloud server. The server analyzes the video and identifies the damaged area. This information is then sent to the relevant authorities, who then arrange for repairs to be made promptly.

[0310] When a user reports an abnormality, the emotion engine detects that the user is stressed and responds to reassure the user, for example by providing an explanation of the situation and real-time updates on the progress of repairs.

[0311] Prompt Sentence Examples

[0312] "We developed a system that detects major road damage while an autonomous vehicle is traveling down a city's main streets and sends the information to a server in real time. The server then identifies the location of the damage and notifies the relevant authorities. It also recognizes the user's emotions and responds flexibly if the user is feeling stressed. What technologies and methods should we use?"

[0313] In this way, the system can not only quickly and efficiently detect traffic accidents and road damage situations and notify the relevant authorities, but also respond flexibly while taking into account the user's emotions.

[0314] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0315] Step 1:

[0316] The device (drive recorder) records high-resolution video of the area in front of and behind the vehicle. Specifically, it captures full HD video data while the vehicle is moving. The input is real-time video of the area around the vehicle, and the output is a high-resolution video file. The video data is temporarily stored in the recording device.

[0317] Step 2:

[0318] The device acquires location information. The drive recorder has a built-in GPS module that acquires the vehicle's current location. The input is a signal from a GPS satellite, and the output is the vehicle's precise location data. This location information is associated with the video data and temporarily saved.

[0319] Step 3:

[0320] The device sends the collected video data and location information to a server. The data is uploaded to a cloud server in real time via Wi-Fi or a mobile network. The input is a high-resolution video file and location data, and the output is a data stream sent to the server.

[0321] Step 4:

[0322] The server receives the video data and location information sent from the device. The received data is stored in a database to ensure safety and redundancy. The input is the data stream sent from the device, and the output is the video data and location information stored in the database.

[0323] Step 5:

[0324] The server analyzes the video data using a machine learning model. Anomalies are detected for each video frame, and traffic accidents and road damage situations are identified. The input is the video data stored in the database, and the output is the identified anomaly information. Specifically, image features of abnormal areas are extracted and analyzed using an anomaly detection algorithm.

[0325] Step 6:

[0326] The server notifies the relevant authorities of detected anomalies. It sends the location, type, severity, and estimated repair time of the anomaly in real time via email or SMS. The input is the identified anomaly information, and the output is the notification sent to the relevant authorities.

[0327] Step 7:

[0328] The server analyzes traffic volume based on the video data it receives. The traffic volume data is used to determine the priority of road repairs. The input is the video data stored in the database, and the output is the analyzed traffic volume data. Specifically, it analyzes information such as the number of vehicles and passing times to calculate traffic volume.

[0329] Step 8:

[0330] The server uses an emotion engine to analyze the user's emotional state. It recognizes the user's emotions from video, audio, and operation patterns, and identifies feelings such as stress or impatience. The input is the user's video and audio data, and the output is the identified emotional state.

[0331] Step 9:

[0332] The server adjusts its response based on the user's emotional state. For example, if the user is feeling extremely stressed, it adjusts the notification content and priority of the response. The input is the identified emotional state, and the output is the adjusted response. Specifically, it performs actions such as sending a message to reassure the user.

[0333] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0334] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0335] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0336] [Second embodiment]

[0337] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0338] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0339] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0340] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0341] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0342] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0343] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0344] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0345] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0346] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0347] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0348] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0349] The system of the present invention collects video footage recorded by a drive recorder and analyzes it on a server to quickly and efficiently detect traffic accidents and road damage. This system notifies relevant authorities of abnormality information, analyzes traffic volume, and prioritizes road repairs. Specific program processing and its implementation are explained in natural language below.

[0350] Video collection and transmission by dashcam

[0351] The device (dashcam) records high-resolution video while the vehicle is driving. For example, if the vehicle is driving throughout the day, it will record road conditions and traffic conditions in full HD resolution. The device also collects GPS data along with the video, linking the location information to the video. This data is sent to a server in real time via Wi-Fi or a mobile network.

[0352] Data reception and analysis by the server

[0353] The server receives the video data and GPS information sent from the device. The received data is stored in a database to ensure security and redundancy. Analysis software on the server then analyzes the video data to identify traffic accident situations and road damage. The analysis is performed using machine learning models to detect anomalies in each frame of the video. For example, if an anomaly such as a large hole, crack, or traffic accident scene is detected, its location and time are identified.

[0354] Notification and Arrangements

[0355] Based on the analysis results, the server notifies relevant authorities of any abnormalities in real time. The notification includes detailed information such as the location, type, severity, and estimated repair time of the abnormality. For example, if a large hole is detected on a specific road, detailed information about it will be sent to the road management bureau via email or SMS.

[0356] Traffic analysis and repair prioritization

[0357] The server analyzes traffic volume based on the video data to understand road usage. The analyzed traffic volume data is used to determine repair priorities. For example, repairs to defects on major roads with very heavy traffic may be planned as a priority, while roads with less traffic may be postponed.

[0358] User interface support

[0359] Users can check notifications from the system through the interface. For example, after a road maintenance officer receives a notification, he or she can check the detailed situation on the system interface and enter the necessary instructions, allowing a repair team to quickly head to the site and begin repair work.

[0360] Specific examples

[0361] For example, one morning, while a vehicle is driving down a major road, the dashcam detects a large pothole ahead and records the video. The video and GPS data are immediately sent to a server. The server receives the data, and analysis software detects the large pothole. The analysis results identify its location and severity, and notify the road management authority. The road management authority then issues repair orders through the system and arranges for the necessary resources, allowing repairs to be carried out quickly.

[0362] Such systems can improve public safety by quickly detecting and responding to road anomalies.

[0363] The processing flow will be explained below.

[0364] Step 1:

[0365] The device records video while the vehicle is in motion.

[0366] The dashcam captures high-resolution footage of the vehicle while driving and saves each frame.

[0367] The GPS sensor records your current location in real time and tags it in the footage.

[0368] Step 2:

[0369] The device sends the collected video and location information to the server.

[0370] The device uploads data to the server at regular intervals via Wi-Fi or mobile network.

[0371] Confirm the data transmission and record the successful transmission in the log.

[0372] Step 3:

[0373] The server stores the received video data and location information.

[0374] The server receives the data and stores it in a database.

[0375] Video files are assigned timestamps and location information and are centrally managed.

[0376] Step 4:

[0377] The server analyzes the stored video data.

[0378] Computer vision technology and machine learning models are used to detect anomalies in the video (e.g., potholes, cracks, accident scenes, etc.).

[0379] The type and severity of the abnormality is evaluated based on the detection results.

[0380] Step 5:

[0381] The server notifies the relevant authorities of the abnormality information.

[0382] Detailed information about the anomaly (location, type, severity, estimated repair time) is sent to relevant authorities via an automated notification system.

[0383] Notification methods include email, SMS, and in-system messaging.

[0384] Step 6:

[0385] The server analyzes traffic volume from video data and determines repair priorities.

[0386] The number of cars in the video and their frequency of passage are measured to estimate traffic volume.

[0387] Traffic volume data is compiled and repair priorities for abnormal areas are determined.

[0388] Step 7:

[0389] The user checks the notification through the interface and takes action.

[0390] Road management personnel check the notification and view detailed information on the system interface.

[0391] Necessary repair instructions and resource arrangements are made via the interface.

[0392] By using the above processing steps, the system of the present invention can quickly detect road abnormalities and take appropriate measures.

[0393] Example 1

[0394] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0395] Conventional systems for detecting traffic accidents and road damage required manual confirmation and reporting, making it difficult to respond quickly. Furthermore, delays in reporting and responding to abnormal information often threatened public safety. Traffic volume analysis and prioritization of road repairs were also inappropriate, preventing efficient repair work. The present invention aims to solve these problems.

[0396] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0397] In this invention, the server includes a means for storing video footage and GPS data recorded by the drive recorder in a database in real time, a means for detecting road damage and abnormalities using a machine learning model, a means for notifying relevant authorities of detected abnormalities, and a means for analyzing traffic volume and determining the priority of road repairs. This enables rapid and efficient detection of road condition abnormalities and real-time notification to relevant authorities, enabling prompt and appropriate response. Furthermore, determining repair priorities based on traffic volume data enables efficient road repairs.

[0398] A "dashcam" is a device that records video and audio while a vehicle is in motion.

[0399] An "external domain" refers to a remote data storage location, such as a server or cloud storage connected via a network.

[0400] "Video data" refers to video information recorded by a drive recorder.

[0401] "GPS Data" means location information collected using the Global Positioning System.

[0402] The "database" is a system for systematically storing and managing received video data and GPS data.

[0403] A "machine learning model" is an algorithm or system that learns from large amounts of data and performs pattern recognition and prediction.

[0404] "Abnormal information" refers to information that deviates from normal conditions, such as traffic accidents or road damage.

[0405] "Related agencies" refer to organizations and institutions that respond to traffic accidents and road damage, such as road management bureaus and traffic police.

[0406] "Traffic analysis" is the process of evaluating road usage and traffic flow through data analysis.

[0407] "Road repair priorities" are criteria for determining which areas should be given priority when carrying out road repairs or maintenance.

[0408] Embodiments of the present invention will be described in detail below.

[0409] The system of the present invention collects video data and GPS data recorded by a drive recorder and transmits them to a server. The device (drive recorder) records video in high resolution and simultaneously collects GPS data. The device transmits this data to the server in real time via Wi-Fi or a mobile network.

[0410] Specifically, when the vehicle starts moving, the drive recorder automatically starts recording video. The device records high-resolution video and GPS data while the vehicle is moving, and associates it with a timestamp and location information. This data is temporarily stored inside the device and periodically sent to the server. Communication errors are checked by receiving a transmission confirmation message from the server.

[0411] The server receives the video data and GPS data sent from the device and stores it in a database. When storing the data, the data is backed up in multiple storage devices to ensure safety and redundancy. The received data is analyzed using machine learning models by the server's analysis software. During the analysis, the video data is analyzed frame by frame to detect traffic accidents and road damage. For example, if an abnormality such as a large hole, crack, or traffic accident site is detected, the location and time of the abnormality are identified.

[0412] Based on the analysis results, the server notifies the relevant authorities of any detected anomalies. The notification includes information such as details of the anomaly, its location, type, severity, and estimated repair time. This information is sent to the relevant authorities (e.g., road management bureau) via email or SMS. In addition, the analyzed traffic volume data is used to determine the priority of repairs. Anomalies on major roads with very heavy traffic volume are given priority for repair planning, while roads with less traffic volume may be postponed.

[0413] The user (e.g., road maintenance personnel) checks the system's notification through the system's interface. After receiving the notification, the user can check the detailed situation in the interface and enter the necessary instructions, allowing repair teams to quickly head to the site and begin repair work.

[0414] Specific examples

[0415] For example, one morning, a vehicle traveling down a major road might detect a large pothole ahead and record video footage of it on its dashcam. The video and GPS data are immediately sent to a server, where analysis software detects the presence of the pothole. The analysis results identify its location and severity, and the road management authority is notified. The road management authority then issues repair orders through the system and arranges for the necessary resources to be used, ensuring prompt repairs.

[0416] Such a system can improve public safety by quickly detecting road anomalies and responding quickly.

[0417] Prompt Sentence Examples

[0418] "Please explain in detail, step by step, how you analyze footage recorded by a dashcam to detect road anomalies. Also, please explain in detail the notification and response process when an anomaly is detected."

[0419] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0420] Step 1:

[0421] The device collects video and GPS data while the vehicle is moving. It receives video of the vehicle moving and its current location as input, and outputs high-resolution video and GPS data. Specifically, the device automatically starts recording when the vehicle's engine starts, recording video and GPS data in full HD. This data is temporarily stored in the device's internal memory.

[0422] Step 2:

[0423] The device sends the collected video data and GPS data to the server. It takes the collected video data and GPS data as input and outputs the data to the server. Specifically, the device periodically uploads data packets to the server using Wi-Fi or a mobile network. It also checks for communication errors by receiving a confirmation message that transmission is complete.

[0424] Step 3:

[0425] The server receives the video data and GPS data sent from the device. It receives the data packets sent as input and outputs them to be safely stored in a database. Specifically, the server monitors the receiving port in real time and promptly stores the received data in the database. It also copies the data to multiple backup storages to ensure redundancy.

[0426] Step 4:

[0427] The server analyzes the received video data and GPS data. Using the stored video data and GPS data as input, it obtains an output that detects traffic accidents and road damage. Specifically, the analysis software on the server uses a machine learning model to detect anomalies in each video frame. For example, large holes, cracks, and traffic accident scenes on the road are analyzed. As a result of the analysis, the location and severity of the detected anomalies are identified.

[0428] Step 5:

[0429] Based on the analysis results, the server notifies relevant organizations of detected anomalies. It receives the analysis result data as input and obtains an output that sends detailed information to relevant organizations. Specifically, the server creates a notification message that includes the location, type, severity, estimated repair time, etc. of the anomaly, and sends it via email or SMS to relevant organizations such as road management bureaus. In addition, images or videos of the anomaly may be attached to the notification email if necessary.

[0430] Step 6:

[0431] The server analyzes traffic volume based on video data and determines repair priorities. It takes in all video data and road condition data as input and outputs a list of repair priorities. Specifically, the server calculates traffic volume for specific road sections from the video data and ranks the repair priority of each road based on the traffic volume information. The more major roads with heavy traffic have defects, the higher the priority for repair plans.

[0432] Step 7:

[0433] The user checks notifications from the system through an interface and inputs the necessary instructions. The input is a confirmation of the notification message, and the system outputs the instructions and resource arrangement information required for repair response. Specifically, the user (road management officer) logs into the system and checks the details of the notification (location, type of anomaly, severity, etc.). After that, the system registers and sends the arrangement of a repair team and detailed instructions for the repair content. This allows the repair team to respond quickly.

[0434] (Application example 1)

[0435] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0436] In modern road traffic, early detection and rapid response to traffic accidents and road damage are important. However, conventional traffic monitoring systems require dedicated infrastructure, which is costly and difficult to operate efficiently. It is also difficult to prioritize road repairs appropriately in response to fluctuations in traffic volume. Furthermore, with the spread of autonomous vehicles, there is a need for real-time understanding of the road conditions these vehicles face and support for safe operation.

[0437] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0438] In this invention, the server includes means for collecting video recorded by the drive recorder, means for transmitting the collected video data to the server, means for analyzing the transmitted video data and detecting traffic accidents and road damage, means for notifying relevant organizations of detected abnormality information, means for analyzing traffic volume and determining the priority of road repairs, and means for collecting and transmitting video and location information during driving in real time via an application installed in the autonomous vehicle, feeding back abnormality detection results to the server, and proposing safe avoidance routes to the autonomous vehicle. This not only enables early detection of traffic accidents and road damage and rapid response, but also supports the safe operation of the autonomous vehicle.

[0439] A "drive recorder" is a device that records video while a vehicle is driving.

[0440] A "server" is a computing system that receives, stores, analyzes, and processes data over a network.

[0441] An "autonomous vehicle" is a vehicle that can drive autonomously using artificial intelligence and sensors.

[0442] "Video data" refers to digital data of visual information recorded by a drive recorder.

[0443] "GPS Data" means location information obtained from the Global Positioning System (GPS) to indicate a precise location on Earth.

[0444] "Analysis Software" means the programs used to interpret collected data and extract specific information on demand.

[0445] "Abnormal information" is data on events that deviate from normal conditions, specifically traffic accidents and road damage.

[0446] "Related organizations" refers to government agencies and companies in charge of road management and traffic management.

[0447] "Traffic analysis" is the process of analyzing the flow of vehicles and traffic on a particular road to identify patterns and trends.

[0448] "Repair priority" is a criterion for determining the order of repair work for road sections where abnormalities have been detected.

[0449] A "user interface" is an interactive platform that allows users to view displayed information and perform necessary operations.

[0450] A "safe avoidance route" is an alternative route for a detected anomaly, including instructions for an autonomous vehicle to safely reach its destination.

[0451] The system of the present invention is mainly composed of a drive recorder, a GPS module, a server, and analysis software that are installed in a vehicle, including an autonomous driving vehicle.

[0452] Data collection and transmission by the device (autonomous vehicle):

[0453] The autonomous vehicle terminal is equipped with a dashcam and a GPS module. The dashcam records high-resolution video while the vehicle is driving, and the GPS module obtains precise location information. The collected video data and GPS data are transmitted to a server in real time. To achieve this, the terminal is connected to the Internet via Wi-Fi or a mobile network.

[0454] Data reception and analysis by the server:

[0455] The server receives the video data and GPS information sent from the device and stores it in a database for security and redundancy. Next, analysis software on the server uses machine learning models to analyze the video data and detect traffic accidents and road damage. This analysis uses software such as OpenCV and TensorFlow. If an abnormality is detected, the location information and details are notified to the relevant authorities.

[0456] Real-time safety assessment and route suggestions:

[0457] Based on the analysis results, the server proposes safe avoidance routes to autonomous vehicles in real time, enabling vehicles in operation to avoid abnormal areas and operate safely. The route proposal algorithm is implemented using Google Maps API, etc.

[0458] Notification and Arrangements:

[0459] Notification of abnormalities is made via communication methods such as email and SMS. Notifications include the location, type, severity, and estimated repair time of the abnormality, allowing relevant organizations to take prompt action based on this information.

[0460] User Interface:

[0461] Personnel from relevant agencies can check notifications from the system through a dedicated user interface, which allows them to check the detailed situation, input necessary instructions, and take prompt action such as dispatching a repair team.

[0462] Specific examples

[0463] For example, one morning, while an autonomous vehicle is driving down a major road, its dashcam detects a large pothole ahead and immediately sends the video and GPS data to a server. The server receives the data, and analysis software detects the large pothole. The analysis identifies its location and severity, and notifies the road management authority. The road management authority then issues repair orders through the system and arranges for the necessary resources, ensuring prompt repairs.

[0464] Example prompts for generative AI models

[0465] "Design an application that automates the response when a dashcam detects road damage ahead while an autonomous vehicle is driving. Please provide specific code examples and the processing flow for building a system that sends video data and GPS data to a server in real time and notifies the relevant authorities if an abnormality is detected."

[0466] In this way, by describing the mode for carrying out the invention in detail, the components of the system, the operation procedures, and specific examples will become clear.

[0467] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0468] Step 1:

[0469] The device activates the dashcam while the vehicle is moving, and collects video data and GPS data synchronously. The input is the vehicle's driving status and current location, and the output is high-resolution video data and location information. Video frames are stored in a buffer in real time, and GPS data is simultaneously linked.

[0470] Step 2:

[0471] The device transmits the collected video data and GPS data to the server in real time. The input is the video data and GPS data acquired in step 1, and the output is the status of successful transmission to the server. This process involves encoding the data using Wi-Fi or mobile network and sending it to the server via the HTTP protocol.

[0472] Step 3:

[0473] The server stores the received video data and GPS data in a database. The input is the data sent from the device, and the output is the status of successful storage in the database. The server performs data consistency checks to ensure redundancy and performs highly reliable storage processing.

[0474] Step 4:

[0475] The analysis software on the server analyzes the received data and detects anomalies such as traffic accidents and road damage. The input is the saved video data and GPS data, and the output is the identification of anomalies and detailed information about them. Here, a machine learning model is used to determine whether there are any anomalies in each video frame and match them with location information.

[0476] Step 5:

[0477] The server notifies the relevant organizations of the anomaly information. The input is the detailed information about the anomaly obtained in step 4, and the output is the status of sending the notification to the relevant organizations. In this process, detailed information including the location, type, severity, and estimated repair time of the anomaly is notified using email and SMS sending functions.

[0478] Step 6:

[0479] The server analyzes traffic volume and determines the priority of road repairs. The input is collected traffic data and anomaly information, and the output is a list of repair priorities. The server analyzes traffic volume fluctuations and calculates repair priorities based on the frequency of use of anomaly points.

[0480] Step 7:

[0481] The device proposes a safe avoidance route to the autonomous vehicle based on feedback from the server. The input is the abnormality information and the avoidance route proposal obtained from the server, and the output is the vehicle's new driving route. The device uses the navigation system to calculate a safe route in real time and update the driving plan.

[0482] This allows the above-mentioned processing steps to work together to create a system that enables safe and efficient operation of autonomous vehicles.

[0483] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0484] The system of the present invention collects video footage recorded by a drive recorder and analyzes it on a server to quickly and efficiently detect traffic accidents and road damage. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, it is possible to respond flexibly according to the user's condition. Below, specific program processing and its implementation are explained in natural language.

[0485] Video collection and transmission by dashcam

[0486] The device (dashcam) records high-resolution video while the vehicle is driving. For example, if the vehicle is driving in an urban area, it records road conditions and traffic conditions in full HD resolution. The device also collects GPS data along with the video, linking the location information to the video. This data is sent to a server in real time via Wi-Fi or a mobile network.

[0487] Data reception and analysis by the server

[0488] The server receives the video data and GPS information sent from the device. The received data is stored in a database to ensure security and redundancy. The analysis software on the server then analyzes the video data to identify traffic accident situations and road damage. The analysis is performed using a machine learning model to detect anomalies in each frame of the video. For example, if an anomaly such as a large hole or crack in the road or the scene of a traffic accident is detected, the location and time of the anomaly are identified.

[0489] Notification and Arrangements

[0490] Based on the analysis results, the server notifies relevant authorities of any abnormalities in real time. The notification includes detailed information such as the location, type, severity, and estimated repair time of the abnormality. For example, if a large hole is detected on a specific road, detailed information about it will be sent to the road management bureau via email or SMS.

[0491] Traffic analysis and repair prioritization

[0492] The server analyzes traffic volume based on the video data to understand road usage. The analyzed traffic volume data is used to determine repair priorities. For example, repairs to defects on major roads with very heavy traffic may be planned as a priority, while roads with less traffic may be postponed.

[0493] User response using an emotion engine

[0494] The server uses an emotion engine to analyze the user's emotional state. The emotion engine recognizes the user's emotions from video, audio, and the user's operation patterns, and can detect, for example, stress or impatience. This allows for flexible responses depending on the user's emotional state when reporting an abnormality. For example, if the user is feeling extremely stressed, the content of the notification and the priority of the response can be adjusted.

[0495] Specific examples

[0496] For example, one morning, while a vehicle is driving down a major street in the city, the dashcam detects a large pothole ahead and records the video. The video and GPS data are immediately sent to the server. The server receives the data, and analysis software detects the large pothole. The analysis results identify the location and severity of the pothole, and notify the road management bureau. The road management bureau then issues repair orders through the system and arranges for the necessary resources, ensuring that repairs are carried out quickly. At the same time, an emotion engine is used to analyze the user's emotional state when they report the abnormality, and further measures can be taken if the abnormality is particularly urgent. This type of system makes it possible to quickly detect road abnormalities and implement appropriate responses.

[0497] The processing flow will be explained below.

[0498] Step 1:

[0499] The device records video while the vehicle is in motion.

[0500] The dashcam captures high-resolution footage of the vehicle while driving and saves each frame.

[0501] The GPS sensor records your current location in real time and tags it in the footage.

[0502] Step 2:

[0503] The device sends the collected video and location information to the server.

[0504] The device uploads data to the server at regular intervals via Wi-Fi or mobile network.

[0505] Confirm the data transmission and record the successful transmission in the log.

[0506] Step 3:

[0507] The server stores the received video data and location information.

[0508] The server receives the data and stores it in a database.

[0509] Video files are assigned timestamps and location information and are centrally managed.

[0510] Step 4:

[0511] The server analyzes the stored video data.

[0512] Computer vision technology and machine learning models are used to detect anomalies in the video (e.g., potholes, cracks, accident scenes, etc.).

[0513] The type and severity of the abnormality is evaluated based on the detection results.

[0514] Step 5:

[0515] The server analyzes the user's emotional state.

[0516] It uses an emotion engine that recognizes user emotions from video and audio data and operation patterns.

[0517] Analyze and evaluate the user's emotional state (e.g., stress, impatience) in real time.

[0518] Step 6:

[0519] The server notifies the relevant authorities of the abnormality information.

[0520] Detailed information about the anomaly (location, type, severity, estimated repair time) is sent to relevant authorities via an automated notification system.

[0521] Notification methods include email, SMS, and in-system messaging.

[0522] Step 7:

[0523] The server reflects the emotion data in traffic volume analysis and repair priority determination.

[0524] The system takes into account the user's emotional state and adjusts the priority of repairs if the urgency is high.

[0525] The final decision is made by combining the priorities calculated based on traffic volume with the analysis results of the emotion engine.

[0526] Step 8:

[0527] The user checks the notification through the interface and takes action.

[0528] Road management personnel check the notification and view detailed information on the system interface.

[0529] Necessary repair instructions and resource arrangements are made via the interface.

[0530] By using the above processing steps, the system of the present invention can quickly detect road abnormalities and take appropriate measures. Furthermore, by taking the user's emotional state into consideration, more flexible and appropriate responses are possible.

[0531] Example 2

[0532] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0533] Traffic accidents and road damage must be detected quickly and accurately, but conventional methods often delay appropriate analysis and notification. Furthermore, responses that ignore the user's emotional state can increase stress and frustration. The purpose of this invention is to solve these problems by efficiently and quickly detecting anomalies, notifying relevant authorities, and enabling flexible responses based on the user's emotional state.

[0534] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing the transmitted video data and detecting the situation of a traffic accident or road damage, a means for notifying the relevant authorities of the detected abnormality information, a means for analyzing the emotional state of the user, and a means for providing a flexible response to the user based on the analysis result. This enables quick detection and notification of a traffic accident or road damage, and also enables a flexible response according to the emotional state of the user.

[0535] A "dashcam" is a device that records images of the front and rear of a vehicle while it is in motion.

[0536] "Video data" refers to video information collected by a drive recorder.

[0537] A "server" is a computer system that receives, analyzes, and stores data over a network and works in conjunction with other systems and devices.

[0538] "Transmitting means" refers to the mechanism by which collected data is sent to other systems via wireless or wired communication.

[0539] "Analysis means" refers to devices and technologies, including software and algorithms, that process collected data and detect abnormalities such as traffic accidents and road damage.

[0540] "Traffic accident" means an accident caused by a collision between vehicles or contact between a vehicle and a person or object.

[0541] "Road damage" refers to conditions that impede vehicle movement, such as cracks or holes in the road surface.

[0542] "Notification means" refers to the mechanism for communicating analysis results to relevant agencies by means of email, SMS, etc.

[0543] "Relevant organizations" refer to organizations that should respond to detected abnormalities, such as road management bureaus and police.

[0544] "Traffic volume analysis means" refers to technology that analyzes the number, speed, flow, etc. of vehicles based on video data.

[0545] "Means for determining repair priority" refers to a mechanism for determining the order and priority of road repairs based on analysis results and traffic volume data.

[0546] "Emotional state analysis means" refers to technology that analyzes the user's tone of voice, operation patterns, etc. to determine the user's emotions.

[0547] "Flexible response means" refers to techniques and processes for adjusting notification content and changing response priorities according to the user's emotional state.

[0548] The system of this invention collects video footage recorded by a dashcam and analyzes it on a server to quickly and efficiently detect traffic accidents and road damage.The system also incorporates an emotion engine that recognizes the user's emotional state and enables flexible responses.

[0549] Video collection and transmission by dashcam

[0550] The device (dashcam) records high-resolution video while the vehicle is driving. For example, when the vehicle is driving in an urban area, the dashcam records video of the road ahead in full HD resolution. This can be done using a commercially available high-performance camera. A GPS device also operates to obtain location information, collecting location data in real time. This data (video data and GPS data) is processed in real time within the device and sent to a server using Wi-Fi or a mobile network.

[0551] Data reception and storage by the server

[0552] The server receives the video data and GPS information sent from the device. The server is equipped with a high-performance data receiving module (e.g., an API server using Python and Flask) and receives data packets arriving via the Internet. It then temporarily stores this data in storage and stores it in a database (e.g., MySQL or PostgreSQL) in a highly redundant format.

[0553] Data analysis by server

[0554] The server analyzes the stored video data and location information. Machine learning models (e.g., deep learning models using TensorFlow or PyTorch) are used for video analysis, with anomaly detection algorithms being particularly important. The video data is processed frame by frame to detect anomalies (e.g., large holes or debris from traffic accidents) within each frame. Any anomalies detected by the analysis software are recorded in a database along with their location information.

[0555] Server notification of abnormal information

[0556] The server notifies the relevant authorities of the anomaly information based on the analysis results. This notification process is automated, and a message containing information such as the location, type, severity, and estimated repair time of the anomaly is generated based on the analysis results. The message is sent to the relevant authorities (e.g., road management authorities or police) via email or SMS. This can be done, for example, using the SMTP protocol or an SMS gateway.

[0557] Traffic analysis and repair priority determination by server

[0558] The server analyzes traffic volume from the video data. During the analysis process, the number of vehicles, speed, flow, etc. are analyzed and recorded in a traffic volume database. Based on these results, an algorithm is run to determine the priority of repair work. For example, the data analysis is performed using the Python Pandas library, and a high priority is assigned to abnormalities on major roads based on a priority algorithm.

[0559] Server-based user emotion analysis and response

[0560] The server uses an emotion engine to analyze the user's emotional state. The emotion engine uses a generative AI model (for example, a large-scale language model such as GPT-3) to process data such as the user's voice and operation patterns to recognize the user's emotional state. For example, it can detect stress or impatience from the user's typing speed and tone of voice when operating the device. Based on the results of this emotion analysis, the server can respond flexibly to the user. For example, for a user who feels stressed, it can simplify the notification content and provide a high-priority response.

[0561] Specific examples

[0562] For example, one morning, while a vehicle is driving down a major street in the city, the dashcam detects a large pothole ahead and records the video. The video and GPS data are immediately sent to the server. The server receives the data, and analysis software detects the large pothole. The analysis results identify the location and severity of the pothole, and notify the road management bureau. The road management bureau then issues repair orders through the system and arranges for the necessary resources, ensuring prompt repairs. At the same time, an emotion engine is used to analyze the user's emotional state when reporting the abnormality, and further measures can be taken if the abnormality is particularly urgent. This type of system makes it possible to quickly detect road abnormalities and implement appropriate responses.

[0563] Prompt Sentence Examples

[0564] "A large hole has been discovered on a major street in the city. We have provided detailed location information and urgent repairs are required."

[0565] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0566] Step 1:

[0567] The device (dashcam) records high-resolution video while the vehicle is driving. The dashcam is equipped with a high-performance camera that captures full HD resolution video of the road ahead, while a GPS device simultaneously collects location data in real time. The input is the vehicle's current driving video and location information, and the output is the video data and GPS data temporarily stored in the dashcam.

[0568] Step 2:

[0569] The device transmits the collected video data and GPS data to a server in real time via Wi-Fi or mobile networks. The input is the data stored in the dashcam, and the output is the video data and location information transferred to the server.

[0570] Step 3:

[0571] The server receives the video data and GPS information sent from the device. It receives the data packets via the Internet through a receiving API server (using Flask, for example) and temporarily stores them in storage. The input is the sent video data and location information, and the output is the raw data stored in temporary storage on the server.

[0572] Step 4:

[0573] The server stores the received data in a database for redundancy, for example, using MySQL or PostgreSQL. The input is video data and GPS information stored in the server's temporary storage, and the output is data stored in the database ready for analysis.

[0574] Step 5:

[0575] The server analyzes the stored video data. It uses a machine learning model (for example, a model using TensorFlow) to process the video data frame by frame to detect anomalies. The input is the video data read from the database, and the output contains anomaly detection information (for example, the occurrence of a large hole or a traffic accident) as the analysis result.

[0576] Step 6:

[0577] The server records the detected anomaly information in a database. The analyzed anomaly information and its location data are stored in the database. The input is the anomaly detection information, and the output is detailed anomaly information recorded in the database.

[0578] Step 7:

[0579] The server notifies relevant organizations based on the detected anomaly information. A notification message containing the location, type, severity, estimated repair time, etc. of the anomaly is generated and sent via email or SMS to the configured email address or phone number. The input is the anomaly information stored in the database, and the output is the notification message sent to the relevant organizations.

[0580] Step 8:

[0581] The server analyzes traffic volume from the video data. It uses Python's Pandas library to extract the number of vehicles, their speeds, and other data, and records them in a traffic volume database. The input is raw video data, and the output is analyzed traffic volume data.

[0582] Step 9:

[0583] The server determines the priority of road repairs based on traffic volume data. It applies a priority algorithm to assign higher priority to defects on major roads with heavy traffic. The input is traffic volume data, and the output is a repair priority list.

[0584] Step 10:

[0585] The server uses an emotion engine to analyze the user's emotional state. It uses a generative AI model (e.g., GPT-3) to analyze the user's tone of voice and operation patterns. The input is the user's voice data and operation patterns, and the output is the analyzed emotional state.

[0586] Step 11:

[0587] The server responds flexibly to users based on their emotional state. If a user is feeling stressed, it simplifies notifications and provides high-priority responses. The input is the emotion analysis results, and the output is tailored notifications and support responses.

[0588] (Application example 2)

[0589] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0590] Conventional systems for detecting traffic accidents and road damage have had the problem of being difficult to respond quickly and efficiently. Furthermore, while it is important for autonomous vehicles to detect road conditions and traffic abnormalities in real time and take appropriate measures, they often lack the ability to respond flexibly while taking into account the emotional state of the user. Furthermore, it is also necessary to accurately notify the location of abnormalities and determine repair priorities based on traffic volume.

[0591] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0592] In this invention, the server includes means for collecting video recorded by the drive recorder, means for transmitting the collected video data to the server, means for analyzing the transmitted video data and detecting traffic accidents and road damage, means for notifying the relevant authorities of the detected abnormality information, means for analyzing traffic volume and determining the priority of road repairs, means for analyzing the emotional state of the user, and means for adjusting the response according to the emotional state. This makes it possible to not only detect road conditions and traffic abnormalities in real time and notify the relevant authorities, but also to respond flexibly according to the emotional state of the user.

[0593] A "dashcam" is a device that records video footage of the front and rear of a vehicle, and is used to analyze traffic accidents and monitor road conditions.

[0594] A "server" is a computer system that stores and processes data via a network, and is responsible for functions such as analyzing video data and notifying users of abnormal information.

[0595] "Video data" refers to video information recorded by a drive recorder, and is data used to analyze traffic accidents and road damage.

[0596] "Traffic accidents" refer to accidents that occur when vehicles collide with each other or when a vehicle comes into contact with a pedestrian or object.

[0597] "Road damage" refers to cracks, holes, or other damage to the road surface that may impede vehicle movement.

[0598] "Relevant authorities" refers to public institutions responsible for responding to traffic accidents and road damage, such as road management authorities and traffic police.

[0599] "Traffic volume" refers to the number of vehicles passing through a particular road within a certain period of time, and is an indicator of road usage.

[0600] The "emotional state of the user" refers to the mental state of the driver and passengers while driving, and includes emotions such as stress and impatience.

[0601] "Flexible response" refers to the ability to change the way you respond depending on the situation or conditions, and to adjust the content of notifications and responses based on the user's emotional state.

[0602] "Location information" is data relating to geographical location obtained using technology such as GPS, and is transmitted to the server together with the video data.

[0603] "Analysis" refers to the process of examining collected data to find meaning and identify anomalies or specific situations.

[0604] "Abnormal information" is data relating to unusual situations such as traffic accidents and road damage, and is notified to relevant authorities.

[0605] "Priority" indicates the order or importance of how things should be handled, and is a criterion for determining the order of repairs based on traffic volume and road conditions.

[0606] MODE FOR CARRYING OUT THE INVENTION

[0607] System Overview

[0608] The system that realizes this invention collects video footage using a drive recorder installed in an autonomous vehicle, transmits the footage to a cloud server, and analyzes it to detect traffic accidents and road damage. It can also analyze the user's emotional state and provide flexible responses accordingly. Each component of the system and its function are described in detail below.

[0609] Device (Dashcam)

[0610] The dashcam records high-resolution footage of the front and rear of the vehicle. The footage is captured in Full HD resolution, providing crystal-clear video data. The dashcam also has a built-in GPS module, which simultaneously collects location information. This data is sent in real time to a cloud server via Wi-Fi or mobile network.

[0611] Data processing and analysis by the server

[0612] The server receives the video data and location information sent from the device. The received data is stored in a database to ensure security and redundancy. The server uses advanced analytics software to analyze the video data and identify traffic accidents and road damage. This analysis utilizes machine learning models to detect anomalies on a frame-by-frame basis.

[0613] For example, if a major road damage or traffic accident is detected in the video, the specific location and time of the incident are identified. When an anomaly is detected, the server immediately notifies the relevant authorities. The notification includes details such as the anomaly's location, type, severity, and estimated repair time.

[0614] User response using an emotion engine

[0615] The server uses an emotion engine to analyze the user's emotional state. This emotion engine recognizes emotions such as stress and impatience from video, audio, and the user's operation patterns. This makes it possible to provide a flexible response according to the user's emotional state when they report an abnormality.

[0616] For example, if it detects that a user is experiencing high levels of stress, it can adjust the content of notifications and the priority of responses, thereby reducing the psychological burden on the user.

[0617] Traffic analysis and repair prioritization

[0618] The server analyzes traffic volume based on the received video data. The analyzed traffic volume data is used to understand road usage. This data is used to determine the priority of road repairs. For example, if there is an abnormality on a major road with very heavy traffic, repairs will be planned as a priority.

[0619] Specific examples

[0620] As a concrete example, consider a case where a vehicle detects a large pothole while driving on a major city street. At this time, the dashcam records the video and location information and immediately sends it to a cloud server. The server analyzes the video and identifies the damaged area. This information is then sent to the relevant authorities, who then arrange for repairs to be made promptly.

[0621] When a user reports an abnormality, the emotion engine detects that the user is stressed and responds to reassure the user, for example by providing an explanation of the situation and real-time updates on the progress of repairs.

[0622] Prompt Sentence Examples

[0623] "We developed a system that detects major road damage while an autonomous vehicle is traveling down a city's main streets and sends the information to a server in real time. The server then identifies the location of the damage and notifies the relevant authorities. It also recognizes the user's emotions and responds flexibly if the user is feeling stressed. What technologies and methods should we use?"

[0624] In this way, the system can not only quickly and efficiently detect traffic accidents and road damage situations and notify the relevant authorities, but also respond flexibly while taking into account the user's emotions.

[0625] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0626] Step 1:

[0627] The device (drive recorder) records high-resolution video of the area in front of and behind the vehicle. Specifically, it captures full HD video data while the vehicle is moving. The input is real-time video of the area around the vehicle, and the output is a high-resolution video file. The video data is temporarily stored in the recording device.

[0628] Step 2:

[0629] The device acquires location information. The drive recorder has a built-in GPS module that acquires the vehicle's current location. The input is a signal from a GPS satellite, and the output is the vehicle's precise location data. This location information is associated with the video data and temporarily saved.

[0630] Step 3:

[0631] The device sends the collected video data and location information to a server. The data is uploaded to a cloud server in real time via Wi-Fi or a mobile network. The input is a high-resolution video file and location data, and the output is a data stream sent to the server.

[0632] Step 4:

[0633] The server receives the video data and location information sent from the device. The received data is stored in a database to ensure safety and redundancy. The input is the data stream sent from the device, and the output is the video data and location information stored in the database.

[0634] Step 5:

[0635] The server analyzes the video data using a machine learning model. Anomalies are detected for each video frame, and traffic accidents and road damage situations are identified. The input is the video data stored in the database, and the output is the identified anomaly information. Specifically, image features of abnormal areas are extracted and analyzed using an anomaly detection algorithm.

[0636] Step 6:

[0637] The server notifies the relevant authorities of detected anomalies. It sends the location, type, severity, and estimated repair time of the anomaly in real time via email or SMS. The input is the identified anomaly information, and the output is the notification sent to the relevant authorities.

[0638] Step 7:

[0639] The server analyzes traffic volume based on the video data it receives. The traffic volume data is used to determine the priority of road repairs. The input is the video data stored in the database, and the output is the analyzed traffic volume data. Specifically, it analyzes information such as the number of vehicles and passing times to calculate traffic volume.

[0640] Step 8:

[0641] The server uses an emotion engine to analyze the user's emotional state. It recognizes the user's emotions from video, audio, and operation patterns, and identifies feelings such as stress or impatience. The input is the user's video and audio data, and the output is the identified emotional state.

[0642] Step 9:

[0643] The server adjusts its response based on the user's emotional state. For example, if the user is feeling extremely stressed, it adjusts the notification content and priority of the response. The input is the identified emotional state, and the output is the adjusted response. Specifically, it performs actions such as sending a message to reassure the user.

[0644] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0645] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0646] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0647] [Third embodiment]

[0648] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0649] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0650] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0651] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0652] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0653] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0654] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0655] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0656] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0657] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0658] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0659] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0660] The system of the present invention collects video footage recorded by a drive recorder and analyzes it on a server to quickly and efficiently detect traffic accidents and road damage. This system notifies relevant authorities of abnormality information, analyzes traffic volume, and prioritizes road repairs. Specific program processing and its implementation are explained in natural language below.

[0661] Video collection and transmission by dashcam

[0662] The device (dashcam) records high-resolution video while the vehicle is driving. For example, if the vehicle is driving throughout the day, it will record road conditions and traffic conditions in full HD resolution. The device also collects GPS data along with the video, linking the location information to the video. This data is sent to a server in real time via Wi-Fi or a mobile network.

[0663] Data reception and analysis by the server

[0664] The server receives the video data and GPS information sent from the device. The received data is stored in a database to ensure security and redundancy. Analysis software on the server then analyzes the video data to identify traffic accident situations and road damage. The analysis is performed using machine learning models to detect anomalies in each frame of the video. For example, if an anomaly such as a large hole, crack, or traffic accident scene is detected, its location and time are identified.

[0665] Notification and Arrangements

[0666] Based on the analysis results, the server notifies relevant authorities of any abnormalities in real time. The notification includes detailed information such as the location, type, severity, and estimated repair time of the abnormality. For example, if a large hole is detected on a specific road, detailed information about it will be sent to the road management bureau via email or SMS.

[0667] Traffic analysis and repair prioritization

[0668] The server analyzes traffic volume based on the video data to understand road usage. The analyzed traffic volume data is used to determine repair priorities. For example, repairs to defects on major roads with very heavy traffic may be planned as a priority, while roads with less traffic may be postponed.

[0669] User interface support

[0670] Users can check notifications from the system through the interface. For example, after a road maintenance officer receives a notification, he or she can check the detailed situation on the system interface and enter the necessary instructions, allowing a repair team to quickly head to the site and begin repair work.

[0671] Specific examples

[0672] For example, one morning, while a vehicle is driving down a major road, the dashcam detects a large pothole ahead and records the video. The video and GPS data are immediately sent to a server. The server receives the data, and analysis software detects the large pothole. The analysis results identify its location and severity, and notify the road management authority. The road management authority then issues repair orders through the system and arranges for the necessary resources, allowing repairs to be carried out quickly.

[0673] Such systems can improve public safety by quickly detecting and responding to road anomalies.

[0674] The processing flow will be explained below.

[0675] Step 1:

[0676] The device records video while the vehicle is in motion.

[0677] The dashcam captures high-resolution footage of the vehicle while driving and saves each frame.

[0678] The GPS sensor records your current location in real time and tags it in the footage.

[0679] Step 2:

[0680] The device sends the collected video and location information to the server.

[0681] The device uploads data to the server at regular intervals via Wi-Fi or mobile network.

[0682] Confirm the data transmission and record the successful transmission in the log.

[0683] Step 3:

[0684] The server stores the received video data and location information.

[0685] The server receives the data and stores it in a database.

[0686] Video files are assigned timestamps and location information and are centrally managed.

[0687] Step 4:

[0688] The server analyzes the stored video data.

[0689] Computer vision technology and machine learning models are used to detect anomalies in the video (e.g., potholes, cracks, accident scenes, etc.).

[0690] The type and severity of the abnormality is evaluated based on the detection results.

[0691] Step 5:

[0692] The server notifies the relevant authorities of the abnormality information.

[0693] Detailed information about the anomaly (location, type, severity, estimated repair time) is sent to relevant authorities via an automated notification system.

[0694] Notification methods include email, SMS, and in-system messaging.

[0695] Step 6:

[0696] The server analyzes traffic volume from video data and determines repair priorities.

[0697] The number of cars in the video and their frequency of passage are measured to estimate traffic volume.

[0698] Traffic volume data is compiled and repair priorities for abnormal areas are determined.

[0699] Step 7:

[0700] The user checks the notification through the interface and takes action.

[0701] Road management personnel check the notification and view detailed information on the system interface.

[0702] Necessary repair instructions and resource arrangements are made via the interface.

[0703] By using the above processing steps, the system of the present invention can quickly detect road abnormalities and take appropriate measures.

[0704] Example 1

[0705] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0706] Conventional systems for detecting traffic accidents and road damage required manual confirmation and reporting, making it difficult to respond quickly. Furthermore, delays in reporting and responding to abnormal information often threatened public safety. Traffic volume analysis and prioritization of road repairs were also inappropriate, preventing efficient repair work. The present invention aims to solve these problems.

[0707] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0708] In this invention, the server includes a means for storing video footage and GPS data recorded by the drive recorder in a database in real time, a means for detecting road damage and abnormalities using a machine learning model, a means for notifying relevant authorities of detected abnormalities, and a means for analyzing traffic volume and determining the priority of road repairs. This enables rapid and efficient detection of road condition abnormalities and real-time notification to relevant authorities, enabling prompt and appropriate response. Furthermore, determining repair priorities based on traffic volume data enables efficient road repairs.

[0709] A "dashcam" is a device that records video and audio while a vehicle is in motion.

[0710] An "external domain" refers to a remote data storage location, such as a server or cloud storage connected via a network.

[0711] "Video data" refers to video information recorded by a drive recorder.

[0712] "GPS Data" means location information collected using the Global Positioning System.

[0713] The "database" is a system for systematically storing and managing received video data and GPS data.

[0714] A "machine learning model" is an algorithm or system that learns from large amounts of data and performs pattern recognition and prediction.

[0715] "Abnormal information" refers to information that deviates from normal conditions, such as traffic accidents or road damage.

[0716] "Related agencies" refer to organizations and institutions that respond to traffic accidents and road damage, such as road management bureaus and traffic police.

[0717] "Traffic analysis" is the process of evaluating road usage and traffic flow through data analysis.

[0718] "Road repair priorities" are criteria for determining which areas should be given priority when carrying out road repairs or maintenance.

[0719] Embodiments of the present invention will be described in detail below.

[0720] The system of the present invention collects video data and GPS data recorded by a drive recorder and transmits them to a server. The device (drive recorder) records video in high resolution and simultaneously collects GPS data. The device transmits this data to the server in real time via Wi-Fi or a mobile network.

[0721] Specifically, when the vehicle starts moving, the drive recorder automatically starts recording video. The device records high-resolution video and GPS data while the vehicle is moving, and associates it with a timestamp and location information. This data is temporarily stored inside the device and periodically sent to the server. Communication errors are checked by receiving a transmission confirmation message from the server.

[0722] The server receives the video data and GPS data sent from the device and stores it in a database. When storing the data, the data is backed up in multiple storage devices to ensure safety and redundancy. The received data is analyzed using machine learning models by the server's analysis software. During the analysis, the video data is analyzed frame by frame to detect traffic accidents and road damage. For example, if an abnormality such as a large hole, crack, or traffic accident site is detected, the location and time of the abnormality are identified.

[0723] Based on the analysis results, the server notifies the relevant authorities of any detected anomalies. The notification includes information such as details of the anomaly, its location, type, severity, and estimated repair time. This information is sent to the relevant authorities (e.g., road management bureau) via email or SMS. In addition, the analyzed traffic volume data is used to determine the priority of repairs. Anomalies on major roads with very heavy traffic volume are given priority for repair planning, while roads with less traffic volume may be postponed.

[0724] The user (e.g., road maintenance personnel) checks the system's notification through the system's interface. After receiving the notification, the user can check the detailed situation in the interface and enter the necessary instructions, allowing repair teams to quickly head to the site and begin repair work.

[0725] Specific examples

[0726] For example, one morning, a vehicle traveling down a major road might detect a large pothole ahead and record video footage of it on its dashcam. The video and GPS data are immediately sent to a server, where analysis software detects the presence of the pothole. The analysis results identify its location and severity, and the road management authority is notified. The road management authority then issues repair orders through the system and arranges for the necessary resources to be used, ensuring prompt repairs.

[0727] Such a system can improve public safety by quickly detecting road anomalies and responding quickly.

[0728] Prompt Sentence Examples

[0729] "Please explain in detail, step by step, how you analyze footage recorded by a dashcam to detect road anomalies. Also, please explain in detail the notification and response process when an anomaly is detected."

[0730] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0731] Step 1:

[0732] The device collects video and GPS data while the vehicle is moving. It receives video of the vehicle moving and its current location as input, and outputs high-resolution video and GPS data. Specifically, the device automatically starts recording when the vehicle's engine starts, recording video and GPS data in full HD. This data is temporarily stored in the device's internal memory.

[0733] Step 2:

[0734] The device sends the collected video data and GPS data to the server. It takes the collected video data and GPS data as input and outputs the data to the server. Specifically, the device periodically uploads data packets to the server using Wi-Fi or a mobile network. It also checks for communication errors by receiving a confirmation message that transmission is complete.

[0735] Step 3:

[0736] The server receives the video data and GPS data sent from the device. It receives the data packets sent as input and outputs them to be safely stored in a database. Specifically, the server monitors the receiving port in real time and promptly stores the received data in the database. It also copies the data to multiple backup storages to ensure redundancy.

[0737] Step 4:

[0738] The server analyzes the received video data and GPS data. Using the stored video data and GPS data as input, it obtains an output that detects traffic accidents and road damage. Specifically, the analysis software on the server uses a machine learning model to detect anomalies in each video frame. For example, large holes, cracks, and traffic accident scenes on the road are analyzed. As a result of the analysis, the location and severity of the detected anomalies are identified.

[0739] Step 5:

[0740] Based on the analysis results, the server notifies relevant organizations of detected anomalies. It receives the analysis result data as input and obtains an output that sends detailed information to relevant organizations. Specifically, the server creates a notification message that includes the location, type, severity, estimated repair time, etc. of the anomaly, and sends it via email or SMS to relevant organizations such as road management bureaus. In addition, images or videos of the anomaly may be attached to the notification email if necessary.

[0741] Step 6:

[0742] The server analyzes traffic volume based on video data and determines repair priorities. It takes in all video data and road condition data as input and outputs a list of repair priorities. Specifically, the server calculates traffic volume for specific road sections from the video data and ranks the repair priority of each road based on the traffic volume information. The more major roads with heavy traffic have defects, the higher the priority for repair plans.

[0743] Step 7:

[0744] The user checks notifications from the system through an interface and inputs the necessary instructions. The input is a confirmation of the notification message, and the system outputs the instructions and resource arrangement information required for repair response. Specifically, the user (road management officer) logs into the system and checks the details of the notification (location, type of anomaly, severity, etc.). After that, the system registers and sends the arrangement of a repair team and detailed instructions for the repair content. This allows the repair team to respond quickly.

[0745] (Application example 1)

[0746] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0747] In modern road traffic, early detection and rapid response to traffic accidents and road damage are important. However, conventional traffic monitoring systems require dedicated infrastructure, which is costly and difficult to operate efficiently. It is also difficult to prioritize road repairs appropriately in response to fluctuations in traffic volume. Furthermore, with the spread of autonomous vehicles, there is a need for real-time understanding of the road conditions these vehicles face and support for safe operation.

[0748] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0749] In this invention, the server includes means for collecting video recorded by the drive recorder, means for transmitting the collected video data to the server, means for analyzing the transmitted video data and detecting traffic accidents and road damage, means for notifying relevant organizations of detected abnormality information, means for analyzing traffic volume and determining the priority of road repairs, and means for collecting and transmitting video and location information during driving in real time via an application installed in the autonomous vehicle, feeding back abnormality detection results to the server, and proposing safe avoidance routes to the autonomous vehicle. This not only enables early detection of traffic accidents and road damage and rapid response, but also supports the safe operation of the autonomous vehicle.

[0750] A "drive recorder" is a device that records video while a vehicle is driving.

[0751] A "server" is a computing system that receives, stores, analyzes, and processes data over a network.

[0752] An "autonomous vehicle" is a vehicle that can drive autonomously using artificial intelligence and sensors.

[0753] "Video data" refers to digital data of visual information recorded by a drive recorder.

[0754] "GPS Data" means location information obtained from the Global Positioning System (GPS) to indicate a precise location on Earth.

[0755] "Analysis Software" means the programs used to interpret collected data and extract specific information on demand.

[0756] "Abnormal information" is data on events that deviate from normal conditions, specifically traffic accidents and road damage.

[0757] "Related organizations" refers to government agencies and companies in charge of road management and traffic management.

[0758] "Traffic analysis" is the process of analyzing the flow of vehicles and traffic on a particular road to identify patterns and trends.

[0759] "Repair priority" is a criterion for determining the order of repair work for road sections where abnormalities have been detected.

[0760] A "user interface" is an interactive platform that allows users to view displayed information and perform necessary operations.

[0761] A "safe avoidance route" is an alternative route for a detected anomaly, including instructions for an autonomous vehicle to safely reach its destination.

[0762] The system of the present invention is mainly composed of a drive recorder, a GPS module, a server, and analysis software that are installed in a vehicle, including an autonomous driving vehicle.

[0763] Data collection and transmission by the device (autonomous vehicle):

[0764] The autonomous vehicle terminal is equipped with a dashcam and a GPS module. The dashcam records high-resolution video while the vehicle is driving, and the GPS module obtains precise location information. The collected video data and GPS data are transmitted to a server in real time. To achieve this, the terminal is connected to the Internet via Wi-Fi or a mobile network.

[0765] Data reception and analysis by the server:

[0766] The server receives the video data and GPS information sent from the device and stores it in a database for security and redundancy. Next, analysis software on the server uses machine learning models to analyze the video data and detect traffic accidents and road damage. This analysis uses software such as OpenCV and TensorFlow. If an abnormality is detected, the location information and details are notified to the relevant authorities.

[0767] Real-time safety assessment and route suggestions:

[0768] Based on the analysis results, the server proposes safe avoidance routes to autonomous vehicles in real time, enabling vehicles in operation to avoid abnormal areas and operate safely. The route proposal algorithm is implemented using Google Maps API, etc.

[0769] Notification and Arrangements:

[0770] Notification of abnormalities is made via communication methods such as email and SMS. Notifications include the location, type, severity, and estimated repair time of the abnormality, allowing relevant organizations to take prompt action based on this information.

[0771] User Interface:

[0772] Personnel from relevant agencies can check notifications from the system through a dedicated user interface, which allows them to check the detailed situation, input necessary instructions, and take prompt action such as dispatching a repair team.

[0773] Specific examples

[0774] For example, one morning, while an autonomous vehicle is driving down a major road, its dashcam detects a large pothole ahead and immediately sends the video and GPS data to a server. The server receives the data, and analysis software detects the large pothole. The analysis identifies its location and severity, and notifies the road management authority. The road management authority then issues repair orders through the system and arranges for the necessary resources, ensuring prompt repairs.

[0775] Example prompts for generative AI models

[0776] "Design an application that automates the response when a dashcam detects road damage ahead while an autonomous vehicle is driving. Please provide specific code examples and the processing flow for building a system that sends video data and GPS data to a server in real time and notifies the relevant authorities if an abnormality is detected."

[0777] In this way, by describing the mode for carrying out the invention in detail, the components of the system, the operation procedures, and specific examples will become clear.

[0778] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0779] Step 1:

[0780] The device activates the dashcam while the vehicle is moving, and collects video data and GPS data synchronously. The input is the vehicle's driving status and current location, and the output is high-resolution video data and location information. Video frames are stored in a buffer in real time, and GPS data is simultaneously linked.

[0781] Step 2:

[0782] The device transmits the collected video data and GPS data to the server in real time. The input is the video data and GPS data acquired in step 1, and the output is the status of successful transmission to the server. This process involves encoding the data using Wi-Fi or mobile network and sending it to the server via the HTTP protocol.

[0783] Step 3:

[0784] The server stores the received video data and GPS data in a database. The input is the data sent from the device, and the output is the status of successful storage in the database. The server performs data consistency checks to ensure redundancy and performs highly reliable storage processing.

[0785] Step 4:

[0786] The analysis software on the server analyzes the received data and detects anomalies such as traffic accidents and road damage. The input is the saved video data and GPS data, and the output is the identification of anomalies and detailed information about them. Here, a machine learning model is used to determine whether there are any anomalies in each video frame and match them with location information.

[0787] Step 5:

[0788] The server notifies the relevant organizations of the anomaly information. The input is the detailed information about the anomaly obtained in step 4, and the output is the status of sending the notification to the relevant organizations. In this process, detailed information including the location, type, severity, and estimated repair time of the anomaly is notified using email and SMS sending functions.

[0789] Step 6:

[0790] The server analyzes traffic volume and determines the priority of road repairs. The input is collected traffic data and anomaly information, and the output is a list of repair priorities. The server analyzes traffic volume fluctuations and calculates repair priorities based on the frequency of use of anomaly points.

[0791] Step 7:

[0792] The device proposes a safe avoidance route to the autonomous vehicle based on feedback from the server. The input is the abnormality information and the avoidance route proposal obtained from the server, and the output is the vehicle's new driving route. The device uses the navigation system to calculate a safe route in real time and update the driving plan.

[0793] This allows the above-mentioned processing steps to work together to create a system that enables safe and efficient operation of autonomous vehicles.

[0794] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0795] The system of the present invention collects video footage recorded by a drive recorder and analyzes it on a server to quickly and efficiently detect traffic accidents and road damage. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, it is possible to respond flexibly according to the user's condition. Below, specific program processing and its implementation are explained in natural language.

[0796] Video collection and transmission by dashcam

[0797] The device (dashcam) records high-resolution video while the vehicle is driving. For example, if the vehicle is driving in an urban area, it records road conditions and traffic conditions in full HD resolution. The device also collects GPS data along with the video, linking the location information to the video. This data is sent to a server in real time via Wi-Fi or a mobile network.

[0798] Data reception and analysis by the server

[0799] The server receives the video data and GPS information sent from the device. The received data is stored in a database to ensure security and redundancy. The analysis software on the server then analyzes the video data to identify traffic accident situations and road damage. The analysis is performed using a machine learning model to detect anomalies in each frame of the video. For example, if an anomaly such as a large hole or crack in the road or the scene of a traffic accident is detected, the location and time of the anomaly are identified.

[0800] Notification and Arrangements

[0801] Based on the analysis results, the server notifies relevant authorities of any abnormalities in real time. The notification includes detailed information such as the location, type, severity, and estimated repair time of the abnormality. For example, if a large hole is detected on a specific road, detailed information about it will be sent to the road management bureau via email or SMS.

[0802] Traffic analysis and repair prioritization

[0803] The server analyzes traffic volume based on the video data to understand road usage. The analyzed traffic volume data is used to determine repair priorities. For example, repairs to defects on major roads with very heavy traffic may be planned as a priority, while roads with less traffic may be postponed.

[0804] User response using an emotion engine

[0805] The server uses an emotion engine to analyze the user's emotional state. The emotion engine recognizes the user's emotions from video, audio, and the user's operation patterns, and can detect, for example, stress or impatience. This allows for flexible responses depending on the user's emotional state when reporting an abnormality. For example, if the user is feeling extremely stressed, the content of the notification and the priority of the response can be adjusted.

[0806] Specific examples

[0807] For example, one morning, while a vehicle is driving down a major street in the city, the dashcam detects a large pothole ahead and records the video. The video and GPS data are immediately sent to the server. The server receives the data, and analysis software detects the large pothole. The analysis results identify the location and severity of the pothole, and notify the road management bureau. The road management bureau then issues repair orders through the system and arranges for the necessary resources, ensuring that repairs are carried out quickly. At the same time, an emotion engine is used to analyze the user's emotional state when they report the abnormality, and further measures can be taken if the abnormality is particularly urgent. This type of system makes it possible to quickly detect road abnormalities and implement appropriate responses.

[0808] The processing flow will be explained below.

[0809] Step 1:

[0810] The device records video while the vehicle is in motion.

[0811] The dashcam captures high-resolution footage of the vehicle while driving and saves each frame.

[0812] The GPS sensor records your current location in real time and tags it in the footage.

[0813] Step 2:

[0814] The device sends the collected video and location information to the server.

[0815] The device uploads data to the server at regular intervals via Wi-Fi or mobile network.

[0816] Confirm the data transmission and record the successful transmission in the log.

[0817] Step 3:

[0818] The server stores the received video data and location information.

[0819] The server receives the data and stores it in a database.

[0820] Video files are assigned timestamps and location information and are centrally managed.

[0821] Step 4:

[0822] The server analyzes the stored video data.

[0823] Computer vision technology and machine learning models are used to detect anomalies in the video (e.g., potholes, cracks, accident scenes, etc.).

[0824] The type and severity of the abnormality is evaluated based on the detection results.

[0825] Step 5:

[0826] The server analyzes the user's emotional state.

[0827] It uses an emotion engine that recognizes user emotions from video and audio data and operation patterns.

[0828] Analyze and evaluate the user's emotional state (e.g., stress, impatience) in real time.

[0829] Step 6:

[0830] The server notifies the relevant authorities of the abnormality information.

[0831] Detailed information about the anomaly (location, type, severity, estimated repair time) is sent to relevant authorities via an automated notification system.

[0832] Notification methods include email, SMS, and in-system messaging.

[0833] Step 7:

[0834] The server reflects the emotion data in traffic volume analysis and repair priority determination.

[0835] The system takes into account the user's emotional state and adjusts the priority of repairs if the urgency is high.

[0836] The final decision is made by combining the priorities calculated based on traffic volume with the analysis results of the emotion engine.

[0837] Step 8:

[0838] The user checks the notification through the interface and takes action.

[0839] Road management personnel check the notification and view detailed information on the system interface.

[0840] Necessary repair instructions and resource arrangements are made via the interface.

[0841] By using the above processing steps, the system of the present invention can quickly detect road abnormalities and take appropriate measures. Furthermore, by taking the user's emotional state into consideration, more flexible and appropriate responses are possible.

[0842] Example 2

[0843] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0844] Traffic accidents and road damage must be detected quickly and accurately, but conventional methods often delay appropriate analysis and notification. Furthermore, responses that ignore the user's emotional state can increase stress and frustration. The purpose of this invention is to solve these problems by efficiently and quickly detecting anomalies, notifying relevant authorities, and enabling flexible responses based on the user's emotional state.

[0845] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing the transmitted video data and detecting the situation of a traffic accident or road damage, a means for notifying the relevant authorities of the detected abnormality information, a means for analyzing the emotional state of the user, and a means for providing a flexible response to the user based on the analysis result. This enables quick detection and notification of a traffic accident or road damage, and also enables a flexible response according to the emotional state of the user.

[0846] A "dashcam" is a device that records images of the front and rear of a vehicle while it is in motion.

[0847] "Video data" refers to video information collected by a drive recorder.

[0848] A "server" is a computer system that receives, analyzes, and stores data over a network and works in conjunction with other systems and devices.

[0849] "Transmitting means" refers to the mechanism by which collected data is sent to other systems via wireless or wired communication.

[0850] "Analysis means" refers to devices and technologies, including software and algorithms, that process collected data and detect abnormalities such as traffic accidents and road damage.

[0851] "Traffic accident" means an accident caused by a collision between vehicles or contact between a vehicle and a person or object.

[0852] "Road damage" refers to conditions that impede vehicle movement, such as cracks or holes in the road surface.

[0853] "Notification means" refers to the mechanism for communicating analysis results to relevant agencies by means of email, SMS, etc.

[0854] "Relevant organizations" refer to organizations that should respond to detected abnormalities, such as road management bureaus and police.

[0855] "Traffic volume analysis means" refers to technology that analyzes the number, speed, flow, etc. of vehicles based on video data.

[0856] "Means for determining repair priority" refers to a mechanism for determining the order and priority of road repairs based on analysis results and traffic volume data.

[0857] "Emotional state analysis means" refers to technology that analyzes the user's tone of voice, operation patterns, etc. to determine the user's emotions.

[0858] "Flexible response means" refers to techniques and processes for adjusting notification content and changing response priorities according to the user's emotional state.

[0859] The system of this invention collects video footage recorded by a dashcam and analyzes it on a server to quickly and efficiently detect traffic accidents and road damage.The system also incorporates an emotion engine that recognizes the user's emotional state and enables flexible responses.

[0860] Video collection and transmission by dashcam

[0861] The device (dashcam) records high-resolution video while the vehicle is driving. For example, when the vehicle is driving in an urban area, the dashcam records video of the road ahead in full HD resolution. This can be done using a commercially available high-performance camera. A GPS device also operates to obtain location information, collecting location data in real time. This data (video data and GPS data) is processed in real time within the device and sent to a server using Wi-Fi or a mobile network.

[0862] Data reception and storage by the server

[0863] The server receives the video data and GPS information sent from the device. The server is equipped with a high-performance data receiving module (e.g., an API server using Python and Flask) and receives data packets arriving via the Internet. It then temporarily stores this data in storage and stores it in a database (e.g., MySQL or PostgreSQL) in a highly redundant format.

[0864] Data analysis by server

[0865] The server analyzes the stored video data and location information. Machine learning models (e.g., deep learning models using TensorFlow or PyTorch) are used for video analysis, with anomaly detection algorithms being particularly important. The video data is processed frame by frame to detect anomalies (e.g., large holes or debris from traffic accidents) within each frame. Any anomalies detected by the analysis software are recorded in a database along with their location information.

[0866] Server notification of abnormal information

[0867] The server notifies the relevant authorities of the anomaly information based on the analysis results. This notification process is automated, and a message containing information such as the location, type, severity, and estimated repair time of the anomaly is generated based on the analysis results. The message is sent to the relevant authorities (e.g., road management authorities or police) via email or SMS. This can be done, for example, using the SMTP protocol or an SMS gateway.

[0868] Traffic analysis and repair priority determination by server

[0869] The server analyzes traffic volume from the video data. During the analysis process, the number of vehicles, speed, flow, etc. are analyzed and recorded in a traffic volume database. Based on these results, an algorithm is run to determine the priority of repair work. For example, the data analysis is performed using the Python Pandas library, and a high priority is assigned to abnormalities on major roads based on a priority algorithm.

[0870] Server-based user emotion analysis and response

[0871] The server uses an emotion engine to analyze the user's emotional state. The emotion engine uses a generative AI model (for example, a large-scale language model such as GPT-3) to process data such as the user's voice and operation patterns to recognize the user's emotional state. For example, it can detect stress or impatience from the user's typing speed and tone of voice when operating the device. Based on the results of this emotion analysis, the server can respond flexibly to the user. For example, for a user who feels stressed, it can simplify the notification content and provide a high-priority response.

[0872] Specific examples

[0873] For example, one morning, while a vehicle is driving down a major street in the city, the dashcam detects a large pothole ahead and records the video. The video and GPS data are immediately sent to the server. The server receives the data, and analysis software detects the large pothole. The analysis results identify the location and severity of the pothole, and notify the road management bureau. The road management bureau then issues repair orders through the system and arranges for the necessary resources, ensuring prompt repairs. At the same time, an emotion engine is used to analyze the user's emotional state when reporting the abnormality, and further measures can be taken if the abnormality is particularly urgent. This type of system makes it possible to quickly detect road abnormalities and implement appropriate responses.

[0874] Prompt Sentence Examples

[0875] "A large hole has been discovered on a major street in the city. We have provided detailed location information and urgent repairs are required."

[0876] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0877] Step 1:

[0878] The device (dashcam) records high-resolution video while the vehicle is driving. The dashcam is equipped with a high-performance camera that captures full HD resolution video of the road ahead, while a GPS device simultaneously collects location data in real time. The input is the vehicle's current driving video and location information, and the output is the video data and GPS data temporarily stored in the dashcam.

[0879] Step 2:

[0880] The device transmits the collected video data and GPS data to a server in real time via Wi-Fi or mobile networks. The input is the data stored in the dashcam, and the output is the video data and location information transferred to the server.

[0881] Step 3:

[0882] The server receives the video data and GPS information sent from the device. It receives the data packets via the Internet through a receiving API server (using Flask, for example) and temporarily stores them in storage. The input is the sent video data and location information, and the output is the raw data stored in temporary storage on the server.

[0883] Step 4:

[0884] The server stores the received data in a database for redundancy, for example, using MySQL or PostgreSQL. The input is video data and GPS information stored in the server's temporary storage, and the output is data stored in the database ready for analysis.

[0885] Step 5:

[0886] The server analyzes the stored video data. It uses a machine learning model (for example, a model using TensorFlow) to process the video data frame by frame to detect anomalies. The input is the video data read from the database, and the output contains anomaly detection information (for example, the occurrence of a large hole or a traffic accident) as the analysis result.

[0887] Step 6:

[0888] The server records the detected anomaly information in a database. The analyzed anomaly information and its location data are stored in the database. The input is the anomaly detection information, and the output is detailed anomaly information recorded in the database.

[0889] Step 7:

[0890] The server notifies relevant organizations based on the detected anomaly information. A notification message containing the location, type, severity, estimated repair time, etc. of the anomaly is generated and sent via email or SMS to the configured email address or phone number. The input is the anomaly information stored in the database, and the output is the notification message sent to the relevant organizations.

[0891] Step 8:

[0892] The server analyzes traffic volume from the video data. It uses Python's Pandas library to extract the number of vehicles, their speeds, and other data, and records them in a traffic volume database. The input is raw video data, and the output is analyzed traffic volume data.

[0893] Step 9:

[0894] The server determines the priority of road repairs based on traffic volume data. It applies a priority algorithm to assign higher priority to defects on major roads with heavy traffic. The input is traffic volume data, and the output is a repair priority list.

[0895] Step 10:

[0896] The server uses an emotion engine to analyze the user's emotional state. It uses a generative AI model (e.g., GPT-3) to analyze the user's tone of voice and operation patterns. The input is the user's voice data and operation patterns, and the output is the analyzed emotional state.

[0897] Step 11:

[0898] The server responds flexibly to users based on their emotional state. If a user is feeling stressed, it simplifies notifications and provides high-priority responses. The input is the emotion analysis results, and the output is tailored notifications and support responses.

[0899] (Application example 2)

[0900] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0901] Conventional systems for detecting traffic accidents and road damage have had the problem of being difficult to respond quickly and efficiently. Furthermore, while it is important for autonomous vehicles to detect road conditions and traffic abnormalities in real time and take appropriate measures, they often lack the ability to respond flexibly while taking into account the emotional state of the user. Furthermore, it is also necessary to accurately notify the location of abnormalities and determine repair priorities based on traffic volume.

[0902] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0903] In this invention, the server includes means for collecting video recorded by the drive recorder, means for transmitting the collected video data to the server, means for analyzing the transmitted video data and detecting traffic accidents and road damage, means for notifying the relevant authorities of the detected abnormality information, means for analyzing traffic volume and determining the priority of road repairs, means for analyzing the emotional state of the user, and means for adjusting the response according to the emotional state. This makes it possible to not only detect road conditions and traffic abnormalities in real time and notify the relevant authorities, but also to respond flexibly according to the emotional state of the user.

[0904] A "dashcam" is a device that records video footage of the front and rear of a vehicle, and is used to analyze traffic accidents and monitor road conditions.

[0905] A "server" is a computer system that stores and processes data via a network, and is responsible for functions such as analyzing video data and notifying users of abnormal information.

[0906] "Video data" refers to video information recorded by a drive recorder, and is data used to analyze traffic accidents and road damage.

[0907] "Traffic accidents" refer to accidents that occur when vehicles collide with each other or when a vehicle comes into contact with a pedestrian or object.

[0908] "Road damage" refers to cracks, holes, or other damage to the road surface that may impede vehicle movement.

[0909] "Relevant authorities" refers to public institutions responsible for responding to traffic accidents and road damage, such as road management authorities and traffic police.

[0910] "Traffic volume" refers to the number of vehicles passing through a particular road within a certain period of time, and is an indicator of road usage.

[0911] The "emotional state of the user" refers to the mental state of the driver and passengers while driving, and includes emotions such as stress and impatience.

[0912] "Flexible response" refers to the ability to change the way you respond depending on the situation or conditions, and to adjust the content of notifications and responses based on the user's emotional state.

[0913] "Location information" is data relating to geographical location obtained using technology such as GPS, and is transmitted to the server together with the video data.

[0914] "Analysis" refers to the process of examining collected data to find meaning and identify anomalies or specific situations.

[0915] "Abnormal information" is data relating to unusual situations such as traffic accidents and road damage, and is notified to relevant authorities.

[0916] "Priority" indicates the order or importance of how things should be handled, and is a criterion for determining the order of repairs based on traffic volume and road conditions.

[0917] MODE FOR CARRYING OUT THE INVENTION

[0918] System Overview

[0919] The system that realizes this invention collects video footage using a drive recorder installed in an autonomous vehicle, transmits the footage to a cloud server, and analyzes it to detect traffic accidents and road damage. It can also analyze the user's emotional state and provide flexible responses accordingly. Each component of the system and its function are described in detail below.

[0920] Device (Dashcam)

[0921] The dashcam records high-resolution footage of the front and rear of the vehicle. The footage is captured in Full HD resolution, providing crystal-clear video data. The dashcam also has a built-in GPS module, which simultaneously collects location information. This data is sent in real time to a cloud server via Wi-Fi or mobile network.

[0922] Data processing and analysis by the server

[0923] The server receives the video data and location information sent from the device. The received data is stored in a database to ensure security and redundancy. The server uses advanced analytics software to analyze the video data and identify traffic accidents and road damage. This analysis utilizes machine learning models to detect anomalies on a frame-by-frame basis.

[0924] For example, if a major road damage or traffic accident is detected in the video, the specific location and time of the incident are identified. When an anomaly is detected, the server immediately notifies the relevant authorities. The notification includes details such as the anomaly's location, type, severity, and estimated repair time.

[0925] User response using an emotion engine

[0926] The server uses an emotion engine to analyze the user's emotional state. This emotion engine recognizes emotions such as stress and impatience from video, audio, and the user's operation patterns. This makes it possible to provide a flexible response according to the user's emotional state when they report an abnormality.

[0927] For example, if it detects that a user is experiencing high levels of stress, it can adjust the content of notifications and the priority of responses, thereby reducing the psychological burden on the user.

[0928] Traffic analysis and repair prioritization

[0929] The server analyzes traffic volume based on the received video data. The analyzed traffic volume data is used to understand road usage. This data is used to determine the priority of road repairs. For example, if there is an abnormality on a major road with very heavy traffic, repairs will be planned as a priority.

[0930] Specific examples

[0931] As a concrete example, consider a case where a vehicle detects a large pothole while driving on a major city street. At this time, the dashcam records the video and location information and immediately sends it to a cloud server. The server analyzes the video and identifies the damaged area. This information is then sent to the relevant authorities, who then arrange for repairs to be made promptly.

[0932] When a user reports an abnormality, the emotion engine detects that the user is stressed and responds to reassure the user, for example by providing an explanation of the situation and real-time updates on the progress of repairs.

[0933] Prompt Sentence Examples

[0934] "We developed a system that detects major road damage while an autonomous vehicle is traveling down a city's main streets and sends the information to a server in real time. The server then identifies the location of the damage and notifies the relevant authorities. It also recognizes the user's emotions and responds flexibly if the user is feeling stressed. What technologies and methods should we use?"

[0935] In this way, the system can not only quickly and efficiently detect traffic accidents and road damage situations and notify the relevant authorities, but also respond flexibly while taking into account the user's emotions.

[0936] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0937] Step 1:

[0938] The device (drive recorder) records high-resolution video of the area in front of and behind the vehicle. Specifically, it captures full HD video data while the vehicle is moving. The input is real-time video of the area around the vehicle, and the output is a high-resolution video file. The video data is temporarily stored in the recording device.

[0939] Step 2:

[0940] The device acquires location information. The drive recorder has a built-in GPS module that acquires the vehicle's current location. The input is a signal from a GPS satellite, and the output is the vehicle's precise location data. This location information is associated with the video data and temporarily saved.

[0941] Step 3:

[0942] The device sends the collected video data and location information to a server. The data is uploaded to a cloud server in real time via Wi-Fi or a mobile network. The input is a high-resolution video file and location data, and the output is a data stream sent to the server.

[0943] Step 4:

[0944] The server receives the video data and location information sent from the device. The received data is stored in a database to ensure safety and redundancy. The input is the data stream sent from the device, and the output is the video data and location information stored in the database.

[0945] Step 5:

[0946] The server analyzes the video data using a machine learning model. Anomalies are detected for each video frame, and traffic accidents and road damage situations are identified. The input is the video data stored in the database, and the output is the identified anomaly information. Specifically, image features of abnormal areas are extracted and analyzed using an anomaly detection algorithm.

[0947] Step 6:

[0948] The server notifies the relevant authorities of detected anomalies. It sends the location, type, severity, and estimated repair time of the anomaly in real time via email or SMS. The input is the identified anomaly information, and the output is the notification sent to the relevant authorities.

[0949] Step 7:

[0950] The server analyzes traffic volume based on the video data it receives. The traffic volume data is used to determine the priority of road repairs. The input is the video data stored in the database, and the output is the analyzed traffic volume data. Specifically, it analyzes information such as the number of vehicles and passing times to calculate traffic volume.

[0951] Step 8:

[0952] The server uses an emotion engine to analyze the user's emotional state. It recognizes the user's emotions from video, audio, and operation patterns, and identifies feelings such as stress or impatience. The input is the user's video and audio data, and the output is the identified emotional state.

[0953] Step 9:

[0954] The server adjusts its response based on the user's emotional state. For example, if the user is feeling extremely stressed, it adjusts the notification content and priority of the response. The input is the identified emotional state, and the output is the adjusted response. Specifically, it performs actions such as sending a message to reassure the user.

[0955] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0956] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0957] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0958] [Fourth embodiment]

[0959] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0960] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0961] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0962] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0963] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0964] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0965] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0966] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0967] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0968] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0969] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0970] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0971] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0972] The system of the present invention collects video footage recorded by a drive recorder and analyzes it on a server to quickly and efficiently detect traffic accidents and road damage. This system notifies relevant authorities of abnormality information, analyzes traffic volume, and prioritizes road repairs. Specific program processing and its implementation are explained in natural language below.

[0973] Video collection and transmission by dashcam

[0974] The device (dashcam) records high-resolution video while the vehicle is driving. For example, if the vehicle is driving throughout the day, it will record road conditions and traffic conditions in full HD resolution. The device also collects GPS data along with the video, linking the location information to the video. This data is sent to a server in real time via Wi-Fi or a mobile network.

[0975] Data reception and analysis by the server

[0976] The server receives the video data and GPS information sent from the device. The received data is stored in a database to ensure security and redundancy. Analysis software on the server then analyzes the video data to identify traffic accident situations and road damage. The analysis is performed using machine learning models to detect anomalies in each frame of the video. For example, if an anomaly such as a large hole, crack, or traffic accident scene is detected, its location and time are identified.

[0977] Notification and Arrangements

[0978] Based on the analysis results, the server notifies relevant authorities of any abnormalities in real time. The notification includes detailed information such as the location, type, severity, and estimated repair time of the abnormality. For example, if a large hole is detected on a specific road, detailed information about it will be sent to the road management bureau via email or SMS.

[0979] Traffic analysis and repair prioritization

[0980] The server analyzes traffic volume based on the video data to understand road usage. The analyzed traffic volume data is used to determine repair priorities. For example, repairs to defects on major roads with very heavy traffic may be planned as a priority, while roads with less traffic may be postponed.

[0981] User interface support

[0982] Users can check notifications from the system through the interface. For example, after a road maintenance officer receives a notification, he or she can check the detailed situation on the system interface and enter the necessary instructions, allowing a repair team to quickly head to the site and begin repair work.

[0983] Specific examples

[0984] For example, one morning, while a vehicle is driving down a major road, the dashcam detects a large pothole ahead and records the video. The video and GPS data are immediately sent to a server. The server receives the data, and analysis software detects the large pothole. The analysis results identify its location and severity, and notify the road management authority. The road management authority then issues repair orders through the system and arranges for the necessary resources, allowing repairs to be carried out quickly.

[0985] Such systems can improve public safety by quickly detecting and responding to road anomalies.

[0986] The processing flow will be explained below.

[0987] Step 1:

[0988] The device records video while the vehicle is in motion.

[0989] The dashcam captures high-resolution footage of the vehicle while driving and saves each frame.

[0990] The GPS sensor records your current location in real time and tags it in the footage.

[0991] Step 2:

[0992] The device sends the collected video and location information to the server.

[0993] The device uploads data to the server at regular intervals via Wi-Fi or mobile network.

[0994] Confirm the data transmission and record the successful transmission in the log.

[0995] Step 3:

[0996] The server stores the received video data and location information.

[0997] The server receives the data and stores it in a database.

[0998] Video files are assigned timestamps and location information and are centrally managed.

[0999] Step 4:

[1000] The server analyzes the stored video data.

[1001] Computer vision technology and machine learning models are used to detect anomalies in the video (e.g., potholes, cracks, accident scenes, etc.).

[1002] The type and severity of the abnormality is evaluated based on the detection results.

[1003] Step 5:

[1004] The server notifies the relevant authorities of the abnormality information.

[1005] Detailed information about the anomaly (location, type, severity, estimated repair time) is sent to relevant authorities via an automated notification system.

[1006] Notification methods include email, SMS, and in-system messaging.

[1007] Step 6:

[1008] The server analyzes traffic volume from video data and determines repair priorities.

[1009] The number of cars in the video and their frequency of passage are measured to estimate traffic volume.

[1010] Traffic volume data is compiled and repair priorities for abnormal areas are determined.

[1011] Step 7:

[1012] The user checks the notification through the interface and takes action.

[1013] Road management personnel check the notification and view detailed information on the system interface.

[1014] Necessary repair instructions and resource arrangements are made via the interface.

[1015] By using the above processing steps, the system of the present invention can quickly detect road abnormalities and take appropriate measures.

[1016] Example 1

[1017] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1018] Conventional systems for detecting traffic accidents and road damage required manual confirmation and reporting, making it difficult to respond quickly. Furthermore, delays in reporting and responding to abnormal information often threatened public safety. Traffic volume analysis and prioritization of road repairs were also inappropriate, preventing efficient repair work. The present invention aims to solve these problems.

[1019] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1020] In this invention, the server includes a means for storing video footage and GPS data recorded by the drive recorder in a database in real time, a means for detecting road damage and abnormalities using a machine learning model, a means for notifying relevant authorities of detected abnormalities, and a means for analyzing traffic volume and determining the priority of road repairs. This enables rapid and efficient detection of road condition abnormalities and real-time notification to relevant authorities, enabling prompt and appropriate response. Furthermore, determining repair priorities based on traffic volume data enables efficient road repairs.

[1021] A "dashcam" is a device that records video and audio while a vehicle is in motion.

[1022] An "external domain" refers to a remote data storage location, such as a server or cloud storage connected via a network.

[1023] "Video data" refers to video information recorded by a drive recorder.

[1024] "GPS Data" means location information collected using the Global Positioning System.

[1025] The "database" is a system for systematically storing and managing received video data and GPS data.

[1026] A "machine learning model" is an algorithm or system that learns from large amounts of data and performs pattern recognition and prediction.

[1027] "Abnormal information" refers to information that deviates from normal conditions, such as traffic accidents or road damage.

[1028] "Related agencies" refer to organizations and institutions that respond to traffic accidents and road damage, such as road management bureaus and traffic police.

[1029] "Traffic analysis" is the process of evaluating road usage and traffic flow through data analysis.

[1030] "Road repair priorities" are criteria for determining which areas should be given priority when carrying out road repairs or maintenance.

[1031] Embodiments of the present invention will be described in detail below.

[1032] The system of the present invention collects video data and GPS data recorded by a drive recorder and transmits them to a server. The device (drive recorder) records video in high resolution and simultaneously collects GPS data. The device transmits this data to the server in real time via Wi-Fi or a mobile network.

[1033] Specifically, when the vehicle starts moving, the drive recorder automatically starts recording video. The device records high-resolution video and GPS data while the vehicle is moving, and associates it with a timestamp and location information. This data is temporarily stored inside the device and periodically sent to the server. Communication errors are checked by receiving a transmission confirmation message from the server.

[1034] The server receives the video data and GPS data sent from the device and stores it in a database. When storing the data, the data is backed up in multiple storage devices to ensure safety and redundancy. The received data is analyzed using machine learning models by the server's analysis software. During the analysis, the video data is analyzed frame by frame to detect traffic accidents and road damage. For example, if an abnormality such as a large hole, crack, or traffic accident site is detected, the location and time of the abnormality are identified.

[1035] Based on the analysis results, the server notifies the relevant authorities of any detected anomalies. The notification includes information such as details of the anomaly, its location, type, severity, and estimated repair time. This information is sent to the relevant authorities (e.g., road management bureau) via email or SMS. In addition, the analyzed traffic volume data is used to determine the priority of repairs. Anomalies on major roads with very heavy traffic volume are given priority for repair planning, while roads with less traffic volume may be postponed.

[1036] The user (e.g., road maintenance personnel) checks the system's notification through the system's interface. After receiving the notification, the user can check the detailed situation in the interface and enter the necessary instructions, allowing repair teams to quickly head to the site and begin repair work.

[1037] Specific examples

[1038] For example, one morning, a vehicle traveling down a major road might detect a large pothole ahead and record video footage of it on its dashcam. The video and GPS data are immediately sent to a server, where analysis software detects the presence of the pothole. The analysis results identify its location and severity, and the road management authority is notified. The road management authority then issues repair orders through the system and arranges for the necessary resources to be used, ensuring prompt repairs.

[1039] Such a system can improve public safety by quickly detecting road anomalies and responding quickly.

[1040] Prompt Sentence Examples

[1041] "Please explain in detail, step by step, how you analyze footage recorded by a dashcam to detect road anomalies. Also, please explain in detail the notification and response process when an anomaly is detected."

[1042] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1043] Step 1:

[1044] The device collects video and GPS data while the vehicle is moving. It receives video of the vehicle moving and its current location as input, and outputs high-resolution video and GPS data. Specifically, the device automatically starts recording when the vehicle's engine starts, recording video and GPS data in full HD. This data is temporarily stored in the device's internal memory.

[1045] Step 2:

[1046] The device sends the collected video data and GPS data to the server. It takes the collected video data and GPS data as input and outputs the data to the server. Specifically, the device periodically uploads data packets to the server using Wi-Fi or a mobile network. It also checks for communication errors by receiving a confirmation message that transmission is complete.

[1047] Step 3:

[1048] The server receives the video data and GPS data sent from the device. It receives the data packets sent as input and outputs them to be safely stored in a database. Specifically, the server monitors the receiving port in real time and promptly stores the received data in the database. It also copies the data to multiple backup storages to ensure redundancy.

[1049] Step 4:

[1050] The server analyzes the received video data and GPS data. Using the stored video data and GPS data as input, it obtains an output that detects traffic accidents and road damage. Specifically, the analysis software on the server uses a machine learning model to detect anomalies in each video frame. For example, large holes, cracks, and traffic accident scenes on the road are analyzed. As a result of the analysis, the location and severity of the detected anomalies are identified.

[1051] Step 5:

[1052] Based on the analysis results, the server notifies relevant organizations of detected anomalies. It receives the analysis result data as input and obtains an output that sends detailed information to relevant organizations. Specifically, the server creates a notification message that includes the location, type, severity, estimated repair time, etc. of the anomaly, and sends it via email or SMS to relevant organizations such as road management bureaus. In addition, images or videos of the anomaly may be attached to the notification email if necessary.

[1053] Step 6:

[1054] The server analyzes traffic volume based on video data and determines repair priorities. It takes in all video data and road condition data as input and outputs a list of repair priorities. Specifically, the server calculates traffic volume for specific road sections from the video data and ranks the repair priority of each road based on the traffic volume information. The more major roads with heavy traffic have defects, the higher the priority for repair plans.

[1055] Step 7:

[1056] The user checks notifications from the system through an interface and inputs the necessary instructions. The input is a confirmation of the notification message, and the system outputs the instructions and resource arrangement information required for repair response. Specifically, the user (road management officer) logs into the system and checks the details of the notification (location, type of anomaly, severity, etc.). After that, the system registers and sends the arrangement of a repair team and detailed instructions for the repair content. This allows the repair team to respond quickly.

[1057] (Application example 1)

[1058] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1059] In modern road traffic, early detection and rapid response to traffic accidents and road damage are important. However, conventional traffic monitoring systems require dedicated infrastructure, which is costly and difficult to operate efficiently. It is also difficult to prioritize road repairs appropriately in response to fluctuations in traffic volume. Furthermore, with the spread of autonomous vehicles, there is a need for real-time understanding of the road conditions these vehicles face and support for safe operation.

[1060] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1061] In this invention, the server includes means for collecting video recorded by the drive recorder, means for transmitting the collected video data to the server, means for analyzing the transmitted video data and detecting traffic accidents and road damage, means for notifying relevant organizations of detected abnormality information, means for analyzing traffic volume and determining the priority of road repairs, and means for collecting and transmitting video and location information during driving in real time via an application installed in the autonomous vehicle, feeding back abnormality detection results to the server, and proposing safe avoidance routes to the autonomous vehicle. This not only enables early detection of traffic accidents and road damage and rapid response, but also supports the safe operation of the autonomous vehicle.

[1062] A "drive recorder" is a device that records video while a vehicle is driving.

[1063] A "server" is a computing system that receives, stores, analyzes, and processes data over a network.

[1064] An "autonomous vehicle" is a vehicle that can drive autonomously using artificial intelligence and sensors.

[1065] "Video data" refers to digital data of visual information recorded by a drive recorder.

[1066] "GPS Data" means location information obtained from the Global Positioning System (GPS) to indicate a precise location on Earth.

[1067] "Analysis Software" means the programs used to interpret collected data and extract specific information on demand.

[1068] "Abnormal information" is data on events that deviate from normal conditions, specifically traffic accidents and road damage.

[1069] "Related organizations" refers to government agencies and companies in charge of road management and traffic management.

[1070] "Traffic analysis" is the process of analyzing the flow of vehicles and traffic on a particular road to identify patterns and trends.

[1071] "Repair priority" is a criterion for determining the order of repair work for road sections where abnormalities have been detected.

[1072] A "user interface" is an interactive platform that allows users to view displayed information and perform necessary operations.

[1073] A "safe avoidance route" is an alternative route for a detected anomaly, including instructions for an autonomous vehicle to safely reach its destination.

[1074] The system of the present invention is mainly composed of a drive recorder, a GPS module, a server, and analysis software that are installed in a vehicle, including an autonomous driving vehicle.

[1075] Data collection and transmission by the device (autonomous vehicle):

[1076] The autonomous vehicle terminal is equipped with a dashcam and a GPS module. The dashcam records high-resolution video while the vehicle is driving, and the GPS module obtains precise location information. The collected video data and GPS data are transmitted to a server in real time. To achieve this, the terminal is connected to the Internet via Wi-Fi or a mobile network.

[1077] Data reception and analysis by the server:

[1078] The server receives the video data and GPS information sent from the device and stores it in a database for security and redundancy. Next, analysis software on the server uses machine learning models to analyze the video data and detect traffic accidents and road damage. This analysis uses software such as OpenCV and TensorFlow. If an abnormality is detected, the location information and details are notified to the relevant authorities.

[1079] Real-time safety assessment and route suggestions:

[1080] Based on the analysis results, the server proposes safe avoidance routes to autonomous vehicles in real time, enabling vehicles in operation to avoid abnormal areas and operate safely. The route proposal algorithm is implemented using Google Maps API, etc.

[1081] Notification and Arrangements:

[1082] Notification of abnormalities is made via communication methods such as email and SMS. Notifications include the location, type, severity, and estimated repair time of the abnormality, allowing relevant organizations to take prompt action based on this information.

[1083] User Interface:

[1084] Personnel from relevant agencies can check notifications from the system through a dedicated user interface, which allows them to check the detailed situation, input necessary instructions, and take prompt action such as dispatching a repair team.

[1085] Specific examples

[1086] For example, one morning, while an autonomous vehicle is driving down a major road, its dashcam detects a large pothole ahead and immediately sends the video and GPS data to a server. The server receives the data, and analysis software detects the large pothole. The analysis identifies its location and severity, and notifies the road management authority. The road management authority then issues repair orders through the system and arranges for the necessary resources, ensuring prompt repairs.

[1087] Example prompts for generative AI models

[1088] "Design an application that automates the response when a dashcam detects road damage ahead while an autonomous vehicle is driving. Please provide specific code examples and the processing flow for building a system that sends video data and GPS data to a server in real time and notifies the relevant authorities if an abnormality is detected."

[1089] In this way, by describing the mode for carrying out the invention in detail, the components of the system, the operation procedures, and specific examples will become clear.

[1090] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1091] Step 1:

[1092] The device activates the dashcam while the vehicle is moving, and collects video data and GPS data synchronously. The input is the vehicle's driving status and current location, and the output is high-resolution video data and location information. Video frames are stored in a buffer in real time, and GPS data is simultaneously linked.

[1093] Step 2:

[1094] The device transmits the collected video data and GPS data to the server in real time. The input is the video data and GPS data acquired in step 1, and the output is the status of successful transmission to the server. This process involves encoding the data using Wi-Fi or mobile network and sending it to the server via the HTTP protocol.

[1095] Step 3:

[1096] The server stores the received video data and GPS data in a database. The input is the data sent from the device, and the output is the status of successful storage in the database. The server performs data consistency checks to ensure redundancy and performs highly reliable storage processing.

[1097] Step 4:

[1098] The analysis software on the server analyzes the received data and detects anomalies such as traffic accidents and road damage. The input is the saved video data and GPS data, and the output is the identification of anomalies and detailed information about them. Here, a machine learning model is used to determine whether there are any anomalies in each video frame and match them with location information.

[1099] Step 5:

[1100] The server notifies the relevant organizations of the anomaly information. The input is the detailed information about the anomaly obtained in step 4, and the output is the status of sending the notification to the relevant organizations. In this process, detailed information including the location, type, severity, and estimated repair time of the anomaly is notified using email and SMS sending functions.

[1101] Step 6:

[1102] The server analyzes traffic volume and determines the priority of road repairs. The input is collected traffic data and anomaly information, and the output is a list of repair priorities. The server analyzes traffic volume fluctuations and calculates repair priorities based on the frequency of use of anomaly points.

[1103] Step 7:

[1104] The device proposes a safe avoidance route to the autonomous vehicle based on feedback from the server. The input is the abnormality information and the avoidance route proposal obtained from the server, and the output is the vehicle's new driving route. The device uses the navigation system to calculate a safe route in real time and update the driving plan.

[1105] This allows the above-mentioned processing steps to work together to create a system that enables safe and efficient operation of autonomous vehicles.

[1106] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1107] The system of the present invention collects video footage recorded by a drive recorder and analyzes it on a server to quickly and efficiently detect traffic accidents and road damage. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, it is possible to respond flexibly according to the user's condition. Below, specific program processing and its implementation are explained in natural language.

[1108] Video collection and transmission by dashcam

[1109] The device (dashcam) records high-resolution video while the vehicle is driving. For example, if the vehicle is driving in an urban area, it records road conditions and traffic conditions in full HD resolution. The device also collects GPS data along with the video, linking the location information to the video. This data is sent to a server in real time via Wi-Fi or a mobile network.

[1110] Data reception and analysis by the server

[1111] The server receives the video data and GPS information sent from the device. The received data is stored in a database to ensure security and redundancy. The analysis software on the server then analyzes the video data to identify traffic accident situations and road damage. The analysis is performed using a machine learning model to detect anomalies in each frame of the video. For example, if an anomaly such as a large hole or crack in the road or the scene of a traffic accident is detected, the location and time of the anomaly are identified.

[1112] Notification and Arrangements

[1113] Based on the analysis results, the server notifies relevant authorities of any abnormalities in real time. The notification includes detailed information such as the location, type, severity, and estimated repair time of the abnormality. For example, if a large hole is detected on a specific road, detailed information about it will be sent to the road management bureau via email or SMS.

[1114] Traffic analysis and repair prioritization

[1115] The server analyzes traffic volume based on the video data to understand road usage. The analyzed traffic volume data is used to determine repair priorities. For example, repairs to defects on major roads with very heavy traffic may be planned as a priority, while roads with less traffic may be postponed.

[1116] User response using an emotion engine

[1117] The server uses an emotion engine to analyze the user's emotional state. The emotion engine recognizes the user's emotions from video, audio, and the user's operation patterns, and can detect, for example, stress or impatience. This allows for flexible responses depending on the user's emotional state when reporting an abnormality. For example, if the user is feeling extremely stressed, the content of the notification and the priority of the response can be adjusted.

[1118] Specific examples

[1119] For example, one morning, while a vehicle is driving down a major street in the city, the dashcam detects a large pothole ahead and records the video. The video and GPS data are immediately sent to the server. The server receives the data, and analysis software detects the large pothole. The analysis results identify the location and severity of the pothole, and notify the road management bureau. The road management bureau then issues repair orders through the system and arranges for the necessary resources, ensuring that repairs are carried out quickly. At the same time, an emotion engine is used to analyze the user's emotional state when they report the abnormality, and further measures can be taken if the abnormality is particularly urgent. This type of system makes it possible to quickly detect road abnormalities and implement appropriate responses.

[1120] The processing flow will be explained below.

[1121] Step 1:

[1122] The device records video while the vehicle is in motion.

[1123] The dashcam captures high-resolution footage of the vehicle while driving and saves each frame.

[1124] The GPS sensor records your current location in real time and tags it in the footage.

[1125] Step 2:

[1126] The device sends the collected video and location information to the server.

[1127] The device uploads data to the server at regular intervals via Wi-Fi or mobile network.

[1128] Confirm the data transmission and record the successful transmission in the log.

[1129] Step 3:

[1130] The server stores the received video data and location information.

[1131] The server receives the data and stores it in a database.

[1132] Video files are assigned timestamps and location information and are centrally managed.

[1133] Step 4:

[1134] The server analyzes the stored video data.

[1135] Computer vision technology and machine learning models are used to detect anomalies in the video (e.g., potholes, cracks, accident scenes, etc.).

[1136] The type and severity of the abnormality is evaluated based on the detection results.

[1137] Step 5:

[1138] The server analyzes the user's emotional state.

[1139] It uses an emotion engine that recognizes user emotions from video and audio data and operation patterns.

[1140] Analyze and evaluate the user's emotional state (e.g., stress, impatience) in real time.

[1141] Step 6:

[1142] The server notifies the relevant authorities of the abnormality information.

[1143] Detailed information about the anomaly (location, type, severity, estimated repair time) is sent to relevant authorities via an automated notification system.

[1144] Notification methods include email, SMS, and in-system messaging.

[1145] Step 7:

[1146] The server reflects the emotion data in traffic volume analysis and repair priority determination.

[1147] The system takes into account the user's emotional state and adjusts the priority of repairs if the urgency is high.

[1148] The final decision is made by combining the priorities calculated based on traffic volume with the analysis results of the emotion engine.

[1149] Step 8:

[1150] The user checks the notification through the interface and takes action.

[1151] Road management personnel check the notification and view detailed information on the system interface.

[1152] Necessary repair instructions and resource arrangements are made via the interface.

[1153] By using the above processing steps, the system of the present invention can quickly detect road abnormalities and take appropriate measures. Furthermore, by taking the user's emotional state into consideration, more flexible and appropriate responses are possible.

[1154] Example 2

[1155] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1156] Traffic accidents and road damage must be detected quickly and accurately, but conventional methods often delay appropriate analysis and notification. Furthermore, responses that ignore the user's emotional state can increase stress and frustration. The purpose of this invention is to solve these problems by efficiently and quickly detecting anomalies, notifying relevant authorities, and enabling flexible responses based on the user's emotional state.

[1157] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing the transmitted video data and detecting the situation of a traffic accident or road damage, a means for notifying the relevant authorities of the detected abnormality information, a means for analyzing the emotional state of the user, and a means for providing a flexible response to the user based on the analysis result. This enables quick detection and notification of a traffic accident or road damage, and also enables a flexible response according to the emotional state of the user.

[1158] A "dashcam" is a device that records images of the front and rear of a vehicle while it is in motion.

[1159] "Video data" refers to video information collected by a drive recorder.

[1160] A "server" is a computer system that receives, analyzes, and stores data over a network and works in conjunction with other systems and devices.

[1161] "Transmitting means" refers to the mechanism by which collected data is sent to other systems via wireless or wired communication.

[1162] "Analysis means" refers to devices and technologies, including software and algorithms, that process collected data and detect abnormalities such as traffic accidents and road damage.

[1163] "Traffic accident" means an accident caused by a collision between vehicles or contact between a vehicle and a person or object.

[1164] "Road damage" refers to conditions that impede vehicle movement, such as cracks or holes in the road surface.

[1165] "Notification means" refers to the mechanism for communicating analysis results to relevant agencies by means of email, SMS, etc.

[1166] "Relevant organizations" refer to organizations that should respond to detected abnormalities, such as road management bureaus and police.

[1167] "Traffic volume analysis means" refers to technology that analyzes the number, speed, flow, etc. of vehicles based on video data.

[1168] "Means for determining repair priority" refers to a mechanism for determining the order and priority of road repairs based on analysis results and traffic volume data.

[1169] "Emotional state analysis means" refers to technology that analyzes the user's tone of voice, operation patterns, etc. to determine the user's emotions.

[1170] "Flexible response means" refers to techniques and processes for adjusting notification content and changing response priorities according to the user's emotional state.

[1171] The system of this invention collects video footage recorded by a dashcam and analyzes it on a server to quickly and efficiently detect traffic accidents and road damage.The system also incorporates an emotion engine that recognizes the user's emotional state and enables flexible responses.

[1172] Video collection and transmission by dashcam

[1173] The device (dashcam) records high-resolution video while the vehicle is driving. For example, when the vehicle is driving in an urban area, the dashcam records video of the road ahead in full HD resolution. This can be done using a commercially available high-performance camera. A GPS device also operates to obtain location information, collecting location data in real time. This data (video data and GPS data) is processed in real time within the device and sent to a server using Wi-Fi or a mobile network.

[1174] Data reception and storage by the server

[1175] The server receives the video data and GPS information sent from the device. The server is equipped with a high-performance data receiving module (e.g., an API server using Python and Flask) and receives data packets arriving via the Internet. It then temporarily stores this data in storage and stores it in a database (e.g., MySQL or PostgreSQL) in a highly redundant format.

[1176] Data analysis by server

[1177] The server analyzes the stored video data and location information. Machine learning models (e.g., deep learning models using TensorFlow or PyTorch) are used for video analysis, with anomaly detection algorithms being particularly important. The video data is processed frame by frame to detect anomalies (e.g., large holes or debris from traffic accidents) within each frame. Any anomalies detected by the analysis software are recorded in a database along with their location information.

[1178] Server notification of abnormal information

[1179] The server notifies the relevant authorities of the anomaly information based on the analysis results. This notification process is automated, and a message containing information such as the location, type, severity, and estimated repair time of the anomaly is generated based on the analysis results. The message is sent to the relevant authorities (e.g., road management authorities or police) via email or SMS. This can be done, for example, using the SMTP protocol or an SMS gateway.

[1180] Traffic analysis and repair priority determination by server

[1181] The server analyzes traffic volume from the video data. During the analysis process, the number of vehicles, speed, flow, etc. are analyzed and recorded in a traffic volume database. Based on these results, an algorithm is run to determine the priority of repair work. For example, the data analysis is performed using the Python Pandas library, and a high priority is assigned to abnormalities on major roads based on a priority algorithm.

[1182] Server-based user emotion analysis and response

[1183] The server uses an emotion engine to analyze the user's emotional state. The emotion engine uses a generative AI model (for example, a large-scale language model such as GPT-3) to process data such as the user's voice and operation patterns to recognize the user's emotional state. For example, it can detect stress or impatience from the user's typing speed and tone of voice when operating the device. Based on the results of this emotion analysis, the server can respond flexibly to the user. For example, for a user who feels stressed, it can simplify the notification content and provide a high-priority response.

[1184] Specific examples

[1185] For example, one morning, while a vehicle is driving down a major street in the city, the dashcam detects a large pothole ahead and records the video. The video and GPS data are immediately sent to the server. The server receives the data, and analysis software detects the large pothole. The analysis results identify the location and severity of the pothole, and notify the road management bureau. The road management bureau then issues repair orders through the system and arranges for the necessary resources, ensuring prompt repairs. At the same time, an emotion engine is used to analyze the user's emotional state when reporting the abnormality, and further measures can be taken if the abnormality is particularly urgent. This type of system makes it possible to quickly detect road abnormalities and implement appropriate responses.

[1186] Prompt Sentence Examples

[1187] "A large hole has been discovered on a major street in the city. We have provided detailed location information and urgent repairs are required."

[1188] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1189] Step 1:

[1190] The device (dashcam) records high-resolution video while the vehicle is driving. The dashcam is equipped with a high-performance camera that captures full HD resolution video of the road ahead, while a GPS device simultaneously collects location data in real time. The input is the vehicle's current driving video and location information, and the output is the video data and GPS data temporarily stored in the dashcam.

[1191] Step 2:

[1192] The device transmits the collected video data and GPS data to a server in real time via Wi-Fi or mobile networks. The input is the data stored in the dashcam, and the output is the video data and location information transferred to the server.

[1193] Step 3:

[1194] The server receives the video data and GPS information sent from the device. It receives the data packets via the Internet through a receiving API server (using Flask, for example) and temporarily stores them in storage. The input is the sent video data and location information, and the output is the raw data stored in temporary storage on the server.

[1195] Step 4:

[1196] The server stores the received data in a database for redundancy, for example, using MySQL or PostgreSQL. The input is video data and GPS information stored in the server's temporary storage, and the output is data stored in the database ready for analysis.

[1197] Step 5:

[1198] The server analyzes the stored video data. It uses a machine learning model (for example, a model using TensorFlow) to process the video data frame by frame to detect anomalies. The input is the video data read from the database, and the output contains anomaly detection information (for example, the occurrence of a large hole or a traffic accident) as the analysis result.

[1199] Step 6:

[1200] The server records the detected anomaly information in a database. The analyzed anomaly information and its location data are stored in the database. The input is the anomaly detection information, and the output is detailed anomaly information recorded in the database.

[1201] Step 7:

[1202] The server notifies relevant organizations based on the detected anomaly information. A notification message containing the location, type, severity, estimated repair time, etc. of the anomaly is generated and sent via email or SMS to the configured email address or phone number. The input is the anomaly information stored in the database, and the output is the notification message sent to the relevant organizations.

[1203] Step 8:

[1204] The server analyzes traffic volume from the video data. It uses Python's Pandas library to extract the number of vehicles, their speeds, and other data, and records them in a traffic volume database. The input is raw video data, and the output is analyzed traffic volume data.

[1205] Step 9:

[1206] The server determines the priority of road repairs based on traffic volume data. It applies a priority algorithm to assign higher priority to defects on major roads with heavy traffic. The input is traffic volume data, and the output is a repair priority list.

[1207] Step 10:

[1208] The server uses an emotion engine to analyze the user's emotional state. It uses a generative AI model (e.g., GPT-3) to analyze the user's tone of voice and operation patterns. The input is the user's voice data and operation patterns, and the output is the analyzed emotional state.

[1209] Step 11:

[1210] The server responds flexibly to users based on their emotional state. If a user is feeling stressed, it simplifies notifications and provides high-priority responses. The input is the emotion analysis results, and the output is tailored notifications and support responses.

[1211] (Application example 2)

[1212] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1213] Conventional systems for detecting traffic accidents and road damage have had the problem of being difficult to respond quickly and efficiently. Furthermore, while it is important for autonomous vehicles to detect road conditions and traffic abnormalities in real time and take appropriate measures, they often lack the ability to respond flexibly while taking into account the emotional state of the user. Furthermore, it is also necessary to accurately notify the location of abnormalities and determine repair priorities based on traffic volume.

[1214] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1215] In this invention, the server includes means for collecting video recorded by the drive recorder, means for transmitting the collected video data to the server, means for analyzing the transmitted video data and detecting traffic accidents and road damage, means for notifying the relevant authorities of the detected abnormality information, means for analyzing traffic volume and determining the priority of road repairs, means for analyzing the emotional state of the user, and means for adjusting the response according to the emotional state. This makes it possible to not only detect road conditions and traffic abnormalities in real time and notify the relevant authorities, but also to respond flexibly according to the emotional state of the user.

[1216] A "dashcam" is a device that records video footage of the front and rear of a vehicle, and is used to analyze traffic accidents and monitor road conditions.

[1217] A "server" is a computer system that stores and processes data via a network, and is responsible for functions such as analyzing video data and notifying users of abnormal information.

[1218] "Video data" refers to video information recorded by a drive recorder, and is data used to analyze traffic accidents and road damage.

[1219] "Traffic accidents" refer to accidents that occur when vehicles collide with each other or when a vehicle comes into contact with a pedestrian or object.

[1220] "Road damage" refers to cracks, holes, or other damage to the road surface that may impede vehicle movement.

[1221] "Relevant authorities" refers to public institutions responsible for responding to traffic accidents and road damage, such as road management authorities and traffic police.

[1222] "Traffic volume" refers to the number of vehicles passing through a particular road within a certain period of time, and is an indicator of road usage.

[1223] The "emotional state of the user" refers to the mental state of the driver and passengers while driving, and includes emotions such as stress and impatience.

[1224] "Flexible response" refers to the ability to change the way you respond depending on the situation or conditions, and to adjust the content of notifications and responses based on the user's emotional state.

[1225] "Location information" is data relating to geographical location obtained using technology such as GPS, and is transmitted to the server together with the video data.

[1226] "Analysis" refers to the process of examining collected data to find meaning and identify anomalies or specific situations.

[1227] "Abnormal information" is data relating to unusual situations such as traffic accidents and road damage, and is notified to relevant authorities.

[1228] "Priority" indicates the order or importance of how things should be handled, and is a criterion for determining the order of repairs based on traffic volume and road conditions.

[1229] MODE FOR CARRYING OUT THE INVENTION

[1230] System Overview

[1231] The system that realizes this invention collects video footage using a drive recorder installed in an autonomous vehicle, transmits the footage to a cloud server, and analyzes it to detect traffic accidents and road damage. It can also analyze the user's emotional state and provide flexible responses accordingly. Each component of the system and its function are described in detail below.

[1232] Device (Dashcam)

[1233] The dashcam records high-resolution footage of the front and rear of the vehicle. The footage is captured in Full HD resolution, providing crystal-clear video data. The dashcam also has a built-in GPS module, which simultaneously collects location information. This data is sent in real time to a cloud server via Wi-Fi or mobile network.

[1234] Data processing and analysis by the server

[1235] The server receives the video data and location information sent from the device. The received data is stored in a database to ensure security and redundancy. The server uses advanced analytics software to analyze the video data and identify traffic accidents and road damage. This analysis utilizes machine learning models to detect anomalies on a frame-by-frame basis.

[1236] For example, if a major road damage or traffic accident is detected in the video, the specific location and time of the incident are identified. When an anomaly is detected, the server immediately notifies the relevant authorities. The notification includes details such as the anomaly's location, type, severity, and estimated repair time.

[1237] User response using an emotion engine

[1238] The server uses an emotion engine to analyze the user's emotional state. This emotion engine recognizes emotions such as stress and impatience from video, audio, and the user's operation patterns. This makes it possible to provide a flexible response according to the user's emotional state when they report an abnormality.

[1239] For example, if it detects that a user is experiencing high levels of stress, it can adjust the content of notifications and the priority of responses, thereby reducing the psychological burden on the user.

[1240] Traffic analysis and repair prioritization

[1241] The server analyzes traffic volume based on the received video data. The analyzed traffic volume data is used to understand road usage. This data is used to determine the priority of road repairs. For example, if there is an abnormality on a major road with very heavy traffic, repairs will be planned as a priority.

[1242] Specific examples

[1243] As a concrete example, consider a case where a vehicle detects a large pothole while driving on a major city street. At this time, the dashcam records the video and location information and immediately sends it to a cloud server. The server analyzes the video and identifies the damaged area. This information is then sent to the relevant authorities, who then arrange for repairs to be made promptly.

[1244] When a user reports an abnormality, the emotion engine detects that the user is stressed and responds to reassure the user, for example by providing an explanation of the situation and real-time updates on the progress of repairs.

[1245] Prompt Sentence Examples

[1246] "We developed a system that detects major road damage while an autonomous vehicle is traveling down a city's main streets and sends the information to a server in real time. The server then identifies the location of the damage and notifies the relevant authorities. It also recognizes the user's emotions and responds flexibly if the user is feeling stressed. What technologies and methods should we use?"

[1247] In this way, the system can not only quickly and efficiently detect traffic accidents and road damage situations and notify the relevant authorities, but also respond flexibly while taking into account the user's emotions.

[1248] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1249] Step 1:

[1250] The device (drive recorder) records high-resolution video of the area in front of and behind the vehicle. Specifically, it captures full HD video data while the vehicle is moving. The input is real-time video of the area around the vehicle, and the output is a high-resolution video file. The video data is temporarily stored in the recording device.

[1251] Step 2:

[1252] The device acquires location information. The drive recorder has a built-in GPS module that acquires the vehicle's current location. The input is a signal from a GPS satellite, and the output is the vehicle's precise location data. This location information is associated with the video data and temporarily saved.

[1253] Step 3:

[1254] The device sends the collected video data and location information to a server. The data is uploaded to a cloud server in real time via Wi-Fi or a mobile network. The input is a high-resolution video file and location data, and the output is a data stream sent to the server.

[1255] Step 4:

[1256] The server receives the video data and location information sent from the device. The received data is stored in a database to ensure safety and redundancy. The input is the data stream sent from the device, and the output is the video data and location information stored in the database.

[1257] Step 5:

[1258] The server analyzes the video data using a machine learning model. Anomalies are detected for each video frame, and traffic accidents and road damage situations are identified. The input is the video data stored in the database, and the output is the identified anomaly information. Specifically, image features of abnormal areas are extracted and analyzed using an anomaly detection algorithm.

[1259] Step 6:

[1260] The server notifies the relevant authorities of detected anomalies. It sends the location, type, severity, and estimated repair time of the anomaly in real time via email or SMS. The input is the identified anomaly information, and the output is the notification sent to the relevant authorities.

[1261] Step 7:

[1262] The server analyzes traffic volume based on the video data it receives. The traffic volume data is used to determine the priority of road repairs. The input is the video data stored in the database, and the output is the analyzed traffic volume data. Specifically, it analyzes information such as the number of vehicles and passing times to calculate traffic volume.

[1263] Step 8:

[1264] The server uses an emotion engine to analyze the user's emotional state. It recognizes the user's emotions from video, audio, and operation patterns, and identifies feelings such as stress or impatience. The input is the user's video and audio data, and the output is the identified emotional state.

[1265] Step 9:

[1266] The server adjusts its response based on the user's emotional state. For example, if the user is feeling extremely stressed, it adjusts the notification content and priority of the response. The input is the identified emotional state, and the output is the adjusted response. Specifically, it performs actions such as sending a message to reassure the user.

[1267] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1268] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1269] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1270] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1271] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1272] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1273] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1274] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1275] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1276] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1277] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1278] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1279] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1280] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1281] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1282] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1283] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1284] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1285] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1286] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1287] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1288] The following is further disclosed regarding the above embodiment.

[1289] (Claim 1)

[1290] A means for collecting footage recorded by a dashcam;

[1291] means for transmitting the collected video data to a server;

[1292] A means for analyzing the transmitted video data and detecting traffic accidents and road damage situations;

[1293] A means for notifying relevant organizations of detected abnormalities;

[1294] A means of analyzing traffic volumes and determining priorities for road repairs;

[1295] A system including:

[1296] (Claim 2)

[1297] 10. The system of claim 1, further comprising means for acquiring location information along with the video and transmitting it to a server.

[1298] (Claim 3)

[1299] 2. The system of claim 1, further comprising means for notifying the user of the location, type, and severity of the anomaly, and an estimated time to repair the anomaly.

[1300] "Example 1"

[1301] (Claim 1)

[1302] A means for collecting footage recorded by a dashcam;

[1303] means for transmitting the collected video data to an external domain;

[1304] A means for analyzing the transmitted video data and detecting traffic accidents and road damage situations;

[1305] A means for notifying relevant organizations of detected abnormalities;

[1306] A means of analyzing traffic volumes and determining priorities for road repairs;

[1307] A means of storing video data and GPS data in a database in real time,

[1308] A means of detecting road damage and anomalies using machine learning models; and

[1309] A system including:

[1310] (Claim 2)

[1311] 10. The system according to claim 1, further comprising means for acquiring location information together with the video and transmitting the information to an external server.

[1312] (Claim 3)

[1313] 2. The system of claim 1, further comprising means for notifying the user of the location, type, and severity of the anomaly, and an estimated time to repair the anomaly.

[1314] "Application Example 1"

[1315] (Claim 1)

[1316] A means for collecting footage recorded by a dashcam;

[1317] means for transmitting the collected video data to a server;

[1318] A means for analyzing the transmitted video data and detecting traffic accidents and road damage situations;

[1319] A means for notifying relevant organizations of detected abnormalities;

[1320] A means of analyzing traffic volumes and determining priorities for road repairs;

[1321] Through an application installed in the autonomous vehicle, video footage and location information are collected and transmitted in real time while the vehicle is in motion.

[1322] A method for feeding back the anomaly detection results to the server and proposing safe avoidance routes to the autonomous driving vehicle.

[1323] A system including:

[1324] (Claim 2)

[1325] 10. The system of claim 1, further comprising means for acquiring location information along with the video and transmitting it to a server.

[1326] (Claim 3)

[1327] 2. The system of claim 1, further comprising means for notifying the user of the location, type, and severity of the anomaly, and an estimated time to repair the anomaly.

[1328] "Example 2: Combining Emotion Engines"

[1329] (Claim 1)

[1330] A means for collecting footage recorded by a dashcam;

[1331] means for transmitting the collected video data to a server;

[1332] A means for analyzing the transmitted video data and detecting traffic accidents and road damage situations;

[1333] A means for notifying relevant organizations of detected abnormalities;

[1334] A means of analyzing traffic volumes and determining priorities for road repairs;

[1335] means for analyzing the emotional state of a user;

[1336] A means for providing flexible support to users based on the analysis results;

[1337] A system including:

[1338] (Claim 2)

[1339] 10. The system of claim 1, further comprising means for acquiring location information along with the video and transmitting it to a server.

[1340] (Claim 3)

[1341] 2. The system of claim 1, further comprising means for notifying the user of the location, type, and severity of the anomaly, and an estimated time to repair the anomaly.

[1342] "Application example 2 when combining emotion engines"

[1343] (Claim 1)

[1344] A means for collecting footage recorded by a dashcam;

[1345] means for transmitting the collected video data to a server;

[1346] A means for analyzing the transmitted video data and detecting traffic accidents and road damage situations;

[1347] A means for notifying relevant organizations of detected abnormalities;

[1348] A means of analyzing traffic volumes and determining priorities for road repairs;

[1349] a means for analyzing the emotional state of a user;

[1350] a means of adjusting responses depending on emotional states;

[1351] A system including:

[1352] (Claim 2)

[1353] 10. The system of claim 1, further comprising means for acquiring location information along with the video and transmitting it to a server.

[1354] (Claim 3)

[1355] 2. The system of claim 1, further comprising means for notifying the user of the location, type, and severity of the anomaly, and an estimated time to repair the anomaly. [Explanation of symbols]

[1356] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for collecting footage recorded by a dashcam; means for transmitting the collected video data to a server; A means for analyzing the transmitted video data and detecting traffic accidents and road damage situations; A means for notifying relevant organizations of detected abnormalities; A means of analyzing traffic volumes and determining priorities for road repairs; A system including:

2. The system according to claim 1 , further comprising means for acquiring location information together with the image and transmitting the information to a server.

3. 2. The system of claim 1, further comprising means for notifying the user of the location, type, and severity of the anomaly, and an estimated time to repair the anomaly.

Images