System

Abstract

The system provided by the embodiment of the invention aims to properly deal with abnormal events in automatic driving. A system according to an embodiment includes an image generation unit, a data set addition unit, a presentation input unit, and a response determination unit. The image generation unit generates an image for an abnormal event. The data set addition unit adds the image generated by the image generation unit to the data set. The cue input unit is configured to input a traveling condition as a cue word. The response determination unit determines an appropriate response measure on the basis of the situation input by the presentation input unit.

Description

Technical Field

[0001] The technology disclosed herein relates to a system. Background Technology

[0002] Patent Document 1 discloses a personalized chatbot control method executed by at least one processor, the method comprising: receiving user speech; adding the user speech to a prompt containing instructions related to a chatbot role; encoding the prompt; and inputting the encoded prompt into a language model to generate chatbot speech in response to the user speech.

[0003] Patent document 1: Japanese Patent Application Publication No. 2022-180282. Summary of the Invention

[0004] In current technologies, autonomous driving is not adequately prepared to handle abnormal events, and there is room for improvement.

[0005] The system involved in this technical solution is designed to respond appropriately to abnormal events in autonomous driving.

[0006] The system involved in this technical solution includes an image generation unit, a dataset appending unit, a prompt input unit, and a response judgment unit. The image generation unit generates images for abnormal events. The dataset appending unit appends the images generated by the image generation unit to the dataset. The prompt input unit inputs the driving situation as a prompt word. The response judgment unit determines appropriate response measures based on the situation input by the prompt input unit.

[0007] The system involved in this technical solution can respond appropriately to abnormal events in autonomous driving. Attached Figure Description

[0008] Figure 1 This is a conceptual diagram illustrating an example of the configuration of a data processing system according to the first embodiment.

[0009] Figure 2 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and smart device according to the first embodiment.

[0010] Figure 3 This is a conceptual diagram illustrating an example of the data processing system configuration in the second embodiment.

[0011] Figure 4 This is a conceptual diagram illustrating an example of the main functions of the data processing device and smart glasses according to the second embodiment.

[0012] Figure 5 This is a conceptual diagram illustrating an example of the data processing system configuration in the third embodiment.

[0013] Figure 6 This is a conceptual diagram illustrating an example of the main functions of the data processing device and head-mounted terminal according to the third embodiment.

[0014] Figure 7 This is a conceptual diagram illustrating an example of the data processing system configuration in the fourth embodiment.

[0015] Figure 8 This is a conceptual diagram illustrating an example of the functions of the main parts of the data processing device and robot according to the fourth embodiment.

[0016] Figure 9 It represents an emotion graph that maps multiple emotions.

[0017] Figure 10 It represents an emotion graph that maps multiple emotions.

[0018] Explanation of reference numerals in the attached figures

[0019] Data processing systems 10, 210, 310, and 410

[0020] 12 Data processing devices

[0021] 14 Smart devices

[0022] 214 Smart Glasses

[0023] 314 Head-mounted terminal

[0024] 414 Robot. Detailed Implementation

[0025] Hereinafter, an example of an implementation of the system involved in this disclosure will be described with reference to the accompanying drawings.

[0026] First, let's explain the terms used in the following description.

[0027] In the following embodiments, the processor (hereinafter referred to as "processor"), as indicated by the reference numerals, can be a single computing device or a combination of multiple computing devices. Furthermore, a processor can be a single computing device or a combination of multiple computing devices. Examples of computing devices include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), or TPU (Tensor Processing Unit), etc.

[0028] In the following embodiments, RAM (Random Access Memory), as indicated in the figures, is a memory for temporary information storage that is used by the processor as working memory.

[0029] In the following embodiments, the memory, as indicated by the reference numerals, is one or more non-volatile storage devices used to store various programs and parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), disks (e.g., hard disks), or magnetic tapes, etc.

[0030] In the following embodiments, the communication I / F (Interface) with reference numerals is an interface including a communication processor and an antenna, etc. The communication I / F is responsible for communication between multiple computers. Examples of communication standards applicable to the communication I / F include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0031] In the following implementation, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. Furthermore, in this specification, when "and / or" connects more than three items, the same approach as "A and / or B" applies.

[0032] First Implementation Method

[0033] Figure 1 An example of the configuration of the data processing system 10 according to the first embodiment is shown.

[0034] like Figure 1 As shown, the data processing system 10 includes a data processing device 12 and an intelligent device 14. An example of the data processing device 12 is a server.

[0035] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. An example of the network 54 includes a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0036] The smart device 14 includes a computer 36, a receiver 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. In addition, the receiver 38, output device 40, and camera 42 are also connected to the bus 52.

[0037] The receiving device 38 includes a touchscreen 38A and a microphone 38B, etc., for receiving user input. The touchscreen 38A receives user input generated by contact with an indicator (e.g., a pen or finger). The microphone 38B receives user input generated by sound by detecting the user's voice. The control unit 46A sends data representing user input received via the touchscreen 38A and microphone 38B to the data processing unit 12. In the data processing unit 12, a specific processing unit 290 (see reference...) Figure 2 Get the data that represents the user input.

[0038] The output device 40 includes a display 40A and a speaker 40B, etc., and presents data to the user by outputting data in a user-perceptible form (e.g., sound and / or text). The display 40A displays visual information such as text and images according to instructions from the processor 46. The speaker 40B outputs sound according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0039] Communication I / F 44 is connected to network 54. Communication I / F 44 and 26 are responsible for sending and receiving various information between processor 46 and processor 28 via network 54.

[0040] Figure 2 An example of the main functions of the data processing device 12 and the smart device 14 is shown.

[0041] like Figure 2 As shown, in the data processing apparatus 12, specific processing is performed by the processor 28. A specific processing program 56 is stored in the memory 32. The specific processing program 56 is an example of a "program" as understood in this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 based on the specific processing program 56 executed on the RAM 30.

[0042] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. The emotion inference function using the emotion-specific model 59 (emotion-specific function) includes inferring and predicting the user's emotions, performing various inferences and predictions related to the user's emotions, but is not limited to this example. Furthermore, emotion inference and prediction also include, for example, emotion analysis (analysis).

[0043] In the smart device 14, specific processing is performed by the processor 46. A specific processing program 60 is stored in the memory 50. The specific processing program 60 is used in conjunction with the data processing system 10. The processor 46 reads the specific processing program 60 from the memory 50 and executes the read specific processing program 60 on the RAM 48. Specific processing is implemented by the processor 46 operating as a control unit 46A based on the specific processing program 60 executed on the RAM 48. Furthermore, the smart device 14 may also have the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and use these models to perform the same processing as the specific processing unit 290.

[0044] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains the processing results (prediction results, etc.) using the data generation model 58 by communicating with the server device that has the data generation model 58. Furthermore, the data processing device 12 may be a server device or a user-owned terminal device (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of the processing performed by the data processing system 10 of the first embodiment will be described.

[0045] Implementation Method 1

[0046] This invention relates to an autonomous driving system and proposes a method to improve the recognition accuracy of AI-powered autonomous driving. While autonomous driving systems can handle driving under normal conditions, their accuracy in recognizing abnormal situations is insufficient. To address this issue, two methods utilizing generative AI are proposed. First, a method for generating images of abnormal events. Generative AI is used to generate images of abnormal situations not typically included in the dataset. For example, generative AI can reproduce situations such as an animal suddenly running out on the road or driving conditions under abnormal weather. In this way, AI can handle abnormal situations it has not previously learned. Second, a method of inputting situations as prompts during driving. Using speech recognition technology, abnormal situations occurring during driving are input as prompts to the generative AI in real time. For example, by inputting "There is an obstacle ahead" via voice, the generative AI determines the appropriate response based on this situation. This allows for rapid response to abnormal situations during driving. These methods can improve the recognition accuracy of AI-powered autonomous driving, achieving a world of almost error-free autonomous driving. Specifically, the method includes the following steps: First, generating images of abnormal events using generative AI; second, appending the generated images to the dataset for AI to learn from. This enables AI to handle abnormal situations. Furthermore, by utilizing speech recognition technology, abnormal situations occurring during driving are input as prompts into the AI ​​generator, which then determines appropriate responses based on these prompts. For example, by inputting "An animal is running out in front" while driving, the AI ​​generator can determine appropriate responses to assist driving. Additionally, by appending images of abnormal situations generated by the AI ​​generator to the dataset, the AI ​​can handle situations it has not previously learned to handle. This mechanism improves the recognition accuracy of AI-powered autonomous driving, achieving a world of virtually error-free autonomous driving. For example, it can handle abnormal situations that AI previously could not handle, such as animals suddenly running out on the road or driving under abnormal weather conditions. This enhances the safety of autonomous driving, allowing more people to use it with peace of mind. Through these methods, autonomous driving systems can improve the recognition accuracy of AI-powered autonomous driving, achieving a world of virtually error-free autonomous driving.

[0047] The autonomous driving system according to this embodiment includes an image generation unit, a dataset addition unit, a prompt input unit, and a response judgment unit. The image generation unit generates images for abnormal events. For example, the image generation unit can use a generation AI to generate images that reproduce abnormal situations not normally included in the dataset. For example, the image generation unit can generate images that reproduce a situation where an animal suddenly runs out on the road. Furthermore, the image generation unit can also generate images that reproduce driving conditions under abnormal weather conditions. For example, the image generation unit can generate images that reproduce driving conditions in heavy rain or snow. Further, the image generation unit can also use a generation AI to generate images that reproduce driving conditions at night or in foggy weather. The dataset addition unit adds the images generated by the image generation unit to the dataset. For example, the dataset addition unit integrates the generated images into an existing dataset, allowing the AI ​​to learn. For example, the dataset addition unit adds the generated images to the dataset, enabling the AI ​​to respond to abnormal situations. In addition, the dataset addition unit can also classify the generated images and add them to appropriate categories. For example, the dataset addition unit can classify the generated images into categories such as traffic accidents, natural disasters, and mechanical failures. The prompt input unit is used to input the driving situation as prompt words. The prompt input unit utilizes speech recognition technology to input abnormal situations occurring during driving as prompts into the generating AI in real time. For example, the prompt input unit can input "There is an obstacle ahead" or similar situations via voice while driving. Furthermore, it can input "The road is slippery" or similar situations via voice while driving. Further, it can input "Poor visibility" or similar situations via voice while driving. The response judgment unit determines appropriate response measures based on the situation input by the prompt input unit. The response judgment unit, for example, utilizes the generating AI to determine appropriate response measures based on the input situation. For example, when there is an obstacle ahead, the response judgment unit instructs the user to take evasive action. Furthermore, when the road is slippery, the response judgment unit can instruct the user to slow down. Further, when visibility is poor, the response judgment unit can instruct the user to turn on the headlights. Therefore, the autonomous driving system according to this embodiment can realize image generation of abnormal events, dataset addition, prompt input, and response judgment.

[0048] The image generation unit generates images for abnormal events. For example, it can utilize a generative AI to generate images that reproduce abnormal situations not typically included in the dataset. Specifically, the generative AI generates images based on a large, pre-learned dataset and specific prompts. For instance, when generating an image reproducing a situation where an animal suddenly runs out onto the road, the AI ​​considers the animal's species, movement, road conditions, etc., to generate a realistic image. Furthermore, it can generate images reproducing driving conditions under abnormal weather conditions. For example, when generating images reproducing driving conditions in heavy rain or snow, the AI ​​details the pattern of rain and snowfall, poor visibility, and slippery roads. Further, when generating images reproducing driving conditions at night or in fog, the AI ​​considers factors such as light reflection, limited visibility, and fog density to generate a realistic image. Thus, the image generation unit can generate images that reproduce diverse abnormal situations not typically included in the dataset, which can be used as learning data for autonomous driving systems.

[0049] The dataset appending unit adds images generated by the image generation unit to the dataset. For example, it integrates the generated images into existing datasets for AI learning. Specifically, this involves converting the generated images to an appropriate format before appending them to the existing dataset. For instance, generated images are categorized into traffic accidents, natural disasters, and mechanical failures, and appended to their respective datasets. Furthermore, the dataset appending unit manages the metadata of the generated images, recording the generation time, generation conditions, and cue words. This allows for efficient management of learning data used by the AI ​​to handle abnormal situations. Moreover, the dataset appending unit can evaluate the quality of the generated images and filter or correct them as necessary. For example, if a generated image is blurry or contains erroneous information, it is appropriately corrected to ensure quality before being appended to the dataset. Thus, the dataset appending unit effectively integrates generated images into the dataset, improving the learning accuracy of the AI.

[0050] The prompt input unit is used to input driving conditions as prompts. For example, it utilizes speech recognition technology to input abnormal driving conditions as prompts into the generating AI in real time. Specifically, when the driver reports a situation via voice, the speech recognition system converts the content into text and inputs it into the generating AI. For instance, when the driver inputs "obstacle ahead" by voice, the speech recognition system accurately recognizes the content and provides it as a prompt to the generating AI. Similarly, when the driver inputs "slippery road" by voice, the speech recognition system also converts the content into text and inputs it into the generating AI. Furthermore, when the driver inputs "poor visibility" by voice, the speech recognition system can also accurately recognize the situation and provide it as a prompt to the generating AI. Thus, the prompt input unit can input abnormal driving conditions into the generating AI in real time, providing the information needed to determine appropriate countermeasures.

[0051] The response judgment unit determines appropriate responses based on the conditions input by the prompt input unit. For example, the response judgment unit utilizes generative AI to determine appropriate responses based on the input prompts. Specifically, the generative AI calculates the optimal response based on the input prompts, referencing past data and learning results. For instance, when there is an obstacle ahead, the generative AI instructs an evasive maneuver based on past data. Furthermore, when the road is slippery, the generative AI may instruct for deceleration. Further, when visibility is poor, the generative AI may instruct for turning on the headlights. Thus, the response judgment unit can determine appropriate responses in real time based on the information provided by the prompt input unit and issue instructions to the autonomous driving system. In addition, the response judgment unit can handle situations where multiple prompts are input simultaneously. For example, when there is an obstacle ahead and the road is slippery, the generative AI will consider both situations to determine the optimal response. Furthermore, the response judgment unit can use past response results as feedback to continuously learn and improve response accuracy. Therefore, the response judgment unit can always provide highly accurate response judgments based on the latest information, improving the safety and reliability of the autonomous driving system.

[0052] The system includes a generative AI unit that generates images using generative AI. For example, the generative AI unit can generate images that reproduce unusual situations not typically included in the dataset. For instance, it can generate images reproducing a situation where an animal suddenly runs out onto the road. Furthermore, it can generate images reproducing driving conditions under abnormal weather conditions. For example, it can generate images reproducing driving conditions in heavy rain or snow. Moreover, it can generate images reproducing driving conditions at night or in foggy weather. Thus, using generative AI can improve the accuracy of image generation. Some or all of the above-described processes in the generative AI unit may be performed using generative AI, or they may not. For example, the generative AI unit can append images generated using generative AI to the dataset for the AI ​​to learn from.

[0053] This includes a voice recognition unit that uses voice recognition technology to input conditions. For example, the voice recognition unit can use voice recognition technology to input abnormal situations occurring while driving as prompts into the generating AI in real time. For instance, the voice recognition unit can input "There is an obstacle ahead" or similar conditions while driving. Furthermore, it can input "The road is slippery" or similar conditions while driving. Even further, it can input "Visibility is poor" or similar conditions while driving. Thus, the accuracy of condition input can be improved using voice recognition technology. Some or all of the above processing in the voice recognition unit can be performed using AI, or it can be performed without AI. For example, the voice recognition unit can input the conditions using voice recognition technology into the generating AI, which then determines the appropriate response based on the conditions.

[0054] The dataset management department, for example, adds the generated images to the dataset. This department can integrate the generated images into existing datasets, allowing AI to learn. For instance, adding generated images to the dataset enables AI to handle abnormal situations. Furthermore, the dataset management department can classify the generated images and add them to appropriate categories. For example, it can classify generated images into categories such as traffic accidents, natural disasters, and mechanical failures. Thus, utilizing the dataset management department can improve the efficiency of dataset management. Some or all of the above processes in the dataset management department can be performed using AI, or they can be performed without AI. For example, the dataset management department can input the generated images into AI, which will then classify and add them to the dataset.

[0055] This includes a response judgment AI unit that determines appropriate response measures based on the driving situation. The response judgment AI unit can, for example, utilize generated AI to determine appropriate response measures based on the input conditions. For instance, when there is an obstacle ahead, the response judgment AI unit instructs the driver to take evasive action. Furthermore, when the road is slippery, the response judgment AI unit can instruct the driver to slow down. Moreover, when visibility is poor, the response judgment AI unit can instruct the driver to turn on the headlights. Thus, the response judgment AI unit can quickly determine appropriate response measures. Some or all of the above processing in the response judgment AI unit can be performed using AI, or it can be performed without AI. For example, the response judgment AI unit can utilize generated AI to determine appropriate response measures based on the input conditions and instruct the driver on the result.

[0056] The image generation unit is capable of generating images that reproduce abnormal situations not typically included in the dataset. For example, the image generation unit can utilize generative AI to generate images that reproduce abnormal situations not typically included in the dataset. For instance, the image generation unit can generate images that reproduce a situation where an animal suddenly runs out onto the road. Furthermore, the image generation unit can generate images that reproduce driving conditions under abnormal weather conditions. For example, the image generation unit can generate images that reproduce driving conditions in heavy rain or snow. Further, the image generation unit can also utilize generative AI to generate images that reproduce driving conditions at night or in foggy weather. Thus, by generating images that reproduce abnormal situations, the recognition accuracy of the AI ​​can be improved. Some or all of the above-described processes in the image generation unit can be performed using generative AI, or they can be performed without using generative AI. For example, the image generation unit can append images generated using generative AI to the dataset, allowing the AI ​​to learn.

[0057] The prompt input unit utilizes speech recognition technology to input abnormal situations occurring while driving as prompts into the generating AI in real time. For example, the prompt input unit can input "Obstacle ahead" via voice while driving. Furthermore, it can input "Slippery road" via voice while driving. Further, it can input "Poor visibility" via voice while driving. Thus, by inputting prompts in real time, rapid response can be achieved. Some or all of the above processing in the prompt input unit can be performed using the generating AI, or it can be performed without it. For example, the prompt input unit can input the situation using speech recognition technology into the generating AI, which then determines the appropriate response based on the situation.

[0058] The response judgment unit can determine appropriate response measures based on the conditions input by the generated AI. For example, the response judgment unit can utilize the generated AI to determine appropriate response measures based on the input conditions. For instance, when there is an obstacle ahead, the response judgment unit instructs the driver to take evasive action. Furthermore, when the road is slippery, the response judgment unit can instruct the driver to slow down. Further, when visibility is poor, the response judgment unit can instruct the driver to turn on the headlights. Thus, the generated AI can be used to quickly determine appropriate response measures. Some or all of the above processing in the response judgment unit can be performed using the generated AI, or it can be performed without using the generated AI. For example, the response judgment unit can use the generated AI to determine appropriate response measures based on the input conditions and instruct the driver on the result.

[0059] The image generation unit can reproduce more diverse anomalous events by incorporating visual information from different viewpoints and angles into the generated images. For example, the image generation unit can utilize generative AI to generate images containing visual information from different viewpoints and angles. For instance, the image generation unit can generate images reproducing traffic congestion from an overhead view of a road. Furthermore, the image generation unit can generate images reproducing parking lot congestion from a vehicle side view. Further, the image generation unit can generate images reproducing crosswalk congestion from a pedestrian's perspective. Thus, by incorporating visual information from different viewpoints and angles, more diverse anomalous events can be reproduced. Some or all of the above processing in the image generation unit can be performed using generative AI, or it can be performed without generative AI. For example, the image generation unit can append images generated using generative AI to a dataset for the AI ​​to learn from.

[0060] The image generation unit can provide more realistic scenes by reflecting changes in different time periods and seasons in the generated images. For example, the image generation unit can utilize generative AI to generate images that reflect changes in different time periods and seasons. For instance, the image generation unit can generate images that reproduce nighttime road conditions. Furthermore, the image generation unit can generate images that reproduce winter snow conditions. Further, the image generation unit can generate images that reproduce autumn road conditions with fallen leaves scattered on them. Thus, by reflecting changes in different time periods and seasons, a more realistic scene can be provided. Some or all of the above processing in the image generation unit can be performed using generative AI, or it can be performed without generative AI. For example, the image generation unit can append images generated using generative AI to a dataset for the AI ​​to learn from.

[0061] The image generation unit can provide more diverse scenarios by reflecting different traffic conditions and road conditions in the generated images. For example, the image generation unit can utilize generative AI to generate images reflecting different traffic conditions and road conditions. For instance, the image generation unit can generate images reproducing congested road conditions. Furthermore, the image generation unit can generate images reproducing road construction conditions. Further, the image generation unit can generate images reproducing road conditions at accident scenes. Thus, by reflecting different traffic conditions and road conditions, more diverse scenarios can be provided. Some or all of the above processing in the image generation unit can be performed using generative AI, or it can be performed without generative AI. For example, the image generation unit can append images generated using generative AI to a dataset, allowing the AI ​​to learn.

[0062] The image generation unit can provide more realistic scenes by reflecting different weather conditions in the generated images. For example, the image generation unit can utilize generative AI to generate images that reflect different weather conditions. For instance, the image generation unit can generate images that reproduce road conditions on rainy days. Furthermore, the image generation unit can generate images that reproduce road conditions on snowy days. Further, the image generation unit can generate images that reproduce road conditions on foggy days. Thus, by reflecting different weather conditions, a more realistic scene can be provided. Some or all of the above processing in the image generation unit can be performed using generative AI, or it can be performed without generative AI. For example, the image generation unit can append images generated using generative AI to a dataset, allowing the AI ​​to learn.

[0063] The dataset appending unit is capable of automatically generating and managing image metadata when appending data to a dataset. For example, it can automatically generate the image's capture time and location as metadata and append it to the dataset. Furthermore, it can automatically generate tags related to the image content and append them to the dataset. Moreover, it can automatically generate image resolution and format information as metadata and append it to the dataset. Thus, by automatically generating and managing image metadata, the efficiency of dataset management can be improved. Some or all of the above processing in the dataset appending unit can be performed using AI, or it can be performed without AI. For example, the dataset appending unit can input image metadata into AI, which will then generate metadata and append it to the dataset.

[0064] The dataset addition unit is capable of evaluating image quality and automatically excluding low-quality images when adding them to the dataset. For example, the dataset addition unit automatically excludes images with low resolution. Furthermore, it can automatically exclude images with high noise levels. Moreover, it can automatically exclude blurry images. Thus, by automatically excluding low-quality images, the quality of the dataset can be improved. Some or all of the above processing in the dataset addition unit may be performed using AI, or it may not. For example, the dataset addition unit can input image quality data into AI, allowing AI to evaluate the quality and exclude low-quality images.

[0065] The dataset addition unit is capable of evaluating image relevance and prioritizing the addition of highly relevant images when adding data to the dataset. For example, the dataset addition unit prioritizes adding images whose content is relevant to the current driving situation. Furthermore, it can also prioritize adding images whose content is relevant to past driving records. Moreover, it can prioritize adding images whose content is relevant to a specific driving scenario. Thus, by prioritizing the addition of highly relevant images, the quality of the dataset can be improved. Some or all of the above processing in the dataset addition unit may be performed using AI, or it may not. For example, the dataset addition unit can input image relevance into AI, which will evaluate the relevance and select highly relevant images to add to the dataset.

[0066] The dataset addition unit is capable of evaluating image diversity and prioritizing the addition of images with high diversity when adding data to the dataset. For example, the dataset addition unit may prioritize adding images taken from different viewpoints and angles. Furthermore, it may prioritize adding images reflecting changes over different time periods and seasons. Moreover, it may prioritize adding images reflecting different traffic conditions and road conditions. Thus, by prioritizing the addition of highly diverse images, the quality of the dataset can be improved. Some or all of the above processing in the dataset addition unit may be performed using AI, or it may not. For example, the dataset addition unit may input image diversity into AI, which will evaluate the diversity and select images with high diversity to add to the dataset.

[0067] The prompt input unit can generate optimal prompts by referencing the driver's past driving records when prompts are entered. For example, the prompt input unit prioritizes prompts used in similar situations in the past. Furthermore, the prompt input unit can also provide prompts used in specific time periods or locations based on past driving records. Further, the prompt input unit can analyze past driving records to provide the most effective prompts. Thus, by referencing past driving records, optimal prompts can be provided. Some or all of the above processing in the prompt input unit can be performed using AI, or it can be performed without AI. For example, the prompt input unit can input the driver's past driving records into AI, which analyzes the records and generates optimal prompts.

[0068] The prompt input unit can analyze the driver's current driving status in real time when prompt words are input, and generate the optimal prompt words. For example, the prompt input unit analyzes the current traffic conditions in real time and provides appropriate prompt words. Furthermore, the prompt input unit can also analyze the current weather conditions in real time and provide appropriate prompt words. Further, the prompt input unit can also analyze the current road conditions in real time and provide appropriate prompt words. Thus, by analyzing the current driving status in real time, the optimal prompt words can be provided. Some or all of the above processing in the prompt input unit can be performed using AI, or it can be performed without AI. For example, the prompt input unit can input the driver's current driving status into the AI, which will analyze the situation in real time and generate the optimal prompt words.

[0069] The prompt input unit can generate optimal prompts by considering the driver's geographical location information when prompts are input. For example, the prompt input unit can provide appropriate prompts based on the current traffic conditions. Furthermore, it can provide appropriate prompts based on the current weather conditions. Even further, it can provide appropriate prompts based on the current road conditions. Thus, by considering geographical location information, optimal prompts can be provided. Some or all of the above processing in the prompt input unit can be performed using AI, or it can be performed without AI. For example, the prompt input unit can input the driver's geographical location information into AI, which can then generate optimal prompts based on that information.

[0070] The prompt input unit can analyze the driver's social media activity and generate relevant prompts when prompts are entered. For example, the prompt input unit can provide appropriate prompts based on information shared by the driver on social media. Furthermore, the prompt input unit can analyze the driver's activity history on social media to provide relevant prompts. Further, the prompt input unit can also provide appropriate prompts based on the accounts the driver follows on social media. Thus, relevant prompts can be provided by analyzing social media activity. Some or all of the above processing in the prompt input unit can be performed using AI, or it can be performed without AI. For example, the prompt input unit can input the driver's social media activity into AI, which can then analyze the activity and generate relevant prompts.

[0071] The response judgment unit can select the optimal response by referring to past response records when determining response measures. For example, the response judgment unit may prioritize response measures taken in similar situations in the past. Furthermore, the response judgment unit can also select response measures taken in specific time periods or locations based on past response records. Further, the response judgment unit can analyze past response records to select the most effective response measures. Thus, by referring to past response records, the optimal response can be provided. Some or all of the above processing in the response judgment unit may be performed using AI, or it may not. For example, the response judgment unit can input past response records into AI, which will analyze the records and select the optimal response.

[0072] The response judgment unit can analyze the current driving situation in real time and select the optimal response when determining the appropriate action. For example, the response judgment unit can analyze the current traffic situation in real time and select appropriate actions. Furthermore, it can analyze the current weather conditions in real time and select appropriate actions. Moreover, it can analyze the current road conditions in real time and select appropriate actions. Thus, by analyzing the current driving situation in real time, the optimal response can be provided. Some or all of the above processing in the response judgment unit can be performed using AI, or it can be performed without AI. For example, the response judgment unit can input the current driving situation into AI, which can then analyze the situation in real time and select the optimal response.

[0073] The response judgment unit can select the optimal response by considering the driver's geographical location information when determining the response measure. For example, the response judgment unit selects an appropriate response measure based on the traffic conditions at the current location. Furthermore, the response judgment unit can also select an appropriate response measure based on the weather conditions at the current location. Further, the response judgment unit can also select an appropriate response measure based on the road conditions at the current location. Thus, by considering geographical location information, an optimal response can be provided. Some or all of the above processing in the response judgment unit can be performed using AI, or it can be performed without AI. For example, the response judgment unit can input the driver's geographical location information into the AI, and the AI ​​can select the optimal response based on that information.

[0074] The response judgment unit can analyze the driver's social media activity and select relevant responses when determining the appropriate action. For example, the unit can select appropriate responses based on information shared by the driver on social media. Furthermore, it can analyze the driver's activity history on social media and select relevant responses. Moreover, it can select appropriate responses based on the accounts the driver follows on social media. Thus, relevant responses can be provided by analyzing social media activity. Some or all of the above processing in the response judgment unit can be performed using AI, or it can be performed without AI. For example, the response judgment unit can input the driver's social media activity into AI, which can then analyze the activity and generate relevant responses.

[0075] The system involved in this embodiment is not limited to the above examples. For example, various modifications can be made as follows.

[0076] Autonomous driving systems can analyze a driver's past driving records and provide optimal driving assistance based on the driver's driving style. For example, for drivers who frequently brake suddenly, the system can provide deceleration assistance in advance. For drivers who frequently drive on highways, it can provide assistance to maintain the optimal following distance on highways. Furthermore, for drivers who frequently drive at night, it can provide assistance to improve nighttime visibility. Thus, by providing assistance tailored to the driver's driving style, driving safety and comfort can be improved. Driving record analysis can be performed using AI, or not. For example, driving record data can be input into AI, which can analyze the data and generate optimal driving assistance.

[0077] Autonomous driving systems can reference a driver's past driving records and propose optimal routes based on the driver's driving patterns. For example, for drivers who frequently use specific routes, those routes are prioritized. For drivers who tend to avoid congestion, routes that avoid congestion are recommended. Furthermore, for drivers who prefer highways, highway routes can also be recommended. Thus, by recommending optimal routes that match the driver's driving patterns, driving efficiency and comfort can be improved. Driving record referencing can be performed using AI, or not. For example, driving record data can be input into AI, which analyzes the data and generates the optimal route.

[0078] Autonomous driving systems can analyze a driver's current driving status in real time to provide optimal driving assistance. For example, real-time analysis of current traffic conditions can provide assistance in avoiding congestion. Furthermore, real-time analysis of current weather conditions can provide driving assistance in adverse weather conditions. Further, real-time analysis of current road conditions can provide assistance in avoiding road construction or accidents. Thus, by analyzing the current driving status in real time, optimal driving assistance can be provided. Real-time analysis of driving status can be performed using AI, or not. For example, driving status data can be input into AI, which can analyze the data in real time to generate optimal driving assistance.

[0079] Autonomous driving systems can take into account the driver's location information to provide optimal driving assistance. For example, based on the current traffic conditions, they can provide assistance to avoid congestion. Furthermore, based on the current weather conditions, they can provide driving assistance in adverse weather conditions. Further, based on the current road conditions, they can provide assistance to avoid road construction or accidents. Thus, by considering location information, optimal driving assistance can be provided. The consideration of location information can be done using AI, or not. For example, location information data can be input into AI, which can analyze the data and generate optimal driving assistance.

[0080] Autonomous driving systems can analyze a driver's social media activity and provide optimal driving assistance based on the driver's interests and concerns. For example, routes to locations of interest can be recommended based on information shared by the driver on social media. Furthermore, the system can analyze the driver's social media activity history to provide relevant driving assistance. Moreover, routes to activities or locations of interest can be recommended based on the accounts the driver follows on social media. Thus, by analyzing social media activity, driving assistance based on the driver's interests and concerns can be provided. Social media activity analysis can be performed using AI, or not. For example, social media activity data can be input into AI, which can then analyze the data to generate optimal driving assistance.

[0081] The following is a brief description of the processing flow of Implementation Method 1.

[0082] Step 1: The image generation unit generates images for the abnormal events. For example, it uses generative AI to generate images that reproduce abnormal situations not typically included in the dataset. Specifically, it generates images that reproduce situations such as an animal suddenly running out on the road or driving under abnormal weather conditions (heavy rain, snow, night, fog).

[0083] Step 2: The dataset appending unit appends the images generated by the image generation unit to the dataset. For example, the generated images are integrated into the existing dataset to allow the AI ​​to learn. Furthermore, the generated images are categorized and appended to appropriate categories such as traffic accidents, natural disasters, and mechanical failures.

[0084] Step 3: The input unit inputs the driving conditions as prompts. For example, using voice recognition technology, abnormal situations occurring while driving are input as prompts into the AI ​​in real time. Specifically, prompts can be input via voice such as "Obstacle ahead," "Slippery road," or "Poor visibility."

[0085] Step 4: The response judgment unit determines the appropriate response based on the situation input by the prompt input unit. For example, using generated AI, the appropriate response is determined based on the input situation, such as instructing to take evasive action when there is an obstacle ahead, instructing to slow down when the road is slippery, and instructing to turn on the headlights when visibility is poor.

[0086] Implementation Method 2

[0087] This invention relates to an autonomous driving system and proposes a method to improve the recognition accuracy of AI-powered autonomous driving. While autonomous driving systems can handle driving under normal conditions, their accuracy in recognizing abnormal situations is insufficient. To address this issue, two methods utilizing generative AI are proposed. First, a method for generating images of abnormal events. Generative AI is used to generate images of abnormal situations not typically included in the dataset. For example, generative AI can reproduce situations such as an animal suddenly running out on the road or driving conditions under abnormal weather. In this way, AI can handle abnormal situations it has not previously learned. Second, a method of inputting situations as prompts during driving. Using speech recognition technology, abnormal situations occurring during driving are input as prompts to the generative AI in real time. For example, by inputting "There is an obstacle ahead" via voice, the generative AI determines the appropriate response based on this situation. This allows for rapid response to abnormal situations during driving. These methods can improve the recognition accuracy of AI-powered autonomous driving, achieving a world of almost error-free autonomous driving. Specifically, the method includes the following steps: First, generating images of abnormal events using generative AI; second, appending the generated images to the dataset for AI to learn from. This enables AI to handle abnormal situations. Furthermore, by utilizing speech recognition technology, abnormal situations occurring during driving are input as prompts into the AI ​​generator, which then determines appropriate responses based on these prompts. For example, by inputting "An animal is running out in front" while driving, the AI ​​generator can determine appropriate responses to assist driving. Additionally, by appending images of abnormal situations generated by the AI ​​generator to the dataset, the AI ​​can handle situations it has not previously learned to handle. This mechanism improves the recognition accuracy of AI-powered autonomous driving, achieving a world of virtually error-free autonomous driving. For example, it can handle abnormal situations that AI previously could not handle, such as animals suddenly running out on the road or driving under abnormal weather conditions. This enhances the safety of autonomous driving, allowing more people to use it with peace of mind. Through these methods, autonomous driving systems can improve the recognition accuracy of AI-powered autonomous driving, achieving a world of virtually error-free autonomous driving.

[0088] The autonomous driving system according to this embodiment includes an image generation unit, a dataset addition unit, a prompt input unit, and a response judgment unit. The image generation unit generates images for abnormal events. For example, the image generation unit can use a generation AI to generate images that reproduce abnormal situations not normally included in the dataset. For example, the image generation unit can generate images that reproduce a situation where an animal suddenly runs out on the road. Furthermore, the image generation unit can also generate images that reproduce driving conditions under abnormal weather conditions. For example, the image generation unit can generate images that reproduce driving conditions in heavy rain or snow. Further, the image generation unit can also use a generation AI to generate images that reproduce driving conditions at night or in foggy weather. The dataset addition unit adds the images generated by the image generation unit to the dataset. For example, the dataset addition unit integrates the generated images into an existing dataset, allowing the AI ​​to learn. For example, the dataset addition unit adds the generated images to the dataset, enabling the AI ​​to respond to abnormal situations. In addition, the dataset addition unit can also classify the generated images and add them to appropriate categories. For example, the dataset addition unit can classify the generated images into categories such as traffic accidents, natural disasters, and mechanical failures. The prompt input unit is used to input the driving situation as prompt words. The prompt input unit utilizes speech recognition technology to input abnormal situations occurring during driving as prompts into the generating AI in real time. For example, the prompt input unit can input "There is an obstacle ahead" or similar situations via voice while driving. Furthermore, it can input "The road is slippery" or similar situations via voice while driving. Further, it can input "Poor visibility" or similar situations via voice while driving. The response judgment unit determines appropriate response measures based on the situation input by the prompt input unit. The response judgment unit, for example, utilizes the generating AI to determine appropriate response measures based on the input situation. For example, when there is an obstacle ahead, the response judgment unit instructs the user to take evasive action. Furthermore, when the road is slippery, the response judgment unit can instruct the user to slow down. Further, when visibility is poor, the response judgment unit can instruct the user to turn on the headlights. Therefore, the autonomous driving system according to this embodiment can realize image generation of abnormal events, dataset addition, prompt input, and response judgment.

[0089] The image generation unit generates images for abnormal events. For example, it can utilize a generative AI to generate images that reproduce abnormal situations not typically included in the dataset. Specifically, the generative AI generates images based on a large, pre-learned dataset and specific prompts. For instance, when generating an image reproducing a situation where an animal suddenly runs out onto the road, the AI ​​considers the animal's species, movement, road conditions, etc., to generate a realistic image. Furthermore, it can generate images reproducing driving conditions under abnormal weather conditions. For example, when generating images reproducing driving conditions in heavy rain or snow, the AI ​​details the pattern of rain and snowfall, poor visibility, and slippery roads. Further, when generating images reproducing driving conditions at night or in fog, the AI ​​considers factors such as light reflection, limited visibility, and fog density to generate a realistic image. Thus, the image generation unit can generate images that reproduce diverse abnormal situations not typically included in the dataset, which can be used as learning data for autonomous driving systems.

[0090] The dataset appending unit adds images generated by the image generation unit to the dataset. For example, it integrates the generated images into existing datasets for AI learning. Specifically, this involves converting the generated images to an appropriate format before appending them to the existing dataset. For instance, generated images are categorized into traffic accidents, natural disasters, and mechanical failures, and appended to their respective datasets. Furthermore, the dataset appending unit manages the metadata of the generated images, recording the generation time, generation conditions, and cue words. This allows for efficient management of learning data used by the AI ​​to handle abnormal situations. Moreover, the dataset appending unit can evaluate the quality of the generated images and filter or correct them as necessary. For example, if a generated image is blurry or contains erroneous information, it is appropriately corrected to ensure quality before being appended to the dataset. Thus, the dataset appending unit effectively integrates generated images into the dataset, improving the learning accuracy of the AI.

[0091] The prompt input unit is used to input driving conditions as prompts. For example, it utilizes speech recognition technology to input abnormal driving conditions as prompts into the generating AI in real time. Specifically, when the driver reports a situation via voice, the speech recognition system converts the content into text and inputs it into the generating AI. For instance, when the driver inputs "obstacle ahead" by voice, the speech recognition system accurately recognizes the content and provides it as a prompt to the generating AI. Similarly, when the driver inputs "slippery road" by voice, the speech recognition system also converts the content into text and inputs it into the generating AI. Furthermore, when the driver inputs "poor visibility" by voice, the speech recognition system can also accurately recognize the situation and provide it as a prompt to the generating AI. Thus, the prompt input unit can input abnormal driving conditions into the generating AI in real time, providing the information needed to determine appropriate countermeasures.

[0092] The response judgment unit determines appropriate responses based on the conditions input by the prompt input unit. For example, the response judgment unit utilizes generative AI to determine appropriate responses based on the input prompts. Specifically, the generative AI calculates the optimal response based on the input prompts, referencing past data and learning results. For instance, when there is an obstacle ahead, the generative AI instructs an evasive maneuver based on past data. Furthermore, when the road is slippery, the generative AI may instruct for deceleration. Further, when visibility is poor, the generative AI may instruct for turning on the headlights. Thus, the response judgment unit can determine appropriate responses in real time based on the information provided by the prompt input unit and issue instructions to the autonomous driving system. In addition, the response judgment unit can handle situations where multiple prompts are input simultaneously. For example, when there is an obstacle ahead and the road is slippery, the generative AI will consider both situations to determine the optimal response. Furthermore, the response judgment unit can use past response results as feedback to continuously learn and improve response accuracy. Therefore, the response judgment unit can always provide highly accurate response judgments based on the latest information, improving the safety and reliability of the autonomous driving system.

[0093] The system includes a generative AI unit that generates images using generative AI. For example, the generative AI unit can generate images that reproduce unusual situations not typically included in the dataset. For instance, it can generate images reproducing a situation where an animal suddenly runs out onto the road. Furthermore, it can generate images reproducing driving conditions under abnormal weather conditions. For example, it can generate images reproducing driving conditions in heavy rain or snow. Moreover, it can generate images reproducing driving conditions at night or in foggy weather. Thus, using generative AI can improve the accuracy of image generation. Some or all of the above-described processes in the generative AI unit may be performed using generative AI, or they may not. For example, the generative AI unit can append images generated using generative AI to the dataset for the AI ​​to learn from.

[0094] This includes a voice recognition unit that uses voice recognition technology to input conditions. For example, the voice recognition unit can use voice recognition technology to input abnormal situations occurring while driving as prompts into the generating AI in real time. For instance, the voice recognition unit can input "There is an obstacle ahead" or similar conditions while driving. Furthermore, it can input "The road is slippery" or similar conditions while driving. Even further, it can input "Visibility is poor" or similar conditions while driving. Thus, the accuracy of condition input can be improved using voice recognition technology. Some or all of the above processing in the voice recognition unit can be performed using AI, or it can be performed without AI. For example, the voice recognition unit can input the conditions using voice recognition technology into the generating AI, which then determines the appropriate response based on the conditions.

[0095] The dataset management department, for example, adds the generated images to the dataset. This department can integrate the generated images into existing datasets, allowing AI to learn. For instance, adding generated images to the dataset enables AI to handle abnormal situations. Furthermore, the dataset management department can classify the generated images and add them to appropriate categories. For example, it can classify generated images into categories such as traffic accidents, natural disasters, and mechanical failures. Thus, utilizing the dataset management department can improve the efficiency of dataset management. Some or all of the above processes in the dataset management department can be performed using AI, or they can be performed without AI. For example, the dataset management department can input the generated images into AI, which will then classify and add them to the dataset.

[0096] This includes a response judgment AI unit that determines appropriate response measures based on the driving situation. The response judgment AI unit can, for example, utilize generated AI to determine appropriate response measures based on the input conditions. For instance, when there is an obstacle ahead, the response judgment AI unit instructs the driver to take evasive action. Furthermore, when the road is slippery, the response judgment AI unit can instruct the driver to slow down. Moreover, when visibility is poor, the response judgment AI unit can instruct the driver to turn on the headlights. Thus, the response judgment AI unit can quickly determine appropriate response measures. Some or all of the above processing in the response judgment AI unit can be performed using AI, or it can be performed without AI. For example, the response judgment AI unit can utilize generated AI to determine appropriate response measures based on the input conditions and instruct the driver on the result.

[0097] The image generation unit is capable of generating images that reproduce abnormal situations not typically included in the dataset. For example, the image generation unit can utilize generative AI to generate images that reproduce abnormal situations not typically included in the dataset. For instance, the image generation unit can generate images that reproduce a situation where an animal suddenly runs out onto the road. Furthermore, the image generation unit can generate images that reproduce driving conditions under abnormal weather conditions. For example, the image generation unit can generate images that reproduce driving conditions in heavy rain or snow. Further, the image generation unit can also utilize generative AI to generate images that reproduce driving conditions at night or in foggy weather. Thus, by generating images that reproduce abnormal situations, the recognition accuracy of the AI ​​can be improved. Some or all of the above-described processes in the image generation unit can be performed using generative AI, or they can be performed without using generative AI. For example, the image generation unit can append images generated using generative AI to the dataset, allowing the AI ​​to learn.

[0098] The prompt input unit utilizes speech recognition technology to input abnormal situations occurring while driving as prompts into the generating AI in real time. For example, the prompt input unit can input "Obstacle ahead" via voice while driving. Furthermore, it can input "Slippery road" via voice while driving. Further, it can input "Poor visibility" via voice while driving. Thus, by inputting prompts in real time, rapid response can be achieved. Some or all of the above processing in the prompt input unit can be performed using the generating AI, or it can be performed without it. For example, the prompt input unit can input the situation using speech recognition technology into the generating AI, which then determines the appropriate response based on the situation.

[0099] The response judgment unit can determine appropriate response measures based on the conditions input by the generated AI. For example, the response judgment unit can utilize the generated AI to determine appropriate response measures based on the input conditions. For instance, when there is an obstacle ahead, the response judgment unit instructs the driver to take evasive action. Furthermore, when the road is slippery, the response judgment unit can instruct the driver to slow down. Further, when visibility is poor, the response judgment unit can instruct the driver to turn on the headlights. Thus, the generated AI can be used to quickly determine appropriate response measures. Some or all of the above processing in the response judgment unit can be performed using the generated AI, or it can be performed without using the generated AI. For example, the response judgment unit can use the generated AI to determine appropriate response measures based on the input conditions and instruct the driver on the result.

[0100] The image generation unit can infer the driver's emotions and adjust the content of the generated images accordingly. For example, when the driver is tense, the AI ​​can generate peaceful landscape images that help relieve tension. Furthermore, when the driver is fatigued, the AI ​​can generate refreshing natural landscape images. Moreover, when the driver is excited, the AI ​​can generate nighttime landscape images that help restore calm. Thus, by generating images that match the driver's emotions, images suitable for the driver's state can be provided. Emotion inference is achieved, for example, through an emotion engine or an AI-generated emotion inference function. The AI-generated image can be text-generated AI (such as LLM) or multimodal AI-generated image, but is not limited to these. Some or all of the above processing in the image generation unit may be performed using AI-generated image, or it may not. For example, the image generation unit can input driver emotion data into the AI-generated image, which then generates an appropriate image based on that emotion.

[0101] The image generation unit can reproduce more diverse anomalous events by incorporating visual information from different viewpoints and angles into the generated images. For example, the image generation unit can utilize generative AI to generate images containing visual information from different viewpoints and angles. For instance, the image generation unit can generate images reproducing traffic congestion from an overhead view of a road. Furthermore, the image generation unit can generate images reproducing parking lot congestion from a vehicle side view. Further, the image generation unit can generate images reproducing crosswalk congestion from a pedestrian's perspective. Thus, by incorporating visual information from different viewpoints and angles, more diverse anomalous events can be reproduced. Some or all of the above processing in the image generation unit can be performed using generative AI, or it can be performed without generative AI. For example, the image generation unit can append images generated using generative AI to a dataset for the AI ​​to learn from.

[0102] The image generation unit can provide more realistic scenes by reflecting changes in different time periods and seasons in the generated images. For example, the image generation unit can utilize generative AI to generate images that reflect changes in different time periods and seasons. For instance, the image generation unit can generate images that reproduce nighttime road conditions. Furthermore, the image generation unit can generate images that reproduce winter snow conditions. Further, the image generation unit can generate images that reproduce autumn road conditions with fallen leaves scattered on them. Thus, by reflecting changes in different time periods and seasons, a more realistic scene can be provided. Some or all of the above processing in the image generation unit can be performed using generative AI, or it can be performed without generative AI. For example, the image generation unit can append images generated using generative AI to a dataset for the AI ​​to learn from.

[0103] The image generation unit can infer the driver's emotions and determine the priority of generated images based on the inferred driver emotions. For example, when the driver is nervous, the AI ​​prioritizes generating images with a relaxing effect. Furthermore, when the driver is fatigued, the AI ​​may prioritize generating images with a refreshing effect. Moreover, when the driver is excited, the AI ​​may prioritize generating images that help restore calm. Thus, by determining the priority of images that match the driver's emotions, images suitable for the driver's state can be provided preferentially. Emotion inference is achieved, for example, through an emotion inference function such as an emotion engine or a generative AI. The generative AI can be text-generated AI (such as LLM) or multimodal generative AI, but is not limited to these. Some or all of the above processing in the image generation unit may be performed using generative AI, or it may not. For example, the image generation unit can input driver emotion data into the generative AI, which then generates appropriate images based on that emotion.

[0104] The image generation unit can provide more diverse scenarios by reflecting different traffic conditions and road conditions in the generated images. For example, the image generation unit can utilize generative AI to generate images reflecting different traffic conditions and road conditions. For instance, the image generation unit can generate images reproducing congested road conditions. Furthermore, the image generation unit can generate images reproducing road construction conditions. Further, the image generation unit can generate images reproducing road conditions at accident scenes. Thus, by reflecting different traffic conditions and road conditions, more diverse scenarios can be provided. Some or all of the above processing in the image generation unit can be performed using generative AI, or it can be performed without generative AI. For example, the image generation unit can append images generated using generative AI to a dataset, allowing the AI ​​to learn.

[0105] The image generation unit can provide more realistic scenes by reflecting different weather conditions in the generated images. For example, the image generation unit can utilize generative AI to generate images that reflect different weather conditions. For instance, the image generation unit can generate images that reproduce road conditions on rainy days. Furthermore, the image generation unit can generate images that reproduce road conditions on snowy days. Further, the image generation unit can generate images that reproduce road conditions on foggy days. Thus, by reflecting different weather conditions, a more realistic scene can be provided. Some or all of the above processing in the image generation unit can be performed using generative AI, or it can be performed without generative AI. For example, the image generation unit can append images generated using generative AI to a dataset, allowing the AI ​​to learn.

[0106] The dataset addition unit can infer the driver's emotions and select images to add to the dataset based on the inferred emotions. For example, when the driver is nervous, images with a relaxing effect are added to the dataset. Furthermore, when the driver is fatigued, images with a refreshing effect are added. Moreover, when the driver is excited, images that help restore calm are added. Thus, by adding images that match the driver's emotions, the quality of the dataset can be improved. Emotion inference is achieved, for example, through emotion inference functions such as an emotion engine or generative AI. Generative AI can be text-generated AI (such as LLM) or multimodal generative AI, but is not limited to these. Some or all of the above processing in the dataset addition unit may be performed using AI, or it may not. For example, the dataset addition unit can input driver emotion data into AI, which then selects appropriate images to add to the dataset based on the emotion.

[0107] The dataset appending unit is capable of automatically generating and managing image metadata when appending data to a dataset. For example, it can automatically generate the image's capture time and location as metadata and append it to the dataset. Furthermore, it can automatically generate tags related to the image content and append them to the dataset. Moreover, it can automatically generate image resolution and format information as metadata and append it to the dataset. Thus, by automatically generating and managing image metadata, the efficiency of dataset management can be improved. Some or all of the above processing in the dataset appending unit can be performed using AI, or it can be performed without AI. For example, the dataset appending unit can input image metadata into AI, which will then generate metadata and append it to the dataset.

[0108] The dataset addition unit is capable of evaluating image quality and automatically excluding low-quality images when adding them to the dataset. For example, the dataset addition unit automatically excludes images with low resolution. Furthermore, it can automatically exclude images with high noise levels. Moreover, it can automatically exclude blurry images. Thus, by automatically excluding low-quality images, the quality of the dataset can be improved. Some or all of the above processing in the dataset addition unit may be performed using AI, or it may not. For example, the dataset addition unit can input image quality data into AI, allowing AI to evaluate the quality and exclude low-quality images.

[0109] The dataset addition unit can infer the driver's emotions and determine the priority of images added to the dataset based on the inferred driver emotions. For example, when the driver is nervous, images with a relaxing effect are prioritized for addition to the dataset. Furthermore, when the driver is fatigued, images with an energizing effect are prioritized. Moreover, when the driver is excited, images that help restore calm are prioritized. Thus, by determining the image priority that matches the driver's emotions, the quality of the dataset can be improved. Emotion inference is achieved, for example, through emotion inference functions such as an emotion engine or generative AI. Generative AI can be text-generated AI (such as LLM) or multimodal generative AI, but is not limited to these. Some or all of the above processing in the dataset addition unit may be performed using AI, or it may not. For example, the dataset addition unit can input driver emotion data into AI, which then selects appropriate images to add to the dataset based on the emotion.

[0110] The dataset addition unit is capable of evaluating image relevance and prioritizing the addition of highly relevant images when adding data to the dataset. For example, the dataset addition unit prioritizes adding images whose content is relevant to the current driving situation. Furthermore, it can also prioritize adding images whose content is relevant to past driving records. Moreover, it can prioritize adding images whose content is relevant to a specific driving scenario. Thus, by prioritizing the addition of highly relevant images, the quality of the dataset can be improved. Some or all of the above processing in the dataset addition unit may be performed using AI, or it may not. For example, the dataset addition unit can input image relevance into AI, which will evaluate the relevance and select highly relevant images to add to the dataset.

[0111] The dataset addition unit is capable of evaluating image diversity and prioritizing the addition of images with high diversity when adding data to the dataset. For example, the dataset addition unit may prioritize adding images taken from different viewpoints and angles. Furthermore, it may prioritize adding images reflecting changes over different time periods and seasons. Moreover, it may prioritize adding images reflecting different traffic conditions and road conditions. Thus, by prioritizing the addition of highly diverse images, the quality of the dataset can be improved. Some or all of the above processing in the dataset addition unit may be performed using AI, or it may not. For example, the dataset addition unit may input image diversity into AI, which will evaluate the diversity and select images with high diversity to add to the dataset.

[0112] The prompt input unit can infer the driver's mood and adjust the prompt content accordingly. For example, when the driver is nervous, concise and clear prompts are provided. Conversely, when the driver is relaxed, prompts containing detailed information can be provided. Furthermore, when the driver is anxious, prompts that allow for quick responses can be provided. Thus, by providing prompts that match the driver's mood, prompts suitable for the driver's state can be provided. Mood inference is achieved, for example, through mood inference functions such as mood engines or generative AI. Generative AI can be text-generated AI (such as LLM) or multimodal generative AI, but is not limited to these. Some or all of the above processing in the prompt input unit may be performed using AI, or it may not. For example, the prompt input unit can input driver mood data into AI, which then generates appropriate prompts based on that mood.

[0113] The prompt input unit can generate optimal prompts by referencing the driver's past driving records when prompts are entered. For example, the prompt input unit prioritizes prompts used in similar situations in the past. Furthermore, the prompt input unit can also provide prompts used in specific time periods or locations based on past driving records. Further, the prompt input unit can analyze past driving records to provide the most effective prompts. Thus, by referencing past driving records, optimal prompts can be provided. Some or all of the above processing in the prompt input unit can be performed using AI, or it can be performed without AI. For example, the prompt input unit can input the driver's past driving records into AI, which analyzes the records and generates optimal prompts.

[0114] The prompt input unit can analyze the driver's current driving status in real time when prompt words are input, and generate the optimal prompt words. For example, the prompt input unit analyzes the current traffic conditions in real time and provides appropriate prompt words. Furthermore, the prompt input unit can also analyze the current weather conditions in real time and provide appropriate prompt words. Further, the prompt input unit can also analyze the current road conditions in real time and provide appropriate prompt words. Thus, by analyzing the current driving status in real time, the optimal prompt words can be provided. Some or all of the above processing in the prompt input unit can be performed using AI, or it can be performed without AI. For example, the prompt input unit can input the driver's current driving status into the AI, which will analyze the situation in real time and generate the optimal prompt words.

[0115] The prompt input unit can infer the driver's mood and determine the priority of prompt words based on the inferred mood. For example, when the driver is nervous, prompt words with a relaxing effect are prioritized. Furthermore, when the driver is tired, prompt words with an energizing effect are prioritized. Moreover, when the driver is excited, prompt words that help restore calm are prioritized. Thus, by determining the priority of prompt words that match the driver's mood, prompt words suitable for the driver's state can be provided preferentially. Mood inference is achieved, for example, through mood inference functions such as mood engines or generative AI. Generative AI can be text-generated AI (such as LLM) or multimodal generative AI, but is not limited to these. Some or all of the above processing in the prompt input unit may be performed using AI, or it may not. For example, the prompt input unit can input driver mood data into AI, which then generates appropriate prompt words based on that mood.

[0116] The prompt input unit can generate optimal prompts by considering the driver's geographical location information when prompts are input. For example, the prompt input unit can provide appropriate prompts based on the current traffic conditions. Furthermore, it can provide appropriate prompts based on the current weather conditions. Even further, it can provide appropriate prompts based on the current road conditions. Thus, by considering geographical location information, optimal prompts can be provided. Some or all of the above processing in the prompt input unit can be performed using AI, or it can be performed without AI. For example, the prompt input unit can input the driver's geographical location information into AI, which can then generate optimal prompts based on that information.

[0117] The prompt input unit can analyze the driver's social media activity and generate relevant prompts when prompts are entered. For example, the prompt input unit can provide appropriate prompts based on information shared by the driver on social media. Furthermore, the prompt input unit can analyze the driver's activity history on social media to provide relevant prompts. Further, the prompt input unit can also provide appropriate prompts based on the accounts the driver follows on social media. Thus, relevant prompts can be provided by analyzing social media activity. Some or all of the above processing in the prompt input unit can be performed using AI, or it can be performed without AI. For example, the prompt input unit can input the driver's social media activity into AI, which can then analyze the activity and generate relevant prompts.

[0118] The response judgment unit can infer the driver's emotions and adjust the response accordingly. For example, when the driver is tense, it provides relaxation-enhancing measures. Furthermore, when the driver is fatigued, it can provide energizing measures. Moreover, when the driver is excited, it can provide measures to help restore calm. Thus, by providing responses that match the driver's emotions, it is possible to provide responses suitable for the driver's state. Emotion inference is achieved, for example, through emotion inference functions such as emotion engines or generative AI. Generative AI can be text-generated AI (such as LLM) or multimodal generative AI, but is not limited to these. Some or all of the above processing in the response judgment unit may be performed using AI, or it may not. For example, the response judgment unit can input driver emotion data into AI, which then generates appropriate responses based on that emotion.

[0119] The response judgment unit can select the optimal response by referring to past response records when determining response measures. For example, the response judgment unit may prioritize response measures taken in similar situations in the past. Furthermore, the response judgment unit can also select response measures taken in specific time periods or locations based on past response records. Further, the response judgment unit can analyze past response records to select the most effective response measures. Thus, by referring to past response records, the optimal response can be provided. Some or all of the above processing in the response judgment unit may be performed using AI, or it may not. For example, the response judgment unit can input past response records into AI, which will analyze the records and select the optimal response.

[0120] The response judgment unit can analyze the current driving situation in real time and select the optimal response when determining the appropriate action. For example, the response judgment unit can analyze the current traffic situation in real time and select appropriate actions. Furthermore, it can analyze the current weather conditions in real time and select appropriate actions. Moreover, it can analyze the current road conditions in real time and select appropriate actions. Thus, by analyzing the current driving situation in real time, the optimal response can be provided. Some or all of the above processing in the response judgment unit can be performed using AI, or it can be performed without AI. For example, the response judgment unit can input the current driving situation into AI, which can then analyze the situation in real time and select the optimal response.

[0121] The response judgment unit can infer the driver's emotions and determine the response priority based on the inferred emotions. For example, when the driver is nervous, relaxation-enhancing responses are prioritized. Similarly, when the driver is fatigued, energizing responses are prioritized. Furthermore, when the driver is excited, responses that help restore calm are prioritized. Thus, by determining the response priority that aligns with the driver's emotions, responses suitable for the driver's state can be provided preferentially. Emotion inference is achieved, for example, through an emotion engine or generative AI. Generative AI can be text-generated AI (such as LLM) or multimodal generative AI, but is not limited to these. Some or all of the above processing in the response judgment unit may be performed using AI, or it may not. For example, the response judgment unit can input driver emotion data into AI, which then generates appropriate responses based on that emotion.

[0122] The response judgment unit can select the optimal response by considering the driver's geographical location information when determining the response measure. For example, the response judgment unit selects an appropriate response measure based on the traffic conditions at the current location. Furthermore, the response judgment unit can also select an appropriate response measure based on the weather conditions at the current location. Further, the response judgment unit can also select an appropriate response measure based on the road conditions at the current location. Thus, by considering geographical location information, an optimal response can be provided. Some or all of the above processing in the response judgment unit can be performed using AI, or it can be performed without AI. For example, the response judgment unit can input the driver's geographical location information into the AI, and the AI ​​can select the optimal response based on that information.

[0123] The response judgment unit can analyze the driver's social media activity and select relevant responses when determining the appropriate action. For example, the unit can select appropriate responses based on information shared by the driver on social media. Furthermore, it can analyze the driver's activity history on social media and select relevant responses. Moreover, it can select appropriate responses based on the accounts the driver follows on social media. Thus, relevant responses can be provided by analyzing social media activity. Some or all of the above processing in the response judgment unit can be performed using AI, or it can be performed without AI. For example, the response judgment unit can input the driver's social media activity into AI, which can then analyze the activity and generate relevant responses.

[0124] The system involved in this embodiment is not limited to the above examples. For example, various modifications can be made as follows.

[0125] Autonomous driving systems can infer a driver's emotions and provide feedback based on those inferences. For example, when a driver is tense, the system can suggest relaxation breathing techniques or music. When a driver is fatigued, the system can prompt them to rest. Furthermore, when a driver is excited, the system can provide suggestions to calm them down. Thus, by providing feedback that aligns with the driver's emotions, the system can optimize the driver's state and support safe driving. Emotion inference can be achieved, for example, through emotion engines or generative AI. Generative AI can be text-generated AI (such as LLM) or multimodal generative AI, but is not limited to these. Some or all of the above processing in the feedback provision unit may be performed using generative AI, or it may not. For example, the feedback provision unit can input driver emotion data into the generative AI, which then generates appropriate feedback based on that emotion.

[0126] Autonomous driving systems can analyze a driver's past driving records and provide optimal driving assistance based on the driver's driving style. For example, for drivers who frequently brake suddenly, the system can provide deceleration assistance in advance. For drivers who frequently drive on highways, it can provide assistance to maintain the optimal following distance on highways. Furthermore, for drivers who frequently drive at night, it can provide assistance to improve nighttime visibility. Thus, by providing assistance tailored to the driver's driving style, driving safety and comfort can be improved. Driving record analysis can be performed using AI, or not. For example, driving record data can be input into AI, which can analyze the data and generate optimal driving assistance.

[0127] Autonomous driving systems can infer a driver's emotions and monitor their stress levels based on these inferred emotions. For example, when a driver experiences high stress, the system can prompt them to rest. When the driver is at a low stress level, the system can suggest continuing to drive. Furthermore, when a driver's stress level changes drastically, the system can identify the cause and propose appropriate countermeasures. Thus, by monitoring the driver's stress levels, the system can support the driver's health and safety. Emotion inference can be achieved, for example, through emotion engines or generative AI. Generative AI can be text-generated AI (such as LLM) or multimodal generative AI, but is not limited to these. Some or all of the above processing in the stress monitoring unit may be performed using generative AI, or it may not. For example, the stress monitoring unit can input driver emotion data into the generative AI, which then performs appropriate monitoring based on that emotion.

[0128] Autonomous driving systems can infer a driver's emotions and assess their attention based on these inferred emotions. For example, when the driver is attentive, the system suggests continuing to drive. When the driver is inattentive, the system can prompt the driver to rest. Furthermore, when the driver's attention drops sharply, the system can identify the cause and propose appropriate countermeasures. Thus, by assessing the driver's attention, driving safety can be improved. Emotion inference can be achieved, for example, through emotion inference functions such as emotion engines or generative AI. Generative AI can be text-generated AI (such as LLM) or multimodal generative AI, but is not limited to these. Some or all of the above processing in the attention assessment unit may be performed using generative AI, or it may not. For example, the attention assessment unit can input driver emotion data into the generative AI, which then performs an appropriate assessment based on that emotion.

[0129] Autonomous driving systems can infer a driver's emotions and adjust their driving style accordingly. For example, when a driver is tense, the system adjusts to a smoother driving style. When the driver is relaxed, the system can adjust to a more dynamic driving style. Furthermore, when a driver is fatigued, the system can reduce speed to improve safety. Thus, by providing a driving style that matches the driver's emotions, driving safety and comfort can be improved. Emotion inference can be achieved, for example, through emotion inference functions such as emotion engines or generative AI. Generative AI can be text-generated AI (such as LLM) or multimodal generative AI, but is not limited to these. Some or all of the above processing in the driving style adjustment unit may be performed using generative AI, or it may not. For example, the driving style adjustment unit can input driver emotion data into the generative AI, which then generates an appropriate driving style based on that emotion.

[0130] Autonomous driving systems can reference a driver's past driving records and propose optimal routes based on the driver's driving patterns. For example, for drivers who frequently use specific routes, those routes are prioritized. For drivers who tend to avoid congestion, routes that avoid congestion are recommended. Furthermore, for drivers who prefer highways, highway routes can also be recommended. Thus, by recommending optimal routes that match the driver's driving patterns, driving efficiency and comfort can be improved. Driving record referencing can be performed using AI, or not. For example, driving record data can be input into AI, which analyzes the data and generates the optimal route.

[0131] Autonomous driving systems can analyze a driver's current driving status in real time to provide optimal driving assistance. For example, real-time analysis of current traffic conditions can provide assistance in avoiding congestion. Furthermore, real-time analysis of current weather conditions can provide driving assistance in adverse weather conditions. Further, real-time analysis of current road conditions can provide assistance in avoiding road construction or accidents. Thus, by analyzing the current driving status in real time, optimal driving assistance can be provided. Real-time analysis of driving status can be performed using AI, or not. For example, driving status data can be input into AI, which can analyze the data in real time to generate optimal driving assistance.

[0132] Autonomous driving systems can take into account the driver's location information to provide optimal driving assistance. For example, based on the current traffic conditions, they can provide assistance to avoid congestion. Furthermore, based on the current weather conditions, they can provide driving assistance in adverse weather conditions. Further, based on the current road conditions, they can provide assistance to avoid road construction or accidents. Thus, by considering location information, optimal driving assistance can be provided. The consideration of location information can be done using AI, or not. For example, location information data can be input into AI, which can analyze the data and generate optimal driving assistance.

[0133] Autonomous driving systems can analyze a driver's social media activity and provide optimal driving assistance based on the driver's interests and concerns. For example, routes to locations of interest can be recommended based on information shared by the driver on social media. Furthermore, the system can analyze the driver's social media activity history to provide relevant driving assistance. Moreover, routes to activities or locations of interest can be recommended based on the accounts the driver follows on social media. Thus, by analyzing social media activity, driving assistance based on the driver's interests and concerns can be provided. Social media activity analysis can be performed using AI, or not. For example, social media activity data can be input into AI, which can then analyze the data to generate optimal driving assistance.

[0134] Autonomous driving systems can infer a driver's emotions and assess driving performance based on these inferred emotions. For example, when a driver is tense, the system assesses a decline in driving performance. When the driver is relaxed, the system assesses improved driving performance. Furthermore, when a driver is fatigued, the system can also assess a decline in driving performance. Thus, by assessing driving performance in accordance with the driver's emotions, driving safety and efficiency can be improved. Emotion inference can be achieved, for example, through emotion inference functions such as emotion engines or generative AI. Generative AI can be text-generated AI (such as LLM) or multimodal generative AI, but is not limited to these. Some or all of the above processing in the driving performance evaluation unit may be performed using generative AI, or it may not. For example, the driving performance evaluation unit can input driver emotion data into the generative AI, which then performs an appropriate assessment based on that emotion.

[0135] The following is a brief description of the processing flow of Implementation Method 2.

[0136] Step 1: The image generation unit generates images for the abnormal events. For example, it uses generative AI to generate images that reproduce abnormal situations not typically included in the dataset. Specifically, it generates images that reproduce situations such as an animal suddenly running out on the road or driving under abnormal weather conditions (heavy rain, snow, night, fog).

[0137] Step 2: The dataset appending unit appends the images generated by the image generation unit to the dataset. For example, the generated images are integrated into the existing dataset to allow the AI ​​to learn. Furthermore, the generated images are categorized and appended to appropriate categories such as traffic accidents, natural disasters, and mechanical failures.

[0138] Step 3: The input unit inputs the driving conditions as prompts. For example, using voice recognition technology, abnormal situations occurring while driving are input as prompts into the AI ​​in real time. Specifically, prompts can be input via voice such as "Obstacle ahead," "Slippery road," or "Poor visibility."

[0139] Step 4: The response judgment unit determines the appropriate response based on the situation input by the prompt input unit. For example, using generated AI, the appropriate response is determined based on the input situation, such as instructing to take evasive action when there is an obstacle ahead, instructing to slow down when the road is slippery, and instructing to turn on the headlights when visibility is poor.

[0140] The specific processing unit 290 sends the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires voice representing the user's input to the result of the specific processing. The control unit 46A sends the voice data representing the user's input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0141] Data generation model 58 is what is known as generative AI (Artificial Intelligence). An example of data generation model 58 includes ChatGPT (registered trademark) (Internet search).<URL:https: / / openai.com / blog / chatgpt> Generative AI, such as data generation model 58, is obtained by deep learning through a neural network. The data generation model 58 is input with a prompt containing instructions, and with inference data such as speech data representing speech, text data representing text, and image data representing images (e.g., data of still images or data of moving images). The data generation model 58 infers the input inference data according to the instructions shown in the prompt, and outputs the inference result in one or more data forms such as speech data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the above-described specific processing while using the data generation model 58. The data generation model 58 can also be a fine-tuned model to output inference results from prompts without instructions; in this case, the data generation model 58 can output inference results from prompts without instructions. The data processing device 12, etc., includes multiple data generation models 58, including AI other than generative AI. AI other than generative AI includes, but is not limited to, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes. Furthermore, AI can also act as an AI agent. Additionally, when the processing described above is performed by AI, this processing may be partially or entirely performed by AI, but is not limited to this example. Moreover, processing performed by AI, including generative AI, can be replaced by rule-based processing, and vice versa.

[0142] Furthermore, the processing performed by the aforementioned data processing system 10 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it can also be executed jointly by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart device 14 or external devices, and the smart device 14 acquires or collects information required for processing from the data processing device 12 or external devices.

[0143] Each of the elements described above—including the image generation unit, dataset appending unit, prompt input unit, and response judgment unit—can be implemented, for example, in at least one of the smart device 14 and the data processing device 12. For example, the image generation unit, implemented by the control unit 46A of the smart device 14, generates an image reproducing the abnormal situation using a generative AI. The dataset appending unit, for example, is implemented by the specific processing unit 290 of the data processing device 12, appending the generated image to the dataset. The prompt input unit, for example, is implemented by the control unit 46A of the smart device 14, using voice recognition technology to input the driving situation as a prompt word. The response judgment unit, for example, is implemented by the specific processing unit 290 of the data processing device 12, determining an appropriate response based on the input situation. The correspondence between each unit and the device or control unit is not limited to the above examples and can be modified in various ways.

[0144] Second Implementation Method

[0145] Figure 3 An example of the configuration of the data processing system 210 according to the second embodiment is shown.

[0146] like Figure 3 As shown, the data processing system 210 includes a data processing device 12 and smart glasses 214. One example of the data processing device 12 is a server.

[0147] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. An example of the network 54 includes a WAN and / or LAN, etc.

[0148] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0149] Microphone 238 receives user commands by receiving the user's voice. Microphone 238 captures the user's voice, converts the captured sound into speech data, and outputs it to processor 46. Speaker 240 outputs sound according to commands from processor 46.

[0150] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, used to capture the user's surroundings (e.g., the shooting range defined by an angle of view equivalent to the field of vision of an average healthy person).

[0151] Communication I / F 44 is connected to network 54. Communication I / F 44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F 44 and 26 is performed in a secure state.

[0152] Figure 4 An example of the main functions of the data processing device 12 and the smart glasses 214 is shown. Figure 4 As shown, in the data processing device 12, specific processing is performed by the processor 28. The memory 32 stores a specific processing program 56.

[0153] The processor 28 reads a specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is implemented by the processor 28 operating as a specific processing unit 290 based on the specific processing program 56 executed on the RAM 30.

[0154] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. The emotion inference function using the emotion-specific model 59 (emotion-specific function) includes inferring and predicting the user's emotions, performing various inferences and predictions related to the user's emotions, but is not limited to this example. Furthermore, emotion inference and prediction also include, for example, emotion analysis (analysis).

[0155] In the smart glasses 214, specific processing is performed by the processor 46. A specific processing program 60 is stored in the memory 50. The processor 46 reads the specific processing program 60 from the memory 50 and executes the read specific processing program 60 on the RAM 48. Specific processing is implemented by the processor 46 operating as a control unit 46A based on the specific processing program 60 executed on the RAM 48. Furthermore, the smart glasses 214 may also have the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and use these models to perform the same processing as the specific processing unit 290.

[0156] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains the processing results (prediction results, etc.) using the data generation model 58 by communicating with the server device that has the data generation model 58. In addition, the data processing device 12 may be a server device or a user-owned terminal device (e.g., a mobile phone, robot, home appliance, etc.).

[0157] The specific processing unit 290 sends the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires voice input representing the user's input to the specific processing result. The control unit 46A sends the voice data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0158] Data generation model 58 is a so-called generative AI. An example of data generation model 58 includes generative AIs such as ChatGPT. Data generation model 58 is obtained by performing deep learning on a neural network. A prompt containing instructions is input to data generation model 58, along with inference data such as speech data representing speech, text data representing text, and image data representing images (e.g., data from still images or moving images). Data generation model 58 infers the inference data based on the instructions shown in the prompt and outputs the inference result in one or more data forms such as speech data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. Specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. Data generation model 58 can also be a fine-tuned model to output inference results from prompts without instructions; in this case, data generation model 58 can output inference results from prompts without instructions. The data processing device 12, etc., includes various data generation models 58, which include AI other than generative AI. These AIs include, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and are capable of various processing methods, but are not limited to these examples. Furthermore, AI can also be an AI agent. Additionally, when the processing described above is performed by AI, this processing may be partially or entirely performed by AI, but is not limited to these examples. Moreover, processing performed by AI including generative AI can be replaced by rule-based processing, and vice versa.

[0159] The data processing system 210 of the second embodiment performs the same processing as the data processing system 10 of the first embodiment. The processing performed by the data processing system 210 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it can also be executed jointly by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or external devices, and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or external devices.

[0160] Each of the aforementioned elements—image generation unit, dataset appending unit, prompt input unit, and response judgment unit—can be implemented, for example, in at least one of the smart glasses 214 and the data processing device 12. For instance, the image generation unit, implemented by the control unit 46A of the smart glasses 214, generates an image reproducing the abnormal situation using AI. The dataset appending unit, for example, is implemented by the specific processing unit 290 of the data processing device 12, appending the generated image to the dataset. The prompt input unit, for example, is implemented by the control unit 46A of the smart glasses 214, using voice recognition technology to input the driving situation as a prompt word. The response judgment unit, for example, is implemented by the specific processing unit 290 of the data processing device 12, determining an appropriate response based on the input situation. The correspondence between each unit and the device or control unit is not limited to the above examples and can be varied.

[0161] Third Implementation Method

[0162] Figure 5 An example of the configuration of the data processing system 310 according to the third embodiment is shown.

[0163] like Figure 5 As shown, the data processing system 310 includes a data processing device 12 and a head-mounted terminal 314. An example of the data processing device 12 is a server.

[0164] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. An example of the network 54 includes a WAN and / or LAN, etc.

[0165] The head-mounted terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0166] Microphone 238 receives user commands by receiving the user's voice. Microphone 238 captures the user's voice, converts the captured sound into speech data, and outputs it to processor 46. Speaker 240 outputs sound according to commands from processor 46.

[0167] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, used to capture the user's surroundings (e.g., the shooting range defined by an angle of view equivalent to the field of vision of an average healthy person).

[0168] Communication I / F 44 is connected to network 54. Communication I / F 44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F 44 and 26 is performed in a secure state.

[0169] Figure 6 An example of the main functions of the data processing device 12 and the head-mounted terminal 314 is shown. Figure 6 As shown, in the data processing device 12, specific processing is performed by the processor 28. The memory 32 stores a specific processing program 56.

[0170] The processor 28 reads a specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is implemented by the processor 28 operating as a specific processing unit 290 based on the specific processing program 56 executed on the RAM 30.

[0171] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. The emotion inference function using the emotion-specific model 59 (emotion-specific function) includes inferring and predicting the user's emotions, performing various inferences and predictions related to the user's emotions, but is not limited to this example. Furthermore, emotion inference and prediction also include, for example, emotion analysis (analysis).

[0172] In the head-mounted terminal 314, specific processing is performed by the processor 46. A specific program 60 is stored in the memory 50. The processor 46 reads the specific program 60 from the memory 50 and executes the read specific program 60 on the RAM 48. Specific processing is implemented by the processor 46 operating as a control unit 46A based on the specific program 60 executed on the RAM 48. Furthermore, the head-mounted terminal 314 may also have the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and use these models to perform the same processing as the specific processing unit 290.

[0173] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains the processing results (prediction results, etc.) using the data generation model 58 by communicating with the server device that has the data generation model 58. In addition, the data processing device 12 may be a server device or a user-owned terminal device (e.g., a mobile phone, robot, home appliance, etc.).

[0174] The specific processing unit 290 sends the result of the specific processing to the head-mounted terminal 314. In the head-mounted terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires voice representing the user's input to the specific processing result. The control unit 46A sends the voice data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0175] Data generation model 58 is a so-called generative AI. An example of data generation model 58 includes generative AIs such as ChatGPT. Data generation model 58 is obtained by performing deep learning on a neural network. A prompt containing instructions is input to data generation model 58, along with inference data such as speech data representing speech, text data representing text, and image data representing images (e.g., data from still images or moving images). Data generation model 58 infers the inference data based on the instructions shown in the prompt and outputs the inference result in one or more data forms such as speech data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. Specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. Data generation model 58 can also be a fine-tuned model to output inference results from prompts without instructions; in this case, data generation model 58 can output inference results from prompts without instructions. The data processing device 12, etc., includes various data generation models 58, which include AI other than generative AI. These AIs include, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and are capable of various processing methods, but are not limited to these examples. Furthermore, AI can also be an AI agent. Additionally, when the processing described above is performed by AI, this processing may be partially or entirely performed by AI, but is not limited to these examples. Moreover, processing performed by AI including generative AI can be replaced by rule-based processing, and vice versa.

[0176] The data processing system 310 of the third embodiment performs the same processing as the data processing system 10 of the first embodiment. The processing performed by the data processing system 310 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the head-mounted terminal 314, but it can also be executed jointly by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the head-mounted terminal 314. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the head-mounted terminal 314 or external devices, and the head-mounted terminal 314 acquires or collects information required for processing from the data processing device 12 or external devices.

[0177] Each of the aforementioned elements—image generation unit, dataset appending unit, prompt input unit, and response judgment unit—can be implemented, for example, in at least one of the head-mounted terminal 314 and the data processing device 12. For example, the image generation unit, implemented by the control unit 46A of the head-mounted terminal 314, generates an image reproducing the abnormal situation using a generative AI. The dataset appending unit, for example, is implemented by the specific processing unit 290 of the data processing device 12, appending the generated image to the dataset. The prompt input unit, for example, is implemented by the control unit 46A of the head-mounted terminal 314, using voice recognition technology to input the driving situation as a prompt word. The response judgment unit, for example, is implemented by the specific processing unit 290 of the data processing device 12, determining an appropriate response based on the input situation. The correspondence between each unit and the device or control unit is not limited to the above examples and can be modified in various ways.

[0178] Fourth Implementation Method

[0179] Figure 7 An example of the configuration of the data processing system 410 according to the fourth embodiment is shown.

[0180] like Figure 7 As shown, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0181] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. An example of the network 54 includes a WAN and / or LAN, etc.

[0182] Robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control object 443. Computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, and control object 443 are also connected to the bus 52.

[0183] Microphone 238 receives user commands by receiving the user's voice. Microphone 238 captures the user's voice, converts the captured sound into speech data, and outputs it to processor 46. Speaker 240 outputs sound according to commands from processor 46.

[0184] The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and an imaging element such as a CMOS image sensor or a CCD image sensor, used to photograph the user's surroundings (e.g., the shooting range defined by an angle of view equivalent to the field of vision of an average healthy person).

[0185] Communication I / F 44 is connected to network 54. Communication I / F 44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F 44 and 26 is performed in a secure state.

[0186] The controlled object 443 includes a display device, LEDs for the eyes, and motors for driving the arms, hands, and feet. The posture and movements of the robot 414 are controlled by controlling the motors for the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. In addition, facial expressions of the robot 414 can also be expressed by controlling the illumination state of the LEDs for the robot 414's eyes.

[0187] Figure 8 An example of the main functions of the data processing device 12 and the robot 414 is shown. For example... Figure 8 As shown, in the data processing device 12, specific processing is performed by the processor 28. The memory 32 stores a specific processing program 56.

[0188] The processor 28 reads a specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is implemented by the processor 28 operating as a specific processing unit 290 based on the specific processing program 56 executed on the RAM 30.

[0189] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. The emotion inference function using the emotion-specific model 59 (emotion-specific function) includes inferring and predicting the user's emotions, performing various inferences and predictions related to the user's emotions, but is not limited to this example. Furthermore, emotion inference and prediction also include, for example, emotion analysis (analysis).

[0190] In robot 414, specific processing is performed by processor 46. A specific program 60 is stored in memory 50. Processor 46 reads the specific program 60 from memory 50 and executes the read specific program 60 on RAM 48. Specific processing is achieved by processor 46 acting as control unit 46A based on the specific program 60 executed on RAM 48. Furthermore, robot 414 may also have the same data generation model and emotion-specific model as data generation model 58 and emotion-specific model 59, and use these models to perform the same processing as specific processing unit 290.

[0191] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains the processing results (prediction results, etc.) using the data generation model 58 by communicating with the server device that has the data generation model 58. In addition, the data processing device 12 may be a server device or a user-owned terminal device (e.g., a mobile phone, robot, home appliance, etc.).

[0192] The specific processing unit 290 sends the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires voice representing the user's input regarding the result of the specific processing. The control unit 46A sends the voice data representing the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0193] Data generation model 58 is a so-called generative AI. An example of data generation model 58 includes generative AIs such as ChatGPT. Data generation model 58 is obtained by performing deep learning on a neural network. A prompt containing instructions is input to data generation model 58, along with inference data such as speech data representing speech, text data representing text, and image data representing images (e.g., data from still images or moving images). Data generation model 58 infers the inference data based on the instructions shown in the prompt and outputs the inference result in one or more data forms such as speech data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. Specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. Data generation model 58 can also be a fine-tuned model to output inference results from prompts without instructions; in this case, data generation model 58 can output inference results from prompts without instructions. The data processing device 12, etc., includes various data generation models 58, which include AI other than generative AI. These AIs include, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and are capable of various processing methods, but are not limited to these examples. Furthermore, AI can also be an AI agent. Additionally, when the processing described above is performed by AI, this processing may be partially or entirely performed by AI, but is not limited to these examples. Moreover, processing performed by AI including generative AI can be replaced by rule-based processing, and vice versa.

[0194] The data processing system 410 of the fourth embodiment performs the same processing as the data processing system 10 of the first embodiment. The processing performed by the data processing system 410 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it can also be executed jointly by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or external devices, and the robot 414 acquires or collects information required for processing from the data processing device 12 or external devices.

[0195] Each of the elements described above—including the image generation unit, dataset appending unit, prompt input unit, and response judgment unit—can be implemented, for example, in at least one of the robot 414 and the data processing device 12. For instance, the image generation unit, implemented by the control unit 46A of the robot 414, generates an image reproducing the abnormal situation using a generative AI. The dataset appending unit, for example, is implemented by the specific processing unit 290 of the data processing device 12, appending the generated image to the dataset. The prompt input unit, for example, is implemented by the control unit 46A of the robot 414, using speech recognition technology to input the driving situation as a prompt word. The response judgment unit, for example, is implemented by the specific processing unit 290 of the data processing device 12, determining an appropriate response based on the input situation. The correspondence between each unit and the device or control unit is not limited to the above examples and can be modified in various ways.

[0196] Furthermore, the emotion-specific model 59, acting as an emotion engine, can determine the user's emotion based on a specific mapping. Specifically, the emotion-specific model 59 can determine the user's emotion based on an emotion graph that serves as a specific mapping (see...). Figure 9 The robot's emotions can be determined by the emotion-specific model 59. In addition, the emotion-specific model 59 can also determine the robot's emotions in the same way, and the specific processing unit 290 can also perform specific processing using the robot's emotions.

[0197] Figure 9 This is a diagram representing an emotion map 400 that maps various emotions. In the emotion map 400, emotions are arranged radially from the center in concentric circles. The closer to the center of the concentric circles, the more primitive the emotion is. Further out on the concentric circles, emotions are arranged representing states or actions arising from mood. Emotion is a concept that includes both feelings and mental states. To the left of the concentric circles, emotions generated by reactions occurring in the brain are arranged roughly. To the right of the concentric circles, emotions guided by situational judgments are arranged roughly. Above and below the concentric circles, emotions generated by reactions occurring in the brain and guided by situational judgments are arranged roughly. Furthermore, the emotion of "pleasure" is arranged above the concentric circles, and the emotion of "unpleasantness" is arranged below. Thus, in the emotion map 400, various emotions are mapped according to the structure of emotion generation, while easily generated emotions are mapped nearby.

[0198] These emotions are distributed at the 3 o'clock position on the Emotion Chart 400, and usually fluctuate between peace and unease. In the right half of the Emotion Chart 400, because situational awareness is more dominant than internal feelings, it gives a sense of calm.

[0199] The inner side of the emotion diagram 400 represents the mind, and the outer side of the emotion diagram 400 represents actions. Therefore, the further you go to the outer side of the emotion diagram 400, the more the emotion can be seen (manifested in actions).

[0200] Here, human emotions are based on a balance of various factors such as posture and blood sugar levels. When these balances deviate from the ideal, it indicates unhappiness; when they approach the ideal, it indicates pleasure. In robots, cars, and motorcycles, emotions can also be created based on a balance of factors such as posture and remaining battery power. When these balances deviate from the ideal, it indicates unhappiness; when they approach the ideal, it indicates pleasure. Emotion maps can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Speech Emotion Recognition and Brain Physiological Signal Analysis Systems for Emotion, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). In the left half of the emotion map, emotions belonging to the "Reaction" domain, where sensation is dominant, are arranged. Furthermore, in the right half of the emotion map, emotions belonging to the "Situation" domain, where situational cognition is dominant, are arranged.

[0201] The emotion map defines two types of emotions that promote learning. One is a negative emotion located near the middle of "repentance" or "reflection" on the situation side. That is, when the robot experiences negative emotions such as "I never want to feel this way again" or "I never want to be scolded again." The other is a positive emotion located near "desire" on the response side. That is, when the robot experiences positive feelings such as "wanting more" or "wanting to know more."

[0202] The emotion-specific model 59 feeds user input into a pre-learned neural network to obtain emotion values ​​representing each emotion shown in the emotion graph 400, and determines the user's emotion. This neural network is pre-learned based on multiple learning data sets that combine user input with emotion values ​​representing each emotion shown in the emotion graph 400. Furthermore, this neural network is learned to... Figure 10 As shown in sentiment graph 900, nearby sentiment values ​​are similar to each other. Figure 10 Examples show how emotions such as "peace of mind," "stability," and "reassurance" can be associated with similar emotional values.

[0203] In the above embodiments, a specific processing is described by a single computer 22, but the technology disclosed herein is not limited to this, and distributed processing by multiple computers, including computer 22, is also possible.

[0204] In the above embodiments, an example of storing a specific processing program 56 in memory 32 is illustrated, but the technology disclosed herein is not limited thereto. For example, the specific processing program 56 may also be stored in a portable computer-readable non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed into the computer 22 of the data processing device 12. The processor 28 performs specific processing according to the specific processing program 56.

[0205] Alternatively, the specific processing program 56 can be stored in a storage device such as a server connected to the data processing device 12 via a network 54, and the specific processing program 56 can be downloaded and installed into the computer 22 upon request from the data processing device 12.

[0206] Furthermore, it is not necessary to store the entire specific process 56 in a storage device such as a server connected to the data processing device 12 via the network 54, nor is it necessary to store the entire specific process 56 in the memory 32; a portion of the specific process 56 may also be stored.

[0207] As a hardware resource for performing specific processing, various processors can be used. For example, a CPU is a general-purpose processor that functions as a hardware resource for performing specific processing by executing software, i.e., programs. Additionally, a dedicated circuit can be listed as a processor; it is a processor with a circuit structure specifically designed for performing specific processing, such as a FPGA (Field-Programmable Gate Array), PLD (Programmable Logic Device), or ASIC (Application Specific Integrated Circuit). Every processor has built-in or connected memory, and every processor executes specific processing by using that memory.

[0208] The hardware resources for performing a specific process can consist of one of these various processors, or a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, the hardware resources for performing a specific process can also be a single processor.

[0209] As an example of a single processor, the first type consists of a combination of one or more CPUs and software, which functions as a hardware resource to perform specific processing. The second type uses a processor, such as a System-on-a-chip (SoC), which implements the entire system functionality, including multiple hardware resources for performing specific processing, using a single IC chip. In this case, the specific processing is implemented using one or more of the aforementioned processors that serve as hardware resources.

[0210] Furthermore, as the hardware architecture of these various processors, more specifically, circuits combining semiconductor elements and other circuit components can be used. Moreover, the specific process described above is merely an example. Therefore, it goes without saying that, without departing from the main point, unnecessary steps can be removed, new steps can be added, or the processing order can be changed.

[0211] Furthermore, although the above examples have been described in terms of first to fourth embodiments, some or all of these embodiments can be combined. Additionally, the smart device 14, smart glasses 214, head-mounted terminal 314, and robot 414 are just examples and can be combined separately, or other devices may be used. Furthermore, although the above examples have been described in terms of morphological example 1 and morphological example 2, these can also be combined.

[0212] The foregoing descriptions and illustrations are detailed explanations of the parts covered by this disclosure and are merely one example of this disclosure. For instance, the descriptions of the above-described structure, function, role, and effect are just one example of the structure, function, role, and effect of the parts covered by this disclosure. Therefore, it goes without saying that, without departing from the spirit of this disclosure, unnecessary parts can be deleted, new elements can be added, or replacements can be made to the foregoing descriptions and illustrations. Furthermore, to avoid confusion and facilitate understanding of the parts covered by this disclosure, explanations of technical common sense that does not require special explanation for implementing this disclosure have been omitted from the foregoing descriptions and illustrations.

[0213] All documents, patent applications and technical standards described in this specification are incorporated herein by reference as if they were specifically and individually described as incorporated by reference.

[0214] [Postscript 1]

[0215] A system characterized in that,

[0216] Includes an image generation unit for generating images for abnormal events;

[0217] A dataset appending unit is used to append the images generated by the image generation unit to the dataset;

[0218] The prompt input section is used to input the driving status as a prompt word;

[0219] The response judgment unit is used to determine an appropriate response based on the situation input by the prompt input unit.

[0220] [Postscript 2]

[0221] The system as described in Appendix 1 is characterized in that,

[0222] This includes a generative AI department that uses generative AI to generate images.

[0223] [Postscript 3]

[0224] The system as described in Appendix 1 is characterized in that,

[0225] This includes a voice recognition unit that uses voice recognition technology to input information.

[0226] [Postscript 4]

[0227] The system as described in Appendix 1 is characterized in that,

[0228] This includes the dataset management department, which appends the generated images to the dataset.

[0229] [Postscript 5]

[0230] The system as described in Appendix 1 is characterized in that,

[0231] This includes an AI-powered response and judgment department that determines appropriate response measures based on the driving conditions.

[0232] [Postscript 6]

[0233] The system as described in Appendix 1 is characterized in that,

[0234] The image generation unit generates images that reproduce abnormal conditions not typically included in the dataset.

[0235] [Postscript 7]

[0236] The system as described in Appendix 1 is characterized in that,

[0237] The prompt input unit uses speech recognition technology to input abnormal situations that occur during driving as prompt words into the AI ​​generation system in real time.

[0238] [Postscript 8]

[0239] The system as described in Appendix 1 is characterized in that,

[0240] The response judgment unit determines the appropriate response measures based on the situation generated by the AI ​​input.

[0241] [Postscript 9]

[0242] The system as described in Appendix 1 is characterized in that,

[0243] The image generation unit estimates the driver's mood and adjusts the content of the generated image based on the estimated driver's mood.

[0244] [Postscript 10]

[0245] The system as described in Appendix 1 is characterized in that,

[0246] The image generation unit reproduces more diverse anomalous events by including visual information from different viewpoints and angles in the generated images.

[0247] [Postscript 11]

[0248] The system as described in Appendix 1 is characterized in that,

[0249] The image generation unit provides a more realistic scene by reflecting changes in different time periods and seasons in the generated images.

[0250] [Postscript 12]

[0251] The system as described in Appendix 1 is characterized in that,

[0252] The image generation unit estimates the driver's mood and determines the priority of the generated images based on the estimated driver's mood.

[0253] [Postscript 13]

[0254] The system as described in Appendix 1 is characterized in that,

[0255] The image generation unit provides more diverse scenarios by reflecting different traffic conditions and road conditions in the generated images.

[0256] [Postscript 14]

[0257] The system as described in Appendix 1 is characterized in that,

[0258] The image generation unit provides a more realistic scene by reflecting different weather conditions in the generated images.

[0259] [Postscript 15]

[0260] The system as described in Appendix 1 is characterized in that,

[0261] The dataset addition unit estimates the driver's emotions and selects images to be added to the dataset based on the estimated driver emotions.

[0262] [Postscript 16]

[0263] The system as described in Appendix 1 is characterized in that,

[0264] When the dataset is added, the dataset addition unit has the function of automatically generating and managing image metadata.

[0265] [Postscript 17]

[0266] The system as described in Appendix 1 is characterized in that,

[0267] The dataset addition unit has the function of evaluating image quality and automatically excluding low-quality images when adding data to the dataset.

[0268] [Postscript 18]

[0269] The system as described in Appendix 1 is characterized in that,

[0270] The dataset addition unit estimates the driver's mood and determines the priority of images added to the dataset based on the estimated driver's mood.

[0271] [Postscript 19]

[0272] The system as described in Appendix 1 is characterized in that,

[0273] The dataset addition unit has the function of evaluating image relevance and prioritizing the addition of highly relevant images when adding data to the dataset.

[0274] [Postscript 20]

[0275] The system as described in Appendix 1 is characterized in that,

[0276] The dataset addition unit has the function of evaluating image diversity and prioritizing the addition of images with high diversity when adding data to the dataset.

[0277] [Postscript 21]

[0278] The system as described in Appendix 1 is characterized in that,

[0279] The prompt input unit estimates the driver's mood and adjusts the prompt content based on the estimated driver mood.

[0280] [Postscript 22]

[0281] The system as described in Appendix 1 is characterized in that,

[0282] When inputting a prompt, the prompt input unit generates the optimal prompt by referring to the driver's past driving records.

[0283] [Postscript 23]

[0284] The system as described in Appendix 1 is characterized in that,

[0285] When a prompt word is entered, the prompt input unit analyzes the driver's current driving status in real time and generates the optimal prompt word.

[0286] [Postscript 24]

[0287] The system as described in Appendix 1 is characterized in that,

[0288] The prompt input unit estimates the driver's mood and determines the priority of the prompt words based on the estimated driver mood.

[0289] [Postscript 25]

[0290] The system as described in Appendix 1 is characterized in that,

[0291] When inputting a prompt, the prompt input unit takes into account the driver's geographical location information to generate the optimal prompt.

[0292] [Postscript 26]

[0293] The system as described in Appendix 1 is characterized in that,

[0294] When a prompt word is entered, the prompt input unit analyzes the driver's social media activity and generates relevant prompt words.

[0295] [Postscript 27]

[0296] The system as described in Appendix 1 is characterized in that,

[0297] The response judgment unit presupposes the driver's emotions and adjusts the response accordingly.

[0298] [Postscript 28]

[0299] The system as described in Appendix 1 is characterized in that,

[0300] When determining the appropriate response, the response judgment unit refers to past response records to select the optimal response.

[0301] [Postscript 29]

[0302] The system as described in Appendix 1 is characterized in that,

[0303] When determining the appropriate response, the response judgment unit analyzes the current driving situation in real time and selects the optimal response.

[0304] [Postscript 30]

[0305] The system as described in Appendix 1 is characterized in that,

[0306] The response judgment unit presupposes the driver's mood and determines the priority of the response based on the presumed driver's mood.

[0307] [Postscript 31]

[0308] The system as described in Appendix 1 is characterized in that,

[0309] When determining the appropriate response, the response judgment unit considers the driver's geographical location information to select the optimal response.

[0310] [Postscript 32]

[0311] The system as described in Appendix 1 is characterized in that,

[0312] When determining the appropriate response, the response judgment unit analyzes the driver's social media activity and selects a relevant response.

Claims

1. A system, characterized in that, include: An image generation unit is used to generate images for abnormal events; A dataset appending unit is used to append the images generated by the image generation unit to the dataset; The prompt input section is used to input the driving status as a prompt word; The response judgment unit is used to determine an appropriate response based on the situation input by the prompt input unit.

2. The system as described in claim 1, characterized in that, This includes a generative AI department that uses generative AI to generate images.

3. The system as described in claim 1, characterized in that, This includes a voice recognition unit that uses voice recognition technology to input information.

4. The system as described in claim 1, characterized in that, This includes the dataset management department, which appends the generated images to the dataset.

5. The system as described in claim 1, characterized in that, This includes an AI-powered response and judgment department that determines appropriate response measures based on the driving conditions.

6. The system as described in claim 1, characterized in that, The image generation unit generates images that reproduce abnormal conditions not typically included in the dataset.

7. The system as described in claim 1, characterized in that, The prompt input unit uses speech recognition technology to input abnormal situations that occur during driving as prompt words into the AI ​​generation system in real time.

8. The system as described in claim 1, characterized in that, The response judgment unit determines the appropriate response measures based on the situation generated by the AI ​​input.

Images