Automotive user status scenario generation system

Abstract

In a scenario generation system for automobile user states, scenarios describing automobile user states can be generated with high accuracy even if some modality data is missing. [Solution] The scenario generation system includes a designation unit (211) that specifies a scenario to be predicted relating to an automobile or a user, an identification unit (212) that, when multimodal data relating to an automobile or a user is input, identifies modality data that is necessary for generating the scenario using multimodal LLM but has not been observed, and an estimation unit (213) that estimates the modality data using other modality data using multimodal LLM.

Description

Technical Field

[0004] , , , ,

[0005] , , ,

[0003] , , , ,

[0001] The present invention relates to the technical field of a scenario generation system for an automobile user state, which generates a scenario of a user state (hereinafter simply referred to as "automobile user state" as appropriate) in an automobile or a vehicle by utilizing a multimodal large language model (LLM).

Background Art

[0002] Regarding information such as the psychological state and emotions of a user or driver that cannot be directly sensed, it is conceivable to observe some of that information by combining various sensing data. For example, even if some of the modality data handled by machine learning is missing, a learning method for predicting with high accuracy has been proposed (see Non-Patent Document 1).

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, according to the above-described background art, even if information that cannot be directly sensed is indirectly observed by combining various sensing data, often only some of that information can be observed. There is a technical problem that even in the case of machine learning that makes predictions when some modality data is missing, further improvement in processing efficiency or prediction accuracy is desired.

[0005] The present invention aims to provide a scenario generation system for automobile user states that can generate scenarios describing automobile user states with high accuracy, even if there are some missing modality data. [Means for solving the problem]

[0006] One aspect of the automobile user state scenario generation system according to the present invention, in order to solve the above problems, includes: a designation unit that designates a scenario to be predicted relating to at least one of an automobile and the user of the automobile; an identification unit that, when multimodal data relating to the automobile and the user is input, identifies modality data that is necessary to generate the designated scenario but has not been observed using a multimodal LLM; an estimation unit that estimates the identified modality data using other modality data using a multimodal LLM; and a prediction unit that predicts the scenario using a multimodal LLM based on the input multimodal data and the estimated modality data. [Effects of the Invention]

[0007] According to one embodiment of the automobile user state scenario generation system of the present invention, even if there are some missing modality data, scenarios describing the automobile user state can be generated with high accuracy by mutually complementing the multimodal data.

[0008] The effects and benefits of the present invention will be further clarified by the embodiments of the invention described below. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing the overall configuration of the scenario generation system according to the embodiment. [Figure 2] This is a schematic conceptual diagram illustrating the scenario estimation and generation process in the scenario generation system according to the embodiment. [Figure 3] This flowchart shows an example of processing in the scenario generation system according to the embodiment. [Modes for carrying out the invention]

[0010] First, with reference to Figure 1, the overall configuration of the automobile user state scenario generation system according to this embodiment (hereinafter simply referred to as the "scenario generation system") will be described. In this embodiment or application, "automobile user state scenario" means a scenario relating to the state of the automobile, a scenario relating to the state of the user, or a scenario relating to both states, and the "scenario generation system" means a system that estimates and generates such scenarios.

[0011] In this embodiment, as will be explained in detail below, even if some modality data is missing, a scenario describing the automobile user state is generated by utilizing a multimodal LLM to supplement or mutually supplement the missing modality data with other modality data. Specifically, the system is configured to input multimodal data into a multimodal LLM, identify modality data that is necessary for scenario generation but has not been observed, and then estimate this identified multimodal data using other modality data.

[0012] In this embodiment, the "multimodal LLM" or "machine learning" employed may include traditional AI learning systems such as supervised learning, unsupervised learning, or reinforcement learning, as well as novel generative AI or multimodal LLM technologies that have recently been put into practical use, are currently under development, or will be developed in the future. For example, the AI ​​learning or multimodal LLM learning described here may be configured using a neural network that performs efficient learning through representation learning, transfer learning, feature selection, fine-tuning or hyperparameter tuning, ensemble learning, etc.

[0013] As shown in Figure 1, the scenario generation system according to this embodiment is configured to include an on-board unit 101 mounted on a vehicle 100 and a server device 200. The on-board unit 101 and the server device 200 are connected to a communication network 10, such as the Internet or a dedicated network line. Other multiple or numerous vehicles 100 are similarly connected to the communication network 10.

[0014] The communication network 10 also includes a user data collection unit 306 that collects data specific to the driver or user of the automobile 100 (i.e., "user data"). The user data collection unit 306 collects user data specific to the user or driver, such as work schedule data, medical visit history data, driver behavior data, and driving scene data, in a state where some of this data may be missing.

[0015] In addition to the user data collection unit 306, the communication network 10 may also house a general knowledge collection unit (not shown) that collects "common sense" useful when making estimations based on verbalized or documented data when utilizing the multimodal LLM, and a map data collection unit (not shown) that collects road maps, multipurpose maps, etc., useful when making estimations based on data related to the state and conditions of the vehicle 100's driving. Furthermore, the communication network 10 may house an external related knowledge collection unit (not shown) that collects information obtained outside the vehicle 100 that can be used to impart domain knowledge by performing fine-tuning or hyper-tuning in the scenario generation system (i.e., "external related knowledge").

[0016] Such a user data collection unit 306, or the general knowledge collection unit and map data collection unit (not shown) described above, may be provided at least partially within the server device 200 or within the facility where the server device 200 is located, or within the vehicle-mounted unit 101 or within the vehicle 100.

[0017] A database (DB) 300, which stores various data including data used in the scenario generation system, is connected to the server device 200. The DB 300 may be connected to the server device 200 or the vehicle-mounted unit 101 via the communication network 10. The server device 200 is composed of various computer-equipped devices and various computer devices that perform centralized or distributed processing. In other words, the scenario generation system is constructed as a system that performs centralized or distributed processing using the large-scale data in the DB 300.

[0018] In Figure 1, the automobile 100 may be, for example, a so-called HEV (Hybrid Electric Vehicle), PHEV (Plugin HEV), FCEV (Fuel Cell EV), BEV (Battery Electric Vehicle), or a vehicle primarily powered by an internal combustion engine, but it is constructed as a so-called connected car.

[0019] The in-vehicle unit 101 comprises an interface unit 20, a vehicle behavior / driving operation (CAN) unit 21, a location information unit 22, an in-vehicle / out-of-vehicle camera video unit 23, a biometric data unit 24, a processing unit 26, and a communication unit 28.

[0020] The interface unit 20 is configured to communicate with the driver or user in the vehicle using voice and images. The interface unit 20 is configured to communicate, for example, when using navigation functions, AV functions, autonomous driving functions, etc., provided in the automobile 100, using voice input or predetermined operations on images. All or part of these functions may be configured to be executed by the processing unit 26, or some or all of them may be configured to be executed by the processing unit 210 on the server device 200 side (in other words, the in-vehicle unit 101 side may exclusively perform browser functions). The interface unit 20 is further configured to output data showing the estimation results from machine learning, received from the server device 200 side, in some format as voice or images as appropriate.

[0021] The vehicle behavior and driving operation (CAN) unit 21 includes various sensors such as a vehicle speed sensor, an acceleration sensor, a distance sensor, an engine speed sensor, a temperature sensor, an altitude sensor, a pressure sensor, a battery remaining amount sensor, etc., which are respectively deployed at predetermined positions within the automobile 100. The vehicle behavior and driving operation (CAN) unit 21 is configured to be able to appropriately transmit CAN (Controlled Area Network) data and the like related to the current driving state of the automobile 100 to the server device 200 side via the communication network 10 from the communication unit 28 under the control of the processing unit 26.

[0022] The position information unit 22 includes a GPS device, an inertial navigation positioning device, etc., and detects the current position of the automobile 100. The position information unit 22 is further configured to be able to appropriately generate raw position information data indicating, for example, latitude and longitude or coordinates, and transmit it to the server device 200 side via the communication network 10 from the communication unit 28 under the control of the processing unit 26.

[0023] The in-vehicle and out-vehicle camera moving image unit 23 includes one or more cameras such as CCDs, and functions as a drive recorder that captures the conditions inside and outside the vehicle. The in-vehicle and out-vehicle camera moving image unit 23 is further configured to be able to appropriately transmit the drive recorder video as image data to the server device 200 side via the communication network 10 from the communication unit 28 under the control of the processing unit 26.

[0024] The biometric data unit 24 directly or indirectly detects the biometric data of the driver or user, for example, by various vital sensors deployed at predetermined positions within the automobile 100 and image analysis devices, voice analysis devices, etc. that analyze the appearance and voice of the driver or user. The biometric data unit 24 is further configured to be able to appropriately transmit the detected biometric data to the server device 200 side via the communication network 10 from the communication unit 28 under the control of the processing unit 26.

[0025] The processing unit 26 includes a CPU, memory, etc., which controls the interface unit 20, the vehicle behavior / driving operation (CAN) unit 21, the location information unit 22, the in-vehicle / out-of-vehicle camera video unit 23, the biometric data unit 24, and the communication unit 28. The processing unit 26 appropriately transmits multimodal data detected or generated at various parts of the automobile 100 and its driver from the communication unit 28 to the server device 200 in a predetermined format. Furthermore, it is configured to appropriately receive data indicating estimation results from the server device 200 via the communication unit 28.

[0026] The communication unit 28 includes a modem or the like that is configured to communicate with the outside of the vehicle via the communication network 10. The communication unit 28 is configured to appropriately transmit various multimodal data collected by the automobile 100 to the server device 200 via the communication network 10 in a predetermined format under the control of the processing unit 26.

[0027] In Figure 1, the server device 200 is configured to include a communication unit 201 and a processing unit 210.

[0028] The communication unit 201, under the control of the processing unit 210, appropriately receives various multimodal data collected by the automobile 100 via the communication network 10, and on the other hand, appropriately receives user data collected by the user data collection unit 306 via the communication network 10, and is configured to pass the received data to the processing unit 210.

[0029] The processing unit 210 comprises a designation unit 211, a specification unit 212, an estimation unit 213, and a prediction unit 214. The various processes performed here will be described in detail later with reference to Figures 2 and 3.

[0030] The designation unit 211 is configured to specify a scenario to be predicted relating to at least one of the automobile 100 and the user of the automobile 100. For example, the scenario is specified in sentence format to the effect of, "Infer the 'user state' step by step based on the input data."

[0031] The identification unit 212 is configured to identify modality data that is necessary to generate a scenario specified by the designation unit 211 but has not been observed, when multimodal data relating to the automobile 100 and the user of the automobile 100 is input, using multimodal LLM. For example, it identifies modality data or multimodal data that is necessary to generate a "user state" but has not been observed. More specifically, for example, it identifies the user's "heart rate," which has not been observed, as information necessary to narrow down the "user state."

[0032] The estimation unit 213 estimates the modality data identified by the identification unit 212 using other modality data via multimodal LLM. The estimation unit 213 is further configured to sequentially estimate the missing modality data via multimodal LLM whenever modality data is insufficient for the prediction unit 214 to predict a scenario. For example, the user's "heart rate" identified by the identification unit 212 is estimated from user data related to other types of vital signs, as well as in-vehicle temperature data, altitude data, speed data, etc., included in the CAN data.

[0033] The prediction unit 214 predicts the scenario specified by the designation unit 211 using a multimodal LLM, based on the input multimodal data and modality data estimated by the estimation unit 213. The prediction unit 214 is further configured to sequentially predict the scenario using a multimodal LLM, based on the input multimodal data and estimated modality data, as well as the sequentially estimated modality data, whenever missing modality data is sequentially estimated by the estimation unit 213. For example, it predicts the "user state" scenario by also taking into account the user's "heart rate" estimated by the estimation unit 213 (in other words, it estimates the "user state" scenario using a multimodal LLM).

[0034] DB300 is configured to include a large-scale and high-speed data input / output storage device that stores various modality data or multimodal data received by the server device 200 via the communication network 10, in particular various data necessary for estimation processing using machine learning, and data related to the estimation results or intermediate results generated by the processing unit 210.

[0035] Next, referring to the block diagram in Figure 1 and the schematic conceptual diagram in Figure 2, we will explain the scenario estimation and generation using multimodal LLM, which is mainly performed in the processing unit 210 according to this embodiment.

[0036] In Figure 2, the multimodal data D includes input data, in other words, data that is either missing or partially complete or observable (data D enclosed by a solid line in Figure 2), and data D estimated from other data, in other words, data that is missing or partially complete or unobservable (data D enclosed by a dotted line in Figure 2).

[0037] In the example shown in Figure 2, the observable data D consists of biometric data, vehicle data, voice interaction data, and navigation setting data, which are either input to the processing unit 210 from the in-vehicle unit 101 or the user data collection unit 306 (see Figure 1), or are input after being temporarily stored in the database DB 300 (see Figure 1). On the other hand, the unobservable data D consists of road condition data, work schedule data, medical visit history data, driver behavior data, and driving scene data.

[0038] In Figure 2, the prediction target 11, which is identified as "user state," is specified by the specification unit 211 (see Figure 1) via verbal instructions in sentence form or other input operations, such as "Please generate a scenario describing the user state from the input multimodal data." Then, the estimation process using the multimodal LLM 12 in the identification unit 212 (see Figure 1) identifies the unobserved data D (data D enclosed by a dotted line in Figure 2). Subsequently, the estimation process using the multimodal LLM 12 in the estimation unit 213 (see Figure 1) estimates the unobserved data D (data D enclosed by a dotted line in Figure 2). In Figure 2, the "Estimate" arrow indicates the relationship between the source and destination of the estimation. Furthermore, based on the input data D and this estimated data D, the estimation process using the multimodal LLM 12 in the prediction unit 214 predicts or estimates scenario 13.

[0039] Furthermore, in the scenario generation process explained with reference to Figure 2, when estimating partially missing modality data D using other modality data D, the uncertainty of each modality data D may also be considered when selecting the multimodal data D to be used. By specifying a condition to the multimodal LLM12 to "consider uncertainty," such selection that takes uncertainty into account can be performed.

[0040] Next, with reference to the block diagram in Figure 1 and the schematic conceptual diagram in Figure 2, as well as the flowchart in Figure 3, an example of processing in the scenario generation system according to this embodiment (in particular, processing performed by the processing unit 210 in the server device 200) will be further explained.

[0041] In this embodiment, the general procedure is as follows: First, multimodal data D in which some modality data D is missing, and the target to be predicted, are input into the multimodal LLM. Subsequently, the modality data D missing for prediction is inferred step by step by the multimodal LLM, and the process of estimating that modality data D using other modality data D is performed sequentially. The accuracy of the scenario describing the user state is improved by executing this sequential process.

[0042] In Figure 3, the prediction target scenario is first specified by the designation unit 211 (see Figure 1) (step S1). The specification here is, for example, a "user state" scenario such as the prediction target 11 shown in Figure 2.

[0043] Next, the identification unit 212 (see Figure 1) determines whether or not there is modality data D that is necessary for predicting the specified target scenario, i.e., for scenario generation, but has not yet been observed (step S2).

[0044] Furthermore, if the determination in step S2 indicates that there is any unobserved modality data D (step S2: Yes), the unobserved modality data D that is most necessary for generating the specified prediction target scenario is identified by the identification unit 212 (see Figure 1) (step S3). Simultaneously with or in parallel with this identification process, other modality data D necessary for generating the specified prediction target scenario are identified by the identification unit 212 (see Figure 1) (step S4).

[0045] Next, the modality data D identified in steps S4 and S5 is estimated or predicted by the estimation process using the multimodal LLM in the estimation unit 213 (step S5), and the process returns to the determination process in step S2.

[0046] On the other hand, if, as a result of the determination in step S2 following step S1, there is no unobserved modality data D (step S2: No), or if, as a result of the determination performed once or repeatedly after the prediction in step S5, there is no unobserved modality data D (step S2: No), the estimation process using LLM in the prediction unit 214 predicts or generates a scenario (step S6), and the series of processes related to the scenario generation is completed. Note that such a series of processes may be called periodically or irregularly as a subroutine that can be executed repeatedly in a short time and executed as appropriate.

[0047] A concrete example of estimation using mutual complementation of multimodal data, as explained above with reference to Figures 1 to 3, is described below.

[0048] The first specific example is the scenario estimation of user state, where a specification such as "Infer the user state step by step based on the input data" is given. In this case, complementary estimations such as "If the user state cannot be narrowed down, please provide the information needed to narrow it down," "Based on biometric data, the heart rate is higher than usual," "Based on vehicle data and weather data, the temperature setting inside the vehicle is appropriate," "Possible causes of a high heart rate include stress, tension, lack of sleep, excitement, caffeine or alcohol, and illness," and "Road traffic conditions are needed to narrow down the state" are performed by the multimodal LLM12 (see Figure 2) in the identification unit 212, estimation unit 213, and prediction unit 214 (see Figure 1).

[0049] The second specific example is the estimation of road traffic conditions from vehicle data. For example, estimation through mutual complementarity, such as "The road appears to be congested. Information on work status is needed to narrow down the condition," is performed by the multimodal LLM12 (see Figure 2) in the identification unit 212, estimation unit 213, and prediction unit 214 (see Figure 1).

[0050] The third specific example is the estimation of working hours from home and work visit times in vehicle GPS data. For example, estimations based on mutual complementarity, such as "It seems that the person has been working late into the night recently. We need information on medical visits to narrow down the situation," are performed by the multimodal LLM12 (see Figure 2) in the identification unit 212, estimation unit 213, and prediction unit 214 (see Figure 1).

[0051] The fourth specific example is the estimation of medical visit history from vehicle GPS data. For example, estimation through mutual complementarity, such as "There is no medical visit history. Driver driving behavior is needed to narrow down the condition," is performed by the multimodal LLM12 (see Figure 2) in the identification unit 212, estimation unit 213, and prediction unit 214 (see Figure 1).

[0052] The fifth specific example is the detection of driver behavior from in-vehicle driving operations. For example, estimations based on mutual complementarity, such as "Sudden braking is frequent and the speed is unstable," and "Information about the driving scene is needed to narrow down the state," are performed by the multimodal LLM12 (see Figure 2) in the identification unit 212, estimation unit 213, and prediction unit 214 (see Figure 1).

[0053] The sixth specific example is the estimation of a driving scene from voice dialogue data and navigation setting data. For example, estimations such as "The user seems to be commuting as usual, but today it appears they will drop off their child at daycare before heading to work," "Recently, they have been busy with work, working from early morning until late at night, and seem to be under a lot of stress," "Their pulse is more unstable than usual, and if this condition continues, they may become ill," and "They are driving on an unfamiliar road for a drop-off / pick-up and are feeling irritated and anxious due to unexpected congestion, so they need to be careful not to cause an accident" are performed by the multimodal LLM12 (see Figure 2) in the identification unit 212, estimation unit 213, and prediction unit 214 (see Figure 1) through mutual complementarity.

[0054] As described in detail above, according to this embodiment, even if there are some missing modality data or multimodal data, scenarios describing the state of the automobile user can be generated with high accuracy by mutually complementing the multimodal data.

[0055] In the identification unit 212, estimation unit 213, and prediction unit 214 (see Figure 1), which perform various estimation processes, for example, in order to efficiently execute the multimodal LLM described above, the scenario may be converted into text entirely using the multimodal LLM at least partially, and then vectorized. Furthermore, in each process using the multimodal LLM described above, a large-scale language model may be fine-tuned using the multimodal LLM, for example, by using a large amount of text data. This makes it adaptable to various natural language processing (NLP) tasks such as text classification, sentiment analysis, information extraction, text summarization, text generation, and question answering.

[0056] The designated unit 211, the specific unit 212, the estimation unit 213, and the prediction unit 214 (see Figure 1) described above are represented as separate parts as functional blocks that perform different processing in this embodiment. However, these may be composed of a single processor or the like as hardware, or they may be configured to be functionally treated separately in software.

[0057] In the scenario generation system according to this embodiment, the identification unit 212, estimation unit 213, and prediction unit 214 (see Figure 1) may be configured to update existing or future improved autoregressive generative models, in addition to multimodal LLMs, models such as chat GPT or Transformer. Alternatively, they may be configured to update a model that generates scenarios as text after estimating individual states and situations using a time series model. The model here may be, for example, one estimated using a state-space model, latent variable model, or time series prediction model (e.g., existing Kalman filters, particle filters, recurrent neural networks, LTSM, HiPPO, LSSL, S4, Mamba, etc.), and obtained by inputting its high-dimensional output value or embedding vector into an autoregressive generative model. In any case, the effect of this embodiment, "being able to generate scenarios describing the state of an automobile user with high accuracy even if there are some missing modality data," is appropriately demonstrated.

[0058] Note The following additional information is disclosed regarding the embodiments described above.

[0059] [Note 1] The automobile user state scenario generation system described in Appendix 1 of the present invention comprises: a designation unit that designates a scenario to be predicted relating to at least one of an automobile and the user of the automobile; an identification unit that, when multimodal data relating to the automobile and the user is input, identifies modality data that is necessary to generate the designated scenario but has not been observed using a multimodal LLM; an estimation unit that estimates the identified modality data using other modality data using a multimodal LLM; and a prediction unit that predicts the scenario using a multimodal LLM based on the input multimodal data and the estimated modality data.

[0060] According to the scenario generation system described in Appendix 1, when multimodal data related to automobiles and users is input, the identification unit identifies modality data that is necessary to generate the specified scenario but has not been observed, using a multimodal LLM. The estimation unit then estimates the identified modality data using a multimodal LLM with other modality data. Then, based not only on the input multimodal data but also on the estimated modality data, the prediction unit predicts the scenario using a multimodal LLM. Therefore, even if there are some missing modality data, scenarios describing the automobile user state can be generated with high accuracy by mutually complementing the multimodal data.

[0061] [Note 2] The scenario generation system described in Appendix 2 of the present invention is a scenario generation system for automobile user states described in Appendix 1, characterized in that the estimation unit further estimates the missing modality data using a multimodal LLM whenever the prediction unit lacks sufficient modality data to predict the scenario.

[0062] According to the scenario generation system described in Appendix 2 of the present invention, whenever modality data is insufficient for the prediction unit to predict a scenario, the estimation unit sequentially estimates the missing modality data using multimodal LLM. Therefore, even if there are some missing modality data, the missing modality data is estimated sequentially without delay each time, enabling the generation of scenarios with increasing accuracy in an incremental manner.

[0063] [Note 3] The scenario generation system described in Appendix 3 of the present invention is a scenario generation system for automobile user states described in Appendix 2, characterized in that the prediction unit further predicts the scenario using a multimodal LLM based on the input multimodal data and the estimated modality data, in addition to the sequentially estimated modality data, each time the missing modality data is sequentially estimated.

[0064] According to the scenario generation system described in Appendix 3 of the present invention, each time missing modality data is sequentially estimated, the prediction unit sequentially predicts a scenario using multimodal LLM, based not only on the input multimodal data and estimated modality data, but also on the sequentially estimated modality data. Therefore, even if there is some missing modality data, the missing modality data is sequentially estimated and the scenario is sequentially predicted without delay each time, so that scenarios can be generated with increasing accuracy in an incremental manner.

[0065] [Note 4] The scenario generation method described in Appendix 4 of the present invention comprises: a designation step of designating a scenario to be predicted relating to at least one of an automobile and a user of the automobile; an identification step of identifying modality data that is necessary to generate the designated scenario but has not been observed by a multimodal LLM when multimodal data relating to the automobile and the user is input; an estimation step of estimating the identified modality data using other modality data by a multimodal LLM; and a prediction step of predicting the scenario by a multimodal LLM based on the input multimodal data and the estimated modality data.

[0066] According to the scenario generation method described in Appendix 4 of the present invention, similar to the scenario generation system described in Appendix 1, even if there are some missing modality data, scenarios describing the state of an automobile user can be generated with high accuracy by mutually complementing the multimodal data.

[0067] The present invention may be modified as appropriate, without contradicting the gist or spirit of the invention as can be inferred from the claims and the specification as a whole, and scenario generation systems and methods involving such modifications are also included in the technical concept of the present invention. [Explanation of Symbols]

[0068] Vehicle behavior / driving operation (CAN) section...21 Location information department...22 In-car and exterior camera video unit...23 Biometric Data Department...24 Server equipment...200 Processing unit...210 Designated section...211 Specific section...212 Estimation department...213 Prediction section... 214 DB...300 User Data Collection Department...306

Claims

1. A designation unit that specifies predictable scenarios relating to at least one of an automobile and the user of the automobile, When multimodal data relating to the vehicle and the user is input, the identification unit identifies modality data that is necessary for generating the specified scenario using multimodal LLM but has not been observed, An estimation unit that estimates the identified modality data using other modality data by multimodal LLM, A prediction unit predicts the scenario using multimodal LLM based on the input multimodal data and the estimated modality data. A scenario generation system for automobile user states, characterized by comprising the following features.

2. The scenario generation system for automobile user states according to claim 1, further characterized in that the estimation unit sequentially estimates the missing modality data using multimodal LLM whenever the prediction unit lacks sufficient modality data to predict the scenario.

3. The scenario generation system for automobile user states according to claim 2, further characterized in that the prediction unit sequentially predicts the scenario using multimodal LLM based on the input multimodal data and the estimated modality data, in addition to the sequentially estimated modality data, each time the missing modality data is sequentially estimated.

4. A specifying step of specifying a scenario to be predicted relating to at least one of the automobile and the user of the automobile, When multimodal data relating to the vehicle and the user is input, the multimodal LLM performs an identification step to identify modality data that is necessary but not observed in order to generate the specified scenario, An estimation step in which the identified modality data is estimated using other modality data by multimodal LLM, A prediction step in which the scenario is predicted by multimodal LLM based on the input multimodal data and the estimated modality data. A method for generating scenarios of automobile user states, characterized by comprising the following features.

Images