Information processing device, information processing method, and program

Abstract

This information processing device comprises a text generation unit and an input processing unit. The text generation unit generates at least one among a first text relating to a condition of a moving body, a second text relating to a driver of the moving body, a third text relating to a passenger of the moving body, and a fourth text relating to an operating condition of an apparatus that is mounted to the moving body. The input processing unit executes processing to generate instruction information for causing a large language model to generate response information relating to the load on the driver of the moving body, said instruction information including at least one among the first text, the second text, the third text, and the fourth text, and to input the instruction information into the large language model.

Description

Information Processing Apparatus, Information Processing Method, and Program

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

[0002] In recent years, technologies for supporting a driver riding in a moving body using information processing technologies have been developed. For example, Patent Document 1 discloses a device that calculates a driving load. This device acquires external information indicating the situation outside the vehicle and state information indicating the state of the driver of this vehicle. Then, based on the acquired external information and state information, this device calculates the driving load of the driver. Patent Document 1 discloses using artificial intelligence such as a neural network for the calculation of the driving load.

[0003] Japanese Unexamined Patent Application Publication No. 2019 - 53436

[0004] In estimating the load applied to the driver, it is required to improve the accuracy while suppressing an increase in the amount of calculation. An example of the object of the present invention is to improve the accuracy while suppressing an increase in the amount of calculation in estimating the load applied to the driver.

[0005] The invention according to claim 1 is an information processing apparatus including: a text generation unit that generates at least one of a first text regarding the situation of a moving body, a second text regarding a driver of the moving body, a third text regarding a passenger of the moving body, and a fourth text regarding the operation status of equipment mounted on the moving body; and an input processing unit that generates instruction information for causing a large - language model to generate response information regarding the load of the driver of the moving body, the instruction information including the at least one of the first text, the second text, the third text, and the fourth text, and executes a process for inputting the instruction information into the large - language model.

[0006] The invention described in claim 8 is an information processing method in which a computer generates at least one of a first text relating to the status of a mobile body, a second text relating to the driver of the mobile body, a third text relating to a passenger of the mobile body, and a fourth text relating to the operating status of equipment mounted on the mobile body, and generates instruction information for a large-scale language model to generate response information relating to the load of the driver of the mobile body, the instruction information comprising at least one of the first text, the second text, the third text, and the fourth text, and performs processing to input the instruction information into the large-scale language model.

[0007] The invention described in claim 9 is a program that provides a computer with: a text generation unit that generates at least one of a first text relating to the status of a mobile body, a second text relating to the driver of the mobile body, a third text relating to a passenger of the mobile body, and a fourth text relating to the operating status of equipment mounted on the mobile body; and an input processing unit that generates instruction information for a large-scale language model to generate response information relating to the load of the driver of the mobile body, the instruction information including at least one of the first text, the second text, the third text, and the fourth text, and performs processing to input the instruction information into the large-scale language model.

[0008] This diagram illustrates the operating environment and functional configuration of the information processing apparatus according to the embodiment. It shows items that may be included in the first text, the second text, the third text, and the fourth text. It also shows an example of instruction information, an example of the hardware configuration of the information processing apparatus, and a flowchart illustrating an example of processing performed by the information processing apparatus.

[0009] Embodiments of the present invention will be described below with reference to the drawings. In all drawings, similar components are denoted by the same reference numerals, and their descriptions are omitted as appropriate.

[0010] (First Embodiment) Figure 1 is a diagram illustrating the usage environment and functional configuration of the information processing device 10 according to this embodiment. The information processing device 10 performs processing to estimate the load of the driver riding in the mobile body 20. The information processing device 10 also performs processing to support the driver using the results of this processing. One example of this processing is controlling a target device 220 mounted on the mobile body 20. The mobile body 20 is, for example, a car or a bus, but is not limited to these.

[0011] The information processing device 10 may be mounted on the mobile device 20 or located outside the mobile device 20. For example, the information processing device 10 can be incorporated as a function of a car navigation system or an in-vehicle device. The information processing device 10 may also communicate with an external device, such as a server, as needed. One example of such an external device is a device that performs route suggestion and route search and stores map information. This map information also contains information about facilities such as stores. Furthermore, the information processing device 10 may be realized by a combination of a device mounted on the mobile device 20 and a device outside the mobile device 20.

[0012] The information processing device 10 is used together with the sensor 210, the target device 220, and the model device 30. The sensor 210 and the target device 220 move together with the mobile body 20. The sensor 210 and the target device 220 are, for example, incorporated into the mobile body 20, but may also be incorporated into a portable communication device held by the driver. Alternatively, one of the sensor 210 and the target device 220 may be incorporated into the mobile body 20, and the other into the communication device. In addition, the mobile body 20 may have multiple sensors 210 and multiple target devices 220 incorporated into it. In this case, some of the multiple sensors 210 may be incorporated into the mobile body 20, and the remaining sensors 210 may be incorporated into the communication device. The same applies to the multiple target devices 220.

[0013] Sensor 210 repeatedly generates data used when generating information about the driver's load and transmits the generated data to the information processing device 10. Hereinafter, the information generated by sensor 210 will be referred to as sensor information. Sensor 210 is, for example, at least one of the following. At least some of these examples may also be handled by the control unit that controls various devices of the mobile body 20. - Speedometer of the mobile body 20 - Acceleration sensor of the mobile body 20 - Sensor that detects the open / closed state of the doors and windows of the mobile body 20 - Sensor that detects the illuminated state of the lights (including fog lamps) for the front lighting of the mobile body 20 - Sensor that detects operations performed on the steering wheel of the mobile body 20 - Sensor that detects the current position of the mobile body 20 (e.g., GPS) - Sensor that acquires or detects information indicating at least one of the type of road the mobile body 20 is currently traveling on and traffic conditions (e.g., congestion status such as whether there is a traffic jam or whether there is construction) (e.g., navigation device) - Imaging device that photographs at least one of the surroundings of the mobile body 20, such as the front, rear, and side. - Imaging device that photographs the passenger space of the mobile body 20. - A sensor that detects the driver's biometric information. The biometric information includes, for example, heart rate, blood pressure, respiratory rate, and sweating status. This sensor may be attached to the driver. - A clock - A thermometer that detects the ambient temperature around the mobile body 20 - A thermometer that detects the ambient temperature in the passenger compartment of the mobile body 20 - A microphone that detects the sound in the passenger compartment of the mobile body 20 - A microphone that detects the ambient sound around the mobile body 20

[0014] The target device 220 is, for example, at least one of the following: • A device that controls the opening and closing state of the windows of the mobile body 20; • Air conditioning equipment with temperature control function; • Navigation device; • Audio device; • Interactive device. It may also have the function of controlling at least one of the navigation device and the audio device. • Lights (including fog lamps) for the forward illumination of the mobile body 20.

[0015] When the information processing device 10 performs processing to estimate the load on the driver, it may also use data indicating the operating status of the target device 220. Hereinafter, this information will be referred to as operating status data. The operating status data is generated, for example, by the target device 220. The target device 220 repeatedly generates the operating status data and transmits it to the information processing device 10.

[0016] The model device 30 performs processing using large language models (LLMs). The information processing device 10 generates instruction information, such as a prompt, which is input to the model device 30, and transmits this instruction information to the model device 30. The model device 30 inputs this instruction information into the large language model, obtains the response information generated by the large language model, and transmits it to the information processing device 10.

[0017] The information processing device 10 then performs processing using the response information. One example of this processing is to cause the target device 220 to perform a predetermined process. A specific example of this predetermined process will be described later.

[0018] The information processing device 10 may also serve as the model device 30.

[0019] The information processing device 10 includes a text generation unit 110, an input processing unit 120, and a response processing unit 130, and can utilize a storage unit 140. The storage unit 140 stores various types of information used by the information processing device 10. The storage unit 140 may be part of the information processing device 10 or may be located outside the information processing device 10.

[0020] The text generation unit 110 generates at least one of the following: a first text relating to the status of the mobile body, a second text relating to the driver of the mobile body, a third text relating to the passengers of the mobile body, and a fourth text relating to the operating status of the equipment mounted on the mobile body. For example, the text generation unit 110 acquires sensor information and processes this sensor information to generate at least one of the first text, second text, third text, and fourth text. Here, an example of equipment mounted on the mobile body is the target device 220.

[0021] As a first example, the text generation unit 110 generates at least one of a first text, a second text, a third text, and a fourth text by processing sensor information using a machine learning model. This machine learning model may be owned by the information processing device 10 or by an external device.

[0022] As a second example, if the sensor information includes a string (e.g., a number), the text generation unit 110 makes at least a part of this string at least a part of the first text. The second example can also be used when generating at least one of the second text, the third text, and the fourth text.

[0023] As a third example, if the sensor information indicates the operation of a device mounted on the mobile body 20, the text generation unit 110 generates a fourth text by processing the sensor information according to predetermined rules.

[0024] As a fourth example, if the sensor information includes an image (e.g., a video), the text generation unit 110 processes this image to detect at least one of the vehicle's occupants, such as the driver and a passenger, and processes this detection result according to predetermined rules to generate at least one of the first text, second text, third text, and fourth text.

[0025] As a fifth example, if the sensor information includes voices based on the occupants of the vehicle, for example, voices based on the speech of at least one of the driver and passengers, the text generation unit 110 generates at least one of the first text, second text, third text, and fourth text by converting this voice into text, or by processing this text according to, for example, a machine learning model or predetermined rules.

[0026] The input processing unit 120 generates instruction information to be input to the large-scale language model used by the model device 30. This instruction information is for causing the large-scale language model to generate response information regarding the load of the mobile device's driver, and includes the text generated by the text generation unit 110 from among the first text, second text, third text, and fourth text.

[0027] The instruction information may further include indicating the load in multiple stages. In this case, the instruction information preferably includes the number of stages. This number is, for example, 2 to 5, but is not limited thereto. This number is, for example, pre-stored in the storage unit 140. The instruction information also preferably includes information indicating at least one criterion for the multiple stages. This information preferably indicates, for example, the criteria for classifying into each stage in text. This information is, for example, pre-stored in the storage unit 140.

[0028] If the instruction information includes "indicating the load in multiple stages," the large-scale language model used by the model device 30 includes information in the response information indicating which stage the driver load is at. Furthermore, if the instruction information includes information indicating at least one criterion for multiple stages, the large-scale language model used by the model device 30 determines which stage the driver load is at according to this information.

[0029] The input processing unit 120 then performs processing to input the instruction information into the large-scale language model. One example of this processing is to send the instruction information to the model device 30. However, if the information processing device 10 also functions as the model device 30, this processing is to input the instruction information into the large-scale language model.

[0030] The text generation unit 110 preferably generates two or more, preferably all, of the first text, second text, third text, and fourth text. Doing so increases the accuracy of the operational load included in the response information generated by the large-scale language model.

[0031] The response processing unit 130 performs processing using the response information. One example of this processing is to cause the target device 220 to perform a predetermined process. The predetermined process is, for example, at least one of the following:

[0032] - Control of the amount of information provided according to the driver load and optimization of interactive devices. When the driver load is high, the response processing unit 130 reduces the amount of information that the target device 220, such as an interactive device, provides to the driver compared to when the driver load is low. By adjusting the amount of information that the target device 220 provides to the driver, the stability of operation by the driver is improved.

[0033] Specific examples of "information quantity control" are as follows: • For audio and text information, change to shorter instructions or summaries when the load is high. • For visual information, display only the minimum necessary information on the screen when the load is high. • Adjust the timing of information delivery. For example, do not provide less urgent information when the load is high, and provide it when the load is low.

[0034] If the target device 220 is an interactive information device, further specific examples of "controlling the amount of information" include: - Adjusting the format of questions. For example, when the load is high, change to closed questions that can be answered with "yes / no". - Adjusting the rhythm of the dialogue. For example, when the load is high, slow down the pace of the dialogue compared to when the load is low. - Providing information according to its importance. For example, navigation information is always provided, but music information is only provided when the load is low.

[0035] Figure 2 shows the items that may be included in the first text, the second text, the third text, and the fourth text. Each of the first to third texts also shows the specific content of these items.

[0036] The information contained in the first text, i.e., information regarding the status of the moving object, may include at least one of the following: time of day, current location of the moving object, weather at the current location, traffic information at the current location, road information at the current location, and information regarding directions from the current location to the destination. More specifically, this information may include at least one of the following items. At least one of these pieces of information may be obtained from an external server, for example, a server that stores weather information or a server that stores road information. Road quality can also be determined, for example, from the magnitude of road noise.・Driving speed ・Traffic information at current location (e.g., traffic volume, degree of congestion (e.g., traffic jam, congestion, or smooth), presence of restrictions (e.g., speed limit, lane restrictions, or road closure)) ・Status of other types of moving objects (e.g., many pedestrians, many cyclists) ・Time of day (e.g., rush hour, late night) ・Road conditions at current location (e.g., under construction, frozen) ・Road quality at current location (e.g., paved, unpaved) ・Type of road at current location (e.g., toll road, highway, general road) ・Surroundings at current location (e.g., urban area, suburbs) ・Weather at current location and its changes ・Temperature at current location ・Humidity at current location ・Lighting conditions at current location (e.g., daytime, nighttime) ・Directions from current location to destination (e.g., distance, type of road) ・Whether the current location is within a living area (e.g., familiar road, unfamiliar road) ・Information about breakdowns (e.g., prone to breakdowns, no problems)

[0037] The information included in the second text, namely the information about the driver of the vehicle, may include, for example, at least one of the following: the driver's biometric information, the driver's emotions, the time since the vehicle was driven, information about breaks taken since the vehicle was driven, and information about the driver's driving experience. More specifically, this information may include at least one of the following items: • Biometric information (e.g., fatigue, vital signs such as heart rate, etc.) • Speech content (e.g., it's getting dark, etc.) • Emotions (e.g., happy, irritated, etc.) • Time since the start of driving, whether or not breaks have been taken, and time since the last break, and driving experience (e.g., 10 years)

[0038] The information contained in the third text, namely information about the passengers of the vehicle, may include at least one of the following items. This information is identified, for example, using images (sometimes video) taken inside the vehicle, and audio information including conversations that took place inside the vehicle. • Number of people • Relationship to the driver (e.g., family, friends, etc.) • Attributes (e.g., age, gender, etc.) • Behavior inside the vehicle (e.g., talking to the driver, etc.)

[0039] The information contained in the fourth text, namely information regarding the operating status of equipment mounted on the mobile device, may include the following items: • The type of equipment operating and its operating status (e.g., car navigation system providing route guidance, audio system operating as radio, etc.)

[0040] Figure 3 shows a first example of instruction information generated by the input processing unit 120. This instruction information includes a first text, i.e., text indicating the "status of the moving object," a second text, i.e., text relating to the "driver," a third text, i.e., text relating to the "passengers," and a fourth text, i.e., text relating to the "status of the equipment." The instruction information shown in Figure 3 further includes information indicating the criteria for each stage of the load information.

[0041] Figure 4 shows an example of the hardware configuration of the information processing device 10. The information processing device 10 includes a bus 1010, a processor 1020, a memory 1030, a storage device 1040, an input / output interface 1050, and a network interface 1060.

[0042] Bus 1010 is a data transmission path for the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060 to send and receive data to and from each other. However, the method of connecting the processor 1020 and the other components to each other is not limited to bus connection.

[0043] Processor 1020 is a processor implemented using components such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit).

[0044] The memory 1030 is a main memory device realized by a RAM (Random Access Memory) or the like.

[0045] The storage device 1040 is an auxiliary storage device realized by an HDD (Hard Disk Drive), an SSD (Solid State Drive), a removable medium such as a memory card, or a ROM (Read Only Memory). The storage device 1040 stores program modules that implement each function of the information processing device 10 (for example, the text generation unit 110, the input processing unit 120, and the response processing unit 130). By the processor 1020 loading and executing these program modules onto the memory 1030, each function corresponding to the program module is realized. The storage device 1040 may further function as the storage unit 140.

[0046] The input / output interface 1050 is an interface for connecting the information processing device 10 and various input / output devices. For example, the information processing device 10 may communicate with at least one of the sensor 210 and the target device 220 via the input / output interface 1050.

[0047] The network interface 1060 is an interface for connecting the information processing device 10 to a network. This network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The method by which the network interface 1060 connects to the network may be a wireless connection or a wired connection. The information processing device 10 may communicate with at least one of the sensor 210 and the target device 220 via the network interface 1060.

[0048] FIG. 5 is a flowchart showing an example of the processing performed by the information processing apparatus 10. The information processing apparatus 10 repeatedly performs the processing shown in this figure. First, the information processing apparatus 10 acquires sensor information from the mobile body 20 (step S10). Then, the text generation unit 110 generates at least one of the first text, the second text, the third text, and the fourth text using this sensor information (step S20). Then, the response processing unit 130 generates instruction information and transmits this instruction information to the model device 30 (step S30).

[0049] The model device 30 inputs the instruction information into a large language model and acquires response information from this large language model. Then, the model device 30 transmits this response information to the information processing apparatus 10. The response processing unit 130 of the information processing apparatus 10 acquires this response information (step S40). Then, the response processing unit 130 executes a predetermined process using this response information (step S50).

[0050] In this way, the information processing apparatus 10 generates at least one of the first text regarding the situation of the mobile body, the second text regarding the driver of the mobile body, the third text regarding the passengers of the mobile body, and the fourth text regarding the operating status of the devices mounted on the mobile body, and generates instruction information including the generated text. Then, the information processing apparatus 10 acquires response information regarding the driver's load by inputting this instruction information into a large language model. Therefore, by using the information processing apparatus 10, the driver's load can be accurately estimated, and the amount of computation required for this estimation can be reduced.

[0051] As described above, the embodiments of the present invention have been described with reference to the drawings, but these are examples of the present invention, and various configurations other than the above can also be adopted.

[0052] Also, in the plurality of flowcharts used in the above description, a plurality of steps (processes) are described in order, but the execution order of the steps executed in each embodiment is not limited to the order described. In each embodiment, the order of the illustrated steps can be changed within a range that does not interfere with the content. Also, the above-described embodiments can be combined within a range where the contents do not conflict.

[0053] This application claims priority based on Japanese Patent Application No. 2024-172085, filed on 1 October 2024, and incorporates all of its disclosures herein.

[0054] 10 Information processing device 20 Mobile device 30 Model device 110 Text generation unit 120 Input processing unit 130 Response processing unit 140 Storage unit 210 Sensor 220 Target device

Claims

A text generation unit that generates at least one of the following: a first text relating to the status of the mobile body, a second text relating to the driver of the mobile body, a third text relating to the passengers of the mobile body, and a fourth text relating to the operating status of equipment mounted on the mobile body. An input processing unit that generates instruction information for a large-scale language model to generate response information regarding the load of the driver of the mobile body, the instruction information comprising at least one of the first text, second text, third text, and fourth text, and performs processing to input the instruction information into the large-scale language model, An information processing device equipped with the following features. 　 In the information processing apparatus according to claim 1, The instruction information further includes indicating the load in multiple stages in the response information, according to the information processing apparatus. 　 In the information processing apparatus according to claim 2, An information processing device comprising the instruction information including information indicating at least one criterion of the plurality of stages. 　 In the information processing apparatus according to any one of claims 1 to 3, Information processing device wherein the first text includes at least one of the following: time of day, current location of the moving object, weather at the current location, traffic information at the current location, road information at the current location, and information regarding directions from the current location to the destination. 　 In the information processing apparatus according to any one of claims 1 to 4, The second text is an information processing device that includes at least one of the driver's biometric information, the driver's emotions, the time since the start of driving the vehicle, information regarding breaks since the start of driving the vehicle, and information regarding the driver's driving experience. 　 In the information processing apparatus according to any one of claims 1 to 5, The text generation unit is an information processing device that acquires sensor information generated by a sensor mounted on the moving body and generates at least one of the first text, second text, third text, and fourth text using the sensor information. 　 In the information processing apparatus according to any one of claims 1 to 6, An information processing device comprising a response processing unit that acquires the aforementioned response information and uses the response information to cause a target device that moves together with the moving body to perform a predetermined process. 　 Computers At least one of the following is generated: a first text concerning the status of the mobile body, a second text concerning the driver of the mobile body, a third text concerning the passengers of the mobile body, and a fourth text concerning the operating status of the equipment mounted on the mobile body. An information processing method that generates instruction information for a large-scale language model to generate response information relating to the load of the driver of the mobile body, the instruction information comprising at least one of the first text, second text, third text, and fourth text, and performs processing to input the instruction information into the large-scale language model. 　 On the computer, A text generation unit that generates at least one of the following: a first text relating to the status of the mobile body, a second text relating to the driver of the mobile body, a third text relating to the passengers of the mobile body, and a fourth text relating to the operating status of equipment mounted on the mobile body. An input processing unit that generates instruction information for a large-scale language model to generate response information regarding the load of the driver of the mobile body, the instruction information comprising at least one of the first text, second text, third text, and fourth text, and performs processing to input the instruction information into the large-scale language model, A program to give it a specific feature.

Images