Data processing device, data processing system, and data processing method

Abstract

A data processing device 1 comprises: a control signal acquisition unit 13 that acquires a control signal of a vehicle; and a prompt generation unit 14 that generates a prompt for a language model for outputting language information. The prompt generation unit 14 generates a prompt including vehicle text obtained by converting the control signal into text, and the prompt includes text indicating the format or range of an answer sentence obtained according to the language model.

Description

Data processing device, data processing system, and data processing method

[0001] The present invention relates to a data processing device, a data processing system, and a data processing method.

[0002] A data generation method using a language model has been known for some time (for example, see Patent Document 1). The data generation method described in Patent Document 1 includes a step of using original data to construct a prompt that serves as an input sentence for the language model, and a step of inputting the prompt into the language model and generating new data and label information for the new data from the language model.

[0003] JP 2023-018624 A

[0004] When a device that executes the above-mentioned data generation method is applied to a vehicle, the information contained in the vehicle's control signal is not used as a prompt, which poses a problem in that an answer corresponding to the vehicle's control signal cannot be obtained from the language model.

[0005] An object of the present invention is to provide a data processing device, a data processing system, and a data processing method for obtaining an answer corresponding to a vehicle control signal from a language model.

[0006] The present invention solves the above problem by acquiring a vehicle control signal, generating a prompt including vehicle text that documents the acquired control information, inputting the prompt into a language model, and including text that indicates the format or range of an answer sentence according to the language model.

[0007] According to the present invention, a response corresponding to a vehicle control signal can be obtained from a language model.

[0008] Fig. 1 is a block diagram of a data processing system according to an embodiment of the present invention. Fig. 2 is a diagram showing a prompt generated by the prompt generation unit of Fig. 1. Fig. 3 is a diagram showing a specific example of a voice uttered by a user and a specific example of linguistic information output from a language model 5. Fig. 4 is a diagram showing a prompt generated by the prompt generation unit of Fig. 1. Fig. 5 is a diagram showing a specific example of a voice uttered by a user and a specific example of linguistic information output from a language model 5. Fig. 6 is a flowchart for explaining a data processing method by the data processing system.

[0009] An embodiment of a data processing device, a data processing system, and a data processing method according to the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram of a data processing system according to the embodiment of the present invention. The data processing system includes a data processing device 1, a microphone 2, an on-board camera 3, a CAN communication network 4, a language model 5, and an IVI system 6. The data processing system is installed in a vehicle. The data processing system inputs a user's voice, generates a prompt using speech text converted from the voice signal as well as information included in a vehicle control signal, outputs the prompt to the language model, and obtains an answer from the language model.

[0010] The data processing device 1 generates prompts for a language model 5, which includes vehicle text that is a written version of a vehicle control signal. The data processing device 1 includes a voice data generation unit 10, a control signal acquisition unit 13, a prompt generation unit 14, and a database 15. The data processing device 1 includes a memory (recording medium) that stores programs, a processor for executing the programs, and the like. The data processing device 1 has functions such as generating voice data based on a voice signal, acquiring a vehicle control signal, and generating a prompt, and includes the voice data generation unit 10, the control signal acquisition unit 13, and the prompt generation unit 14 as functional blocks for executing these functions. The processor included in the data processing device 1 executes the programs to perform processing of the functional blocks, such as the voice data generation unit 10. Details of each component included in the data processing device 1 will be described later.

[0011] The microphone 2 is provided in the vehicle and converts the voice of a user inside the vehicle into an audio signal. The on-board camera 3 captures images of the surroundings of the vehicle and outputs the captured image data (captured images) to the data processing device 1. The on-board camera 3 is used, for example, to detect obstacles in autonomous driving control, detect lanes in lane keeping, and display images of the surroundings of the vehicle while parking. The CAN communication network 4 is a network (communication network) for transmitting and receiving control signals between multiple on-board devices such as ECUs, motors, sensors, etc.

[0012] The language model 5 is a machine learning natural language processing model (large-scale language model: LLM) that is built using large amounts of text data and deep learning technology, such as Transformer and BERT. The language model 5 receives a prompt from the data processing device 1, assigns probabilities to sentences and words included in the prompt, and outputs an appropriate response to the input. In other words, the language model 5 inputs a prompt and outputs linguistic information.

[0013] The IVI system 6 is a system that provides information and entertainment to users in the vehicle through an in-vehicle display, speaker, etc. The IVI system 6 includes a text-to-speech system (TTS) that converts the language data generated by the language model 5 into speech and realizes a conversation with the user through display on the display or speech output from the speaker. Note that the IVI system 6 does not necessarily have to include TTS; the speech synthesis included in the language model 5 may be used directly for speech output.

[0014] Next, we will explain each component included in the data processing device 1. The voice data generation unit 10 generates voice data from a voice signal (user's voice) input from the microphone 2. The voice data generation unit 10 has a voice recognition unit 11 and a language understanding unit 12. The voice recognition unit 11 has an automatic speech recognition (ASR) function and converts the voice signal into voice text (text sentence). The voice text is a written version of the voice.

[0015] The language understanding unit 12 analyzes the speech text converted by the speech recognition unit 11 to identify the intent and / or named entity of the user who uttered the speech. The user's intent is what the user is thinking, such as confirming how to operate an in-vehicle device, issuing operational commands to the in-vehicle device, or obtaining information such as a destination or point of interest (POI). The named entity is a predefined language such as the name of the in-vehicle device, a proper noun such as a place name or facility name. For example, if the user utters "set destination," the language understanding unit 12 identifies the user's intention to operate something and extracts the destination as a named entity. The language understanding unit 12 may also analyze characters included in the speech text to analyze what genre the user is referring to. Genres are classified, for example, by concepts related to vehicle operation instructions, such as how to drive a vehicle or operate an in-vehicle device, or by concepts related to a navigation system, such as route guidance and destination setting. The language understanding unit 12 outputs the speech text converted by the speech recognition unit 11 to the prompt generation unit 14, and outputs the intention and / or the named entity to the control signal acquisition unit 13 and the prompt generation unit 14. The language understanding unit 12 may identify either the intention or the named entity. The speech data generation unit 10 does not necessarily have to include the language understanding unit 12. If the speech data generation unit 10 does not include the language understanding unit 12, the output signal of the speech recognition unit 11 may be output to the control signal acquisition unit 13.

[0016] The control signal acquisition unit 13 acquires a vehicle control signal. The control signal corresponds to an image signal or a CAN signal containing image data. The control signal acquisition unit 13 stores information contained in the CAN signal and image data captured by the on-board camera 3 in the database 15, thereby retaining vehicle information and environmental information surrounding the vehicle. The vehicle information indicates the vehicle speed, the current vehicle position, whether autonomous driving is on or off, and the status of lamps and switches contained in the on-board device. The environmental information includes information regarding traffic laws such as no overtaking or no parking, points of interest (POIs) such as shops and facilities around the vehicle, and road information such as stop lines and lane boundaries. The control signal acquisition unit 13 retains, for example, the current vehicle speed and past vehicle speeds (e.g., the vehicle speed two minutes ago) from the vehicle speed information contained in the CAN signal. The control signal acquisition unit 13 may also retain the status (on or off) of various lamps, such as a seat belt warning light, a VDC OFF indicator light, a headlamp upward indicator light, and a turn signal light, from lamp status signals or on / off switching signals. The control signal acquisition unit 13 does not need to store information such as vehicle information in the database 15, and may acquire the vehicle information in real time from the imaging data or the CAN signal.

[0017] The control signal acquirer 13 transmits at least vehicle information to the prompt generator 14. The control signal acquirer 13 may transmit environmental information to the prompt generator 14. The vehicle information and environmental information are information included in the control signal. The control signal acquirer 13 may extract information about the user's voice from the intent and the named entity included in the control signal and the information stored in the database 15, and transmit the extracted information to the prompt generator 14. The control signal acquirer 13 may output the acquired control signal to the prompt generator 14.

[0018] The prompt generation unit 14 identifies vehicle information related to the voice uttered by the user based on the control signal, and generates a prompt including vehicle text. FIG. 2 is a diagram showing a prompt generated by the prompt generation unit 14. In the example of FIG. 2, the user utters a voice such as "What does the light that's on mean now?". As shown in FIG. 2, the prompt is displayed as text. The prompt generation unit 14 includes voice text (text included in voice data) converted by the voice recognition unit 11 in the prompt. In the example of FIG. 2, "What does the light that's on mean now?" corresponds to the voice text. The prompt generation unit 14 identifies vehicle information related to the voice from the vehicle information input from the control signal acquisition unit 13. The vehicle information related to the voice is vehicle information related to the voice uttered by the user. For example, vehicle information related to the user's intention, vehicle information including a named entity, and vehicle information related to the named entity are vehicle information related to the voice.

[0019] When the prompt generation unit 14 can identify the content of the user's question from the intent, the prompt generation unit 14 may identify information that may be an answer to the question or information related to the information that may be an answer as vehicle information related to the voice. For example, when the user utters, "Please explain VDC (Vehicle Dynamics Control)," the prompt generation unit 14 identifies the VDC as an in-vehicle device that is related to the answer and determines that the question is about the functions of the VDC.

[0020] The prompt generation unit 14 may identify an in-vehicle device that has the same name as a named entity (Entity) in the speech text or an in-vehicle device that is related to the named entity, and identify vehicle information related to the identified in-vehicle device as vehicle information related to the speech. For example, in the speech text "What does the light that's on now mean?" shown in FIG. 2, "on" and "lamp" correspond to words in the speech text. The prompt generation unit 14 identifies an in-vehicle device that corresponds to a lamp (warning light) from the words in the speech text ("on" or "lamp") as an in-vehicle device related to the user's speech. Examples of such in-vehicle devices include a seat belt warning light, a VDC OFF indicator light, a headlamp upward indicator light, and a warning light for fuel or remaining battery capacity. Furthermore, the prompt generation unit 14 identifies information indicating the status of the in-vehicle device from the vehicle information input from the control signal acquisition unit 13. If the in-vehicle device is a warning light such as a seat belt warning light, the status of the in-vehicle device corresponds to whether the lamp is on or off.

[0021] The prompt generation unit 14 also extracts information about the identified in-vehicle device from information about the vehicle's instruction manual. Information about the instruction manual is stored in the database 15. The instruction manual contains information such as the specifications and operation methods of the in-vehicle device, a description of the in-vehicle device, and how to deal with problems when they occur. For example, the instruction manual for the seat belt warning light contains information such as the conditions under which the seat belt warning light turns on, an explanation of the state when the seat belt warning light is on, and how to turn off the seat belt warning light.

[0022] The prompt generation unit 14 refers to an answer table stored in the database 15 and selects an answer format of the language model 5 based on the intent. The answer table stores text indicating the answer format of the language information output from the language model 5 according to the intent. The answer format indicates the format or range of the answer sentence by the language model 5, and the information included in the answer format specifies the answer method and answer content for the language model 5. The answer format is composed of text including variables and character strings. The variables are given words and character strings such as the name of the in-vehicle device, the status of the in-vehicle device, a description of the in-vehicle device, the name of an object inside or outside the vehicle, and the status of the object. The answer format also indicates the sentence structure when the language model 5 answers. The prompt generation unit 14 may select an answer format from the answer table based on the genre analyzed by the language understanding unit 12. In the example of FIG. 2 , the word “following” in the sentence “Follow the following answer format for the question” indicating the answer format corresponds to the range of the answer sentence. As another example of the range of the answer sentence, the range of the answer sentence may be expressed by a specific numerical value such as the number of characters or sentences, such as an answer of 100 characters or less.

[0023] The prompt generation unit 14 identifies various intentions based on the intent, such as a user asking about the functions of an in-vehicle device, a user asking about the current driving status, a user asking about the navigation system (e.g., setting a destination or searching for POIs), a user asking about vehicle operation, etc. The answer table defines answer formats corresponding to various intentions. For example, if the user asks about automated driving based on the intent, a first sentence is defined as the answer format for answering whether the driver or the system has the authority to operate the vehicle. The first sentence is formed by a character string indicating the name and / or status of the in-vehicle device. The answer format is defined as text such as "Currently, the {driver or system} is driving." Here, {} is a variable, and "{driver or system} is driving" corresponds to a character string indicating the status of the in-vehicle device. The answer format also includes multiple sentences, such as a first sentence that directly responds to the user's voice, a second sentence that supplements the first sentence, and a third sentence that is different from the first and second sentences. The answer formats for constructing the second and third sentences are also defined in the table.

[0024] When generating a prompt, the prompt generation unit 14 includes an answer format in the prompt. That is, the prompt generation unit 14 expresses the answer format in text. Furthermore, the prompt generation unit 14 generates vehicle text that is useful in the processing of the language model 5 (that increases the accuracy of the linguistic information output from the language model 5) so that the language model 5 can create speech information in accordance with the answer defined in the answer format. As described above, the answer format is defined by text that includes variables, and the prompt generation unit 14 converts vehicle information that may be input to the variables into text and includes it in the prompt.

[0025] When the answer format is composed of a first sentence and a second sentence that supplements the first sentence, the prompt generation unit 14 generates vehicle text that is useful in the processing of the language model 5 so that the language model 5 can create the second sentence. When the prompt generation unit 14 has extracted information about the specified in-vehicle device from information about the vehicle's instruction manual, it is preferable that the prompt generation unit 14 further extracts information that may be input to the variables of the second sentence, converts the information into text, and includes it in the prompt.

[0026] An example of a prompt generated by the prompt generation unit 14 will be described below with reference to Fig. 2. The prompt generation unit 14 includes in the prompt the voice text "Tell me the meaning of the light that is currently on" converted by the voice recognition unit 11. In the example of Fig. 2, the prompt generation unit 14 determines that the user is asking a question about an in-vehicle device from the voice text or intent.

[0027] The prompt generation unit 14 refers to the answer table and selects an answer format from the language model 5 based on the intent (Intent) and includes it in the prompt. In the example of FIG. 2 , the sentence delimited by the item "Answer Format" corresponds to the answer format. In the example of FIG. 2 , the answer format is composed of a first sentence, "The {lamp name} of {color} is on." and a second sentence, "{Explanation of name}." The first sentence is an answer to the voice text and is formed from the name of the in-vehicle device {lamp name} and a string indicating the state of the in-vehicle device ({color} and "on"). The second sentence is a sentence that supplements the first sentence. Note that in the example of FIG. 2 , the answer format includes the text in the latter half of the second sentence, "Add at least one sentence describing the on-light lamp after the answer sentence." The language model 5 can output linguistic information that provides additional information to the answer sentence using the second sentence and the format sentence following the second sentence. Note that the prompt does not need to include a format sentence following the second sentence.

[0028] The prompt generation unit 14 may add text to supplement the explanation of the in-vehicle device. In the example of Fig. 2, the prompt generation unit 14 includes the text "{additional information about the lamp name}" and "when the lamp with the {name} is lit, perform the {measure} to deal with the problem" in the prompt.

[0029] The prompt generation unit 14 identifies in-vehicle devices associated with the named entity "lamp" in the speech text. In the example of FIG. 2 , the in-vehicle devices associated with "lamp" are the seat belt warning light, the VDC OFF indicator light, and the headlamp upward indicator light. The prompt generation unit 14 identifies the states of the seat belt warning light, the VDC OFF indicator light, and the headlamp upward indicator light from the vehicle information input from the control signal acquisition unit 13, i.e., the vehicle information included in the control signal. In the example of FIG. 2 , the seat belt warning light is on, and the VDC OFF indicator light and the headlamp upward indicator light are off. The prompt generation unit 14 generates vehicle text (the states of the seat belt warning light, the VDC OFF indicator, and the headlamp upward indicator light = "on," "off," and "off") as shown in FIG. 2 and includes it in the prompt. At this time, the prompt generation unit 14 may generate text to be assigned to variables ({on state}, {name}) included in the answer format as the prompt.

[0030] The prompt generation unit 14 extracts information about the identified in-vehicle devices from information about the vehicle's instruction manual and generates a prompt. In the example of FIG. 2 , the prompt generation unit 14 extracts information about each of the identified in-vehicle devices (seat belt warning light, VDC OFF indicator light, and headlamp upward indicator light). The prompt generation unit 14 may generate, as a prompt, text to be assigned to variables included in the answer format and variables included in vehicle text ("{supplementary information about the lamp name}" and "When the {name} lamp is lit, perform {measures}"). In the example of FIG. 2 , the prompt generation unit 14 generates text to be assigned to the variables ({supplementary information about the lamp name} and {measures}). As a result, the prompt generation unit 14 generates a prompt as shown in FIG. 2 .

[0031] 2, the prompt generation unit 14 generates text to be assigned to variables to supplement the description of the in-vehicle equipment, but it may also generate prompts other than text to be assigned to variables. For example, if the prompt generation unit 14 can identify a specific in-vehicle equipment from a named entity (Entity) in the speech text, it may extract information about the identified in-vehicle equipment from information about the vehicle's instruction manual and convert the extracted information into text to include in the prompt. The prompt generation unit 14 may include the extracted information directly in the prompt, or may convert the extracted information into text and include it in the prompt.

[0032] The prompt generation unit 14 outputs the generated prompt to the language model 5. The language model 5 outputs linguistic information in response to the input prompt in accordance with the response method specified by the response format. Furthermore, if the prompt specifies a sentence structure for the response, the language model 5 outputs linguistic information in accordance with the sentence structure.

[0033] The prompt generation unit 14 may identify at least one target control signal from the vehicle control signals input from the control signal acquisition unit 13 and generate a prompt including a vehicle text that documents the target control signal. For example, the control signal acquisition unit 13 transmits various control signals, including information indicating imaging data, sensor information, the status of in-vehicle devices, etc., to the prompt generation unit 14, and the prompt generation unit 14 identifies a control signal that is a candidate for prompt generation as a target control signal. The prompt generation unit 14 identifies the target control signal based on a user input. For example, the prompt generation unit 14 identifies a control signal associated with the voice text from the voice text converted by the voice recognition unit 11. For example, in the voice text example shown in FIG. 2 , “What does the light that’s on right now mean?”, the prompt generation unit 14 identifies, from the word “lamp,” a control signal of an in-vehicle signal associated with “lamp,” such as a brake light, a seat belt warning light, or a room light, as the target control signal. In another example, when a user operates a fog lamp, the prompt generation unit 14 identifies the fog lamp control signal as the target control signal from the fog lamp operation input input by the user. The prompt generator 14 may then generate a prompt including a vehicle text that is a written version of the identified target control signal.

[0034] FIG. 3 shows a specific example of a speech uttered by a user and a specific example of linguistic information output from the language model 5. "Question" indicates the user's speech, and "Answer" indicates the linguistic information output from the language model 5. The prompt input to the language model 5 includes an answer format. The language model 5 creates linguistic information according to the first sentence, "The {light name} of {color} is on." The language model 5 creates an answer sentence to the speech text, "The red seat belt warning light is on," by assigning appropriate text to the variables of the first sentence from the vehicle text other than the answer format.

[0035] Furthermore, the language model 5 generates linguistic information to supplement the answer sentence, "The seat belt warning light warns that the seat belt is not fastened," by providing appropriate text to the variable of the second sentence "{description of name}." Furthermore, the language model 5 generates linguistic information to supplement the answer sentence, "Please fasten your seat belt." The answer format includes "Add at least one sentence explaining the light that is on after the answer sentence," and two supplementary sentences are added in the example answer in Figure 3.

[0036] An example of a prompt generated by the prompt generation unit 14 will be described with reference to Fig. 4. The prompt generation unit 14 includes the voice text "Is it okay to overtake here?" converted by the voice recognition unit 11 in the prompt. In the example of Fig. 4, the prompt generation unit 14 determines from the voice text or the intent that the user is asking a question about a driving operation. Because possible driving operations differ depending on the environment surrounding the vehicle, the prompt generation unit 14 may determine that the question is about the state of the environment surrounding the vehicle.

[0037] The prompt generation unit 14 refers to the answer table and selects an answer format from the language model 5 based on the intent. The prompt generation unit 14 includes the selected answer format as text in the prompt. In the example of FIG. 4, the answer format is composed of a first sentence "1. Select one of the {answer forms} (1) to (3) listed below." and a second sentence "2. Add at least one sentence explaining the selected {driving operation} after the answer sentence." In the example of FIG. 4, the answer sentence to the speech text is in the form of a selection.

[0038] The prompt generator 14 may add text to supplement the explanation of the driving operation. In the example of Fig. 4, the prompt generator 14 includes the text "{supplementary information on driving operation}" and "{attention}" in the prompt.

[0039] The prompt generation unit 14 generates text to be assigned to the variable of the first sentence of the answer format. The variable of the first sentence is a driving maneuver and specifies vehicle information related to the driving maneuver. The database 15 stores information related to traffic laws. The information related to traffic laws is traffic information established in each country, such as road traffic laws, and includes, for example, information on road signs and information on vehicle driving regulations. The information related to traffic laws indicates various road conditions, such as road conditions in which overtaking is permitted, road conditions in which overtaking is prohibited, and road conditions in which parking and stopping are prohibited. For example, if the road sign is a "white dotted line," the permitted driving maneuver is "overtaking permitted." Note that the information related to traffic laws may be included in map information.

[0040] The prompt generation unit 14 generates vehicle text (use the following to describe the current state of the surrounding environment: {road sign} = {driving maneuver}) as shown in Fig. 4 based on information about traffic laws and regulations recorded in the database 15, and includes it in the prompt. The prompt generation unit 14 includes in the prompt, as text to be given to each variable of {road sign} = {driving maneuver}, such as "Dotted white line = 'Overtaking permitted'" as shown in Fig. 4.

[0041] The prompt generation unit 14 may also identify possible driving maneuvers based on vehicle information and / or environmental information. The vehicle information includes information such as the vehicle's current position and speed measured by GPS. The environmental information includes image data from the onboard camera 3, sensor detection results included in the CAN signal, and road information. The road information may be extracted from map information recorded in the database 15. For example, if the prompt generation unit 14 can detect a dotted white line from the image data from the onboard camera 3, it identifies "dotted white line" as a road sign and identifies "overtaking possible" as a possible driving maneuver. The prompt generation unit 14 then generates "overtaking possible" as text to be assigned to the variable ({driving maneuver}) included in the first sentence. In other words, the prompt generation unit 14 converts the information included in the image data into a sentence and includes the converted text in the prompt.

[0042] The prompt generation unit 14 also generates text to be assigned to the variables ({supplementary information about driving operations} and {attention}) based on information about traffic laws and regulations recorded in the database 15. In the example of FIG. 4 , the prompt generation unit 14 generates text such as "Overtaking is permitted in white dotted line sections" as a supplementary text for the driving operation "Overtaking is permitted" and "When overtaking, please check the safety of your surroundings before overtaking" as a text for "attention." The prompt generation unit 14 similarly generates text to supplement driving operations and text to alert drivers for "No parking" and "Speed ​​limit." That is, the prompt generation unit 14 converts character strings indicating vehicle driving restrictions based on traffic laws and regulations into sentences. Information about traffic laws and regulations may be stored in the database 15. In the example of FIG. 4 , the prompt generation unit 14 generates a prompt that includes character strings related to traffic laws and regulations.

[0043] FIG. 5 shows a specific example of a speech uttered by a user and a specific example of linguistic information output from the language model 5. When the user utters the speech "Is it okay to overtake here?" as shown in the "Question" section of FIG. 5, the prompt generation unit 14 generates a prompt as shown in FIG. 4 and outputs it to the language model 5. The language model 5 selects an answer form appropriate to the current vehicle state according to the first sentence of the answer format. The language model 5 generates an answer sentence for the speech text, "You can overtake at this location," by assigning appropriate text to the variables. The language model 5 generates linguistic information for explaining driving operations, "In the white dotted line section, you can overtake by crossing the white line," and "When overtaking, please check the safety of the surrounding area before overtaking." according to the second sentence.

[0044] As described above, in this embodiment, when answering a question from a user, the prompt generator 14 identifies vehicle information related to the user's voice based on a control signal (an image signal including imaging data or a CAN signal), converts the vehicle information into a vehicle text, and generates a prompt including the vehicle text, so that an answer corresponding to the vehicle state and / or the vehicle's surrounding environment can be obtained from the language model 5. This allows the user to obtain an answer corresponding to the vehicle state and / or the vehicle's surrounding environment.

[0045] Next, the flow of a data processing method by the data processing system will be described with reference to Figure 6. In step S1, a user's voice is input to the voice recognition unit 11 via the microphone 2. In step S2, the voice recognition unit 11 converts the voice signal into voice text. In step S3, the language understanding unit 12 analyzes the voice text to identify the intent and / or named entity of the user who uttered the voice. In step S4, the control signal acquisition unit 13 acquires a control signal from the in-vehicle camera 3 and / or the CAN communication network 4.

[0046] In step S5, the prompt generation unit 14 identifies vehicle information related to the voice based on the control signal. In step S6, the prompt generation unit 14 generates a prompt including vehicle text that is a written version of the vehicle information. In step S7, the prompt generation unit 14 inputs the prompt to the language model 5. In step S8, the language model 5 receives the prompt and outputs a response to the user's voice. Then, the data processing device 1 ends the control flow of the data processing method.

[0047] As described above, the data processing device or data processing method according to this embodiment acquires a vehicle control signal, generates a prompt including a vehicle text that is a written version of the control signal, and inputs the prompt into a language model, thereby making it possible to obtain a response corresponding to the vehicle control signal from the language model.

[0048] The prompt also includes a text answer format indicating the format or range of an answer sentence according to the language model. This allows the language model to answer according to the answer format. Furthermore, by having the language model answer according to the answer format, an answer that is in line with the current vehicle state can be obtained. If the language model 5 outputs linguistic information without an answer format, there is a risk that the language model 5 will output an incorrect answer. As in this embodiment, the prompt includes text indicating the format or range of an answer sentence according to the language model, thereby making it possible to prevent an incorrect answer from being output.

[0049] In this embodiment, the prompt generation unit 14 identifies at least one target control signal from among the control signals acquired by the control signal acquisition unit 13 based on a user input, and generates a prompt including text that documents the target control signal. This allows an answer corresponding to the vehicle control signal to be obtained from the language model.

[0050] In this embodiment, speech data is generated from the user's speech, a vehicle control signal is acquired, vehicle information related to the speech is identified based on the control signal, a prompt including a vehicle text that documents the vehicle information is generated, and the prompt is input to a language model, thereby enabling a response corresponding to the vehicle state and / or the state of the vehicle's surrounding environment to be obtained from the language model.

[0051] In this embodiment, the vehicle text includes a first sentence formed of a character string indicating the name of the in-vehicle device and / or the status of the in-vehicle device, so that information about the in-vehicle device can be used in the prompt.

[0052] In this embodiment, the vehicle text also includes a second sentence that supplements the first sentence, thereby enabling the content that supplements the first sentence to be added to the linguistic information output from the language model 5.

[0053] In this embodiment, the vehicle text also includes character strings related to traffic laws, which allows the content related to traffic laws to be added to the linguistic information output from the language model 5.

[0054] In this embodiment, the vehicle text includes text indicating information included in the imaging data, so that information identified from the imaging data can be added to the linguistic information output from the language model 5.

[0055] In this embodiment, the prompt generator 14 outputs a prompt including the voice text and the vehicle text to the language model 5. This allows additional information to be output from the language model 5 in addition to a response to the user's voice.

[0056] In this embodiment, the prompt generation unit 14 identifies at least one of vehicle information related to the user's intention, vehicle information including a named entity, and the vehicle information related to the named entity from the information included in the control signal. This allows the user's intention to be reflected in the linguistic information output from the language model 5. Alternatively, the user's intention can be identified from a named entity included in the speech, and the user's intention can be reflected in the linguistic information of the language model 5.

[0057] In this embodiment, the database 15 may store past vehicle text indicating past vehicle states, and the prompt generation unit 14 may generate a prompt based on the past vehicle text. For example, when a user asks a question about the past state of an in-vehicle device, such as "What was that warning light that was on two minutes ago?", the prompt generation unit 14 may generate a prompt using the past vehicle text. Specifically, the prompt generation unit 14 includes in the prompt a vehicle text that describes the state of the warning light two minutes ago in written form. This allows past information about the in-vehicle device to be added to the linguistic information output from the language model 5. Furthermore, storing the control signal as text reduces the memory capacity required for storage compared to storing the control signal directly in the database 15.

[0058] In this embodiment, the language model 5 may generate linguistic information not only by providing text included in the prompt but also by providing text included in data such as training data and variables included in the answer format. Furthermore, the language model 5 may generate linguistic information based on the prompt without providing text to variables included in the prompt. Furthermore, the language model 5 is not necessarily provided in the vehicle, and may be provided in a server or the like connected to the vehicle so as to be able to communicate with it.

[0059] In this embodiment, the language model 5 may be provided in a communication terminal that can communicate with the data processing device 1 .

[0060] REFERENCE SIGNS LIST 1 Data processing device 2 Microphone 3 In-vehicle camera 4 CAN communication network 5 Language model 6 IVI system 10 Voice data generation unit 11 Voice recognition unit 12 Language understanding unit 13 Control signal acquisition unit 14 Prompt generation unit 15 Database

Claims

1. A control signal acquisition unit that acquires control signals for the vehicle, A prompt generation unit that generates prompts for a language model that outputs language information, It includes an audio data generation unit that generates audio data including audio text, which is a written representation of the user's input voice. The aforementioned audio data generation unit, The user's voice is analyzed to identify named entities contained in the voice. The prompt generation unit, Based on the aforementioned named entity, the in-vehicle equipment is identified, From the information regarding the vehicle's owner's manual recorded in the database, additional information concerning the identified in-vehicle equipment is extracted. The control signal acquisition unit generates a vehicle text containing the control signal acquired by the control signal acquisition unit and the prompt including the additional information. The prompt is a data processing device that includes text indicating the format or range of the response sentence by the language model.

2. In the data processing apparatus according to claim 1, The prompt generation unit, Based on user input, at least one target control signal is identified from the control signals acquired by the control signal acquisition unit. A data processing device that generates the prompt, which includes vehicle text that is a written representation of the target control signal.

3. In the data processing apparatus according to claim 2, The vehicle text is a data processing device that includes a first sentence formed by a string of characters indicating the name and status of an in-vehicle device.

4. In the data processing apparatus according to claim 3, The vehicle text is a data processing device that includes a second sentence that supplements the first sentence.

5. In the data processing apparatus according to any one of claims 1 to 4, The aforementioned vehicle text is a data processing device that includes text related to traffic laws and regulations.

6. In the data processing apparatus according to claim 3, The name of the in-vehicle device is the name of the vehicle's warning light. The status of the in-vehicle device is a data processing device that indicates whether the warning light is on or off.

7. In the data processing device according to any one of claims 1 to 6, It includes a memory unit that stores past vehicle text, The prompt generation unit is a data processing device that generates the prompt based on the past vehicle text.

8. In the data processing device according to any one of claims 1 to 7, The aforementioned control signal includes image data from an in-vehicle camera. The vehicle text is a data processing device that includes text indicating information contained in the imaging data.

9. A data processing device according to any one of claims 1 to 8, The prompt generation unit identifies vehicle information related to the voice based on the control signal, The vehicle text is a data processing device that includes text that represents the vehicle information in written form.

10. In the data processing apparatus according to any one of claims 1 to 8, The prompt generation unit is a data processing device that outputs the prompt, which includes the voice text and the vehicle text, to the language model.

11. In the data processing apparatus according to any one of claims 1 to 8, The aforementioned audio data generation unit, Based on the aforementioned audio text, the intent of the user who uttered the voice, and / or the named entity contained in the audio text are identified. The prompt generation unit, A data processing device that identifies at least one of the following from the information included in the control signal: vehicle information related to the user's intent, vehicle information including the named entity, and vehicle information related to the named entity.

12. A data processing device according to any one of claims 1 to 10, A data processing system comprising a communication terminal capable of communicating with the aforementioned data processing device and having the aforementioned language model.

13. A data processing method performed by a processor, The aforementioned processor, Obtain the vehicle's control signals, It generates audio data that includes speech-to-text transcription of the user's input. The user's voice is analyzed to identify named entities contained in the voice. Based on the aforementioned named entity, the in-vehicle equipment is identified, From the information regarding the vehicle's owner's manual recorded in the database, additional information concerning the identified in-vehicle equipment is extracted. The system generates a vehicle text containing the control signals and a prompt including the additional information. The aforementioned prompt is input to the language model that outputs language information, The prompt is a data processing method that includes text indicating the format or range of the response sentence by the language model.