Signal generation system, computing device, and signal generation method

Abstract

The system appropriately generates output language signals that represent linguistic information related to the vehicle's operation. [Solution] The signal generation system comprises a first arithmetic unit configured to generate a route signal representing the planned route of a vehicle based on sensor signals output from sensors mounted on the vehicle, and a second arithmetic unit configured to generate an output language signal representing language information to be provided to the occupants by inputting an input language signal representing the content of speech by the vehicle occupants in natural language form into a trained language model. At least one of the first and second arithmetic units has a conversion unit that converts at least one of the sensor signal and the signal generated based on the sensor signal into a route data signal expressed in a format that can be input into the trained language model. The second arithmetic unit generates a route language signal relating to the planned route as an output language signal based on the route data signal.

Description

Technical Field

[0001] The present disclosure relates to a signal generation system, a computing device, and a signal generation method for generating an output language signal representing language information to be provided to a vehicle occupant.

Background Art

[0002] There are autonomous vehicles in which a driving control device executes at least part of driving operations such as acceleration, deceleration, and steering of a vehicle. If there is a notification in natural language regarding the driving control by the driving control device, it is possible to give a sense of security to the vehicle occupant (for example, the driver) even when a driving operation different from the intention of the vehicle occupant is performed.

[0003] Patent Document 1 describes a control device that recognizes an object outside a moving body based on an image captured by an imaging device and notifies a user of a risk target having a risk of approaching the moving body in natural language.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The control device described in Patent Document 1 notifies the user independently of the driving control. Therefore, the content of the notification to the user may not correspond to the actual driving control and may rather cause confusion to the user.

[0006] An object of the present disclosure is to provide a signal generation system that appropriately generates an output language signal representing language information related to the running of a vehicle.

Means for Solving the Problems

[0007] The gist of the present disclosure is as follows.

[0008] (1) A first computing device configured to generate a route signal representing the planned route of the vehicle based on sensor signals output from sensors mounted on the vehicle, The vehicle comprises a second processing unit configured to generate an output language signal representing language information to be provided to the occupant by inputting an input language signal representing the content of speech uttered by the occupant of the vehicle in natural language form into a trained language model, At least one of the first and second computing units has a conversion unit that converts at least one of the sensor signal and the signal generated based on the sensor signal into a path data signal expressed in a format that can be input to the trained language model, The second computing unit generates a route language signal relating to the planned route as the output language signal based on the route data signal. Signal generation system.

[0009] (2) The signal generated based on the sensor signal includes the route signal or a control signal that controls the vehicle based on the route signal. The signal generation system described in (1) above.

[0010] (3) The signal generation system described in (2) above, wherein the format that can be input to the trained language model is natural language format.

[0011] (4) If the route signal or the control signal includes a signal relating to a predetermined operation of the vehicle, the conversion unit adds a request signal to the route data signal that requests the addition of additional information corresponding to the predetermined operation to the route language signal. The signal generation system described in (3) above.

[0012] (5) The trained language model includes an embedding unit that converts the input language signal into vector-formatted input vector data, and an output unit that outputs the output language signal based on the input vector data, The signal generation system described in (1) or (2) above, wherein the input format to the trained language model is a vector format.

[0013] (6) At least one of the first and second arithmetic units has an adjustment unit that adjusts the trained language model to fit the path data signal, A signal generation system as described in any one of the above (1)-(5).

[0014] (7) The signal generated based on the sensor signal includes an intermediate signal generated in the process of generating the path signal, A signal generation system as described in any one of the above (1)-(6).

[0015] (8) A first output language signal representing the linguistic information to be provided to the occupant is generated by inputting an input language signal representing the content of the occupant's speech in natural language form into a trained language model, At least one of the sensor signals output from the sensors mounted on the vehicle and the signals generated based on the sensor signals is converted into path data signals in a format that can be input to the trained language model. By inputting the aforementioned route data signal as the input language signal to the trained language model, a route language signal relating to the vehicle's planned route is generated as a second output language signal. A computing device having a processor configured in such a way.

[0016] (9) Based on the sensor signals output from the sensors mounted on the vehicle, a route signal representing the planned route of the vehicle is generated. Convert at least one of the sensor signal and the signal generated based on the sensor signal into a path data signal represented in natural language form or vector form. A computing device having a processor configured in such a way.

[0017] Based on a sensor signal output from a sensor mounted on a vehicle, generating a path signal representing a planned travel route of the vehicle; Inputting an input language signal representing the speech content of the vehicle occupants in a natural language format into a learned language model to generate an output language signal representing the language information to be provided to the vehicle occupants; Converting at least one of the sensor signal and a signal generated based on the sensor signal into a path data signal represented in a format that can be input into the learned language model; Inputting the path data signal as the input language signal into the learned language model to generate a path language signal related to the planned travel route as the output language signal; A signal generation method including the above.

[0018] According to the signal generation system according to the present disclosure, an output language signal representing the language information to be provided to the vehicle occupants can be generated without providing a dedicated ECU.

Brief Description of the Drawings

[0019] [Figure 1] It is a schematic configuration diagram of a vehicle in which a signal generation system is implemented. [Figure 2] It is a hardware schematic diagram of a first arithmetic unit. [Figure 3] It is a hardware schematic diagram of a second arithmetic unit. [Figure 4] It is a diagram for explaining an operation outline of a signal generation system. [Figure 5] It is a diagram for explaining a planned travel route. [Figure 6] It is a diagram showing an example of a display based on a path language signal.

Embodiments for Carrying Out the Invention

[0020] Hereinafter, referring to the drawings, a signal generation system that appropriately generates an output language signal representing language information related to the travel of a vehicle will be described in detail. <0,000110>

[0021] The signal generation system of this embodiment comprises a first arithmetic unit and a second arithmetic unit. The first arithmetic unit is configured to generate a route signal representing the planned route of a vehicle based on sensor signals output from sensors mounted on the vehicle. The second arithmetic unit is configured to generate an output language signal representing linguistic information to be provided to the occupants by inputting an input language signal, which represents the content of speech by the vehicle occupants in natural language form, into a trained language model. The trained language model is configured to generate a series of words (sentences) based on the probability of word occurrence in a sentence, and is pre-built using a large amount of language data. The second arithmetic unit inputs a route data signal, expressed in a format that can be input to the trained language model and converted from at least one of the sensor signals and signals generated based on the sensor signals, as an input language signal to the trained language model. In this case, the second arithmetic unit generates a route language signal relating to the planned route as an output language signal.

[0022] Figure 1 is a schematic diagram of a vehicle in which a signal generation system is implemented.

[0023] The vehicle 1 of this embodiment includes a peripheral camera 2, a microphone device 3, a display device 4, a speaker device 5, a first arithmetic unit 6, and a second arithmetic unit 7. The signal generation system 100 is configured to have the first arithmetic unit 6 and the second arithmetic unit 7.

[0024] The surrounding camera 2 and the first computing unit 6 are connected to each other via an in-vehicle network compliant with standards such as a controller area network. The microphone device 3, display device 4, speaker device 5, and the first computing unit 6 are connected to the second computing unit 7 via an in-vehicle network.

[0025] The peripheral camera 2 is an example of a sensor that generates peripheral images representing the surrounding conditions in front of the vehicle 1 in a time series. The peripheral camera 2 has a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to visible light, such as a CCD or C-MOS, and an imaging optical system that forms an image of the area to be photographed on the two-dimensional detector. The peripheral camera 2 is mounted, for example, in the upper front of the vehicle interior, facing the direction of travel of the vehicle 1. At predetermined shooting intervals (e.g., 1 / 30 second to 1 / 10 second), the peripheral camera 2 photographs the surrounding conditions in front of the vehicle 1 through the windshield and outputs the peripheral images representing the surrounding conditions as data to the first processing unit 6. The peripheral images may be output in video format.

[0026] Vehicle 1 may further include at least one of the following sensors: a LiDAR (Light Detection And Ranging) sensor, a RADAR (Radio Detection And Ranging) sensor, a millimeter-wave sensor, an ultrasonic sensor, a GNSS (Global Navigation Satellite System) receiver, an IMU (Inertial Measurement Unit), an external microphone, and a vehicle condition sensor. The vehicle condition sensor is a sensor that outputs data according to the state of the vehicle, such as vehicle speed, steering angle, brake pedal depression amount, and accelerator pedal depression amount.

[0027] The microphone device 3 is an example of an in-vehicle sensor that generates audio data in a time series corresponding to the sound inside the vehicle 1. The sound inside the vehicle may be, for example, the speech of the occupants of vehicle 1. The microphone device 3 outputs audio data corresponding to the content of the speech made by the occupants of the vehicle to the second processing unit 7.

[0028] Display device 4 is an example of an output device, and for example, has a liquid crystal display. Display device 4 displays various visual information in a way that is visible to the occupants of vehicle 1. The information displayed on display device 4 includes language information represented in the output language signal received from the second processing unit 7 via the in-vehicle network. Display device 4 may be positioned, for example, in front of the driver's seat where the driver of vehicle 1 is seated, so that it is visible to the driver. Display device 4 can also be called a meter display.

[0029] Speaker device 5 is another example of an output device, and for example, has an amplifier and a speaker unit. Speaker device 5 outputs various audio information so that it can be heard by the occupants of vehicle 1. The information output by speaker device 5 includes language information represented in the output language signal received from the second processing unit 7 via the in-vehicle network. Speaker device 5 may be positioned, for example, in front of the driver's seat where the driver of vehicle 1 is seated, so that it can be heard by the driver.

[0030] The first computing unit 6 detects objects around the vehicle 1 from surrounding images acquired from the surrounding camera 2, generates a route signal representing the planned route of the vehicle 1 based on the positions of the detected objects, and outputs it to the driving control device (not shown) via the in-vehicle network. The driving control device generates a control signal to drive the vehicle 1 along the planned route represented by the route signal acquired from the first computing unit 6, and outputs it to the vehicle 1's driving mechanism (not shown). The driving mechanism includes, for example, an engine or motor that supplies power to the vehicle 1, brakes that reduce the vehicle 1's speed, and a steering mechanism that steers the vehicle 1.

[0031] Figure 2 is a schematic hardware diagram of the first arithmetic unit 6. The first arithmetic unit 6 includes a communication interface 61, memory 62, and a processor 63. The first arithmetic unit 6 may be implemented as an ECU (Electronic Control Unit).

[0032] The communication interface 61 is an example of a communication unit and has a communication interface circuit for connecting the first arithmetic unit 6 to the in-vehicle network. The communication interface 61 supplies received data to the processor 63. The communication interface 61 also outputs data supplied from the processor 63 to the outside.

[0033] Memory 62 includes volatile semiconductor memory and non-volatile semiconductor memory. Memory 62 stores various data used in processing by processor 63, such as parameters of a neural network that operates as a classifier for detecting objects from surrounding images. Memory 62 also stores various application programs executed by processor 63, such as a path generation computer program that performs path generation processing.

[0034] The processor 63 is an example of a control unit and has one or more processors and their peripheral circuits. The processor 63 may further have other arithmetic circuits such as a logic unit, a numerical unit, or a graphics processing unit.

[0035] The second processing unit 7 generates an output language signal by inputting audio data acquired from the microphone device 3 into a trained language model, and outputs it to at least one of the display device 4 and the speaker device 5 via the in-vehicle network. The display device 4 and the speaker device 5 can output visual information and audio information, respectively, based on the output language signal acquired from the second processing unit 7.

[0036] Figure 3 is a schematic hardware diagram of the second arithmetic unit 7. The second arithmetic unit 7 includes a communication interface 71, memory 72, and a processor 73. The second arithmetic unit 7 may be implemented as an ECU.

[0037] The communication interface 71 is an example of a communication unit and has a communication interface circuit for connecting the second processing unit 7 to the in-vehicle network. The communication interface 71 supplies received data to the processor 73. The communication interface 71 also outputs data supplied from the processor 73 to the outside.

[0038] Memory 72 includes volatile semiconductor memory and non-volatile semiconductor memory. Memory 72 stores various data used in processing by processor 73, such as parameters of a trained language model for generating output language signals based on input language signals. Memory 72 also stores various application programs executed by processor 73, such as a route generation computer program that performs route generation processing.

[0039] The processor 73 is an example of a control unit and has one or more processors and their peripheral circuits. The processor 73 may further have other arithmetic circuits such as a logic unit, a numerical unit, or a graphics processing unit.

[0040] Figure 4 is a diagram illustrating the operation overview of the signal generation system 100.

[0041] The processor 63 of the first arithmetic unit 6 of the signal generation system 100 includes, as functional blocks, an object detection unit 631 and a path creation unit 632. Each of these parts of the processor 63 is a functional module implemented by a program executed on the processor 63. The computer program that realizes the functions of each part of the processor 63 may be provided in the form of a computer-readable portable recording medium, such as a semiconductor memory, a magnetic recording medium, or an optical recording medium. Alternatively, each of these parts of the processor 63 may be implemented in the first arithmetic unit 6 as an independent integrated circuit, a microprocessor, or firmware.

[0042] The object detection unit 631 detects objects from sensor signals by inputting sensor signals output from sensors mounted on the vehicle 1 (for example, surrounding images output from the surrounding camera 2) into a classifier that has been pre-trained to detect objects from sensor signals.

[0043] The classifier can be a convolutional neural network (CNN) having multiple convolutional layers connected in series from the input side to the output side, such as a Single Shot MultiBox Detector or Faster R-CNN. By pre-training the CNN using images containing the object to be detected as training data and following a predetermined learning method such as backpropagation, the CNN operates as a classifier that detects objects from sensor signals.

[0044] Furthermore, the object detection unit 631 estimates the real-space position of the detected object using the current position and orientation of the vehicle 1, the direction of the detected object as seen from the vehicle 1, and the estimated distance from the vehicle 1 to the surrounding object.

[0045] The object detection unit 631 can determine the current position of the vehicle 1, for example, by using positioning generated based on GNSS signals from GNSS satellites received at predetermined intervals by a GNSS (Global Navigation Satellite System) receiver (not shown) mounted on the vehicle 1.

[0046] The object detection unit 631 can, for example, detect features such as lane markings from the surrounding image, and determine the orientation of the vehicle 1 by matching the detected features with the corresponding features represented in map data stored in a storage device (not shown).

[0047] The object detection unit 631 can determine the direction of the detected object as seen from the vehicle 1 by using, for example, the position of the detected object on the surrounding image, the focal length of the imaging optical system of the peripheral camera 2, and the inclination of the optical axis of the imaging optical system of the peripheral camera 2 with respect to the direction of travel of the vehicle 1.

[0048] The object detection unit 631 can estimate the distance from the vehicle 1 to the object based, for example, on the reference size of the object in real space, the size of the object region represented in the surrounding image, and the internal parameters of the surrounding camera 2. The reference size of the object in real space may be pre-stored in memory 62 for each type of object. The object detection unit 631 can search memory 62 using the type of object output from the classifier that receives the surrounding image as input, and determine the size of the object in real space represented in the surrounding image. The internal parameters of the surrounding camera 2 include, for example, the focal length of the imaging optical system of the surrounding camera 2 and the pixel size of the surrounding image.

[0049] The route creation unit 632 creates a planned travel route such that the distance to the detected object is greater than a predetermined interval, generates a route signal representing the planned travel route, and outputs it to the travel control device (not shown) via the in-vehicle network.

[0050] The processor 73 of the second arithmetic unit 7 of the signal generation system 100 has, as functional blocks, a language generation unit 731, a conversion unit 732, and an adjustment unit 733. Each of these units of the processor 73 is a functional module implemented by a program executed on the processor 73. The computer program that realizes the functions of each unit of the processor 73 may be provided in the form of a computer-readable portable recording medium such as semiconductor memory, a magnetic recording medium, or an optical recording medium. Alternatively, each of these units of the processor 73 may be implemented in the second arithmetic unit 7 as an independent integrated circuit, a microprocessor, or firmware.

[0051] The language generation unit 731 generates an output language signal by inputting an input language signal (for example, text data converted from audio data acquired from the microphone device 3), which represents the content of speech uttered by the occupants of the vehicle 1 in natural language format, into a trained language model 721 defined by parameters stored in memory 72.

[0052] The input language signal is coded data in which the content of the crew's speech has been converted into text. The language generation unit 731 can convert the content of the speech data into text by inputting the speech data acquired from the microphone device 3 into a speech recognition unit. The speech recognition unit can be configured using a speech recognition model such as GMM-HMM (Gaussian Mixture Model - Hidden Markov Model) or DNN-HMM (Deep Neural Network - Hidden Markov Model). Parameters defining the speech recognition model may be stored in memory 72.

[0053] The trained language model 721 is configured to generate a series of words (sentences) in response to an input language signal. The trained language model 721 may include a deep learning architecture that has an attention mechanism for identifying noteworthy parts of the input data, such as a Transformer.

[0054] The language generation unit 731 outputs the output language signal obtained by the trained language model 721 to at least one of the display device 4 and the speaker device 5 via the in-vehicle network.

[0055] The conversion unit 732 converts at least one of the sensor signals output from the sensors mounted on the vehicle 1 and the signals generated based on the sensor signals into route data signals that can be input into the trained language model 721.

[0056] The sensor signal is, for example, a peripheral image generated by peripheral camera 2. Alternatively, the sensor signal may be a signal generated by sensors such as a LiDAR sensor, RADAR sensor, millimeter-wave sensor, ultrasonic sensor, GNSS receiver, IMU, external microphone, and vehicle status sensor.

[0057] The signal generated based on the sensor signal may be an intermediate signal generated during the process by which the first arithmetic unit 6 generates a route signal based on the sensor signal. The intermediate signal may represent information such as the position of objects detected around the vehicle 1 (e.g., structures, other vehicles, pedestrians), the position of lane markings, road surface conditions, and weather. The intermediate signal can also be said to be recognition data of the surrounding conditions of the vehicle 1 by the first arithmetic unit 6. By converting the signal including the intermediate signal into a route data signal, the signal generation system 100 can more reliably include information regarding the reason for generating the planned route in the route language signal generated by the trained language model 721.

[0058] The signal generated based on the sensor signal may be a route signal generated by the first arithmetic unit 6, or a control signal generated by the driving control device based on the route signal. By converting the signal containing the route signal or control signal into a route data signal, the signal generation system 100 can more reliably include information about the planned driving route in the route language signal generated by the trained language model 721.

[0059] The conversion unit 732 may select signals from the sensor signal and signals generated based on the sensor signal to be used for conversion to a route data signal. Alternatively, if the system is configured so that only the signals used for conversion to a route data signal are input to the conversion unit 732 from the sensor signal and signals generated based on the sensor signal, the conversion unit 732 does not need to make such a selection.

[0060] The conversion unit 732 may convert at least one of the sensor signal and the signal generated based on the sensor signal into a path data signal expressed in natural language. The natural language format is an example of a format that can be input to the trained language model 721.

[0061] The conversion unit 732 converts the acquired data into natural language and generates an instruction sentence in natural language format from the words and numerical information (e.g., location information) contained in the converted natural language. If the acquired data is numerical, the conversion unit 732 may convert the acquired data into text using an expression that represents the correspondence between the numerical value and the text in the data. Alternatively, the conversion unit 732 may convert the acquired data into text using a table that represents such a correspondence. The instruction sentence is a sentence that instructs the output language signal to be output based on the words and numerical information, and can also be called a "prompt".

[0062] The conversion unit 732 can create instructional text by, for example, replacing the parts of a sentence used to describe the driving conditions of a vehicle that describe individual situations with specific words or numbers that correspond to the driving conditions of vehicle 1.

[0063] For example, the conversion unit 732 may create an instruction sentence by changing the words in parentheses of an explanatory template such as, "The vehicle is traveling at (vehicle speed) in (location of the driving lane) on (road type). The road ahead is (straight / left curve / right curve). (Several) (object types) have been recognized at a position (direction) (distance) from the vehicle. (object types) are moving in (direction) at (speed). The vehicle is scheduled to travel as follows (content of the planned route)." into words or numbers obtained using at least one of the sensor signal and the signal generated based on the sensor signal.

[0064] Furthermore, in order to generate a more accurate route language signal, the conversion unit 732 may add predetermined sentences to the instruction sentence. For example, the conversion unit 732 may add a sentence such as "Briefly describe the driving plan for a few seconds from now." to the instruction sentence.

[0065] Furthermore, the conversion unit 732 may add sentences corresponding to the acquired data to the instruction sentence so that the trained language model 721 outputs a route language signal with more detailed content. For example, if the future actions of the vehicle represented by the acquired data include braking, the conversion unit 732 may add a sentence such as "Describe the driving plan in detail a few seconds from now." or a sentence such as "Explain the reason for braking." to the instruction sentence. This makes it easier for the signal generation system 100 to have the trained language model 721 generate a route language signal that includes an explanation of the reason for performing a predetermined action that the occupants of vehicle 1 will perform in the future, such as braking or sharp turns. A signal corresponding to a sentence added to an instruction sentence to request the addition of information to such a route language signal (addition of additional information) can also be called a request signal.

[0066] The conversion unit 732 may also create instruction sentences so that route language signals including the reasons for all operations of the vehicle 1 are generated. Alternatively, the conversion unit 732 may create such instruction sentences only when there is specific control related to a predetermined operation of the vehicle 1 (when the acquired data includes signals related to the predetermined operation). By creating instruction sentences only when there is specific control, the signal generation system 100 can suppress redundancy of route language signals.

[0067] The output language signal generated based on the speech content of the occupants of vehicle 1 may be referred to as the first output language signal. The path language signal generated based on the sensor signal and the signal generated based on the sensor signal may be referred to as the second output language signal.

[0068] The adjustment unit 733 adjusts the trained language model 721 to match the route data signal. The adjustment unit 733 places a difference model in parallel with the trained language model 721 that can output the difference between when the output language signal is output according to the input language signal corresponding to the content of speech by the occupants of vehicle 1, and when the output language signal is output according to the input language signal of the route data signal generated by the conversion unit 732. In this case, the language generation unit 731 inputs the input language signal to the trained language model 721 and the difference model, and the sum of their outputs becomes the output language signal.

[0069] The parameters defining the differential model may be stored in memory 72. The differential model can be constructed, for example, by LoRA (Low-Rank Adaptation). Alternatively, the differential model may be constructed by other methods such as MoRA or DoRA (Weight-Decomposed Low-Rank Adaptation).

[0070] In this embodiment, the conversion unit 732 and the adjustment unit 733 are configured as functional blocks of the processor 73 of the second arithmetic unit 7, but at least one of them may be configured as a functional block of the processor 63 of the first arithmetic unit 6.

[0071] The trained language model 721 adjusted by the adjustment unit 733 can output a more appropriate output language signal based on the path data signal. If the trained language model 721 before adjustment can output a sufficiently appropriate output language signal based on the path data signal, the signal generation system 100 does not need to have the adjustment unit 733.

[0072] The signal generation system 100 can appropriately generate output language signals representing language information related to vehicle operation by implementing the signal generation method described above.

[0073] Figure 5 is a diagram illustrating the planned route. Figure 5 shows a bird's-eye view of vehicle 1 traveling on road RD.

[0074] The object detection unit 631 of the first computing unit 6 detects an object OB on the road RD in front of the vehicle 1 in the direction of travel from the surrounding image output from the surrounding camera 2. The route creation unit 632 of the first computing unit 6 creates a planned travel route TJ such that the distance to the detected object OB is greater than a predetermined interval.

[0075] The conversion unit 732 converts at least one of the surrounding image and the signal generated based on the surrounding image into a route data signal. The language generation unit 731 of the second arithmetic unit 7 generates a route language signal relating to the planned route based on the route data signal.

[0076] Figure 6 shows an example of a display based on routing language signals.

[0077] The display device 4 visually displays the language information represented by the path language signal received as an output language signal from the second arithmetic unit 7. For example, the language information is displayed in the center of the lower edge of the display of the display device 4, but the display position and display format are not limited to this.

[0078] Such a signal generation system can appropriately generate output language signals that represent linguistic information related to the vehicle's operation, thereby appropriately communicating information about the vehicle's operation to the vehicle's driver.

[0079] According to the modified version, the trained language model 721 includes an embedding unit that converts an input language signal into vector-formatted input vector data, and an output unit that outputs an output language signal based on the input vector data.

[0080] The embedding unit can convert the input language signal into input vector data by dividing it into tokens, which are the smallest units of linguistic information, and then converting the tokens into numerical values. Note that vector data can also be called tensor data.

[0081] The output unit can output the language signal by understanding the context of the input language signal based on the input vector data, generating output vector data in vector format, and converting the output vector data into text data.

[0082] The conversion unit 732 converts at least one of the sensor signal and the signal generated based on the sensor signal into a vector-format path data signal. The vector-format path data signal is represented in a format that can be input to the trained language model 721. The language generation unit 731 inputs the vector-format path data signal as input vector data to the output unit of the trained language model 721.

[0083] For example, by inputting multiple instruction sentences, each in which part of the explanation template is modified with different words or numbers, into the embedding unit and comparing the resulting multiple input vector data, the portion of the input vector data corresponding to the word or number can be identified. Using a pre-created conversion pattern based on the correspondence between the signals obtained in this way and the input vector data, the conversion unit 732 can convert at least one of the sensor signal and the signal generated based on the sensor signal into a vector-format path data signal.

[0084] By having the conversion unit 732 generate path data signals in this way, the second arithmetic unit can generate path language signals without increasing the processing load on the trained language model 721.

[0085] The trained language model 721 is configured to generate an output language signal from an input language signal that represents the content of speech by the occupants of vehicle 1 in natural language form. It can also generate a route language signal based on the route data signal converted by the conversion unit 732. According to the signal generation system of this disclosure, it is not necessary to provide a dedicated trained language model to generate the route language signal.

[0086] Those skilled in the art should understand that various changes, substitutions, and modifications can be made to this disclosure without departing from its spirit and scope. [Explanation of Symbols]

[0087] 1 vehicle 6 1st arithmetic unit 631 Object detection unit 632 Route Creation Unit 7 Second arithmetic unit 721 Pre-trained language models 731 Language generation section 732 Conversion Unit 733 Adjustment section

Claims

1. A first arithmetic unit configured to generate a route signal representing the planned route of the vehicle based on sensor signals output from sensors mounted on the vehicle, The vehicle comprises a second computing device configured to generate an output language signal representing language information to be provided to the occupant by inputting an input language signal representing the content of speech uttered by the occupant of the vehicle in natural language form into a trained language model, At least one of the first and second arithmetic units has a conversion unit that converts at least one of the sensor signal and the signal generated based on the sensor signal into a path data signal expressed in a format that can be input to the trained language model, The second computing unit generates a route language signal relating to the planned travel route as the output language signal based on the route data signal. Signal generation system.

2. The signal generated based on the sensor signal includes the route signal or a control signal that controls the vehicle based on the route signal. The signal generation system according to claim 1.

3. The signal generation system according to claim 2, wherein the format that can be input to the trained language model is a natural language format.

4. The conversion unit adds a request signal to the route data signal that requests the addition of additional information corresponding to the predetermined operation to the route language signal if the route signal or the control signal includes a signal relating to a predetermined operation of the vehicle. The signal generation system according to claim 3.

5. The trained language model includes an embedding unit that converts the input language signal into vector-formatted input vector data, and an output unit that outputs the output language signal based on the input vector data. The signal generation system according to claim 2, wherein the format that can be input to the trained language model is in vector format.

6. At least one of the first and second arithmetic units has an adjustment unit that adjusts the trained language model to fit the path data signal. A signal generation system according to any one of claims 1 to 5.

7. The signal generated based on the sensor signal includes an intermediate signal generated during the process of generating the path signal. A signal generation system according to any one of claims 1 to 5.

8. By inputting an input language signal representing the content of a vehicle occupant's speech in natural language format into a trained language model, a first output language signal representing the language information to be provided to the occupant is generated. At least one of the sensor signals output from the sensors mounted on the vehicle and the signals generated based on the sensor signals is converted into path data signals in a format that can be input to the trained language model. By inputting the aforementioned route data signal as the input language signal to the trained language model, a route language signal relating to the vehicle's planned route is generated as a second output language signal. A computing device having a processor configured in such a way.

9. Based on sensor signals output from sensors mounted on the vehicle, a route signal representing the planned route of the vehicle is generated. Convert at least one of the sensor signal and the signal generated based on the sensor signal into a path data signal represented in natural language form or vector form. A computing device having a processor configured in such a way.

10. Based on sensor signals output from sensors mounted on the vehicle, a route signal representing the planned route of the vehicle is generated. The process involves inputting an input language signal, which represents the content of speech uttered by the occupants of the vehicle in natural language form, into a trained language model to generate an output language signal that represents the language information to be provided to the occupants. Converting at least one of the sensor signal and the signal generated based on the sensor signal into a path data signal expressed in a format that can be input to the trained language model, By inputting the aforementioned route data signal as the input language signal to the trained language model, a route language signal relating to the planned travel route is generated as the output language signal. A signal generation method including the following.

Images