Signal generation system, arithmetic device, and signal generation method

Abstract

This disclosure provides a signal generation system, a computing device, and a signal generation method. The signal generation system includes: a first computing device configured to generate a route signal representing a predetermined driving route of the vehicle based on sensor signals output from sensors mounted on a vehicle; and a second computing device configured to input an input language signal, representing speech content delivered by a occupant of the vehicle in natural language form, to a learned language model, thereby generating an output language signal representing language information to be provided to the occupant. At least one of the first and second computing devices has a conversion unit, wherein the conversion unit converts at least one of the sensor signal and a signal generated based on the sensor signal into a route data signal represented in a form that can be input to the learned language model. The second computing device generates a route language signal related to the predetermined driving route as the output language signal based on the route data signal.

Description

Technical Field

[0001] This disclosure relates to a signal generation system, a computing device, and a signal generation method for generating output language signals, wherein the output language signals represent language information to be provided to the occupants of a vehicle. Background Technology

[0002] An automated vehicle is one in which at least some of its driving operations, such as acceleration, deceleration, and steering, are performed by a driving control unit. When there are notifications using natural language regarding the driving controls performed by the driving control unit, the occupants of the vehicle can feel reassured even when driving operations are performed that are different from the intentions of the vehicle's passengers (such as the driver).

[0003] Patent document 1 describes a control device that identifies objects outside a moving body based on images captured by an imaging device and notifies the user of risky objects that pose a risk of approaching the moving body using natural language.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: International Publication No. 2024 / 202905

[0007] The control device described in Patent Document 1 notifies the user independently of the driving control. Therefore, sometimes the content of the notification to the user does not correspond to the actual driving control, which may cause confusion for the user. Summary of the Invention

[0008] The purpose of this disclosure is to provide a signal generation system that appropriately generates output language signals representing language information related to the driving of a vehicle.

[0009] The main purpose of this disclosure is as follows.

[0010] (1) A signal generation system comprising: a first computing unit configured to generate a route signal representing a predetermined driving route of the vehicle based on sensor signals output from sensors mounted on a vehicle; and a second computing unit configured to input an input language signal representing speech content made by a passenger of the vehicle in natural language form to a learned language model, thereby generating an output language signal representing language information to be provided to the passenger, wherein at least one of the first computing unit and the second computing unit has a conversion unit, wherein the conversion unit converts at least one of the sensor signal and a signal generated based on the sensor signal into a route data signal represented in a form that can be input to the learned language model, and the second computing unit generates a route language signal related to the predetermined driving route as the output language signal based on the route data signal.

[0011] (2) The signal generation system according to (1) above, wherein the signal generated based on the sensor signal includes the route signal or a control signal for controlling the vehicle based on the route signal.

[0012] (3) According to the signal generation system described in (2) above, the form that can be input to the learned language model is natural language form.

[0013] (4) According to the signal generation system described in (3) above, when the route signal or the control signal contains a signal related to a specified action of the vehicle, the conversion unit adds a request signal to the route data signal, wherein the request signal is a signal that requests additional information corresponding to the specified action to be assigned to the route language signal.

[0014] (5) According to the signal generation system described in (1) or (2) above, the learned language model includes: an embedding unit that converts the input language signal into input vector data in vector form; and an output unit that outputs the output language signal based on the input vector data, wherein the form that can be input to the learned language model is in vector form.

[0015] (6) The signal generation system according to any one of (1) to (5) above, wherein at least one of the first computing device and the second computing device has an adjustment unit, wherein the adjustment unit adjusts the learned language model in a manner suitable for the route data signal.

[0016] (7) The signal generation system according to any one of (1) to (6) above, wherein the signal generated based on the sensor signal includes an intermediate signal generated during the process of generating the route signal.

[0017] (8) A computing device having a processor, wherein the processor is configured to: input an input language signal representing, in natural language form, a speech made by a passenger of a vehicle to a learned language model, thereby generating a first output language signal representing language information to be provided to the passenger; convert at least one of a sensor signal output from a sensor mounted on the vehicle and a signal generated based on the sensor signal into a route data signal in a form that can be input to the learned language model; and input the route data signal as the input language signal to the learned language model, thereby generating a route language signal related to a predetermined driving route of the vehicle as a second output language signal.

[0018] (9) A computing device having a processor, wherein the processor is configured to: generate a route signal representing a predetermined driving route of the vehicle based on sensor signals output from sensors mounted on the vehicle; and convert at least one of the sensor signals and a signal generated based on the sensor signals into a route data signal represented in natural language or vector form.

[0019] (10) A signal generation method, comprising: generating a route signal representing a predetermined driving route of the vehicle based on sensor signals output from sensors mounted on a vehicle; inputting an input language signal representing speech content made by a passenger of the vehicle in natural language form to a learned language model, thereby generating an output language signal representing language information to be provided to the passenger; converting at least one of the sensor signal and a signal generated based on the sensor signal into a route data signal in a form that can be input to the learned language model; and inputting the route data signal as the input language signal to the learned language model, thereby generating a route language signal related to the predetermined driving route as the output language signal.

[0020] According to the signal generation system disclosed herein, an output speech signal representing the speech information to be provided to the occupants of a vehicle can be generated without the need for a dedicated ECU (Electronic Control Unit). Attached Figure Description

[0021] Figure 1 This is a schematic diagram of a vehicle equipped with a signal generation system.

[0022] Figure 2 This is a hardware schematic diagram of the first computing device.

[0023] Figure 3This is a hardware schematic diagram of the second computing device.

[0024] Figure 4 This is a diagram that illustrates the general operation of the signal generation system.

[0025] Figure 5 It is a diagram illustrating the planned driving route.

[0026] Figure 6 This is a diagram illustrating an example of a display based on route-based language signals.

[0027] Explanation of reference numerals in the attached figures

[0028] 1: Vehicle; 6: First computing unit; 631: Object detection unit; 632: Route creation unit; 7: Second computing unit; 721: Language model after learning; 731: Language generation unit; 732: Conversion unit; 733: Adjustment unit. Detailed Implementation

[0029] Hereinafter, with reference to the accompanying drawings, a signal generation system for appropriately generating output language signals that represent language information related to the driving of a vehicle will be described in detail.

[0030] The signal generation system of this embodiment includes a first computing unit and a second computing unit. The first computing unit is configured to generate a route signal representing a predetermined driving route of the vehicle based on sensor signals output from sensors mounted on the vehicle. The second computing unit is configured to input an input language signal, representing the speech content of a passenger in the vehicle in natural language form, to a learned language model, thereby generating an output language signal representing language information to be provided to the passenger. The learned language model is configured to generate a series of words (sentences) based on the occurrence probabilities of words in sentences, and is pre-constructed using a large amount of language data. The second computing unit inputs a route data signal, represented in a form that can be input to the learned language model after conversion from at least one of the sensor signals and signals generated based on the sensor signals, as the input language signal to the learned language model. In this case, the second computing unit generates a route language signal related to the predetermined driving route as the output language signal.

[0031] Figure 1 This is a schematic diagram of a vehicle equipped with a signal generation system.

[0032] The vehicle 1 in this embodiment includes a peripheral camera 2, a microphone device 3, a display device 4, a speaker device 5, a first processing unit 6, and a second processing unit 7. The signal generation system 100 is configured to have the first processing unit 6 and the second processing unit 7.

[0033] The peripheral camera 2 and the first computing device 6 are communicatively connected via an in-vehicle network that follows a standard such as a controller area network. The microphone device 3, the display device 4, the speaker device 5, and the first computing device 6 and the second computing device 7 are communicatively connected via the in-vehicle network.

[0034] The peripheral camera 2 is an example of a sensor that generates peripheral images representing the surrounding conditions in front of the vehicle 1 in a time-series manner. The peripheral camera 2 has a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to visible light, such as a CCD (Charge-Coupled Device) or C-MOS (Complementary Metal-Oxide-Semiconductor), and an imaging optical system that images the area to be photographed onto this two-dimensional detector. The peripheral camera 2 is mounted, for example, in the upper front part of the vehicle compartment facing the direction of travel of the vehicle 1. The peripheral camera 2 photographs the surrounding conditions in front of the vehicle 1 through the windshield at each predetermined photographing cycle (e.g., 1 / 30 to 1 / 10 of a second), and outputs the peripheral images representing the surrounding conditions as data to the first processing unit 6. Alternatively, the peripheral images may be output in the form of moving images.

[0035] Alternatively, vehicle 1 may also be equipped with at least one of the following sensors: LiDAR (Light Detection and Ranging) sensor, RADAR (Radio Detection and Ranging) sensor, millimeter-wave sensor, ultrasonic sensor, GNSS (Global Navigation Satellite System) receiver, IMU (Inertial Measurement Unit), external microphone, and vehicle status sensor. The vehicle status sensor outputs data corresponding to the vehicle's state, such as vehicle speed, steering angle, brake pedal depressure, and accelerator pedal depressure.

[0036] The microphone device 3 is an example of an in-car sensor that generates sound data corresponding to the sounds inside the vehicle 1 in a time sequence. The sounds inside the vehicle can be, for example, the speech of the passengers in the vehicle 1. The microphone device 3 outputs the sound data corresponding to the speech made by the passengers in the vehicle to the second processing unit 7.

[0037] Display device 4 is an example of an output device, such as having a liquid crystal display. Display device 4 displays various visual information in a manner that is visually recognizable by the occupants of vehicle 1. The information displayed by display device 4 includes language information represented by output language signals received from the second computing unit 7 via the in-vehicle network. Display device 4 may, for example, be positioned in front of the driver's seat of vehicle 1 in a manner that is visually recognizable by the driver. Display device 4 may also be referred to as an instrument display.

[0038] The speaker device 5 is another example of an output device, such as having an amplifier and a speaker unit. The speaker device 5 outputs various sound information in a manner audible to the occupants of the vehicle 1. The information output by the speaker device 5 includes speech information represented by output speech signals received from the second processing unit 7 via the in-vehicle network. The speaker device 5 may, for example, be positioned in front of the driver's seat of the vehicle 1 in a manner audible to the driver.

[0039] The first computing unit 6 detects objects around the vehicle 1 based on surrounding images acquired from the surrounding camera 2. Based on the positions of the detected objects, it generates a route signal representing a predetermined driving route for the vehicle 1 and outputs this route signal to a driving control unit (not shown) via an in-vehicle network. The driving control unit generates a control signal to cause the vehicle 1 to travel along the predetermined driving route represented by the route signal acquired from the first computing unit 6 and outputs this control signal to the driving mechanism (not shown) of the vehicle 1. The driving mechanism includes, for example, an engine or motor that supplies power to the vehicle 1, a brake that reduces the speed of the vehicle 1, and a steering mechanism that steers the vehicle 1.

[0040] Figure 2 This is a hardware schematic diagram of the first arithmetic unit 6. The first arithmetic unit 6 includes a communication interface 61, a memory 62, and a processor 63. The first arithmetic unit 6 can be implemented as an ECU (Electronic Control Unit).

[0041] Communication interface 61 is an example of a communication unit, having a communication interface circuit for connecting the first computing device 6 to an in-vehicle network. Communication interface 61 supplies received data to processor 63. Furthermore, communication interface 61 outputs data supplied from processor 63 to an external source.

[0042] The memory 62 includes both volatile semiconductor memory and non-volatile semiconductor memory. The memory 62 stores various data for processing performed by the processor 63, such as parameters of a neural network that performs actions as a classifier for detecting objects based on surrounding images. Furthermore, the memory 62 stores various application programs executed by the processor 63, such as a computer program for route generation that performs route generation processing.

[0043] Processor 63 is an example of a control unit, having more than one processor and its peripheral circuitry. Alternatively, processor 63 may also have other arithmetic circuitry such as a logic unit, a numerical arithmetic unit, or a graphics processing unit.

[0044] The second processing unit 7 inputs the sound data acquired from the microphone device 3 into the learned language model, thereby generating an output language signal, and outputs the output language signal to at least one of the display device 4 and the speaker device 5 via the in-vehicle network. The display device 4 and the speaker device 5 can output visual information and sound information respectively based on the output language signal acquired from the second processing unit 7.

[0045] Figure 3 This is a hardware schematic diagram of the second arithmetic unit 7. The second arithmetic unit 7 includes a communication interface 71, a memory 72, and a processor 73. The second arithmetic unit 7 can be implemented as an ECU.

[0046] Communication interface 71 is an example of a communication unit, having a communication interface circuit for connecting the second computing device 7 to an in-vehicle network. Communication interface 71 supplies received data to processor 73. Furthermore, communication interface 71 outputs data supplied from processor 73 to an external source.

[0047] The memory 72 includes both volatile semiconductor memory and non-volatile semiconductor memory. The memory 72 stores various data for processing performed by the processor 73, such as parameters of a learned language model used to generate an output language signal based on the input language signal. Furthermore, the memory 72 stores various application programs executed by the processor 73, such as a language generation computer program that performs language generation processing.

[0048] Processor 73 is an example of a control unit, having one or more processors and their peripheral circuitry. Alternatively, processor 73 may also have other arithmetic circuitry such as logic operation units, numerical operation units, or graphics processing units.

[0049] Figure 4 This is a diagram illustrating the general operation of the signal generation system 100.

[0050] The processor 63 of the first arithmetic unit 6 in the signal generation system 100 has an object detection unit 631 and a route creation unit 632 as functional blocks. These units of the processor 63 are functional modules implemented by a program executed on the processor 63. Alternatively, the computer program implementing the functions of the units of the processor 63 may be provided in the form of a computer-readable portable recording medium such as a semiconductor memory, magnetic recording medium, or optical recording medium. Or, these units of the processor 63 may be installed in the first arithmetic unit 6 as separate integrated circuits, microprocessors, or firmware.

[0051] The object detection unit 631 inputs sensor signals (e.g., peripheral images output from the peripheral camera 2) from the sensors mounted on the vehicle 1 to a classifier that has been pre-learned to detect objects based on the sensor signals, thereby detecting objects based on the sensor signals.

[0052] The classifier can be, for example, a Single-Shot MultiBox Detector or a Convolutional Neural Network (CNN) with multiple convolutional layers connected in series from the input to the output, such as Faster R-CNN (Faster Region-based Convolutional Neural Network). Images containing the objects to be detected are used as training data, and the CNN is pre-learned using a prescribed learning method such as backpropagation. The CNN then functions as a classifier that detects objects based on sensor signals.

[0053] Furthermore, the object detection unit 631 uses the current position and posture of the vehicle 1, the direction of the detected object as seen from the vehicle 1, and the estimated distance from the vehicle 1 to surrounding objects to estimate the actual spatial position of the detected object.

[0054] The object detection unit 631 may, for example, use a positioning signal to determine the current position of the vehicle 1, wherein the positioning signal is generated based on GNSS signals received from GNSS satellites by a GNSS (Global Navigation Satellite System) receiver (not shown) mounted on the vehicle 1 at each predetermined period.

[0055] The object detection unit 631 can detect ground objects such as lane markings based on surrounding images, and match the detected ground objects with the corresponding ground objects shown in map data stored in a storage device (not shown), thereby determining the posture of the vehicle 1.

[0056] The object detection unit 631 can, for example, use the position of the detected object on the surrounding image, the focal length of the imaging optical system of the peripheral camera 2, and the slope of the optical axis of the imaging optical system of the peripheral camera 2 relative to the direction of travel of the vehicle 1 to determine the direction of the detected object as observed by the vehicle 1.

[0057] The object detection unit 631 can, for example, estimate the distance from the vehicle 1 to the object based on the reference size of the object in the actual space, the size of the object region shown in the surrounding image, and the internal parameters of the surrounding camera 2. The reference size of the object in the actual space can be pre-stored in the memory 62 according to the category of each object. The object detection unit 631 can retrieve the object category from the classifier output from the input surrounding image from the memory 62 to determine the size of the object in the actual space shown in the surrounding image. The internal parameters of the surrounding camera 2 include, for example, the focal length of the imaging optical system of the surrounding camera 2 and the pixel size of the surrounding image.

[0058] The route creation unit 632 creates a predetermined driving route in such a way that the distance to the detected object is greater than a predetermined interval, generates a route signal representing the predetermined driving route, and outputs the route signal to the driving control device (not shown) via the in-vehicle network.

[0059] The processor 73 of the second arithmetic unit 7 in the signal generation system 100 has a language generation unit 731, a conversion unit 732, and an adjustment unit 733 as functional blocks. These units of the processor 73 are functional modules implemented by a program executed on the processor 73. Alternatively, the computer program implementing the functions of the units of the processor 73 may be provided in the form of a computer-readable portable recording medium such as a semiconductor memory, magnetic recording medium, or optical recording medium. Alternatively, these units of the processor 73 may be installed in the second arithmetic unit 7 as independent integrated circuits, microprocessors, or firmware.

[0060] The language generation unit 731 inputs the input language signal (e.g., text data converted from sound data acquired from the microphone device 3) representing the speech content made by the occupant of the vehicle 1 in natural language form to the learned language model 721 defined by parameters stored in the memory 72, thereby generating an output language signal.

[0061] The input speech signal is coded data representing the textualized content of the passenger's speech. The speech generation unit 731, by inputting the sound data acquired from the microphone device 3 into the speech recognizer, can textualize the speech content contained in the sound data. The speech recognizer can be configured using a speech recognition model such as GMM-HMM (Gaussian Mixture Model-Hidden Markov Model) or DNN-HMM (Deep Neural Network-Hidden Markov Model). The parameters defining the speech recognition model can be stored in memory 72.

[0062] Once learned, the language model 721 is configured to generate a series of words (sentences) based on the input language signal. The learned language model 721 may, for example, include a deep learning architecture such as a Transformer that has an attention mechanism for determining the parts of the input data that should be of interest.

[0063] The language generation unit 731 outputs the output language signal obtained from the learned language model 721 to at least one of the display device 4 and the speaker device 5 via the in-vehicle network.

[0064] The conversion unit 732 converts at least one of the sensor signal output from the sensor mounted on the vehicle 1 and the signal generated based on the sensor signal into a route data signal that can be input to the learned language model 721.

[0065] Sensor signals can be, for example, peripheral images generated by the peripheral camera 2. Alternatively, sensor signals can be generated by sensors such as LiDAR sensors, RADAR sensors, millimeter-wave sensors, ultrasonic sensors, GNSS receivers, IMUs, external microphones, and vehicle status sensors.

[0066] The signal generated based on sensor signals can also be an intermediate signal generated during the processing of route signals by the first processing unit 6 based on sensor signals. The intermediate signal can represent information such as the position of objects detected around the vehicle 1 (e.g., structures, other vehicles, pedestrians), the position of lane markings, road conditions, and weather. The intermediate signal can also be described as identification data of the surrounding conditions of the vehicle 1 obtained by the first processing unit 6. By converting the signal containing the intermediate signal into a route data signal through the conversion unit 732, the signal generation system 100 can more reliably include information related to the reasons for generating the predetermined driving route in the route language signal generated by the learned language model 721.

[0067] The signal generated based on sensor signals can be a route signal generated by the first processing unit 6, or a control signal generated by the driving control unit based on the route signal. By converting the signal containing the route signal or control signal into a route data signal through the conversion unit 732, the signal generation system 100 can more reliably include information related to the predetermined driving route in the route language signal generated by the learned language model 721.

[0068] The conversion unit 732 can select the signal for conversion into a route data signal from the sensor signal and the signal generated based on the sensor signal. Furthermore, if the conversion unit 732 is configured to receive only the signal for conversion into a route data signal from the sensor signal and the signal generated based on the sensor signal, the conversion unit 732 may not need to perform such a selection.

[0069] The conversion unit 732 can convert at least one of the sensor signal and the signal generated based on the sensor signal into a route data signal represented in natural language form. Natural language form is an example of a form that can be input to the learned language model 721.

[0070] The conversion unit 732 converts the acquired data into natural language and generates instruction sentences in natural language form based on the words and numerical information (such as location information) contained in the converted natural language. When the acquired data is numerical, the conversion unit 732 can use a formula representing the correspondence between numerical values ​​and text to convert the acquired data into text. Alternatively, the conversion unit 732 can use a table representing such correspondences to convert the acquired data into text. The instruction sentence is a sentence that indicates the output of language signals based on word and numerical information; it can also be called a "prompt word".

[0071] The conversion unit 732 can, for example, set the part of the sentence used to describe the individual conditions in the sentence used to describe the driving conditions of the vehicle to specific words or numbers corresponding to the driving conditions of the vehicle 1, thereby creating an instruction sentence.

[0072] For example, the conversion unit 732 can change the statements within parentheses in an explanatory template such as "This vehicle is traveling at (vehicle speed) in (road category) at (position of driving lane). The road ahead is (straight road / left curve / right curve). (Several) (object categories) are identified at (direction) (distance) of this vehicle. (Object categories) are moving in (direction) at (speed). This vehicle is scheduled to travel as (content of the scheduled driving route)." to words or numbers obtained using at least one of sensor signals and signals generated based on the sensor signals, thereby creating an instruction sentence.

[0073] Furthermore, in order to generate route language signals more accurately, the conversion unit 732 can also append a predefined sentence to the instruction sentence. For example, the conversion unit 732 can also append a sentence such as "Please briefly explain the driving plan for a few seconds later" to the instruction sentence.

[0074] Furthermore, the conversion unit 732 can also append sentences corresponding to the acquired data to the instruction sentences to output more detailed route language signals from the learned language model 721. For example, if the acquired data indicates that the vehicle's future actions include braking, the conversion unit 732 can append sentences such as "Please explain in detail the driving plan in a few seconds" to the instruction sentences, or sentences such as "Please explain the reason for braking." Thus, the signal generation system 100 can easily enable the learned language model 721 to generate route language signals containing explanations of reasons for actions that the vehicle 1 will perform in the future, such as braking or sharp turns, which are easily recognizable by the occupants of the vehicle 1. Such signals, corresponding to sentences appended to the instruction sentences for requesting information (assigning additional information), can also be called request signals.

[0075] It should be noted that the conversion unit 732 can also create instruction sentences by generating route language signals containing the reasons for all actions of the vehicle 1. Furthermore, the conversion unit 732 can also create such instruction sentences only when specific control related to the prescribed actions of the vehicle 1 exists (where the acquired data includes signals related to the prescribed actions). By creating instruction sentences only when specific control exists, the signal generation system 100 can suppress redundancy in the route language signals.

[0076] The output language signal generated based on the speech made by the occupants of vehicle 1 can also be referred to as the first output language signal. The route language signal generated based on sensor signals and signals generated based on sensor signals can also be referred to as the second output language signal.

[0077] The adjustment unit 733 adjusts the learned language model 721 in a manner suitable for the route data signal. For example, the adjustment unit 733 configures a difference model—which outputs a language signal based on an input language signal corresponding to the speech of a passenger in vehicle 1—in parallel with the learned language model 721, and then outputs a language signal based on the route data signal generated by the conversion unit 732. In this case, the language generation unit 731 inputs the route data signal to both the learned language model 721 and the difference model, and sums their respective outputs to form the output language signal.

[0078] The parameters of the difference model can be stored in memory 72. The difference model can be constructed, for example, using LoRA (Low-Rank Adaptation). Alternatively, the difference model can also be constructed using other methods such as MoRA and DoRA (Weight-Decomposed Low-Rank Adaptation).

[0079] In this embodiment, the conversion unit 732 and the adjustment unit 733 are respectively configured as functional blocks of the processor 73 of the second arithmetic device 7, but at least one of them may also be configured as a functional block of the processor 63 of the first arithmetic device 6.

[0080] The language model 721, after being adjusted by the adjustment unit 733, can output a more appropriate output language signal based on the route data signal. If the language model 721 before adjustment can output a sufficiently appropriate output language signal based on the route data signal, the signal generation system 100 may not need to have the adjustment unit 733.

[0081] By implementing the signal generation method described above, the signal generation system 100 can appropriately generate output language signals that represent language information related to the driving of the vehicle.

[0082] Figure 5 It is a diagram illustrating the planned driving route. Figure 5 An overhead view shows vehicle 1 traveling on road RD.

[0083] The object detection unit 631 of the first computing device 6 detects an object OB on the road RD ahead of the vehicle 1 in the direction of travel based on the surrounding image output from the peripheral camera 2. The route creation unit 632 of the first computing device 6 creates a predetermined driving route TJ such that the distance to the detected object OB is greater than a predetermined interval.

[0084] The conversion unit 732 converts at least one of the surrounding image and the signal generated based on the surrounding image into a route data signal. The language generation unit 731 of the second processing device 7 generates a route language signal related to the predetermined driving route based on the route data signal.

[0085] Figure 6 This is a diagram illustrating an example of a display based on route-based language signals.

[0086] The display device 4 can visually display the language information represented by the route language signal received as an output language signal from the second processing unit 7. As an example, the language information is displayed at the lower center of the display of the display device 4, but the display position and display format are not limited to this.

[0087] According to such a signal generation system, output language signals that represent language information related to vehicle driving can be appropriately generated, and the information related to vehicle driving can be appropriately transmitted to the driver of the vehicle.

[0088] According to a variation, the learned language model 721 includes: an embedding section that converts the input language signal into input vector data in vector form; and an output section that outputs the output language signal based on the input vector data.

[0089] The embedding unit segments the input language signal into tokens, which are the smallest units of language information, and converts these tokens into numerical values, thereby converting them into input vector data. It should be noted that vector data can also be called tensor data.

[0090] The output unit can grasp the context of the input language signal based on the input vector data, generate output vector data in vector form, and convert the output vector data into text data, thereby outputting the output language signal.

[0091] The conversion unit 732 converts at least one of the sensor signal and the signal generated based on the sensor signal into a route data signal in vector form. The route data signal in vector form is represented in a form that can be input to the learned language model 721. The language generation unit 731 inputs the route data signal in vector form as input vector data to the output unit of the learned language model 721.

[0092] For example, by comparing multiple instruction sentences obtained by inputting them into the embedding unit after changing a portion of the explanatory template using different words or numbers, the part of the input vector data corresponding to the word or number can be determined. The conversion unit 732 can use a conversion pattern pre-created based on the correspondence between the obtained signal and the input vector data to convert at least one of the sensor signal and the signal generated based on the sensor signal into a route data signal in vector form.

[0093] In this way, the conversion unit 732 generates route data signals, thereby enabling the second processing unit to generate route language signals without increasing the processing load of the learned language model 721.

[0094] The learned language model 721 is configured to generate an output language signal based on an input language signal representing the speech content delivered by the occupants of vehicle 1 in natural language form, and can also generate a route language signal based on the route data signal converted by the conversion unit 732. According to the signal generation system of this disclosure, a dedicated learned language model is not required for generating route language signals.

[0095] It is understood that those skilled in the art can make various changes, substitutions and modifications to this disclosure without departing from the spirit and scope thereof.

Claims

1. A signal generation system, comprising: A first computing device is configured to generate a route signal representing a predetermined driving route of the vehicle based on sensor signals output from sensors mounted on the vehicle; and The second processing unit is configured to input an input language signal, representing the speech content of a passenger in the vehicle in natural language form, to a learned language model, thereby generating an output language signal representing the language information to be provided to the passenger. At least one of the first arithmetic device and the second arithmetic device has a conversion unit, wherein, The conversion unit converts at least one of the sensor signal and the signal generated based on the sensor signal into a route data signal that can be input to the learned language model. The second computing device generates a route language signal related to the predetermined driving route based on the route data signal, which is then used as the output language signal.

2. The signal generation system according to claim 1, wherein, The signals generated based on the sensor signals include the route signals or control signals for controlling the vehicle based on the route signals.

3. The signal generation system according to claim 2, wherein, The form that can be input into the learned language model is natural language.

4. The signal generation system according to claim 3, wherein, When the route signal or the control signal contains a signal related to a prescribed action of the vehicle, the conversion unit adds a request signal to the route data signal, wherein the request signal is a signal requesting additional information corresponding to the prescribed action to be assigned to the route language signal.

5. The signal generation system according to claim 1 or 2, wherein, The learned language model includes: an embedding unit that converts the input language signal into vector data; and an output unit that outputs the output language signal based on the input vector data. The input that can be given to the learned language model is in vector form.

6. The signal generation system according to any one of claims 1 to 5, wherein, At least one of the first computing device and the second computing device has an adjustment unit, wherein the adjustment unit adjusts the learned language model in a manner suitable for the route data signal.

7. The signal generation system according to any one of claims 1 to 6, wherein, The signal generated based on the sensor signal is an intermediate signal generated during the process of generating the route signal.

8. A computing device having a processor, wherein, The processor is configured to: The input language signal, which represents the speech of the vehicle occupant in natural language form, is input into the learned language model, thereby generating a first output language signal representing the language information to be provided to the occupant. At least one of the sensor signals output from the sensors mounted on the vehicle and the signals generated based on the sensor signals is converted into a route data signal that can be input to the learned language model; as well as The route data signal is input as the input language signal to the learned language model, thereby generating a route language signal related to the vehicle's predetermined driving route as the second output language signal.

9. A computing device having a processor, wherein, The processor is configured to: Based on sensor signals output from sensors mounted on the vehicle, a route signal representing the vehicle's predetermined driving route is generated; as well as At least one of the sensor signal and the signal generated based on the sensor signal is converted into a route data signal represented in natural language or vector form.

10. A signal generation method, comprising: Based on sensor signals output from sensors mounted on the vehicle, a route signal representing the vehicle's predetermined driving route is generated; The input language signal, which represents the speech of the vehicle occupant in natural language form, is input into the learned language model, thereby generating an output language signal representing the language information to be provided to the occupant. At least one of the sensor signal and the signal generated based on the sensor signal is converted into a route data signal in a form that can be input to the learned language model; as well as The route data signal is input as the input language signal to the learned language model, thereby generating a route language signal related to the predetermined driving route as the output language signal.

Images