System and System Operation Method

Abstract

To improve user convenience. [Solution] The system includes a first in-vehicle device mounted in a first vehicle, which has a proposed model generated by machine learning a combination of a first input from a first user of the first vehicle and a first operation of the first vehicle in response to the first input; and a second in-vehicle device mounted in a second vehicle of a different type than the first vehicle, which uses the proposed model acquired from the first in-vehicle device to determine a second operation of the second vehicle in response to a second input from a second user of the second vehicle.

Description

Technical Field

[0001] The present disclosure relates to a system and a method for operating the system.

Background Art

[0002] Techniques for controlling the operation of a vehicle according to the preferences of a user of the vehicle have been proposed. For example, Patent Document 1 discloses an example of a system that associates setting information of an in-vehicle environment with identification information of a user and performs setting of an in-vehicle environment preferred by the user.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When in-vehicle devices installed vary depending on the vehicle type, there are variations in the convenience enjoyed by the user from the operations of the vehicle controlled by the in-vehicle devices, so there is room to further improve and enhance the convenience for the user.

[0005] Hereinafter, a system and the like that enable improvement of user convenience will be disclosed.

Means for Solving the Problems

[0006] The system in the present disclosure includes a first in-vehicle device mounted on a first vehicle, having a proposed model generated by machine learning a combination of a first input by a first user of the first vehicle and a first operation of the first vehicle corresponding to the first input, and a second in-vehicle device mounted on a second vehicle of a vehicle type different from the first vehicle, determining a second operation of the second vehicle corresponding to a second input by a second user of the second vehicle using the proposed model acquired from the first in-vehicle device.

[0007] Another aspect of the present disclosure is a method for operating a system having first and second in-vehicle devices installed in first and second vehicles of different types, wherein the first in-vehicle device has a proposed model generated by machine learning a combination of a first input from a first user of the first vehicle and a first operation of the first vehicle in response to said first input, and the second in-vehicle device uses the proposed model acquired from the first in-vehicle device to determine a second operation of the second vehicle in response to a second input from a second user of the second vehicle. [Effects of the Invention]

[0008] The systems and other features described in this disclosure will enable improved user convenience. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram showing an example of the configuration of an information provision system. [Figure 2] This is a diagram showing an example of the configuration of an in-vehicle device. [Figure 3] This is a flowchart illustrating an example of the operation procedure for an in-vehicle device. [Figure 4] This is a flowchart illustrating an example of the operation procedure for an in-vehicle device. [Modes for carrying out the invention]

[0010] The embodiments will be described below with reference to the drawings.

[0011] Figure 1 shows an example of the configuration of a vehicle control system in one embodiment. The vehicle control system 1 has one or more server devices 10 connected to each other via a network 11 so as to be able to communicate information, and in-vehicle devices 12-1 and 12-2 mounted on vehicles 13-1 and 13-2, respectively. The server devices 10 are, for example, one or more server computers belonging to a cloud computing system or other computing system and functioning as servers that implement various functions. The vehicles 13-1 and 13-2 are passenger cars, commercial vehicles, etc. of different types, and include internal combustion engine vehicles, hybrid vehicles (HEV; Hybrid Electric Vehicle), plug-in hybrid vehicles (PHEV; Plug-in Hybrid Electric Vehicle), etc. The in-vehicle devices 12-1 and 12-2 are computers having communication functions and information processing functions, and control the operation of vehicles 13-1 and 13-2, respectively. Network 11 includes, for example, mobile communication networks, the Internet, ad hoc networks, LANs (Local Area Networks), MANs (Metropolitan Area Networks), or other networks, or any combination thereof.

[0012] In this embodiment, vehicle 13-1 is a vehicle of a higher class than vehicle 13-2. For example, vehicle 13-1 belongs to the category of so-called luxury cars, while vehicle 13-2 belongs to the category of so-called mass-market cars that are less expensive than luxury cars. Vehicle 13-1 is equipped with features that provide a more refined user experience than vehicle 13-2, and the in-vehicle device 12-1 is configured to perform controls that are more convenient for the user than the in-vehicle device 12-2. In this embodiment, the imbalance in user convenience between vehicles 13-1 and 13-2 is reduced, and the user convenience in vehicle 13-2 is raised, thereby improving the overall user convenience of vehicles 13-1 and 13-2.

[0013] In this embodiment, the in-vehicle device 12-1 has a proposed model that has been machine-learned by combining input from the user of vehicle 13-1 (one or more users including the driver and passengers, hereinafter referred to as the luxury car user) and the operation of vehicle 13-1 in response to that input (hereinafter referred to as the luxury car operation). The in-vehicle device 12-1 also updates the proposed model using combinations of luxury car operations and the luxury car user's response to them. Then, the in-vehicle device 12-2 uses the proposed model acquired from the in-vehicle device 12-1 to determine the operation of vehicle 13-2 in response to input from the user of vehicle 13-2 (one or more users including the driver and passengers, hereinafter referred to as the mass-market car user) (hereinafter referred to as the mass-market car operation). Here, the input from the luxury car user or the mass-market car user is input made intentionally by each user, and is the content of the utterance corresponding to each user's wishes and preferences. Such input will be referred to as intentional input below. Furthermore, the responses of luxury car users to luxury car operations include various actions taken by the luxury car user, such as physical conditions like body temperature or pulse, signs of sleepiness or discomfort in captured images, and operations on air conditioning or sound. According to the vehicle control system 1, by using the proposed model updated in the in-vehicle device 12-1 through reinforcement learning using the responses of luxury car users in the in-vehicle device 12-2, it is possible to realize more sophisticated mass-market car operations in response to input from mass-market car users. Therefore, overall, it is possible to improve user convenience.

[0014] Figure 2 shows an example configuration of in-vehicle devices 12-1 and 12-2. In-vehicle devices 12-1 and 12-2 are configured similarly in the following respects. Specifically, in-vehicle devices 12-1 and 12-2 have a communication unit 121, a storage unit 122, a control unit 123, a positioning unit 124, an input unit 125, an output unit 126, and a detection unit 127. These may be configured as a single control device, or as two or more control devices, or as a control device and other devices such as a communication device. The control device includes, for example, an ECU (Electronic Control Unit). The communication device includes, for example, a DCM (Data Communication Module). Each part is connected to each other or to the equipment of the vehicle 15 via an in-vehicle network compliant with standards such as CAN (Controller Area Network) to enable information communication. In addition, in-vehicle devices 12-1 and 12-2 may include an information processing device such as a smartphone or tablet terminal as part of their configuration.

[0015] The communication unit 121 includes modules that support mobile communication standards such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation), and modules that support in-vehicle LANs such as CAN. The in-vehicle device 12 is connected to the network 11 via the communication unit 121 through a nearby router device or a mobile communication base station, and communicates information with other devices via the network 11, or communicates information with various parts of the vehicle 13-1 and 13-2 via the in-vehicle LAN.

[0016] The storage unit 122 includes one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or a combination of at least two of these. Semiconductor memories are, for example, RAM (Random Access Memory) or ROM (Read Only Memory). RAM is, for example, SRAM (Static RAM) or DRAM (Dynamic RAM). ROM is, for example, EEPROM (Electrically Erasable Programmable ROM). The storage unit 122 functions, for example, as a main memory, auxiliary memory, or cache memory. The storage unit 122 stores information used in the operation of the control unit 123 and information obtained by the operation of the control unit 123. In this embodiment, the storage unit 122 stores the proposed model 21 and the language model 22.

[0017] The proposed model 21 is generated by machine learning using combinations of intentional inputs from luxury car users and luxury car operations as training data, and is stored in the in-vehicle device 12-1. In vehicle 13-1, the proposed model 21 proposes luxury car operations corresponding to the intentional inputs of the luxury car user. Here, the proposal includes instructions for vehicle 13-1 to execute the luxury car operations. The in-vehicle device 12-1 updates the proposed model 21 by reinforcement learning using the responses of the luxury car user according to the procedure described later. The updated proposed model 21 is sent from the in-vehicle device 12-1 to the in-vehicle device 12-2 via the server device 10 and stored in the in-vehicle device 12-2. In vehicle 13-2, the proposed model 21 proposes mass-market car operations corresponding to the intentional inputs of the mass-market car user.

[0018] Language model 22 is a model that performs natural language text generation, summarization, and analysis, generated by machine learning natural language patterns, structures, and meanings using a large amount of text data. For example, language model 22 is a relatively small language model based on a transformer architecture and has several million to several hundred million parameters. The form and size of language model 22 may differ in the in-vehicle devices 12-1 and 12-2.

[0019] The control unit 123 includes one or more processors, one or more dedicated circuits, or a combination thereof. The processor is a general-purpose processor such as a CPU (Central Processing Unit) or a dedicated processor such as a GPU (Graphics Processing Unit) specialized for specific processing. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or the like. The control unit 123 executes information processing related to the operation of the in-vehicle device 12 while controlling each part of the in-vehicle device 12.

[0020] The functions of the control unit 123 are realized by executing a control / processing program on the processor included in the control unit 123. The control / processing program is a program for causing a computer to execute the processing of the steps included in the operation of the control unit 123, so as to cause the computer to realize the functions corresponding to the processing of those steps. That is, the control / processing program is a program for causing a computer to function as the control unit 123. Also, some or all of the functions of the control unit 123 may be realized by the dedicated circuit included in the control unit 123.

[0021] The positioning unit 124 includes one or more GNSS (Global Navigation Satellite System) receivers. The GNSS includes, for example, at least any one of GPS (Global Positioning System), QZSS (Quasi-Zenith Satellite System), BeiDou, GLONASS (Global Navigation Satellite System), and Galileo. The positioning unit 124 sends the positioning result to the control unit 123, and the control unit 123 obtains the position information of the in-vehicle device 12.

[0022] The input unit 125 includes one or more input interfaces. The input interfaces are, for example, a microphone for receiving voice input, physical keys, capacitive keys, a pointing device, a touch screen provided integrally with a display, and the like. Further, the input interface includes a camera for imaging the interior of the vehicle. The input unit 125 receives operations for inputting various types of information including the user's spoken voice, and sends the input information to the control unit 123 or sends the captured image to the control unit 123.

[0023] The output unit 126 includes one or more output interfaces. The output interfaces are, for example, a speaker or a display. The display is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro - Luminescence) display. The output unit 126 outputs information obtained by the operation of the control unit 123.

[0024] The detection unit 127 is an interface with one or more sensors for detecting the states of each part of the vehicles 13 - 1 and 13 - 2, or has one or more sensors. The sensors include, for example, sensors such as vehicle speed and acceleration sensors, and sensors such as temperature and humidity sensors inside and outside the vehicle cabin. The sensors of the detection unit 127 in the in - vehicle device 12 - 1 further include sensors for measuring and detecting the physical states such as the body temperature and pulse of a luxury car user by infrared rays or the like, and sensors for detecting the amount of operation on in - vehicle equipment such as air conditioning and audio by the luxury car user.

[0025] FIG. 3 is a flowchart for explaining the operation procedure of the in - vehicle device 12 - 1 related to the reinforcement learning of the proposed model 21. Each step in FIG. 3 is a step of information processing executed by the control unit 123 of the in - vehicle device 13 - 1.

[0026] In S31, the control unit 123 acquires intentional input. The control unit 123 acquires the spoken voice of the luxury car user from the input unit 125, converts the spoken content into text using, for example, speech recognition processing, and inputs it into the language model 22, which then acquires intentional input corresponding to the spoken content. The intentional input corresponds to spoken content that suggests the luxury car user's state of mind, such as impatience, tension, drowsiness, boredom, concentration, or discomfort, such as "I want to go to ○○," "I want to listen to ○○'s music," "I want to concentrate on driving," or "I want a change of pace."

[0027] In S32, the control unit 123 acquires user attribute information. The control unit 123 acquires attribute information of luxury car users from, for example, the server device 10. The attribute information includes, for example, information such as the age, gender, place of residence, and occupation of the luxury car user. The attribute information is sent to and stored in the server device 10 in advance from any information processing device, for example, when purchasing a vehicle 13-1. The attribute information may be stored in the server device 10 in any format such as text. Alternatively, the attribute information may be acquired from the content of posts made by luxury car users on SNS (Social Network Service). The control unit 123 can collect various texts and posts from various servers and analyze them using the language model 22 or the large-scale language model of the server device 10 to acquire attribute information.

[0028] In S33, the control unit 123 determines the operation of the vehicle 13-1. The control unit 123 inputs the intentional input from the luxury car user to the proposed model 21 and uses the proposed model 21 to determine the corresponding operation of the vehicle 13-1, i.e., the luxury car operation. The determined luxury car operation includes, for example, adjusting the temperature and humidity of the air conditioning, adjusting the sound (music selection, volume adjustment), and recommending parking and stopping points (voice output) in response to the intentional input from the luxury car user. Furthermore, the control unit 123 may also take into account the attribute information of the luxury car user when determining the operation of the vehicle 13-1. For example, the control unit 123 sets the air conditioning and selects music according to the age and gender of the luxury car user. Information on appropriate air conditioning settings and music for each age group and gender is stored in the storage unit 122 in advance. The control unit 123 also selects parking and stopping points close to the luxury car user's place of residence or workplace based on map information. Map information is stored in the storage unit 122 in advance. Furthermore, for example, if the user of a luxury car is a male office worker in his 50s and it's the afternoon, he's likely to be wearing a suit, so it's possible to set the air conditioning to a lower temperature and a higher fan speed. Conversely, if it's a woman in her 60s and it's summer, she's likely to be wearing light clothing, so it's possible to set the air conditioning to a temperature that isn't too low and reduce the fan speed.

[0029] In S34, the control unit 123 instructs the operation of the vehicle 13-1. The control unit 123 sends instructions to the equipment and actuators of the vehicle 13-1 to perform luxury car operations. These instructions include, for example, instructions to adjust the temperature and humidity of the air conditioning system, instructions to select music and adjust the volume of the sound system, and instructions for the output unit 126 to output voice messages recommending parking or stopping.

[0030] In S35, the control unit 123 acquires and analyzes the user's response. The control unit 123 acquires the response of the luxury car user from the user's physical condition and operation of the vehicle 13-1 acquired by the detection unit 127, and from captured images acquired by the input unit 125. The control unit 123 also analyzes the luxury car user's response and determines whether it is a positive or negative response to the previous luxury car operation. For example, when the temperature and humidity are changed, if the appropriate body temperature and pulse rate corresponding to the changed temperature and humidity are obtained, it can be classified as a positive response; otherwise, it can be classified as a negative response. Information on the appropriate body temperature and pulse rate for each temperature and humidity is arbitrarily set in advance and stored in the memory unit 122. In addition, in the captured images, if the luxury car user who was showing signs of sleepiness or discomfort becomes awake or shows signs of comfort, it is determined as a positive response; otherwise, it is determined as a negative response. The control unit 123 derives patterns of the luxury car user's movements, facial expressions, and gestures through image processing of consecutive frames, and determines whether the gestures indicate alertness or comfort by matching the derived patterns with pre-set patterns. Alternatively, if the luxury car user is detected to cancel or reverse a change in temperature, humidity, or sound, it is determined to be a negative reaction.

[0031] In S36, the control unit 123 performs reinforcement learning of the proposed model 21. The control unit 123 executes steps S31 to S34 any number of times at any interval, for example, a few seconds, to acquire combinations of intentional inputs and corresponding luxury car actions, and combinations of luxury car actions and responses thereto. The control unit 123 uses positive responses as rewards and performs reinforcement learning of the proposed model 21 using any algorithm. Step S34 can be executed at any frequency. For example, step S34 may be executed after each run of the vehicle 13-1, or it may be executed at any interval, for example, every few days.

[0032] In S37, the control unit 123 updates the proposed model 21 with the reinforced learning version and stores it in the memory unit 122.

[0033] In S38, the control unit 123 duplicates the proposed model 21 and transmits the information of the duplicated proposed model 21 to the in-vehicle device 12-2. The in-vehicle device 12-2 receives the information of the duplicated proposed model 21 via the server device 10 and stores it in its own memory unit 122, thereby sharing the updated proposed model 21 between the in-vehicle devices 12-1 and 12-2. Step S37 may be executed each time the proposed model 21 is updated in step S36, or it may be executed when the proposed model 21 has been updated any number of times.

[0034] Figure 4 is a flowchart illustrating the operation procedure of the in-vehicle device 12-2 after obtaining the proposed model 21. Each step in Figure 4 is an information processing step performed by the control unit 123 of the in-vehicle device 13-2.

[0035] In S41, the control unit 123 acquires intentional input. The control unit 123 acquires the spoken voice of a mass-market car user from the input unit 125, converts the spoken content into text using, for example, speech recognition processing, and inputs it into the language model 22, which then acquires intentional input corresponding to the spoken content. The intentional input corresponds to spoken content that suggests states such as impatience, tension, drowsiness, boredom, concentration, or discomfort of a luxury car user, such as "I want to go to ○○", "I want to listen to ○○'s music", "I want to concentrate on driving", or "I want a change of pace".

[0036] In S42, the control unit 123 acquires user attribute information. The control unit 123 acquires attribute information of mass-market car users, for example, from the server device 10. The attribute information includes, for example, information such as the age, gender, place of residence, and occupation of the mass-market car user. The attribute information is sent to and stored in the server device 10 in advance from any information processing device, for example, when purchasing a vehicle 13-2. The attribute information may be stored in the server device 10 in any format such as text. In addition, the attribute information may be acquired from the content of posts made by mass-market car users on social media. The control unit 123 can collect various texts and posts from various servers and analyze them using the language model 22 or the large-scale language model of the server device 10 to acquire attribute information.

[0037] In S43, the control unit 123 determines the operation of vehicle 13-2. The control unit 123 inputs the intentional input from the mass-market vehicle user to the proposed model 21 and uses the proposed model 21 to determine the corresponding operation of vehicle 13-2, i.e., the mass-market vehicle operation. The determined mass-market vehicle operation includes, for example, adjusting the temperature and humidity of the air conditioning, adjusting the sound (music selection, volume adjustment), and recommending parking and stopping points (voice output) in response to the intentional input from the mass-market vehicle user. Furthermore, the control unit 123 may also determine the operation of vehicle 13-1 by taking into account the attribute information of the mass-market vehicle user. For example, the control unit 123 sets the air conditioning and selects music according to the age and gender of the mass-market vehicle user. Information on appropriate air conditioning settings and music for each age group and gender is stored in the storage unit 122 in advance. The control unit 123 also selects parking and stopping points close to the mass-market vehicle user's place of residence or workplace based on map information. Map information is stored in the storage unit 122 in advance.

[0038] In S44, the control unit 123 instructs the operation of the vehicle 13-2. The control unit 123 sends instructions to the equipment and actuators of the vehicle 13-2 to perform the mass-market vehicle operation. These instructions include, for example, instructions to adjust the temperature and humidity of the air conditioning system, instructions to select music and adjust the volume of the sound system, and instructions for the output unit 126 to output voice messages recommending parking or stopping.

[0039] In this way, in vehicle 13-2, mass-market vehicle operation based on the intentional input of a mass-market vehicle user is executed using the proposed model 21 acquired from vehicle 13-1. Therefore, according to this embodiment, it is possible to improve user convenience.

[0040] As described above, embodiments have been explained based on various drawings and examples, but it should be noted that those skilled in the art will find it easy to make various modifications and alterations based on this disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of this disclosure. For example, the functions, etc., included in each means, each step, etc., can be rearranged in a logically consistent manner, and multiple means, steps, etc., can be combined into one or divided. [Explanation of Symbols]

[0041] 1. Information Provision System 10 Server devices 11 Network 12-1, 12-2 On-vehicle equipment Vehicles 13-1 and 13-2 121 Communications Department 122 Storage section 123 Control Unit 124 Positioning Unit 125 Input section 126 Output section 127 Detection unit

Claims

1. A first in-vehicle device mounted on a first vehicle, which generates a proposed model by machine learning a combination of a first input from a first user of the first vehicle and a first operation of the first vehicle in response to the first input, and a combination of the first operation and the first user's response to the first operation, A second in-vehicle device, mounted in a second vehicle of a different type than the first vehicle, determines a second operation of the second vehicle corresponding to a second input from a second user of the second vehicle, using the proposed model acquired from the first in-vehicle device. A system that has

2. A first in-vehicle device mounted on a first vehicle, having a proposed model generated by machine learning a combination of a first input from a first user of the first vehicle and a first operation of the first vehicle in response to the first input, A second in-vehicle device, mounted in a second vehicle of a different type than the first vehicle, determines a second operation of the second vehicle corresponding to a second input from a second user of the second vehicle, using the proposed model acquired from the first in-vehicle device. A system that has

3. In claim 2, The first in-vehicle device updates the proposed model by further machine learning the combination of the first operation and the first user's response to the first operation. system.

4. In claim 2, The first in-vehicle device updates the proposed model using the attribute information of the first user. system.

5. In claim 4, The first in-vehicle device is a system that acquires attribute information of the first user using a language model.

6. In claim 2, The second in-vehicle device further determines the second operation using the attribute information of the second user. system.

7. In claim 6, The second in-vehicle device is a system that acquires attribute information of the second user using a language model.

8. In claim 2, the first input includes the content of a speech made by the first user. system.

9. In claim 2, the second input includes the content of the second user's speech. system.

10. A method for operating a system having first and second in-vehicle devices installed in first and second vehicles of different types, respectively. The first in-vehicle device has a proposed model generated by machine learning a combination of a first input from a first user of the first vehicle and a first operation of the first vehicle in response to said first input. The second in-vehicle device uses the proposed model obtained from the first in-vehicle device to determine a second operation of the second vehicle in response to a second input from a second user of the second vehicle. How the system works.

11. In claim 10, The first in-vehicle device further updates the proposed model by machine learning the combination of the first operation and the first user's response to the first operation. How the system works.

12. In claim 10, The first in-vehicle device updates the proposed model using the attribute information of the first user. How the system works.

13. In claim 12, A method of operating the system in which the first in-vehicle device acquires attribute information of the first user using a language model.

14. In claim 10, The second in-vehicle device further uses the attribute information of the second user to determine the second operation. How the system works.

15. In claim 14, A method of operating the system in which the second in-vehicle device acquires attribute information of the second user using a language model.

16. In claim 10, the first input includes the content of a speech made by the first user. How the system works.

17. In claim 10, the second input includes the content of the second user's speech. How the system works.

Images