Vehicle-mounted apparatus and control method thereof

Abstract

An in-vehicle device is installed in a vehicle in which a user is riding, the in-vehicle device performing the following operations: capturing an environmental image of the vehicle's surroundings; inputting the environmental image into a composite model to generate corresponding environmental text, wherein the environmental text describes the environmental image; inputting the environmental text into a language model to generate a response text corresponding to the vehicle; and performing an interactive operation corresponding to the user based on the response text.

Description

Technical Field

[0001] The present disclosure relates to an in-vehicle device and a control method thereof, and more particularly to an in-vehicle device for generating text and a control method thereof. Prior Art

[0002] According to research, conversations can help reduce driver fatigue and improve their concentration. However, in existing in-vehicle technologies, there is a lack of assisted driving technologies that can interact with the driver based on the vehicle's surrounding environment.

[0003] In view of this, how to provide in-vehicle technologies that interact based on the vehicle's surrounding environment and the driver is an urgent goal that the industry needs to strive for. Summary of the Invention

[0004] To solve the above problems, the present disclosure provides an in-vehicle device disposed in a vehicle in which a user is seated. The in-vehicle device includes a camera, an output interface, and a processor. The camera is used to capture an environmental image around the vehicle. The processor is communicatively connected to the camera and the output interface respectively. The processor inputs the environmental image into a composite model to generate a corresponding environmental text, where the environmental text is used to describe the environmental image; the processor inputs the environmental text into a first language model to generate a first response text corresponding to the vehicle; and the processor generates a control signal corresponding to the first response text to control the output interface to perform an interactive operation corresponding to the user.

[0005] The present disclosure also provides a control method applicable to an in-vehicle device disposed in a vehicle in which a user is seated. The steps include: capturing an environmental image around the vehicle; inputting the environmental image into a composite model to generate a corresponding environmental text, where the environmental text is used to describe the environmental image; inputting the environmental text into a first language model to generate a first response text corresponding to the vehicle; and performing an interactive operation corresponding to the user based on the first response text.

[0006] It should be understood that the foregoing general description and the following detailed description are merely exemplary and explanatory and are intended to provide further explanation of the claimed present disclosure. Brief Description of the Drawings

[0007] To make the above and other objects, features, advantages, and embodiments of the present disclosure more obvious and understandable, the description of the accompanying drawings is as follows: FIG. 1 is a schematic diagram of an in-vehicle device according to a first embodiment of the present disclosure; FIG. 2 is a flowchart of an in-vehicle device generating a response text based on an environmental image according to some embodiments of the present disclosure; FIG. 3 is a schematic diagram of a composite model according to some embodiments of the present disclosure; FIG. 4 is a flowchart of generating a response text using a language model according to some embodiments of the present disclosure; and FIG. 5 is a flowchart of a method for controlling an in-vehicle device according to a second embodiment of the present disclosure. Embodiment

[0008] To make the description of the present disclosure more detailed and complete, reference may be made to the accompanying drawings and the following various embodiments, in which the same numbers represent the same or similar elements.

[0009] Please refer to FIG. 1, which is a schematic diagram of an in-vehicle device 1 according to a first embodiment of the present disclosure. The in-vehicle device 1 is disposed in a vehicle in which a user (e.g., a driver, a passenger) is riding, and includes a processor 12, a camera 14, and an output interface 16, wherein the processor 12 is communicatively connected to the camera 14 and the output interface 16 respectively. The in-vehicle device 1 is configured to provide content for interacting with the user according to the surrounding environment of the vehicle.

[0010] In some embodiments, the processor 12 may include a central processing unit (CPU), a graphics processing unit (GPU), a multiprocessor, a distributed processing system, an application specific integrated circuit (ASIC), and / or a suitable computing unit.

[0011] The camera 14 is configured to capture an environmental image of the vehicle surroundings. In some embodiments, the camera 14 may include one or more image capture components installed on the vehicle for capturing images of the surrounding environment. In practical applications, the in-vehicle device 1 may select the number and location of the image capture components of the camera 14 installed on the vehicle according to requirements. The environmental image is used to record the environment around the vehicle, such as other vehicles, road markings, signals, obstacles, scenery, weather, and other objects or information around the vehicle, and various objects in the environment and the relative relationships such as the relative distance and position distribution between the vehicles can also be recognized according to the environmental image.

[0012] The output interface 16 is used to output the interactive content generated by the processor 12. In some embodiments, the output interface 16 may include a display screen, a speaker, a lamp, and / or other components installed in the vehicle.

[0013] In some examples, based on the positions where the multiple image capture components are installed, the camera 14 stitches together the multiple images captured by the multiple image capture components into an environmental image. In other examples, the camera 14 directly uses the multiple images captured by the multiple image capture components as the environmental image, and marks the images respectively based on the positions where the individual image capture components are installed to indicate the relative positions of the images corresponding to the vehicle, such as: front image, rear image, right-side image, left-side image, etc.

[0014] Regarding the operation of the in-vehicle device 1 to generate text for interacting with the user based on images, please refer to FIG. 2, which is a flowchart of the in-vehicle device 1 generating a response text RT based on the environmental image EI in some embodiments of the present disclosure.

[0015] As shown in FIG. 2, first, the processor 12 inputs the environmental image EI into the composite model CM to generate a corresponding environmental text ET, where the environmental text ET is used to describe the environmental image EI.

[0016] Next, the processor 12 inputs the environmental text ET into the language model LM1 to generate a response text RT corresponding to the vehicle.

[0017] After generating the response text RT, the processor 12 generates a control signal corresponding to the response text RT to control the output interface 16 to perform an interactive operation corresponding to the user.

[0018] Specifically, in order to be able to input it into the language model LM1 for processing, the processor 12 first uses the composite model CM to convert the environmental image EI captured by the camera 14 into an environmental text ET in text form.

[0019] Thereafter, the processor 12 performs calculations using the language model LM1 to generate a response text RT corresponding to the environmental text ET, where the response text RT is a text generated based on the environmental image EI.

[0020] Finally, the processor 12 controls the output interface 16 to play, display, or output the response text RT in other ways to interact with the user.

[0021] For details about the composite model CM, please refer to FIG. 3, which is a schematic diagram of the composite model CM in some embodiments of the present disclosure. The composite model CM includes an image encoder EC, a transformer model TF, and a language model LM2.

[0022] As shown in FIG. 3, in some embodiments, the composite model CM first inputs the environmental image EI into the image encoder EC to generate a plurality of image features IF.

[0023] Specifically, the image encoder EC is an encoder of the transformer model TF, which is used to extract features in the environmental image EI (i.e., image features IF).

[0024] Next, the composite model CM inputs the image features IF and a query element into the transformer model TF to generate a plurality of extracted features EF corresponding to the feature vector FV, where the query element contains the feature vector FV.

[0025] Specifically, the transformer model TF is a model with multiple layers of transformers. Each layer of the transformer may include a self-attention operation for the image features IF and a cross-attention operation based on the image features IF and the query element. After the operation is completed, the operation result is fed forward to the next layer.

[0026] After multiple layers of transformer operations, the transformer model TF can generate the extracted features EF. After referring to the parameter of the feature vector FV in the image features IF, the features of interest to the composite model CM are further extracted (i.e., the extracted features EF). The parameter of the feature vector FV can adjust the weights for different types of features in the extracted features EF, and after multiple operations, the parameter will also be adjusted accordingly.

[0027] Finally, the composite model CM inputs the extracted features EF into the language model LM2 to generate the environmental text ET.

[0028] In some embodiments, the language model LM2 is a decoder corresponding to the image encoder EC and the transformer model TF. In other words, the language model LM2 is used to convert the extraction feature EF extracted by the transformer model TF into the environmental text ET in natural language form. In some examples, the language model LM2 is a trained large language model (LLM).

[0029] The environmental text ET generated by the language model LM2 is a text describing the environmental image EI. For example, it may include the states of other objects around the vehicle: "There is a truck parked by the roadside, and there is a white van and a black vehicle driving on the road."

[0030] In some embodiments, the environmental text ET includes one of an object marker, a direction marker, or a combination thereof. For example, since the environmental image EI has a marker corresponding to the relative position of the vehicle, the corresponding generated environmental text ET includes the attributes of the surrounding objects and their relative positions with respect to the vehicle. For example: "In front of the vehicle, there is a truck parked by the roadside; in the front left of the vehicle, a row of green grass and a concrete curb in front of a building can be seen; in the rear right of the vehicle, two people can be seen walking."

[0031] It should be noted that the in-vehicle device 1 can set the language of the environmental text ET output by the language model LM2 according to requirements, and the present disclosure is not limited thereto.

[0032] In some embodiments, the in-vehicle device 1 also inputs the position coordinates when the camera 14 captures the environmental image EI as a parameter into the composite model CM as a basis for extracting features. Please refer to FIG. 3 again. In response to receiving the vehicle coordinates VC corresponding to the vehicle for capturing the environmental image EI, the composite model CM inputs the vehicle coordinates VC, the image feature IF, and the query element into the transformer model TF to generate the extraction feature EF. Accordingly, the composite model CM can refer to the position of the vehicle to generate the corresponding environmental text ET.

[0033] In some embodiments, the in-vehicle device 1 further includes an input interface (not shown in the figure), such as a microphone, a touch screen, a keyboard, a touchpad, etc., and the in-vehicle device 1 can receive input information from the user accordingly, such as voice input and typing input.

[0034] When the in-vehicle device 1 receives an input from the user through the input interface, for example, when the user asks the in-vehicle device 1 a question about the vehicle in motion, the in-vehicle device 1 inputs the user's input information into the composite model CM to extract corresponding features and generate corresponding environmental text ET. For example, if the user asks a question about surrounding vehicles, the composite model CM can extract more features about surrounding vehicles from the environmental image EI.

[0035] Specifically, as shown in Figure 3, in response to receiving the input information IN, the composite model CM generates the query element based on the input information IN and the feature vector FV; and the composite model CM inputs the image feature IF and the query element into the transformer model TF to generate the extraction feature EF.

[0036] For example, after the in-vehicle device 1 receives the input information IN, the processor 12 converts the input information IN into the form of a query element, and concatenates (appends) the input information IN and the feature vector FV and inputs them into the transformer model TF for operation as the query element. In this way, the composite model CM can generate the relevant environmental text ET based on the user's input information IN to provide information for subsequent operations of the language model LM1.

[0037] For details about the language model LM1, please refer to Figure 4, which is a flowchart for generating the response text RT using the language model LM1 in some embodiments of the present disclosure. In some embodiments, the language model LM1 is a large language model trained to provide instructions.

[0038] In some embodiments, the in-vehicle device 1 also inputs the current position coordinates of the vehicle as a parameter into the language model LM1 as a basis for generating the response text RT. Since the language model LM1 has the characteristic of interpreting text, the processor 12 first converts the vehicle coordinates VC (for example, GPS positioning coordinates) into text representing the vehicle position (for example, address, road name), and then inputs the text into the language model LM1 as an operation parameter.

[0039] Specifically, in response to receiving the vehicle coordinates VC, the processor 12 converts the vehicle coordinates VC into position text PT; and the processor 12 inputs the position text PT and the environmental text ET into the language model LM1 to generate the response text RT.

[0040] As shown in Figure 4, in some embodiments, when the in-vehicle device 1 receives input information from a user (for example, received through the aforementioned input interface), the input information is also input into the language model LM1 in text form (i.e., input text IT) to generate a corresponding response text RT.

[0041] Specifically, in response to receiving the input text IT, the language model LM1 generates the response text RT based on the input text IT, where the response text RT is used to reply to the input text IT.

[0042] For example, when the user asks the in-vehicle device 1 questions such as "How can I reach the destination faster?", the in-vehicle device 1 inputs the user's input in text form into the language model LM1, and the language model LM1 can interpret the text and reply to the question raised by the user accordingly, such as a reply like "Continue to accelerate, but pay attention to maintaining an appropriate distance from the blue car on the right."

[0043] Regarding the response text RT generated by the in-vehicle device 1, its content can be adjusted according to actual needs. In addition to replying to the questions raised by the user as described above, the in-vehicle device 1 can also proactively provide driving suggestions based on the environment around the vehicle or initiate a topic.

[0044] In some embodiments, the language model LM1 provides driving suggestions according to the environment around the vehicle. Specifically, the language model LM1 generates a driving suggestion for operating the vehicle based on the environment text ET; and the language model LM1 uses the driving suggestion as the response text RT.

[0045] For example, when an obstacle approaches, the response text RT generated by the language model LM1 includes instructions to remind the user to decelerate or avoid.

[0046] In some embodiments, the language model LM1 can initiate a conversation with the user to assist the user in concentrating. Specifically, the language model LM1 generates an interaction message corresponding to the vehicle based on the environment text ET; and the language model LM1 uses the interaction message as the response text RT.

[0047] Furthermore, in some embodiments, when the in-vehicle device 1 receives a reply from the user to the interaction message (for example, received through the aforementioned input interface), the language model LM1 generates a response text RT corresponding to the reply to continue the conversation.

[0048] Specifically, after generating the interaction information, in response to receiving the input text IT, the language model LM1 generates a response text RT based on the input text IT and the interaction information, where the response text RT is used to reply to the input text IT.

[0049] For example, the language model LM1 starts a conversation with an interesting question: "Do you know what the slowest car in the world is?" Simulating such a scenario where the in-vehicle device 1 chats with the user (e.g., the driver) helps reduce fatigue.

[0050] Further, when the user replies: "Tell me what the slowest car in the world is", the language model LM1 simultaneously refers to the question it proposed in the previous conversation (i.e., the previously generated response text RT) and the user's reply (i.e., the currently received input text IT), and generates a reply such as "It's the blue car on your right! It is recommended that you accelerate to overtake it."

[0051] In summary, the in-vehicle device 1 proposed in this disclosure can not only provide driving suggestions based on the surrounding environment of the vehicle, but also actively start a conversation with the user for interaction, and the conversation content can be combined with the current driving scenario, which helps reduce the driver's fatigue and concentration, and increases the safety of vehicle driving.

[0052] Please refer to FIG. 5, which is a flowchart of the in-vehicle device control method 200 in the second embodiment of this disclosure. The in-vehicle device control method 200 includes steps S201 to S204. The in-vehicle device control method 200 is used to provide content for interacting with the user according to the surrounding environment of the vehicle. The in-vehicle device control method 200 can be executed by an in-vehicle device (e.g., the in-vehicle device 1 in the first embodiment), where the in-vehicle device is disposed in a vehicle in which a user is riding.

[0053] First, in step S201, the in-vehicle device captures an environmental image around the vehicle.

[0054] Next, in step S202, the in-vehicle device inputs the environmental image into a composite model to generate a corresponding environmental text, where the environmental text is used to describe the environmental image.

[0055] Next, in step S203, the in-vehicle device inputs the environmental text into a first language model to generate a first response text corresponding to the vehicle.

[0056] Finally, in step S204, the in-vehicle device performs an interaction operation corresponding to the user based on the first response text.

[0057] In some embodiments, the composite model is used to input the environmental image into an image encoder to generate a plurality of image features; input the image features and a query element into a transformer model to generate a plurality of extracted features corresponding to a feature vector, where the query element includes the feature vector; and input the extracted features into a second language model to generate the environmental text.

[0058] In some embodiments, the composite model is further used to, in response to receiving an instant coordinate for capturing the environmental image of the vehicle, input the instant coordinate, the image features, and the query element into the transformer model to generate the extracted features.

[0059] In some embodiments, the composite model is further used to, in response to receiving input information of the user, generate the query element based on the input information and the feature vector; and input the image features and the query element into the transformer model to generate the extracted features.

[0060] In some embodiments, the second language model is a decoder corresponding to the image encoder and the transformer model.

[0061] In some embodiments, the in-vehicle device control method 200 further includes the in-vehicle device converting an instant coordinate for capturing the environmental image of the vehicle into a position text; and the in-vehicle device inputting the position text and the environmental text into the first language model to generate the first response text.

[0062] In some embodiments, the first language model is further used to, in response to receiving input text of the user, generate the first response text based on the input text, where the first response text is used to reply to the input text.

[0063] In some embodiments, the first language model is further used to generate a driving recommendation for operating the vehicle based on the environmental text; and use the driving recommendation as the first response text.

[0064] In some embodiments, the first language model is further used to generate interaction information corresponding to the vehicle based on the environmental text; and use the interaction information as the first response text.

[0065] In some embodiments, the first language model is further configured to, after generating the interaction information, in response to receiving an input text corresponding to the user, generate a second response text based on the input text and the interaction information, wherein the second response text is used to reply to the input text.

[0066] In summary, the in-vehicle device control method 200 proposed by the present disclosure can not only provide driving suggestions based on the surrounding environment of the vehicle, but also actively initiate a conversation with the user for interaction, and the conversation content can be combined with the current driving situation, which helps to reduce the driver's fatigue and concentration, and increase the safety of vehicle driving.

[0067] Although several embodiments are described in detail above as examples, the in-vehicle device and method proposed by the present disclosure can also be implemented by other systems, hardware, software, storage media, or combinations thereof. Therefore, the protection scope of the present disclosure should not be limited to the specific implementation manners described in the embodiments of the present disclosure, and should be determined by the scope defined in the appended patent application.

[0068] It is obvious to those of ordinary skill in the art to which the present disclosure pertains that various modifications and changes can be made to the structure of the present disclosure without departing from the scope or spirit of the present disclosure. In view of the foregoing, the protection scope of the present disclosure also covers the modifications and changes made within the scope of the appended patent application.

[0069] 1: In-vehicle device 12: Processor 14: Camera 16: Output interface EI: Environmental image CM: Composite model ET: Environmental text LM1, LM2: Language model RT: Response text EC: Image encoder IF: Image feature FV: Feature vector IN: Input information VC: Vehicle coordinates TF: Transformer model EF: Extracted feature PT: Location text IT: Input text 200: In-vehicle device control method S201~S204: Steps

[0070] Domestic deposit information (Please note in the order of deposit institution, date, and number) None Foreign deposit information (Please note in the order of deposit country, institution, date, and number) None

Claims

1. An in-vehicle device, installed in a vehicle in which a user is seated, comprising: a camera for capturing an environmental image of the area surrounding the vehicle; an output interface; and a processor communicatively connected to the camera and the output interface, for performing the following operations: inputting the environmental image into a composite model to generate corresponding environmental text, wherein the environmental text describes the environmental image; inputting the environmental text into a first language model to generate a first response text corresponding to the vehicle; and generating a control signal corresponding to the first response text to control the output interface to perform an interactive operation corresponding to the user.

2. The vehicle-mounted device as claimed in claim 1, wherein the composite model is used to: input the environmental image into an image encoder to generate a plurality of image features; input the image features and a query element into a transformer model to generate a plurality of extracted features corresponding to a feature vector, wherein the query element contains the feature vector; and input the extracted features into a second language model to generate the environmental text.

3. The vehicle-mounted device as claimed in claim 2, wherein the composite model is further configured to: in response to receiving a real-time coordinate corresponding to the environmental image captured by the vehicle, input the real-time coordinate, the image features, and the query element into the transformer model to generate the extracted features.

4. The vehicle-mounted device as claimed in claim 2, wherein the vehicle-mounted device further includes an input interface for generating input information corresponding to one of the users, and the composite model is further configured to: generate the query element based on the input information and the feature vector in response to receiving the input information; and input the image features and the query element into the transformer model to generate the extracted features.

5. The vehicle-mounted device as claimed in claim 1, wherein the processor is further configured to perform the following operations: converting one of the real-time coordinates of the environmental image captured by the vehicle into a location text; and inputting the location text and the environmental text into the first language model to generate the first response text.

6. The vehicle-mounted device as claimed in claim 1, wherein the vehicle-mounted device further includes an input interface for generating input text corresponding to one of the users, and the first language model is further configured to: generate a first response text based on the input text in response to receiving the input text, wherein the first response text is used to reply to the input text.

7. The in-vehicle device as claimed in claim 1, wherein the first language model is further configured to: generate a driving suggestion for operating the vehicle based on the environmental text; and use the driving suggestion as the first response text.

8. The in-vehicle device as claimed in claim 1, wherein the first language model is further configured to: generate interactive information corresponding to the vehicle based on the environmental text; and use the interactive information as the first response text.

9. The vehicle-mounted device as claimed in claim 8, wherein the vehicle-mounted device further includes an input interface for generating input text corresponding to a user, and the first language model is further configured to: upon generating the interaction information, in response to receiving the input text, generate a second response text based on the input text and the interaction information, wherein the second response text is used to reply to the input text.

10. A control method applicable to an in-vehicle device installed in a vehicle in which a user is riding, the steps comprising: capturing an environmental image of the vehicle's surroundings; inputting the environmental image into a composite model to generate corresponding environmental text, wherein the environmental text describes the environmental image; inputting the environmental text into a first language model to generate a first response text corresponding to the vehicle; and performing an interactive operation corresponding to the user based on the first response text.

Images