Vehicle-mounted device and control method thereof
By integrating the camera, output interface and processor in the vehicle-mounted device, and using composite models and language models to convert the environmental images around the vehicle into interactive text, the problem of assisted driving lack of interaction with the driver in the prior art is solved, and the effect of reducing driving fatigue and improving driving safety is achieved.
Patent Information
- Application Number
- CN202411379456.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-10-04
- Filing Date
- 2024-09-30
- Publication Date
- 2025-04-08
AI Technical Summary
The existing on-board technology lacks assisted driving technology that can interact with the driver based on the vehicle's surrounding environment, making it difficult to effectively reduce the driver's fatigue and improve concentration.
Design an on-board device that includes a camera, an output interface and a processor. It captures the environmental images around the vehicle through the camera, and uses a composite model and language model to convert the images into environmental text and response text to achieve interaction with users.
By interacting with the driver, the on-board device can provide driving advice and chat content, helping to reduce the driver's fatigue, improve concentration, and increase the safety of the vehicle's driving.
Smart Images
Figure CN119782453A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to a vehicle-mounted device and a control method thereof, and more particularly to a vehicle-mounted device for generating text and a control method thereof. Background Art
[0002] According to research, conversations can help reduce driver fatigue and improve driver concentration, but existing in-vehicle technologies lack assisted driving technology that can interact with the driver based on the vehicle's surroundings.
[0003] In view of this, how to provide in-vehicle technology based on the environment around the vehicle and the interaction with the driver is a goal that the industry urgently needs to work hard on. Summary of the invention
[0004] In order to solve the above problems, the present disclosure proposes a vehicle-mounted device, which is arranged in a vehicle in which a user rides, and the vehicle-mounted device includes a camera, an output interface and a processor. The camera is used to capture the environmental image around the vehicle. The processor is respectively connected to the camera and the output interface for communication. The processor inputs the environmental image into a composite model to generate a corresponding environmental text, wherein the environmental text is used to describe the environmental image; the processor inputs the environmental text into a first language model to generate a first response text corresponding to the vehicle; and the processor generates a control signal corresponding to the first response text to control the output interface to perform an interactive operation corresponding to the user.
[0005] In one embodiment of the present invention, the composite model is used to: input the environment image into an image encoder to generate a plurality of image features; input the plurality of image features and a query element into a transformer model to generate a plurality of extracted features corresponding to the feature vector, wherein the query element includes a feature vector; and input the plurality of extracted features into a second language model to generate the environment text.
[0006] In one embodiment of the present invention, the composite model is further configured to: in response to receiving a real-time coordinate corresponding to the environment image captured by the vehicle, input the real-time coordinate, the plurality of image features and the query element into the transformer model to generate the plurality of extracted features.
[0007] In one embodiment of the present invention, the vehicle-mounted device also includes an input interface, which is used to generate input information corresponding to the user, and the composite model is further used to: generate the query element based on the input information and the feature vector in response to receiving the input information; and input the multiple image features and the query element into the transformer model to generate the multiple extracted features.
[0008] In one embodiment of the present invention, the processor is further used to perform the following operations: converting the real-time coordinates corresponding to the environmental image captured by the vehicle into position text; and inputting the position text and the environmental text into the first language model to generate the first response text.
[0009] In one embodiment of the present invention, the vehicle-mounted device further includes an input interface, which is used to generate input text corresponding to the user, and the first language model is further used to: in response to receiving the input text, generate the first response text based on the input text, wherein the first response text is used to reply to the input text.
[0010] In one embodiment of the present invention, the first language model is further used to: generate driving advice for operating the vehicle based on the environment text; and use the driving advice as the first response text.
[0011] In one embodiment of the present invention, the first language model is further used to: generate interactive information corresponding to the vehicle based on the environmental text; and use the interactive information as the first response text.
[0012] In one embodiment of the present invention, the vehicle-mounted device also includes an input interface, which is used to generate input text corresponding to the user, and the first language model is further used to: after generating the interactive information, in response to receiving the input text, generate a second response text based on the input text and the interactive information, wherein the second response text is used to reply to the input text.
[0013] The present disclosure also provides a control method applicable to a vehicle-mounted device, wherein the vehicle-mounted device is arranged in a vehicle in which a user is riding, and the steps include: capturing an environmental image around the vehicle; inputting the environmental image into a composite model to generate a corresponding environmental text, wherein the environmental text is used to describe the environmental image; inputting the environmental text into a first language model to generate a first response text corresponding to the vehicle; and executing an interactive operation corresponding to the user based on the first response text.
[0014] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are intended to provide further explanation of the disclosure as claimed. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to make the above and other objects, features, advantages and embodiments of the present disclosure more clearly understood, the accompanying drawings are described as follows:
[0016] Figure 1This is a schematic diagram of a vehicle-mounted device in a first embodiment of the present disclosure;
[0017] Figure 2 A flow chart of a vehicle-mounted device generating a response text based on an environmental image in some embodiments of the present disclosure;
[0018] Figure 3 A schematic diagram of a composite model in some embodiments of the present disclosure;
[0019] Figure 4 A flowchart of using a language model to generate a response text in some embodiments of the present disclosure; and
[0020] Figure 5 Flow chart of the vehicle-mounted device control method in the second embodiment of the present disclosure. DETAILED DESCRIPTION
[0021] In order to make the description of the present disclosure more detailed and complete, reference may be made to the attached drawings and various embodiments described below, in which the same numbers in the drawings represent the same or similar elements.
[0022] Please refer to Figure 1 , which is a schematic diagram of the vehicle-mounted device 1 in the first embodiment of the present disclosure. The vehicle-mounted device 1 is disposed in a vehicle in which a user (e.g., a driver, a passenger) rides, and includes a processor 12, a camera 14, and an output interface 16, wherein the processor 12 is communicatively connected to the camera 14 and the output interface 16 respectively. The vehicle-mounted device 1 is used to provide content for interaction with the user according to the surrounding environment of the vehicle.
[0023] In some embodiments, the processor 12 may include a central processing unit (CPU), a graphics processing unit (GPU), a multi-processor, a distributed processing system, an application specific integrated circuit (ASIC) and / or a suitable computing unit.
[0024] The camera 14 is used to capture the environment image around the vehicle. In some embodiments, the camera 14 may include one or more image capture elements installed on the vehicle to capture the environment image. In practical applications, the vehicle-mounted device 1 can select the number and position of the image capture elements of the camera 14 installed on the vehicle according to the needs. The environment image is used to record the environment around the vehicle, such as other vehicles around the vehicle, road markings, signs, obstacles, scenes, weather and other objects or information, and the relative distance, position distribution and other relative relationships between various objects in the environment and the vehicle can also be identified based on the environment image.
[0025] The output interface 16 is used to output the interactive content generated by the processor 12. In some embodiments, the output interface 16 may include a display screen, a speaker, a lamp, and / or other components installed in the vehicle.
[0026] In some examples, the camera 14 stitches the multiple images captured by the multiple image capture elements into one environment image based on the positions where the multiple image capture elements are installed. In other examples, the camera 14 directly uses the multiple images captured by the multiple image capture elements as the environment image, and marks the images based on the positions where the image capture elements are installed to indicate the relative position of the image mapping vehicle, such as: front image, rear image, right image, left image, etc.
[0027] For the operation of the vehicle-mounted device 1 to generate text based on images and interact with the user, please refer to Figure 2 , which is a flow chart of the vehicle-mounted device 1 generating a response text RT based on the environment image EI in some embodiments of the present disclosure.
[0028] like Figure 2 As shown, first, the processor 12 inputs the environment image EI into the composite model CM to generate a corresponding environment text ET, wherein the environment text ET is used to describe the environment image EI.
[0029] Next, the processor 12 inputs the environment text ET into the language model LM1 to generate a response text RT corresponding to the vehicle.
[0030] After generating the response text RT, the processor 12 generates a control signal corresponding to the response text RT to control the output interface 16 to execute an interactive operation corresponding to the user.
[0031] Specifically, in order to input the language model LM1 for processing, the processor 12 first converts the environment image EI captured by the camera 14 into the environment text ET in text form using the composite model CM.
[0032] Thereafter, the processor 12 performs calculations using the language model LM1 to generate a response text RT corresponding to the environment text ET, wherein the response text RT is a text generated based on the environment image EI.
[0033] Finally, the processor 12 controls the output interface 16 to play, display or otherwise output the response text RT to interact with the user.
[0034] For details on the composite model CM, please refer to Figure 3 , which is a schematic diagram of a composite model CM in some embodiments of the present disclosure, wherein the composite model CM includes an image encoder EC, a transformer model TF and a language model LM2.
[0035] like Figure 3 As shown, in some embodiments, the composite model CM first inputs the environment image EI into the image encoder EC to generate a plurality of image features IF.
[0036] Specifically, the image encoder EC is an encoder of the transformer model TF, and is used to extract features (ie, image features IF) from the environment image EI.
[0037] Next, the composite model CM inputs the image features IF and a query element (query) into the transformer model TF to generate a plurality of extracted features EF corresponding to the feature vector FV, wherein the query element includes the feature vector FV.
[0038] Specifically, the transformer model TF is a model with multiple layers of transformers, wherein each layer of transformers may include a self-attention operation for the image feature IF and a cross-attention operation based on the image feature IF and the query element, and after the operation is completed, the operation result is fed forward to the next layer.
[0039] After multiple layers of transformer operations, the transformer model TF can generate extracted features EF, which are the parameters of the reference feature vector FV in the image feature IF, and further extract the features of interest to the composite model CM (i.e., extracted features EF). The parameters of the feature vector FV can adjust the weights of different types of features in the extracted features EF, and after multiple operations, the parameters will also be adjusted accordingly.
[0040] Finally, the composite model CM inputs the extracted features EF into the language model LM2 to generate the environment text ET.
[0041] In some embodiments, the language model LM2 is a decoder corresponding to the image encoder EC and the transformer model TF. In other words, the language model LM2 is used to convert the extracted features EF captured by the transformer model TF into the environment text ET in natural language form. In some examples, the language model LM2 is a trained large language model (Large Language Model; LLM).
[0042] The environment text ET generated by the language model LM2 is a text describing the environment image EI, for example, it may include the status of other objects around the vehicle: "There is a truck parked on the side of the road, and there is a white van and a black vehicle driving on the road."
[0043] In some embodiments, the environment text ET includes one of an object tag and a direction tag or a combination thereof. For example, since the environment image EI has a tag corresponding to the relative position of the vehicle, the corresponding generated environment text ET includes the attributes of the surrounding objects and the relative position between them and the vehicle, for example: "In front of the vehicle, there is a truck parked on the side of the road; in front of the left side of the vehicle, a row of green grass and a concrete curb in front of a building can be seen; in the right rear of the vehicle, two people can be seen walking."
[0044] It should be noted that the vehicle-mounted device 1 can set the language of the environment text ET output by the language model LM2 according to needs, but the present disclosure is not limited thereto.
[0045] In some embodiments, the vehicle-mounted device 1 also uses the position coordinates of the camera 14 when shooting the environment image EI as parameters to input into the composite model CM as a basis for extracting features. Figure 3 In response to receiving the vehicle coordinates VC corresponding to the vehicle captured environment image EI, the composite model CM inputs the vehicle coordinates VC, the image features IF and the query element into the transformer model TF to generate the extracted features EF. Accordingly, the composite model CM can refer to the position of the vehicle to generate the corresponding environment text ET.
[0046] In some embodiments, the vehicle-mounted device 1 further includes an input interface (not shown in the figure), such as a microphone, a touch screen, a keyboard, a touchpad, etc., and the vehicle-mounted device 1 can receive input information from the user, such as voice input and typing input.
[0047] When the vehicle-mounted device 1 receives input from the user through the input interface, for example, the user asks the vehicle-mounted device 1 questions about the driving vehicle, the vehicle-mounted device 1 inputs the user's input information into the composite model CM to extract corresponding features and generate corresponding environment text ET. For example, if the user asks questions about surrounding vehicles, the composite model CM can extract more features about surrounding vehicles from the environment image EI.
[0048] Specifically, if Figure 3 As shown, in response to receiving input information IN, the composite model CM generates the query element based on the input information IN and the feature vector FV; and the composite model CM inputs the image feature IF and the query element into the transformer model TF to generate the extracted feature EF.
[0049] For example, after the vehicle-mounted device 1 receives the input information IN, the processor 12 converts the input information IN into the form of a query element, and concatenates (appends) the input information IN and the feature vector FV and inputs them into the transformer model TF as the query element for operation. In this way, the composite model CM can generate the relevant environment text ET based on the user's input information IN to provide the subsequent language model LM1 with relevant information for operation.
[0050] For details about the language model LM1, please refer to Figure 4 , which is a flow chart of using language model LM1 to generate response text RT in some embodiments of the present disclosure. In some embodiments, language model LM1 is a large language model trained to provide instructions.
[0051] In some embodiments, the vehicle-mounted device 1 also inputs the current position coordinates of the vehicle as parameters into the language model LM1 as a basis for generating the response text RT. Since the language model LM1 has the characteristic of interpreting text, the processor 12 first converts the vehicle coordinates VC (e.g., GPS positioning coordinates) into text representing the vehicle position (e.g., address, road name), and then inputs the text into the language model LM1 as a calculation parameter.
[0052] Specifically, in response to receiving the vehicle coordinates VC, the processor 12 converts the vehicle coordinates VC into a position text PT; and the processor 12 inputs the position text PT and the environment text ET into the language model LM1 to generate the response text RT.
[0053] like Figure 4 As shown, in some embodiments, when the vehicle-mounted device 1 receives input information from the user (for example, through the aforementioned input interface), it also inputs the input information in text form (ie, input text IT) into the language model LM1 to generate a corresponding response text RT.
[0054] Specifically, in response to receiving the input text IT, the language model LM1 generates a response text RT based on the input text IT, wherein the response text RT is used to reply to the input text IT.
[0055] For example, when a user asks the vehicle-mounted device 1 questions such as "How to reach the destination faster?", the vehicle-mounted device 1 inputs the user's input in the form of text into the language model LM1, and the language model LM1 can interpret the text and respond to the user's questions accordingly, such as "Continue to accelerate, but be careful to keep a proper distance from the blue car on the right."
[0056] As for the response text RT generated by the vehicle-mounted device 1, its content can be adjusted according to actual needs. In addition to the aforementioned responses to the questions raised by the user, the vehicle-mounted device 1 can also actively provide driving suggestions based on the environment around the vehicle, or actively start a topic.
[0057] In some embodiments, the language model LM1 provides driving advice based on the environment around the vehicle. Specifically, the language model LM1 generates driving advice for operating the vehicle based on the environment text ET; and the language model LM1 uses the driving advice as the response text RT.
[0058] For example, when an obstacle approaches, the response text RT generated by the language model LM1 includes instructions to remind the user to slow down or dodge.
[0059] In some embodiments, the language model LM1 can actively communicate with the user to help the user concentrate. Specifically, the language model LM1 generates interactive information corresponding to the vehicle based on the environment text ET; and the language model LM1 uses the interactive information as the response text RT.
[0060] Furthermore, in some embodiments, when the vehicle-mounted device 1 receives a reply from the user to the interactive information (eg, receiving the reply through the aforementioned input interface), the language model LM1 generates a response text RT corresponding to the reply to continue the conversation.
[0061] Specifically, after generating the interactive information, in response to receiving the input text IT, the language model LM1 generates a response text RT based on the input text IT and the interactive information, wherein the response text RT is used to reply to the input text IT.
[0062] For example, the language model LM1 starts a conversation with an interesting question: "Do you know what the slowest car in the world is?" This simulates the chat situation between the vehicle-mounted device 1 and the user (eg, driver), which helps reduce fatigue.
[0063] Furthermore, if the user replies: "Tell me what is the slowest car in the world", the language model LM1 will simultaneously refer to the question asked in the previous conversation (i.e., the previously generated response text RT) and the user's reply (i.e., the currently received input text IT), and generate a reply such as "It's the blue car on your right! I suggest you speed up and overtake it."
[0064] In summary, the vehicle-mounted device 1 proposed in the present disclosure can not only provide driving suggestions based on the vehicle's surrounding environment, but also actively initiate dialogue with the user for interaction, and the content of the dialogue can be combined with the current driving situation, which helps to reduce the driver's fatigue and concentrate attention, thereby increasing the safety of vehicle driving.
[0065] Please refer to Figure 5 , which is a flow chart of the vehicle-mounted device control method 200 in the second embodiment of the present disclosure. The vehicle-mounted device control method 200 includes steps S201 to S204. The vehicle-mounted device control method 200 is used to provide content that interacts with the user according to the surrounding environment of the vehicle. The vehicle-mounted device control method 200 can be executed by a vehicle-mounted device (for example: the vehicle-mounted device 1 in the first embodiment), wherein the vehicle-mounted device is disposed in the vehicle in which the user is riding.
[0066] First, in step S201 , the vehicle-mounted device captures an environmental image around the vehicle.
[0067] Next, in step S202 , the vehicle-mounted device inputs the environment image into the composite model to generate corresponding environment text, wherein the environment text is used to describe the environment image.
[0068] Next, in step S203 , the vehicle-mounted device inputs the environment text into a first language model to generate a first response text corresponding to the vehicle.
[0069] Finally, in step S204, the vehicle-mounted device executes an interactive operation corresponding to the user based on the first response text.
[0070] In some embodiments, the composite model is used to input the environment image into an image encoder to generate multiple image features; input the image features and a query element into a transformer model to generate multiple extracted features corresponding to a feature vector, wherein the query element includes the feature vector; and input the extracted features into a second language model to generate the environment text.
[0071] In some embodiments, the composite model is further configured to, in response to receiving a real-time coordinate corresponding to the environment image captured by the vehicle, input the real-time coordinate, the image features and the query element into the transformer model to generate the extracted features.
[0072] In some embodiments, the composite model is further configured to generate the query element based on the input information and the feature vector in response to receiving input information corresponding to the user; and input the image features and the query element into the transformer model to generate the extracted features.
[0073] In some embodiments, the second language model is a decoder corresponding to the image encoder and the transformer model.
[0074] In some embodiments, the vehicle-mounted device control method 200 further includes the vehicle-mounted device converting the real-time coordinates of the environmental image captured by the vehicle into location text; and the vehicle-mounted device inputting the location text and the environmental text into the first language model to generate the first response text.
[0075] In some embodiments, the first language model is further configured to generate the first response text based on the input text in response to receiving the input text corresponding to the user, wherein the first response text is used to reply to the input text.
[0076] In some embodiments, the first language model is further used to generate driving suggestions for operating the vehicle based on the environment text; and use the driving suggestions as the first response text.
[0077] In some embodiments, the first language model is further used to generate interaction information corresponding to the vehicle based on the environmental text; and use the interaction information as the first response text.
[0078] In some embodiments, the first language model is further used to generate a second response text based on the input text and the interaction information in response to receiving input text corresponding to the user after generating the interaction information, wherein the second response text is used to reply to the input text.
[0079] In summary, the vehicle-mounted device control method 200 proposed in the present disclosure can not only provide driving suggestions based on the vehicle's surrounding environment, but also actively initiate a dialogue with the user for interaction, and the content of the dialogue can be combined with the current driving situation, which helps to reduce the driver's fatigue and concentrate attention, thereby increasing the safety of vehicle driving.
[0080] Although several embodiments are described above as examples, the vehicle-mounted device and method proposed in the present disclosure may also be implemented by other systems, hardware, software, storage media or a combination thereof. Therefore, the protection scope of the present disclosure should not be limited to the specific implementation methods described in the embodiments of the present disclosure, but should be determined by the attached claims.
[0081] It is obvious to those skilled in the art that various modifications and changes can be made to the structure of the present disclosure without departing from the scope or spirit of the present disclosure. In view of the foregoing, the protection scope of the present disclosure also covers modifications and changes made within the scope of the attached claims.
[0082]
Explanation of symbols
[0083] 1: Vehicle-mounted device
[0084] 12: Processor
[0085] 14: Camera
[0086] 16: Output interface
[0087] EI: Environmental Image
[0088] CM: Composite Model
[0089] ET: Environmental Text
[0090] LM1,LM2: Language Model
[0091] RT: Response text
[0092] EC: Video Encoder
[0093] IF: Image features
[0094] FV: Feature Vector
[0095] IN: Input information
[0096] VC: Vehicle Coordinates
[0097] TF: Transformer Model
[0098] EF: Extraction Features
[0099] PT: Position text
[0100] IT: Input Text
[0101] 200: Vehicle-mounted device control method
[0102] S201~S204: steps.
Claims
1. A vehicle-mounted device, characterized in that: The vehicle is provided for the user to ride in, including: A camera for capturing images of the environment around the vehicle; Output interface; as well as The processor is respectively connected to the camera and the output interface for performing the following operations: Inputting the environmental image into a composite model to generate corresponding environmental text, wherein the environmental text is used to describe the environmental image; Inputting the environment text into a first language model to generate a first response text corresponding to the vehicle; as well as A control signal corresponding to the first response text is generated to control the output interface to execute an interactive operation corresponding to the user.
2. The vehicle-mounted device according to claim 1, characterized in that: The composite model is used to: Inputting the environment image into an image encoder to generate a plurality of image features; Inputting the plurality of image features and a query element into a transformer model to generate a plurality of extracted features corresponding to a feature vector, wherein the query element includes the feature vector; and The plurality of extracted features are input into a second language model to generate the environment text.
3. The vehicle-mounted device according to claim 2, characterized in that: The composite model is further used to: In response to receiving a real-time coordinate corresponding to the environment image captured by the vehicle, the real-time coordinate, the plurality of image features and the query element are input into the transformer model to generate the plurality of extracted features.
4. The vehicle-mounted device according to claim 2, characterized in that: The vehicle-mounted device further comprises an input interface, the input interface is used to generate input information corresponding to the user, and the composite model is further used to: In response to receiving the input information, generating the query element based on the input information and the feature vector; as well as The plurality of image features and the query element are input into the transformer model to generate the plurality of extracted features.
5. The vehicle-mounted device according to claim 1, characterized in that: The processor is further configured to perform the following operations: Converting the real-time coordinates corresponding to the environment image captured by the vehicle into position text; and The location text and the environment text are input into the first language model to generate the first response text.
6. The vehicle-mounted device according to claim 1, characterized in that: The vehicle-mounted device further includes an input interface, the input interface is used to generate input text corresponding to the user, and the first language model is further used to: In response to receiving the input text, the first response text is generated based on the input text, wherein the first response text is used to reply to the input text.
7. The vehicle-mounted device according to claim 1, characterized in that: The first language model is further used to: generating driving advice for operating the vehicle based on the environmental context; and The driving suggestion is used as the first response text.
8. The vehicle-mounted device according to claim 1, characterized in that: The first language model is further used to: generating interactive information corresponding to the vehicle based on the environmental text; and The interactive information is used as the first response text.
9. The vehicle-mounted device according to claim 8, characterized in that: The vehicle-mounted device further includes an input interface, the input interface is used to generate input text corresponding to the user, and the first language model is further used to: After the interactive information is generated, in response to receiving the input text, a second response text is generated based on the input text and the interactive information, wherein the second response text is used to reply to the input text.
10. A control method, characterized in that: The invention is applicable to a vehicle-mounted device, wherein the vehicle-mounted device is arranged in a vehicle in which a user rides, and the steps include: Capturing an environmental image around the vehicle; Inputting the environmental image into a composite model to generate corresponding environmental text, wherein the environmental text is used to describe the environmental image; Inputting the environment text into a first language model to generate a first response text corresponding to the vehicle; as well as Based on the first response text, an interactive operation corresponding to the user is executed.