Human-computer interaction method and device, related equipment and computer program product

Abstract

The invention discloses a man-machine interaction method and device, related equipment and a computer program product, and relates to the technical field of artificial intelligence. The man-machine interaction method comprises the steps that an interaction problem proposed by a target user in a vehicle is acquired, an image of the target user is acquired through shooting of a first camera in the vehicle, sight line direction information of the target user is determined according to the image, and therefore the intention and requirements of the user can be better understood. The sight direction information of a target user is determined by utilizing the first camera, the visual angle of the second camera is adjusted according to the sight direction information, and the out-of-vehicle image shot by the second camera is obtained, so that the system can see an object or scene concerned by the user, and the interaction problem of the out-of-vehicle image and the user is solved by calling the multi-modal large model. According to the method, deeper information processing and understanding can be carried out, and the corresponding reply result is generated according to the image outside the vehicle by understanding the interaction problem of the user, so that the reply is more accurate and related, and the interaction efficiency and quality are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a human-computer interaction method, apparatus, related equipment and computer program product. Background Art

[0002] With the development of intelligent transportation systems, in-vehicle driving assistance systems have improved driving and comfort by integrating sensors, cameras and advanced algorithms, making in-vehicle driving assistance systems play an increasingly important role.

[0003] However, existing in-vehicle driving assistance systems mainly rely on in-vehicle DMS (driver monitoring system) cameras to monitor the driver's status, and fixed external cameras to perceive the external environment. These systems usually work independently and lack effective information fusion and interaction understanding mechanisms, resulting in inaccurate recognition of the driver's intentions. Especially in complex traffic scenarios, it is difficult for the in-vehicle driving assistance system to accurately understand the driver's line of sight and interaction intentions, which limits the environmental perception and interaction feedback capabilities of the in-vehicle driving assistance system. Summary of the invention

[0004] In view of the above problems, this application is proposed to provide a human-computer interaction method, device, equipment and storage medium to enhance the interactive capability of the vehicle-mounted driving assistance system. The specific solution is as follows:

[0005] In a first aspect, a human-computer interaction method is provided, comprising:

[0006] Obtain interactive questions raised by target users in the car;

[0007] Acquire an image of the target user captured by a first camera in the vehicle, and determine the sight direction information of the target user according to the image;

[0008] According to the line of sight direction information, adjust the viewing angle of the second camera on the vehicle so that the viewing angle of the second camera is consistent with the line of sight direction of the target user, and obtain the image outside the vehicle taken by the second camera;

[0009] The multimodal large model is called to instruct the multimodal large model to generate a response result corresponding to the interactive question in combination with the external vehicle image, and output the response result.

[0010] In a possible design, in another implementation manner of the first aspect of the embodiment of the present application, calling the multimodal large model to instruct the multimodal large model to generate a response result corresponding to the interactive question in combination with the external vehicle image, and outputting the response result, includes:

[0011] Obtain a first prompt instruction prompt format template, wherein the first prompt instruction prompt format template includes a first task instruction, an interactive question slot, and an external vehicle image slot, wherein the first task instruction is used to instruct the multimodal large model to combine the external vehicle image in the external vehicle image slot to generate the reply result corresponding to the interactive question in the interactive question slot;

[0012] Fill the interactive question into the interactive question slot, fill the outside-vehicle image into the outside-vehicle image slot, obtain a first prompt instruction prompt, input the first prompt instruction prompt into the pre-trained multimodal large model, and obtain the reply result output by the multimodal large model.

[0013] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the target user is a driver, and before calling the multimodal large model, the method further includes:

[0014] Acquire a task perception result of at least one configured active safety detection task, where the task perception result of the active safety detection task is determined based on the driver's state information, vehicle external environment information and / or vehicle driving state information;

[0015] The multimodal large model is called to instruct the multimodal large model to generate a response result corresponding to the interactive question in combination with the vehicle exterior image, including:

[0016] The multimodal large model is called to instruct the multimodal large model to comprehensively consider the task perception result and the external vehicle image to generate the response result corresponding to the interactive question.

[0017] In a possible design, in another implementation of the first aspect of the embodiment of the present application, calling the multimodal large model to instruct the multimodal large model to comprehensively consider the task perception result and the external vehicle image to generate the response result corresponding to the interactive question includes:

[0018] Obtain a second prompt instruction prompt format template, wherein the second prompt instruction prompt format template includes a second task instruction, an interactive question slot, a task perception result slot, and an external vehicle image slot, wherein the second task instruction is used to instruct the multimodal large model to comprehensively consider the external vehicle image in the external vehicle image slot and the task perception result in the task perception result slot, and generate the reply result corresponding to the interactive question in the interactive question slot;

[0019] Fill the interactive question into the interactive question slot, fill the outside-vehicle image into the outside-vehicle image slot, fill the task perception result into the task perception result slot, obtain a second prompt instruction prompt, input the second prompt instruction prompt into the pre-trained multimodal large model, and obtain the reply result output by the multimodal large model.

[0020] In a possible design, in another implementation of the first aspect of the embodiment of the present application, before acquiring the image outside the vehicle captured by the second camera, the method further includes:

[0021] The interaction question is parsed, and if distance information can be extracted from the interaction question, a shooting focal length is determined according to the distance information, and the focus of the second camera is adjusted to the shooting focal length.

[0022] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the process of determining the sight direction information of the target user according to the image includes:

[0023] Locating an eye region of the target user in the image;

[0024] Acquire the spatial position of the origin of the human eye and the direction of eye movement in the eye area;

[0025] The sight line direction information of the target person is determined according to the spatial position of the human eye origin and the eye movement direction.

[0026] In a possible design, in another implementation manner of the first aspect of the embodiment of the present application, the sight direction information is obtained based on a first coordinate system where the first camera is located;

[0027] The process of adjusting the viewing angle of the second vehicle-mounted camera according to the sight direction information includes:

[0028] Using the coordinate system mapping relationship between the first camera and the second camera, the sight line direction information is converted into a second coordinate system where the second camera is located to obtain target sight line direction information;

[0029] The rotation angle of the second camera is calculated according to the target sight direction information, and the rotation of the second camera is controlled according to the rotation angle.

[0030] In a second aspect, a human-computer interaction device is provided, comprising:

[0031] A question acquisition unit, used to acquire interactive questions raised by a target user in the car;

[0032] an image acquisition unit, configured to acquire an image of the target user captured by a first camera in the vehicle, and determine the sight direction information of the target user according to the image;

[0033] a viewing angle determining unit, configured to adjust the viewing angle of a second camera mounted on the vehicle according to the sight line direction information, so that the viewing angle of the second camera is consistent with the sight line direction of the target user, and obtain an image outside the vehicle taken by the second camera;

[0034] A reply output unit is used to call the multimodal large model to instruct the multimodal large model to generate a reply result corresponding to the interactive question in combination with the external vehicle image, and output the reply result.

[0035] In a third aspect, an electronic device is provided, comprising: a memory and a processor;

[0036] The memory is used to store programs;

[0037] The processor is used to execute the program to implement each step of the human-computer interaction method described in any one of the first aspects of the present application.

[0038] In a fourth aspect, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the various steps of the human-computer interaction method described in any one of the first aspects of the present application are implemented.

[0039] In a fifth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the various steps of the human-computer interaction method described in any one of the aforementioned first aspects of the present application.

[0040] By means of the above technical scheme, a human-computer interaction method proposed in the present application obtains the interactive questions raised by the target user in the car, uses the first camera in the car to obtain the image of the target user, and determines the sight direction information of the target user according to the image, so as to better understand the user's intention and needs, and provide a more personalized and intuitive interactive experience. The sight direction information of the target user is determined by the first camera, and the viewing angle of the second camera is adjusted accordingly, and the image outside the car taken by the second camera is obtained, so that the system can see the object or scene that the user is concerned about, and enhance the system's perception of the environment outside the car. And by calling a multimodal large model, combining the image outside the car and the user's interactive questions, it is possible to perform deeper information processing and understanding, and by understanding the user's interactive questions, the corresponding reply results are generated according to the image outside the car, so that the reply is more accurate and relevant, and the efficiency and quality of the interaction are improved. The human-computer interaction method of the present application realizes the accurate capture and response of the user's intention, improves the naturalness, accuracy and efficiency of the interaction, and enhances the driving safety and the intelligence of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0042] Figure 1 A schematic diagram of the system architecture of a human-computer interaction method provided in an embodiment of the present application;

[0043] Figure 2 A flowchart of a human-computer interaction method provided in an embodiment of the present application;

[0044] Figure 3 A flowchart of another human-computer interaction method provided in an embodiment of the present application;

[0045] Figure 4 A schematic diagram of the working principle of a human-computer interaction system provided in an embodiment of the present application;

[0046] Figure 5 A schematic diagram of the working principle of another human-computer interaction system provided in an embodiment of the present application;

[0047] Figure 6 A schematic diagram of the structure of a human-computer interaction device provided in an embodiment of the present application;

[0048] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] Before introducing this application, the English terms involved in this article are explained first:

[0050] DMS camera: The full name is Driver Monitoring System camera, which is a vehicle-mounted camera system specially designed to monitor the driver's behavior and status. The main purpose of the DMS camera is to improve road safety by monitoring the driver's physiological and behavioral status in real time to ensure that the driver remains alert and focused while driving.

[0051] prompt: Instructions. When interacting with AI (such as an artificial intelligence model), you need to send instructions to AI. It can be a text description, such as "Please recommend me a pop song" when you interact with AI, or it can be a parameter description in a certain format. For example, if you want AI to draw in a certain format, you need to describe the relevant drawing parameters.

[0052] Multimodal large models: Multimodal large models are an advanced AI technology that can process and understand data from different sources, such as text, images, sound, and video. By integrating these different types of information, the model can more fully understand the context and improve the performance of tasks such as recognition, classification, prediction, and generation. These models usually adopt complex neural network architectures, such as Transformer, which are pre-trained on large-scale multimodal datasets and then fine-tuned for specific tasks to achieve more accurate understanding and more natural interaction. Multimodal large models have a wide range of applications, from autonomous driving to smart assistants, from content recommendation to health diagnosis, and they all play an important role in improving robustness, enhanced learning, and interactive learning.

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] See also Figure 1 , Figure 1 This is a schematic diagram of the system architecture of a human-computer interaction method provided in an embodiment of the present application. The system structure described in this embodiment is a human-computer interaction system that integrates a first camera 101, a second camera 102, an onboard processor (CPU) 103, and a control module 104, aiming to improve driving safety and the interactivity of the auxiliary system. The following is an introduction to each part:

[0055] The first camera 101 can be a vehicle-mounted DMS camera. The first camera can be installed inside the vehicle and is responsible for real-time monitoring of the real-time image of the target user, who can be not only the driver but also other passengers in the vehicle.

[0056] The second camera 102 can be a rotatable camera. As a physical device, the second camera can be installed inside or outside the vehicle. The second camera can adjust the viewing angle according to the line of sight direction and interaction intention of the target user, and focus on the vicinity of the line of sight of the target user to track the object that the target user is paying attention to and capture the image.

[0057] On-board processor (CPU) 103: As the brain of the system, the on-board processor is responsible for running the algorithm. The on-board processor can be used to process and integrate data from the first camera and the second camera, and analyze the line of sight and intention of the target user. The on-board processor 103 can also include a multimodal large model, so as to use the multimodal large model to process and analyze the outside image of the vehicle taken by the second camera and the questions raised by the target user, and generate response results corresponding to the questions.

[0058] The control module 104 controls the rotation and zoom of the second camera according to the analysis result of the vehicle-mounted processor 103 to ensure that the second camera can track the sight direction of the target user or the object of interest.

[0059] In one possible implementation, the multimodal large model may not be configured in the vehicle processor (CPU) 103, but the multimodal large model may be placed in a remote server 105. In this case, the system architecture may also include a server 105, and the processing of interactive tasks is realized through the communication between the vehicle processor and the server. The vehicle processor transmits the collected data to the server, and the server uses its powerful processing power to call the multimodal large model to realize the processing of interactive tasks. This allows the system to use more powerful computing resources for data processing and analysis, and also provides scalability, allowing the system to dynamically adjust resources as needed, and can more easily perform software updates and maintenance.

[0060] The present application provides a human-computer interaction method, which obtains the interactive questions raised by the target user in the car, uses the first camera in the car to capture the image of the target user, and determines the line of sight information of the target user based on the image, so as to better understand the user's intentions and needs. The line of sight direction information of the target user is determined by the first camera, and the viewing angle of the second camera is adjusted accordingly, and the image outside the car taken by the second camera is obtained, so that the system can see the object or scene that the user is concerned about, and by calling a multimodal large model, combining the image outside the car and the user's interactive questions, it can perform deeper information processing and understanding, and by understanding the user's interactive questions, generate corresponding reply results based on the image outside the car, so that the reply is more accurate and relevant, and the efficiency and quality of the interaction are improved.

[0061] Next, see Figure 2 , Figure 2 A flowchart of a human-computer interaction method provided in an embodiment of the present application is provided, and the method is applied to a computer device as an example. The computer device may be Figure 1 The vehicle-mounted processor 103 or the system consisting of the vehicle-mounted processor 103 and the server 105. As described below, the specific steps include:

[0062] Step S100: Obtaining interactive questions raised by the target user in the car.

[0063] Specifically, the onboard voice recognition module can actively capture the voice commands or questions of the target user in the car. The voice data can be converted into text information for further processing and analysis, or the voice data can be kept in its state and processed and analyzed. The following is an example of this process:

[0064] Questions asked by the target user in the car, for example, when the target user sees an attractive building outside the car and curiously asks "What is this building?", this question can be captured by the in-car microphone.

[0065] Interactive questions are not limited to specific types. They can be inquiries about the external environment or knowledge questions. For example, assuming that the target user wants to know the current speed of the vehicle or the external temperature, he may ask: "How fast are we going now?" or "What is the temperature outside?" The present application can capture these questions through the on-board microphone and can provide corresponding speed or temperature information by combining it with the vehicle's sensor data through the on-board processor.

[0066] During these interactive processes, the design of this application ensures that it can flexibly respond to the diverse interactive needs of the target users in the car, provide timely and accurate feedback, and enhance the driving experience. This allows both the driver and other passengers to communicate with the vehicle system through natural language to obtain the information they need or perform specific operations, which not only improves the convenience of driving, but also provides passengers with a richer interactive experience.

[0067] Step S110: Acquire an image of the target user captured by a first camera in the vehicle, and determine the sight direction information of the target user according to the image.

[0068] Specifically, the first camera may be a vehicle-mounted DMS camera. The present application utilizes the first camera to capture facial images of a target user in the vehicle, especially detailed images of the eye area, and can extract the target user's line of sight direction information from these images.

[0069] This application can identify the interests or needs of target users in the car by analyzing their line of sight, thereby providing more thoughtful services. For example, if a passenger's line of sight stays on a landmark outside the car window, relevant information about the landmark can be proactively provided to enhance the passenger's riding experience. In this way, this application can monitor and understand the behavior of target users in the car, and achieve more comprehensive in-car interaction and environmental perception.

[0070] Step S120: adjusting the viewing angle of the second vehicle-mounted camera according to the line of sight direction information so that the viewing angle of the second camera is consistent with the line of sight direction of the target user, and acquiring an image outside the vehicle taken by the second camera.

[0071] Specifically, the present application monitors the sight direction of the target user in the car, whether the driver or other passengers, through the first camera. After the sight direction of the target user is determined through the eye tracking algorithm, it will use the control module to adjust the viewing angle of the second camera to match the sight direction of the target user, ensuring that the viewing angle of the second camera is consistent with the sight direction of the target user, so as to capture the corresponding image outside the car.

[0072] The second camera can be a rotatable camera, which has a high degree of flexibility and configurability. The design of the second camera allows it to be installed inside or outside the vehicle. This flexibility enables the second camera to be optimally configured according to actual application requirements and vehicle design.

[0073] When the second camera is configured inside the vehicle, such as the location of a driving recorder, when the lens of the second camera faces outside the vehicle, it can capture the target object that the user is concerned about based on the target user's line of sight information captured by the first camera (the on-board DMS camera). On the other hand, when the second camera is installed outside the vehicle, it can serve as an external sensor to enhance the vehicle's environmental perception capabilities.

[0074] Whether inside or outside the car, the second camera can be linked with the first camera and adjust the viewing angle according to the driver's or passenger's line of sight information to obtain images of the outside environment. This is essential for improving driving safety and providing richer information outside the car. For example, if the target user is interested in a landmark or event outside the car, the second camera can rotate and focus on the outside of the car to capture relevant images in real time. This enables a quick response to the target user's line of sight and request, and provides timely visual information feedback. Regardless of whether the second camera is installed inside or outside the car, the present application can be controlled by the on-board processor and control module to ensure that the camera's viewing angle is consistent with the target user's line of sight and obtain images outside the car when needed, which not only improves the response speed and accuracy, but also enhances the user experience, allowing the vehicle to interact more intelligently with the target user in the car.

[0075] One possible implementation method is that during normal driving, the second camera may remain in a standby state, continuously tracking the target user's line of sight, but not actually taking images. This design can reduce unnecessary data processing and storage requirements while maintaining real-time monitoring of the target user's line of sight. Only when the voice recognition module captures the question raised by the target user, the second camera will be activated to shoot, thereby capturing the most relevant image outside the car at the critical moment.

[0076] For example, if the target user asks "What is that building?", the second camera will be activated at the same time as this question is captured to capture the image of the building in the target user's sight direction, and the image will be taken at the moment the question is asked or a short time later. This design makes the use of the second camera more efficient and purposeful, and it only takes pictures when the target user needs to obtain external information, which not only responds to user needs but also optimizes resource usage.

[0077] Another possible implementation is that during normal driving, the second camera can continuously track and capture images in the direction of the user's sight, but these images can be saved only as previews or caches without further processing. However, once the user asks a specific question through voice recognition or other interactive methods, such as asking "What does that sign mean?" or "What happened to the car in front?", the second camera will be triggered to acquire and process the target image within a preset time. This image can be an instant target image at the moment when the user asks the question. In addition, considering that the process of the user asking a question is a continuous time period, that is, from the beginning of the question to the end of the question, multiple images within the time period can be acquired. Further, considering the possible delay between the user seeing the target and asking the question, in this embodiment, the set time can be extended forward from the time point when the user starts asking the question, and multiple images within the period from the extended start time to the end time of the question are acquired. Thereby ensuring that the captured image is most relevant to the user's question. This design makes the use of the second camera more efficient, and it can respond to the user's sight and questions in real time to ensure that the captured image is most relevant to the user's question.

[0078] Another possible implementation is that the viewing angle of the second camera can remain unchanged under normal conditions until the user asks a question. Once the user asks a question, the second camera will be quickly activated to obtain the visual direction information of the target user, and the viewing angle of the second camera will be adjusted to accurately align with the line of sight of the target user, and start capturing images outside the vehicle. For example, if the driver notices a traffic jam ahead while driving and asks, "What happened ahead?" Upon receiving this request, the second camera will be immediately directed to rotate to the corresponding direction to capture real-time images of the road ahead. In this way, the second camera can provide a detailed view of the scene that the user is concerned about, helping the driver to better understand the situation. This design makes the use of the second camera more efficient and on-demand. The viewing angle will be adjusted and shooting will be performed only when the user actually asks a question or needs it, thereby ensuring the rational allocation of resources and timely response to user needs.

[0079] Step S130: calling the multimodal large model to instruct the multimodal large model to generate a response result corresponding to the interactive question in combination with the external vehicle image, and output the response result.

[0080] Specifically, after acquiring the images outside the vehicle, the present application will further use the multimodal large model to process and understand these images and the interaction problems related thereto. When the second camera captures the image outside the vehicle and the user asks a specific interaction question, the multimodal large model will be called. The multimodal large model can combine the image outside the vehicle and the user's voice command or question for in-depth analysis and understanding, and generate a response result corresponding to the question. For example, if a user asks a question when seeing an unfamiliar traffic sign: "What does this sign mean?" The image of the sign and the question will be input into the multimodal large model together.

[0081] After generating the corresponding response results, the on-board processor will output this information to the user. This may include explanations of signs, relevant traffic rules, or direct answers to user questions. The on-board processor can display this information through the on-board display system or directly tell the user through the voice interaction interface to ensure that the user can get the answers they need in the most intuitive and convenient way.

[0082] The above method of this application can not only capture and understand the external environment of the vehicle, but also provide timely and accurate feedback, enhancing the interactivity and practicality of the driving assistance system. By calling a multimodal large model, this application can handle complex interaction problems and provide users with richer and deeper information, improving the intelligent level of the driving experience.

[0083] This embodiment provides a human-computer interaction method, which obtains the interactive questions raised by the target user in the car, uses the first camera in the car to capture the image of the target user, and determines the sight direction information of the target user based on the image, so as to better understand the user's intentions and needs. The sight direction information of the target user is determined by the first camera, and the viewing angle of the second camera is adjusted accordingly, and the image outside the car captured by the second camera is obtained, so that the object or scene that the user is concerned about can be seen, and by calling a multimodal large model, combining the image outside the car and the user's interactive questions, a deeper level of information processing and understanding can be performed, and by understanding the user's interactive questions, a corresponding reply result is generated based on the image outside the car, so that the reply is more accurate and relevant, and the efficiency and quality of the interaction are improved.

[0084] Furthermore, the process of calling the multimodal large model in step S130 in the aforementioned embodiment to instruct the multimodal large model to generate a response result corresponding to the interactive question in combination with the external vehicle image and output the response result can be described in detail.

[0085] First, a predefined first prompt format template is obtained. This template contains three main parts: the first task instruction, the interactive question slot, and the external image slot. The first task instruction is a key component in the template, which explicitly instructs the multimodal large model how to combine the external image in the external image slot to generate a response result corresponding to the interactive question in the interactive question slot.

[0086] Fill the interactive question slot: When the target user asks an interactive question, this question can be filled into the interactive question slot in the template. For example, if the target user asks: "What does the traffic sign at the intersection ahead mean?" this question will be placed in the interactive question slot.

[0087] Filling the vehicle exterior image slot: At the same time, the vehicle exterior image captured by the second camera is filled into the vehicle exterior image slot in the template, so that the template contains real-time visual information related to the target user's question.

[0088] Task instructions: Instructions or instructions that guide the multimodal model to perform specific tasks, which are used to tell the multimodal model how to combine the external image and the interactive question to generate accurate response results. For example, "Please combine the image in the external image, answer the question in the interactive question, and output the response result."

[0089] Generate the first prompt: After completing the above filling, a complete first prompt can be generated, which includes task instructions, interaction questions and corresponding external images.

[0090] Input to the multimodal big model: Input this first prompt command (prompt) into the pre-trained multimodal big model. The multimodal big model will analyze and understand the content of the question based on this structured input, and refer to the image outside the car to generate an accurate response result. Finally, the multimodal big model will output the response result for the interactive question raised by the target user.

[0091] This embodiment can ensure that the multimodal large model receives clear, comprehensive, and structured input, thereby improving the accuracy and relevance of the response results, so that the target user's questions can be processed and responded to more efficiently while maintaining the fluency and naturalness of the interaction.

[0092] Furthermore, in some embodiments of the present application, in view of the situation where the driver is the target user, in addition to the aforementioned solution, another interaction solution is provided during the interaction with the driver to further focus on driving safety. Figure 3 , Figure 3 A flow chart of another human-computer interaction method provided in an embodiment of the present application is provided below, and this part of the content is introduced in detail.

[0093] Step S200: Obtaining interactive questions raised by the target user in the car.

[0094] Step S210: Acquire an image of the target user captured by a first camera in the vehicle, and determine the sight direction information of the target user according to the image.

[0095] Step S220: adjust the viewing angle of the second vehicle-mounted camera according to the line of sight direction information so that the viewing angle of the second camera is consistent with the line of sight direction of the target user, and obtain the image outside the vehicle taken by the second camera.

[0096] Specifically, the above steps S200-S220 correspond one-to-one to the steps S100-S120 in the aforementioned embodiment, and please refer to the above introduction for details, which will not be repeated here.

[0097] Step S230: Obtain a task perception result of at least one configured active safety detection task.

[0098] Specifically, when the target user is a driver, before preparing to call the multimodal large model to handle the driver's interaction problem, at least one configured active safety detection task is first executed. These tasks are designed to monitor and analyze the driver's state information, the external environment information, and the vehicle's driving state information in real time to ensure driving safety.

[0099] Driver status information: The on-board DMS camera can monitor the driver's physiological and behavioral status, such as fatigue level, concentration, head and eye movement, etc., to assess whether the driver is able to drive safely.

[0100] External environmental information: External sensors and cameras can be used to collect environmental data, including traffic flow, weather conditions, road conditions, etc., which helps to determine the impact of the external environment on driving safety.

[0101] Vehicle driving status information: It can also monitor the real-time driving status of the vehicle, such as speed, acceleration, braking status, etc., which helps to understand the dynamic behavior of the vehicle and prevent potential driving risks.

[0102] This application combines this information to determine the perception results of active safety detection tasks, which reflect the current driving environment and the driver's status, thereby providing a basis for whether safety measures need to be taken. Among them, active safety detection tasks can include fatigue driving detection, speed monitoring and warning, pedestrian detection, and blind spot information monitoring, etc. For example, if signs of driver fatigue driving are detected, it may remind the driver to rest or take other preventive measures.

[0103] After obtaining these task perception results, they can be combined with the interactive questions raised by the driver, and the multimodal large model can be called for in-depth analysis and processing. The multimodal large model can comprehensively consider the user's interactive questions and the perception results of the active safety detection task. In this way, this application can not only respond to the driver's information needs, but also ensure that driving safety is taken into account while providing answers, thereby providing a more comprehensive and reliable driving assistance experience.

[0104] Step S240: calling the multimodal large model to instruct the multimodal large model to comprehensively consider the task perception result and the image outside the vehicle to generate the response result corresponding to the interactive question.

[0105] Specifically, when dealing with interactive questions raised by the driver, multiple factors can be comprehensively considered. Before preparing to call the multimodal large model, the perception results of at least one active safety detection task can be obtained and analyzed. These results are based on the driver's status information, the external environment information and the vehicle driving status information. These perception results are input into the multimodal large model together with the external image taken by the second camera, instructing the multimodal large model to comprehensively consider this information to generate a response result corresponding to the driver's interactive question. Based on the comprehensive consideration of the perception results of the safety detection task, the multimodal large model can output a response result while ensuring driving safety. This response result not only takes into account accuracy, but also driving safety factors. There can be many situations for the response result. The following are examples of some situations:

[0106] Handling of safety issues: The multimodal large model will determine whether to output the answer to the interactive question based on the input information. If there are safety issues in the perception results of the multimodal large model analysis task, such as driver fatigue or distraction, these safety issues will be handled first. In this case, the multimodal large model will not output the specific questions raised by the user, but will issue a safety reminder requiring the driver to take immediate action, such as "Please note that the system has detected a pedestrian ahead. Please concentrate on driving and interact with you later."

[0107] Continuous safety monitoring: During the period when the safety issue is not resolved, such as when the driver continues to drive fatigued, the multimodal large model will suspend outputting the answers to the user's questions until the safety issue is resolved. This is to prevent the driver from ignoring safety warnings due to distraction and ensure driving safety.

[0108] Response when there is no safety issue: When there is no safety issue in the task perception results, the multimodal large model will process the user's question normally and output the corresponding answer. For example, if the driver asks about the landmark information ahead, the name of the landmark and related information can be given by combining the external image and the analysis results of the multimodal large model.

[0109] This embodiment implements a comprehensive human-computer interaction system, which improves driving safety and interactive experience by analyzing the driver's state, the external environment and the driving state of the vehicle. First, at least one configured active safety detection task is executed to collect the driver's state information, the external environment information and the vehicle driving state information to determine the task perception results. These perception results provide the system with important data about the driver's behavior and the external environment, which is the basis for the system to proceed to the next step. Then, the multimodal large model is called to combine the task perception results with the external image of the vehicle to generate specific response results for the interactive questions raised by the driver. This embodiment can not only understand the driver's questions, but also provide accurate feedback based on the real-time environment and vehicle status. In this way, this embodiment can not only provide information assistance, but also remind the driver in time when potential safety issues are detected to ensure safety during driving. This design reflects the importance of driving safety, and also ensures that the driver can get timely and effective information feedback in a safe situation, enhancing the practicality and reliability of the driving assistance system.

[0110] Furthermore, in some embodiments of the present application, the process of calling the multimodal large model in step S240 in the aforementioned embodiment to instruct the multimodal large model to comprehensively consider the task perception results and the external vehicle image to generate the response result corresponding to the interactive question is described in detail.

[0111] One possible implementation method is that when calling the multimodal large model, if the target user is detected as a driver, a more detailed second prompt instruction format template can be used. The second prompt instruction format template not only includes interactive questions and external vehicle images, but also integrates task perception results. First, obtain a second prompt instruction format template. This template is more complex in design because it can include four main parts: second task instructions, interactive question slots, task perception result slots, and external vehicle image slots. The second task instruction is the guiding part of the template, which explicitly instructs the multimodal large model how to comprehensively consider external vehicle images and task perception results to generate response results corresponding to interactive questions.

[0112] The specific steps are as follows:

[0113] Fill the Interaction Question Slot: Fill the Interaction Question Slot in the template with the specific question the driver asked. For example, if the driver asks, "What is the building ahead?" the question will be placed in the Interaction Question Slot.

[0114] Filling the vehicle exterior image slot: At the same time, the vehicle exterior image taken by the second camera can be filled into the vehicle exterior image slot in the template to provide visual information for model analysis.

[0115] Filling the task perception result slot: You can also fill the task perception results obtained from the active safety detection task into the task perception result slot in the template. These results may include key information such as the driver's fatigue state and the current speed of the vehicle.

[0116] Generate a second prompt: After completing the above filling, a complete second prompt will be generated, which includes task instructions, specific interaction questions, relevant external images and task perception results.

[0117] Input to the multimodal big model: Input this second prompt command (prompt) into the pre-trained multimodal big model. Based on this structured input, the model will analyze and understand the content of the question, while referring to the external image and task perception results to generate an accurate response result. Finally, the multimodal big model will output the response result for the interactive question raised by the driver, and comprehensively consider the external environment and the driver's status to provide safer and more accurate driving assistance.

[0118] This embodiment further designs a second prompt instruction (prompt) template based on the target user (driver) with a specific identity, so as to ensure driving safety while completing the interaction. The second prompt instruction (prompt) template not only includes interaction questions and images outside the vehicle, but also integrates task perception results, such as the driver's fatigue state and the current speed of the vehicle and other key information. In this way, it is possible to comprehensively consider the external environment and the driver's state to generate safer and more accurate driving assistance response results. It not only improves the responsiveness and interaction quality of the driving assistance system, but also enhances adaptability and interactivity, providing users with a more intelligent, efficient and safe driving experience.

[0119] Furthermore, in some embodiments of the present application, in order to solve the problem of the accuracy and clarity of the second camera in capturing the image of the target object that the target user is concerned about, a method can be adopted, that is, to determine the focal length of the second camera by extracting distance information, so as to obtain a clearer image outside the vehicle. This part is introduced in detail below.

[0120] Specifically, after obtaining the interactive question raised by the target user in the car, the interactive question is parsed to determine whether there is distance information in the interactive question. When receiving the user's interactive question, the user's voice command can be converted into text through voice recognition technology, and the specific content in the question can be parsed using natural language processing technology to determine whether there is distance information in the interactive question. If there is, the distance information is extracted and the distance information is used to determine the focal length of the second camera. If there is no distance information, the default or preset focal length setting can be used, and the control module can be used to command the second camera to adjust its viewing angle to match the sight direction of the target user.

[0121] In one possible implementation, various technical means can be used to extract distance information. Some examples are given below. Natural language processing (NLP) technology can be used to identify and extract distance information from text, a series of rules can be designed to match and identify common distance expressions, and a machine learning model can be trained to identify and extract distance information. The distance information can be in various forms, including specific numerical distances, such as "100 meters" or "200 kilometers"; descriptions of relative distances, such as "nearby" or "far away"; distances based on reference objects, such as "at the next traffic light" or "opposite the supermarket"; estimated distances based on time, such as "5 minutes by car" or "10 minutes on foot"; and common phrases in navigation systems, such as "3 kilometers away from the destination". Through voice recognition and natural language processing technology, these different types of distance information can be parsed and converted into specific values, which are used to adjust the focus and viewing angle of the second camera to ensure that a clear image of the target object that the user is concerned about is captured.

[0122] Once the distance information is successfully obtained from the interactive question, the appropriate shooting focal length will be determined based on the distance information. The system's on-board processor will calculate the focal length value that matches the distance information and use the control module to control the focal length of the second camera. The second camera will then adjust its focus to ensure that the captured image is both clear and in line with the distance range that the user is concerned about. And obtain the line of sight direction information of the target user. The system's on-board processor will combine these two pieces of information to use the control module to adjust the second camera.

[0123] For example, if a user asks "What is the traffic sign 100 meters ahead?" the line of sight and the distance information "100 meters" can be parsed. Based on this information, the viewing angle and focal length of the second camera are adjusted to focus on the target 100 meters in the direction of the user's line of sight, so that the second camera can capture the specific target that the user is concerned about and provide a clear image.

[0124] This embodiment analyzes the user's interactive questions, determines and extracts the distance information in the questions, and if the interactive questions contain distance information, this information can be analyzed and the shooting focal length of the second camera can be adjusted accordingly to ensure that the acquired image matches the target distance that the driver is concerned about. This not only enhances the understanding and response capabilities, but also can integrate the user's line of sight direction and distance information to adjust the viewing angle and focal length of the second camera, thereby being able to more comprehensively understand the user's intentions and provide more accurate visual feedback.

[0125] Furthermore, the present application can use the image acquired by the first camera and the eye tracking method to determine the sight direction information of the target user, that is, the sight direction information of the driver or the passenger in the car. The following is a detailed description of this process:

[0126] Locate the eye area: First, identify and locate the target user's eye area in the acquired image. This step can use image processing technology to detect and track facial features, and then determine the position of the eyes, laying the foundation for further analysis of the direction of sight.

[0127] Obtain the spatial position of the eye origin and the direction of eye movement: Once the eye area is located, the eye tracking algorithm can be used to identify and track the target user's eye movements, which can include eye rotation, blinking, and other eye movements. By analyzing these eye movements, the algorithm can determine the origin of the target user's eyes, which can be the center point of the pupil or the pupil position of both eyes.

[0128] Determine the direction of sight: Using the spatial position of the eye origin and the data on the direction of eye movement, the onboard processor can calculate the direction of the target user's sight. It is usually located on a straight line between the eye and the object being looked at. By calculating the relative position between the origin position of the eye and the gaze point, the algorithm can construct a virtual ray representing the direction of the target user's sight. This ray not only indicates the general direction of the target user's sight, but also provides angle information between the sight and the horizontal and vertical directions, which is crucial for understanding the target user's intentions and attention allocation. This calculation can involve in-depth analysis of eye image data, as well as possible geometric and trigonometric function calculations to determine the precise direction of the sight.

[0129] This embodiment can accurately capture and analyze the line of sight of the target user, providing important input information for subsequent vehicle operations. The application of this line of sight tracking technology enhances the interaction between the vehicle and the driver, and improves the personalization and responsiveness of the driving experience.

[0130] Furthermore, in some embodiments of the present application, the first camera (vehicle-mounted DMS camera) can be used to obtain the sight direction information of the driver or passenger, and the viewing angle of the second camera (rotatable camera) can be adjusted based on the sight direction information. The following is a detailed description of this process:

[0131] First, the first camera is used to capture images of the target user's eye area, and these images are analyzed in the first coordinate system where the first camera is located to determine the target user's line of sight direction information. Since the first camera and the second camera are located at different positions in the vehicle, they each have independent coordinate systems. The system's on-board processor uses the known coordinate system mapping relationship to convert the line of sight direction information in the first coordinate system to the second coordinate system where the second camera is located. This step may involve spatial geometry calculations to ensure accurate transmission and adaptation of line of sight direction information.

[0132] The system's onboard processor calculates the angle that the second camera needs to rotate according to the converted target sight direction information. Based on the spatial relationship between the second camera's current position and the target sight direction, the second camera's viewing angle is ensured to be consistent with the target user's sight direction. After determining the rotation angle, the system's control module controls the rotation of the second camera according to the calculated rotation angle.

[0133] This embodiment can ensure that the viewing angle of the second camera is consistent with the sight direction of the target user, whether in the first coordinate system or the converted second coordinate system. This viewing angle adjustment capability can capture the object or scene that the user is concerned about and provide the corresponding image outside the vehicle, thereby enhancing interactivity and functionality.

[0134] Further, see Figure 4, Figure 4 This is a schematic diagram of the working principle of a human-computer interaction system provided in an embodiment of the present application, and the picture is introduced in detail below.

[0135] Figure 4 The working principle of a human-computer interaction system is shown in Figure 1. First, the first camera installed inside the vehicle, namely the on-board DMS camera, is used to obtain real-time images of the target user. The on-board processor of the system captures and analyzes the driver's eye movements through an eye tracking algorithm, thereby determining the origin position and line of sight of the driver's eyes, and can monitor the driver's attention focus in real time.

[0136] The system's onboard processor can then send instructions to the control module based on the driver's line of sight, and the control module can adjust the second camera installed on the outside of the vehicle, which can rotate the camera to capture real-time images of the environment outside the vehicle. These images are then transmitted to the onboard processor for further analysis.

[0137] When the target user asks an interactive question, such as "What is the building 100 meters ahead?", the question and the image outside the vehicle are input into a pre-trained multimodal large model. This large model can comprehensively consider multiple data types such as text and images, analyze and understand the content of the question through deep learning algorithms, and refer to the image outside the vehicle to generate an accurate response result.

[0138] Finally, the response results output by the multimodal large model, such as "The building 100 meters ahead is the Guangzhou Tower", can be fed back to the driver through the on-board display system or voice interaction interface.

[0139] This embodiment can not only monitor the driver's state and the external environment in real time, but also enhance the driving experience because it can provide timely and accurate feedback to help the driver better understand. Through this precise line of sight tracking and analysis, it is possible to accurately understand the driver's intentions, thereby improving the interactive quality of the driving assistance system.

[0140] Further, see Figure 5 , Figure 5 This is a schematic diagram of the working principle of another human-computer interaction system provided in an embodiment of the present application, and the picture is introduced in detail below.

[0141] Figure 5The diagram shows the workflow of the system when the target person is the driver. The first camera, the on-board DMS camera, captures the driver's image, and the on-board processor applies the eye tracking algorithm to determine the spatial position of the origin of the driver's eyes and the direction of sight. This information is used to guide the second camera, which can be a rotating camera, to adjust its viewing angle to capture real-time images of the environment outside the vehicle. Then, the system's on-board processor can send instructions to the control module based on the driver's line of sight direction information, and use the control module to adjust the second camera installed on the outside of the vehicle, which can be a rotating camera, to capture real-time images of the environment outside the vehicle. These images are then transmitted to the on-board processor for further analysis.

[0142] When the driver asks an interactive question, such as "What is the building 100 meters ahead?", this question, the image outside the vehicle, and the task perception result are input into a pre-trained multimodal large model. This model can comprehensively process data such as text and images, analyze the content of the question, and generate accurate response results based on the image information outside the vehicle. The task perception result is determined based on the driver's status information, the external environment information, and the vehicle driving status information. This information may include key data such as the driver's fatigue status and the current speed of the vehicle.

[0143] Finally, if there is a safety issue in the task perception result, the response result output by the multimodal large model will stop responding to the user's questions and instead send a prompt message to the user, such as "Please note that the system has detected pedestrians ahead. Please concentrate on driving and we will communicate with you later." This can be fed back to the driver through the on-board display system or voice interaction interface.

[0144] This embodiment not only improves driving safety and can monitor the driver's status and the external environment in real time, but also enhances the driving experience and helps the driver better understand and deal with complex traffic conditions. Through this precise line of sight tracking and analysis of task perception results, an accurate understanding of the driver's intention is achieved, thereby improving the interactive quality and safety of the driving assistance system.

[0145] The human-computer interaction device provided in the embodiment of the present application is described below. The human-computer interaction device described below and the human-computer interaction method described above can be referenced to each other.

[0146] See also Figure 6 , Figure 6 A schematic diagram of the structure of a human-computer interaction device provided in an embodiment of the present application.

[0147] like Figure 6 As shown, the device may include:

[0148] A question acquisition unit 11 is used to acquire interactive questions raised by a target user in the vehicle;

[0149] An image acquisition unit 12 is used to acquire an image of the target user taken by a first camera in the vehicle, and determine the sight direction information of the target user according to the image;

[0150] A viewing angle determining unit 13, configured to adjust the viewing angle of a second camera mounted on the vehicle according to the sight line direction information, so that the viewing angle of the second camera is consistent with the sight line direction of the target user, and obtain an image outside the vehicle taken by the second camera;

[0151] The reply output unit 14 is used to call the multimodal large model to instruct the multimodal large model to generate a reply result corresponding to the interactive question in combination with the external vehicle image, and output the reply result.

[0152] In a possible implementation, the response output unit 14 calls the multimodal large model to instruct the multimodal large model to generate a response result corresponding to the interactive question in combination with the vehicle exterior image, and outputs the response result, including:

[0153] Obtain a first prompt instruction prompt format template, wherein the first prompt instruction prompt format template includes a first task instruction, an interactive question slot, and an external vehicle image slot, wherein the first task instruction is used to instruct the multimodal large model to combine the external vehicle image in the external vehicle image slot to generate the reply result corresponding to the interactive question in the interactive question slot;

[0154] Fill the interactive question into the interactive question slot, fill the outside-vehicle image into the outside-vehicle image slot, obtain a first prompt instruction prompt, input the first prompt instruction prompt into the pre-trained multimodal large model, and obtain the reply result output by the multimodal large model.

[0155] In one possible implementation, when the target user is a driver, the human-computer interaction device of the embodiment of the present application further includes:

[0156] The task perception result acquisition unit is used to acquire the task perception result of at least one configured active safety detection task before the response output unit 14 processes it, wherein the task perception result of the active safety detection task is determined based on the driver's state information, the vehicle's external environment information and / or the vehicle's driving state information. On this basis, the response output unit 14 calls the multimodal large model to instruct the multimodal large model to generate a response result corresponding to the interactive question in combination with the vehicle's external image, including:

[0157] The multimodal large model is called to instruct the multimodal large model to comprehensively consider the task perception result and the external vehicle image to generate the response result corresponding to the interactive question.

[0158] In a possible implementation manner, the response output unit 14 calls the multimodal large model to instruct the multimodal large model to comprehensively consider the task perception result and the external vehicle image to generate the response result corresponding to the interactive question, including:

[0159] Obtain a second prompt instruction prompt format template, wherein the second prompt instruction prompt format template includes a second task instruction, an interactive question slot, a task perception result slot, and an external vehicle image slot, wherein the second task instruction is used to instruct the multimodal large model to comprehensively consider the external vehicle image in the external vehicle image slot and the task perception result in the task perception result slot, and generate the reply result corresponding to the interactive question in the interactive question slot;

[0160] Fill the interactive question into the interactive question slot, fill the outside-vehicle image into the outside-vehicle image slot, fill the task perception result into the task perception result slot, obtain a second prompt instruction prompt, input the second prompt instruction prompt into the pre-trained multimodal large model, and obtain the reply result output by the multimodal large model.

[0161] In a possible implementation manner, a human-computer interaction device in an embodiment of the present application further includes:

[0162] The focal length determination unit is used to analyze the interaction problem before the viewing angle determination unit 13 processes it. If distance information can be extracted from the interaction problem, the shooting focal length is determined according to the distance information, and the focus of the second camera is adjusted to the shooting focal length.

[0163] In a possible implementation manner, the process in which the image acquisition unit 12 determines the sight line direction information of the target user according to the image includes:

[0164] Locating an eye region of the target user in the image;

[0165] Acquire the spatial position of the origin of the human eye and the direction of eye movement in the eye area;

[0166] The sight line direction information of the target person is determined according to the spatial position of the human eye origin and the eye movement direction.

[0167] In a possible implementation manner, the sight line direction information is obtained based on the first coordinate system where the first camera is located; and the viewing angle determination unit 13 adjusts the viewing angle of the vehicle-mounted second camera according to the sight line direction information, including:

[0168] Using the coordinate system mapping relationship between the first camera and the second camera, the sight line direction information is converted into a second coordinate system where the second camera is located to obtain target sight line direction information;

[0169] The rotation angle of the second camera is calculated according to the target sight direction information, and the rotation of the second camera is controlled according to the rotation angle.

[0170] The present application also provides an electronic device in an embodiment. Figure 7 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to a fixed terminal such as a vehicle central processor and the like. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0171] like Figure 7 As shown, the electronic device may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603 to implement the human-computer interaction method of the aforementioned embodiment of the present application. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0172] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0173] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the human-computer interaction method of the aforementioned embodiment of the present application.

[0174] The present application also provides a computer program product, which includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the present application is generated in whole or in part.

[0175] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0176] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0177] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0178] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

Claims

1. A human-computer interaction method, characterized in that: include: Obtain interactive questions raised by target users in the car; Acquire an image of the target user captured by a first camera in the vehicle, and determine the sight direction information of the target user according to the image; According to the line of sight direction information, adjust the viewing angle of the second camera on the vehicle so that the viewing angle of the second camera is consistent with the line of sight direction of the target user, and obtain the image outside the vehicle taken by the second camera; The multimodal large model is called to instruct the multimodal large model to generate a response result corresponding to the interactive question in combination with the external vehicle image, and output the response result.

2. The method according to claim 1, characterized in that The calling of the multimodal large model to instruct the multimodal large model to generate a response result corresponding to the interactive question in combination with the vehicle exterior image and output the response result includes: Obtain a first prompt instruction prompt format template, wherein the first prompt instruction prompt format template includes a first task instruction, an interactive question slot, and an external vehicle image slot, wherein the first task instruction is used to instruct the multimodal large model to combine the external vehicle image in the external vehicle image slot to generate the reply result corresponding to the interactive question in the interactive question slot; Fill the interactive question into the interactive question slot, fill the outside-vehicle image into the outside-vehicle image slot, obtain a first prompt instruction prompt, input the first prompt instruction prompt into the pre-trained multimodal large model, and obtain the reply result output by the multimodal large model.

3. The method according to claim 1, characterized in that: If the target user is a driver, before calling the multimodal large model, the following steps are also included: Acquire a task perception result of at least one configured active safety detection task, where the task perception result of the active safety detection task is determined based on the driver's state information, vehicle external environment information and / or vehicle driving state information; The multimodal large model is called to instruct the multimodal large model to generate a response result corresponding to the interactive question in combination with the vehicle exterior image, including: The multimodal large model is called to instruct the multimodal large model to comprehensively consider the task perception result and the external vehicle image to generate the response result corresponding to the interactive question.

4. The method according to claim 3, characterized in that The calling of the multimodal large model to instruct the multimodal large model to comprehensively consider the task perception result and the external vehicle image to generate the response result corresponding to the interactive question includes: Obtain a second prompt instruction prompt format template, wherein the second prompt instruction prompt format template includes a second task instruction, an interactive question slot, a task perception result slot, and an external vehicle image slot, wherein the second task instruction is used to instruct the multimodal large model to comprehensively consider the external vehicle image in the external vehicle image slot and the task perception result in the task perception result slot, and generate the reply result corresponding to the interactive question in the interactive question slot; Fill the interactive question into the interactive question slot, fill the outside-vehicle image into the outside-vehicle image slot, fill the task perception result into the task perception result slot, obtain a second prompt instruction prompt, input the second prompt instruction prompt into the pre-trained multimodal large model, and obtain the reply result output by the multimodal large model.

5. The method according to claim 1, characterized in that Before acquiring the image outside the vehicle captured by the second camera, the method further includes: The interaction question is parsed, and if distance information can be extracted from the interaction question, a shooting focal length is determined according to the distance information, and the focus of the second camera is adjusted to the shooting focal length.

6. The method according to claim 1, characterized in that The process of determining the sight direction information of the target user according to the image includes: Locating an eye region of the target user in the image; Acquire the spatial position of the origin of the human eye and the direction of eye movement in the eye area; The sight line direction information of the target person is determined according to the spatial position of the human eye origin and the eye movement direction.

7. The method according to any one of claims 1 to 6, characterized in that: The sight direction information is obtained based on the first coordinate system where the first camera is located; The process of adjusting the viewing angle of the second vehicle-mounted camera according to the sight direction information includes: Using the coordinate system mapping relationship between the first camera and the second camera, the sight line direction information is converted into a second coordinate system where the second camera is located to obtain target sight line direction information; The rotation angle of the second camera is calculated according to the target sight direction information, and the rotation of the second camera is controlled according to the rotation angle.

8. A human-computer interaction device, characterized in that: include: A question acquisition unit, used to acquire interactive questions raised by a target user in the car; an image acquisition unit, configured to acquire an image of the target user captured by a first camera in the vehicle, and determine the sight direction information of the target user according to the image; a viewing angle determining unit, configured to adjust the viewing angle of a second camera mounted on the vehicle according to the sight line direction information, so that the viewing angle of the second camera is consistent with the sight line direction of the target user, and obtain an image outside the vehicle taken by the second camera; A reply output unit is used to call the multimodal large model to instruct the multimodal large model to generate a reply result corresponding to the interactive question in combination with the external vehicle image, and output the reply result.

9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the human-computer interaction method according to any one of claims 1 to 7.

10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the human-computer interaction method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, each step in the human-computer interaction method as described in any one of claims 1 to 7 is implemented.

Images