Vehicle-mounted interaction method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to a vehicle-mounted interaction method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of vehicle-mounted interaction.The method in one embodiment comprises the steps that a triggering instruction for triggering a vehicle-mounted interaction auxiliary mode is received, and the vehicle-mounted interaction auxiliary mode is entered in response to the triggering instruction; in the vehicle-mounted interaction auxiliary mode, a first interaction operation instruction of a driver is obtained, and first interaction content information of the first interaction operation instruction is obtained based on a preset first corresponding relation between the interaction operation instruction and the interaction content information; and outputting the first interaction content information. By adopting the scheme of the embodiment, the communication requirements of drivers with language barriers such as deaf-mute people can be met, and the vehicle-mounted interaction performance is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle interaction, and in particular to a vehicle interaction method, a vehicle interaction device, an electronic device, a computer readable storage medium and a computer program product. BACKGROUND

[0002] During the driving of a vehicle by a driver, there is a need for a passenger in a passenger seat of the vehicle to exchange information with the driver. In general, the passenger and the driver can directly exchange information through voice interaction such as speaking. However, for a person with a language barrier, it is difficult for him to clearly express through voice. For example, a deaf-mute person usually has impaired language ability due to hearing impairment. The deaf-mute person has difficulty in hearing or clearly hearing the voice of others and expressing content through voice. Therefore, the deaf-mute person and other persons with language barriers usually express content through sign language.

[0003] During actual driving, the driver needs to actively initiate communication with the passenger. However, in the case of a driver with a language barrier, the driver cannot express interactive content through sign language during driving to ensure driving safety. Therefore, the driver with a language barrier will have difficulty in information interaction with the passenger during driving, and the communication needs of the driver with a language barrier cannot be met. SUMMARY

[0004] Therefore, it is necessary to provide a vehicle interaction method, a vehicle interaction device, an electronic device, a computer readable storage medium and a computer program product capable of meeting the communication needs of a driver with a language barrier and improving vehicle interaction performance.

[0005] In a first aspect, the present application provides a vehicle interaction method. The method comprises:

[0006] receiving a trigger instruction for triggering a vehicle interaction assistance mode, and entering the vehicle interaction assistance mode in response to the trigger instruction;

[0007] In the vehicle interaction assistance mode, a first interaction operation instruction of the driver is obtained, and first interaction content information of the first interaction operation instruction is obtained based on a first correspondence relationship between a preset interaction operation instruction and interaction content information;

[0008] outputting the first interaction content information.

[0009] In some embodiments, receiving the trigger instruction for triggering the vehicle interaction assistance mode comprises:

[0010] In a case where an operation on the first preset button of the vehicle is received, it is determined that the trigger instruction triggering the in-vehicle interaction assistance mode is received.

[0011] In some embodiments, the first preset button comprises a voice trigger button.

[0012] In a case where an operation on the first preset button of the vehicle is received, it is determined that the trigger instruction triggering the in-vehicle interaction assistance mode is received, comprising:

[0013] In a case where an operation on the voice trigger button of the vehicle is received, it is determined that the trigger instruction triggering the in-vehicle interaction assistance mode is received.

[0014] In some embodiments, receiving the trigger instruction triggering the in-vehicle interaction assistance mode comprises:

[0015] Obtaining a first image obtained by photographing a driver's seat;

[0016] In a case where a first body language action is identified from the first image, it is determined that the trigger instruction triggering the in-vehicle interaction assistance mode is received.

[0017] In some embodiments, in response to the trigger instruction, entering the in-vehicle interaction assistance mode comprises: in response to the trigger instruction, entering the in-vehicle interaction assistance mode, and displaying a plurality of first preset dialogue information through a vehicle display device.

[0018] Obtaining a first interaction operation instruction of the driver, and obtaining first interaction content information of the first interaction operation instruction based on a first correspondence relationship between preset interaction operation instructions and interaction content information, comprising:

[0019] Receiving and responding to a selection operation on the displayed first preset dialogue information, determining the first preset dialogue information selected by the selection operation as the first interaction content information, wherein the first interaction operation instruction comprises a selection instruction corresponding to the selection operation on the displayed first preset dialogue information.

[0020] In some embodiments, displaying a plurality of first preset dialogue information through a vehicle display device comprises:

[0021] Obtaining first vehicle state information;

[0022] From a plurality of first preset dialogue information set in advance, one or more first preset dialogue information conforming to the first vehicle state information is filtered out;

[0023] The filtered first preset dialogue information is displayed through the vehicle display device.

[0024] In some embodiments, obtaining the first interaction operation instruction of the driver comprises:

[0025] obtaining a gaze direction of the driver;

[0026] In a case where the driver's gaze direction is determined based on the driver's gaze direction, it is determined that the first interaction operation instruction of the driver is received.

[0027] In some embodiments, in a case where the driver's gaze direction is determined based on the driver's gaze direction, it is determined that the first interaction operation instruction of the driver is received, including:

[0028] In a case where the driver's gaze direction is determined based on the driver's gaze direction, and the second interaction operation instruction of the driver is received, it is determined that the first interaction operation instruction of the driver is received.

[0029] In some embodiments, the method further includes any one of the following two:

[0030] The first one:

[0031] In a case where the operation of the second preset button of the vehicle is received, it is determined that the second interaction operation instruction of the driver is received;

[0032] The second one:

[0033] Obtaining a second image obtained by photographing the driver's seat;

[0034] In a case where the second body language action is recognized from the second image, it is determined that the second interaction operation instruction of the driver is received.

[0035] In some embodiments, based on the first correspondence relationship between the preset interaction operation instruction and the interaction content information, the first interaction content information of the first interaction operation instruction is obtained, including:

[0036] Based on the type of the rearview mirror of the driver's gaze, the second preset dialogue information corresponding to the type of the rearview mirror is obtained, wherein the first correspondence relationship between the preset interaction operation instruction and the interaction content information includes the correspondence relationship between the type of the rearview mirror and the second preset dialogue information.

[0037] The obtained second preset dialogue information is determined as the first interaction content information.

[0038] In some embodiments, based on the first correspondence relationship between the preset interaction operation instruction and the interaction content information, the first interaction content information of the first interaction operation instruction is obtained, including at least one of the following two:

[0039] The first one:

[0040] obtaining second preset dialogue information corresponding to the second interaction operation instruction, the first corresponding relationship between the preset interaction operation instruction and the interaction content information including the corresponding relationship between the second interaction operation instruction and the second preset dialogue information;

[0041] determining the obtained second preset dialogue information as the first interaction content information;

[0042] The second item is:

[0043] obtaining second preset dialogue information corresponding to the second interaction operation instruction and the type of the rearview mirror based on the driver's gaze, the first corresponding relationship between the preset interaction operation instruction and the interaction content information including the corresponding relationship between the type of the rearview mirror and the second preset dialogue information and the second interaction operation instruction;

[0044] determining the obtained second preset dialogue information as the first interaction content information.

[0045] In some embodiments, obtaining the first interaction content information of the first interaction operation instruction based on the corresponding relationship between the preset interaction operation instruction and the interaction content information includes:

[0046] obtaining second vehicle state information;

[0047] obtaining second preset dialogue information corresponding to the second interaction operation instruction and the type of the rearview mirror based on the driver's gaze, the first corresponding relationship between the preset interaction operation instruction and the interaction content information including the corresponding relationship between the type of the rearview mirror and the second preset dialogue information and the second interaction operation instruction;

[0048] determining the obtained second preset dialogue information as the first interaction content information.

[0049] In some embodiments, the second vehicle state information includes at least one of a vehicle speed and a vehicle gear.

[0050] In some embodiments, outputting the first interaction content information includes at least one of:

[0051] The first item is:

[0052] voice broadcasting the first interaction content information;

[0053] The second item is:

[0054] displaying the first interaction content information on a display screen corresponding to the target passenger seat.

[0055] In some embodiments, displaying the first interaction content information on a display screen corresponding to the target passenger seat includes at least one of:

[0056] The first item:

[0057] The first interactive content information is displayed in text on the display screen corresponding to the target passenger seat.

[0058] The second item:

[0059] The sign language video of the first interactive content information is obtained and displayed on the display screen corresponding to the target passenger seat.

[0060] In some embodiments, the output of the first interactive content information further includes at least one of the following:

[0061] The first item:

[0062] Based on the first interactive content information, the target passenger seat is controlled to vibrate;

[0063] The second item:

[0064] Based on the first interactive content information, the ambient light of the target passenger seat is controlled to flash.

[0065] In some embodiments, the method further includes:

[0066] Obtaining second interactive content information initiated by the passenger;

[0067] Displaying the second interactive content information through the vehicle-mounted display device.

[0068] In some embodiments, displaying the second interactive content information through the vehicle-mounted display device includes:

[0069] Obtaining third vehicle state information;

[0070] If it is determined based on the third vehicle state information that the vehicle is in a safe driving state, the second interactive content information is displayed through the vehicle-mounted display device.

[0071] In some embodiments, displaying the second interactive content information through the vehicle-mounted display device includes:

[0072] Determining the emergency type of the second interactive content information;

[0073] If the emergency type of the second interactive content information is emergency, the second interactive content information is displayed in real time through the vehicle-mounted display device;

[0074] If the emergency type of the second interactive content information is non-emergency, if it is determined based on the third vehicle state information that the vehicle is in a safe driving state, the second interactive content information is displayed through the vehicle-mounted display device.

[0075] In some embodiments, the method further comprises at least one of:

[0076] The first item:

[0077] The third vehicle state information comprises a vehicle gear, and in a case where the vehicle gear is a first preset gear, the safe driving state is determined, and the first preset gear comprises a P gear or an N gear.

[0078] The second item:

[0079] The third vehicle state information comprises a vehicle speed, and in a case where a duration in which the vehicle speed is less than or equal to a preset vehicle speed is greater than or equal to a first preset duration, the safe driving state is determined.

[0080] In some embodiments, the method further comprises:

[0081] Obtaining a third interaction operation instruction of the driver, and obtaining third interaction content information of the third interaction operation instruction based on a second correspondence relationship between preset interaction operation instructions and interaction content information;

[0082] Outputting the third interaction content information.

[0083] In some embodiments, the method further comprises:

[0084] Obtaining recommended dialogue information generated by performing intent understanding on the second interaction content information, and displaying the recommended dialogue information through the vehicle-mounted display device;

[0085] Obtaining a third interaction operation instruction of the driver, comprising: obtaining a selection operation on the recommended dialogue information, the third interaction operation instruction comprising a selection instruction corresponding to the selection operation on the displayed recommended dialogue information, and the third interaction content information comprising the recommended dialogue information selected by the selection operation on the displayed recommended dialogue information.

[0086] In some embodiments, obtaining a third interaction operation instruction of the driver comprises:

[0087] Obtaining a third image obtained by photographing the driver's seat;

[0088] In a case where a third body language action is recognized from the third image, it is determined that the third interaction operation instruction of the driver is received.

[0089] In some embodiments, the method further comprises:

[0090] In a case where no interaction operation instruction is received within a second preset duration, the vehicle-mounted interaction assistance mode is exited.

[0091] In some embodiments, the method can further comprise:

[0092] displaying the second interaction content information through the vehicle display device, and entering a vehicle interaction assistance mode.

[0093] In some embodiments, determining the emergency type of the second interaction content information comprises:

[0094] determining an information category of the second interaction content information;

[0095] if the information category of the second interaction content information belongs to the emergency category set, determining that the emergency type of the second interaction content information is emergency; otherwise, determining that the emergency type of the second interaction content information is non-emergency.

[0096] In some embodiments, the method further comprises:

[0097] if the information category of the second interaction content information does not belong to the emergency category set and does not belong to the non-emergency category set, displaying the second interaction content information through the vehicle display device in real time, or, in the case of determining that the vehicle is in a safe driving state based on the third vehicle state information, displaying the second interaction content information through the vehicle display device again.

[0098] In some embodiments, determining the emergency type of the second interaction content information comprises:

[0099] processing the second interaction content information using the information emergency type classification model obtained through training to obtain the emergency type of the second interaction content information.

[0100] In some embodiments, the second vehicle state information includes vehicle speed, and based on the second vehicle state information and the type of the rearview mirror at which the driver's gaze is fixed, the corresponding second preset dialogue information is obtained, including at least one of the following:

[0101] The first item:

[0102] In the case where the vehicle speed is 0 and the rearview mirror at which the driver's gaze is fixed is the inside rearview mirror, the second preset dialogue information obtained includes prompt information for temporary parking;

[0103] The second item:

[0104] In the case where the vehicle speed is 0 and the rearview mirror at which the driver's gaze is fixed is the right rearview mirror, the second preset dialogue information obtained includes prompt information for observing the door opening after a car arrives from the rear;

[0105] The third item:

[0106] In the case where the vehicle speed is 0 and the rearview mirror at which the driver's gaze is fixed is the left rearview mirror, the second preset dialogue information obtained includes prompt information for getting off the vehicle from the right side door;

[0107] The fourth item is:

[0108] In a case where the vehicle speed is less than the preset vehicle speed threshold, and the rearview mirror at which the driver's gaze is directed is the interior rearview mirror, the obtained second preset dialogue information includes prompt information that the vehicle is not parked stably.

[0109] In some embodiments, the second vehicle state information includes a gear of the vehicle, and the corresponding second preset dialogue information is obtained based on the second vehicle state information and the type of the rearview mirror at which the driver's gaze is directed, and includes at least one of the following:

[0110] The first item is:

[0111] In a case where the gear of the vehicle is P, and the rearview mirror at which the driver's gaze is directed is the right rearview mirror, the obtained second preset dialogue information includes prompt information that there is a vehicle coming from behind and that the driver should pay attention to observation when opening the door.

[0112] The second item is:

[0113] In a case where the gear of the vehicle is P, and the rearview mirror at which the driver's gaze is directed is the left rearview mirror, the obtained second preset dialogue information includes prompt information that the driver should get off from the right door.

[0114] In a case where the gear of the vehicle is P, and the rearview mirror at which the driver's gaze is directed is the left rearview mirror, the obtained second preset dialogue information includes prompt information that the driver should get off from the right door.

[0115] The auxiliary triggering module is configured to receive a triggering instruction for triggering the vehicle-mounted interaction auxiliary mode, and enter the vehicle-mounted interaction auxiliary mode in response to the triggering instruction.

[0116] The information obtaining module is configured to, in the vehicle-mounted interaction auxiliary mode, obtain a first interaction operation instruction of the driver, and obtain first interaction content information of the first interaction operation instruction based on a preset first correspondence relationship between an interaction operation instruction and interaction content information.

[0117] The information output module is configured to output the first interaction content information.

[0118] The third aspect of the present application further provides an electronic device. The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the vehicle-mounted interaction method in any of the above embodiments when executing the computer program.

[0119] The fourth aspect of the present application further provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the vehicle-mounted interaction method in any of the above embodiments.

[0120] In a fifth aspect, the present application also provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the steps of the vehicle interaction method in any of the above embodiments.

[0121] The vehicle interaction method, the vehicle interaction device, the electronic device, the storage medium and the computer program product described above provide the vehicle interaction assistance mode for a vehicle, and in the case that a trigger instruction triggering the vehicle interaction assistance mode is received, the vehicle interaction assistance mode is entered, and in the vehicle interaction assistance mode, a first interaction operation instruction of a driver is acquired, and first interaction content information of the first interaction operation instruction is acquired and output based on a preset first correspondence relationship between the interaction operation instruction and the interaction content information. By providing the vehicle interaction assistance mode and receiving the first interaction operation instruction of the driver in the vehicle interaction assistance mode, the interaction operation of the driver can be obtained while reducing the misoperation of the driver in the normal driving process, and the interaction content information is acquired based on the interaction operation of the driver, and the first interaction content information is output, so that other members in the vehicle can know the content to be expressed by the driver. Since the first correspondence relationship between the interaction operation instruction and the interaction content information is preset, the driver can output relatively complex first interaction content information by performing relatively simple first interaction operation, which can assist the driver with language barriers to communicate with passengers, meet the communication needs of the driver with language barriers, and improve the vehicle interaction performance. BRIEF DESCRIPTION OF DRAWINGS

[0122] Figure 1 A flowchart of a vehicle interaction method in an embodiment is shown;

[0123] Figure 2 A flowchart of acquiring a first interaction operation instruction in an example is shown;

[0124] Figure 3 A flowchart of acquiring a first interaction operation instruction in another example is shown;

[0125] Figure 4 A flowchart of a vehicle interaction method in another embodiment is shown;

[0126] Figure 5 A flowchart of displaying second interaction content information in an embodiment is shown;

[0127] Figure 6 A flowchart of a vehicle interaction method in another embodiment is shown;

[0128] Figure 7 A flowchart of a vehicle interaction method in a specific example is shown;

[0129] Figure 8For another specific example, a flowchart of a process of receiving and displaying second interaction content information. DETAILED DESCRIPTION

[0130] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the protection scope of the present application.

[0131] The embodiments of the technical solutions of the present application will be described in detail below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and therefore only serve as examples, and cannot be used to limit the protection scope of the present application.

[0132] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of this application; the terms "include" and "comprise" and variations thereof, as used in the specification and in the claims and the aforementioned summary, are intended to cover a non-exclusive inclusion.

[0133] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly and specifically limited.

[0134] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearance of the phrase in various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily independent or alternative embodiments to each other. Those skilled in the art explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0135] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents a "or" relationship between the front and rear associated objects.

[0136] In the description of the embodiments of the present application, the term "a plurality of" refers to two or more (including two), and similarly, "a plurality of groups" refers to two or more groups (including two groups), and "a plurality of pieces" refers to two or more pieces (including two pieces).

[0137] It should be noted that the information (including but not limited to interactive operation instructions, interactive content information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties, and the acquisition, transmission, storage, use and processing of the relevant data comply with the relevant provisions of national laws and regulations. In the embodiments of the present application, some industry existing solutions such as software, components, models, etc. may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility in the implementation of the technical solutions of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0138] Currently, taking the hearing-impaired as an example, with the development of technology, it is possible for the hearing-impaired to drive a vehicle, for example, to drive the vehicle as a driver while wearing a hearing aid device. The hearing-impaired wearing a hearing aid device can hear external sounds with the assistance of the hearing aid device when driving the vehicle. However, in the driving process, although the external sounds can be heard, in order to ensure driving safety, both hands need to control the steering wheel, and it is not possible to type or express interactive content through sign language during the driving process, making it difficult to interact with passengers and difficult to meet the in-vehicle interaction needs of drivers with language barriers such as the hearing-impaired.

[0139] It is found through research that, in the process of driving a vehicle by a driver, under the condition of ensuring driving safety, the driver can perform some simple interactive operations, such as clicking or touching the display screen of a vehicle display device, operating vehicle buttons, etc., and can also perform some simple body language actions, such as head actions (such as nodding, shaking the head), hand gestures (such as spreading fingers, making a fist, etc.), facial actions (such as blinking, etc.). Therefore, a vehicle can be provided with an in-vehicle interaction assistance system or an in-vehicle interaction assistance mode, which, in combination with the interactive operation of the driver, can assist in determining the corresponding interactive content information and outputting the interactive content information, so that the driver can output relatively complex interactive content information by performing relatively simple interactive operations, and other members in the vehicle can know the content expressed by the driver, thereby assisting the driver with language barriers to perform in-vehicle interaction, meeting the communication needs of drivers with language barriers such as the hearing-impaired, and improving the in-vehicle interaction performance.

[0140] Accordingly, the embodiments of the present application provide an in-vehicle interaction method, which can be applied to a vehicle. The in-vehicle interaction method can be executed by a controller of the vehicle. The controller can be a dedicated controller of the vehicle, or can be a vehicle control unit (VCU) of the vehicle, or can be executed by the vehicle control unit in cooperation with other controllers in the vehicle, and the embodiments of the present application do not make specific limitations in this regard.

[0141] In one embodiment, as shown in Figure 1 A vehicle-mounted interaction method is provided, comprising the following steps:

[0142] Step S101: receiving a trigger instruction for triggering a vehicle-mounted interaction assistance mode, and entering the vehicle-mounted interaction assistance mode in response to the trigger instruction.

[0143] The vehicle-mounted interaction assistance mode refers to a mode for assisting vehicle-mounted interaction, in particular, a mode for assisting drivers and passengers with language barriers such as the deaf and mute to communicate. The vehicle-mounted interaction assistance mode can be a working mode of the vehicle, or a vehicle-mounted interaction assistance function set for the vehicle, and the specific form is not limited. The name of the mode or system provided by the vehicle-mounted interaction assistance mode in the vehicle is not limited, such as an intelligent cockpit multi-modal social assistance mode, but is not limited thereto.

[0144] The trigger instruction for triggering the vehicle-mounted interaction assistance mode is an instruction that can trigger the entry into the vehicle-mounted interaction assistance mode or start the vehicle-mounted interaction assistance mode. After receiving the trigger instruction for triggering the vehicle-mounted interaction assistance mode, the vehicle-mounted interaction assistance mode is entered.

[0145] Step S102: In the vehicle-mounted interaction assistance mode, obtaining a first interaction operation instruction of the driver, and obtaining first interaction content information of the first interaction operation instruction based on a first correspondence relationship between the preset interaction operation instruction and the interaction content information.

[0146] In the vehicle-mounted interaction assistance mode, the driver can perform an interaction operation, and the instruction obtained by the driver performing the interaction operation after entering the vehicle-mounted interaction assistance mode is referred to as the first interaction operation instruction in the embodiments of the present application. It can be understood that the interaction operation performed by the driver is an interaction operation performed under the condition that the driving safety can be met, such as the operation of clicking the options on the button or the touch screen, the simple body language action, etc., but not limited thereto.

[0147] The first correspondence relationship between the interaction operation instruction and the interaction content information is pre-set in the memory of the vehicle, which is referred to as the first correspondence relationship in the embodiments of the present application. After obtaining the first interaction operation instruction, the interaction operation instruction consistent with the first interaction operation instruction can be queried in each interaction operation instruction of the first correspondence relationship, and the interaction content information having the correspondence relationship with the queried interaction operation instruction is taken as the first interaction content information of the first interaction operation instruction.

[0148] Step S103: outputting the first interaction content information.

[0149] After obtaining the first interaction content information, the first interaction content information can be outputted to output the first interaction content information of the driver for the passengers in the vehicle to know the interaction content expressed by the driver.

[0150] Based on the vehicle interaction method of the embodiment, by providing the vehicle interaction auxiliary mode, and entering the vehicle interaction auxiliary mode when the trigger instruction triggering the vehicle interaction auxiliary mode is received, obtaining the first interaction operation instruction of the driver in the vehicle interaction auxiliary mode, and obtaining and outputting the first interaction content information of the first interaction operation instruction based on the first correspondence relationship between the preset interaction operation instruction and the interaction content information, the interaction operation of the driver can be obtained while reducing the misoperation of the driver in the normal driving process, and the interaction content information can be obtained based on the interaction operation of the driver, and the first interaction content information is outputted to enable other members in the vehicle to know the content expressed by the driver. Since the first correspondence relationship between the interaction operation instruction and the interaction content information is preset, the driver can output relatively complex first interaction content information by performing relatively simple first interaction operation, which can assist the driver and passengers with language barriers such as the deaf and mute to communicate, meet the communication needs of the driver with language barriers such as the deaf and mute, and improve the vehicle interaction performance.

[0151] The way of receiving the trigger instruction triggering the vehicle interaction auxiliary mode is not limited, and some of the ways are exemplified below.

[0152] In some embodiments, receiving the trigger instruction triggering the vehicle interaction auxiliary mode includes:

[0153] In a case where the operation of the first preset button of the vehicle is received, it is determined that the trigger instruction triggering the vehicle interaction auxiliary mode is received.

[0154] The first preset button is a button for receiving the trigger instruction triggering the vehicle interaction auxiliary mode, which can be a function button reserved by the vehicle, such as a function button reserved in the steering wheel control system, or a custom button reserved by the vehicle. After the custom button is set as the first preset button, in some examples, the first preset button can also be a button of the vehicle, such as a button in the steering wheel control system, which is redefined as the first preset button.

[0155] Based on the scheme of the embodiment, the trigger instruction triggering the vehicle interaction auxiliary mode can be received through the first preset button of the vehicle, and the vehicle interaction auxiliary mode is triggered in a simple and convenient way.

[0156] In some specific examples, the first preset key comprises a voice trigger key.

[0157] At this time, in the case where the operation of the first preset key of the vehicle is received, it is determined that the trigger instruction triggering the vehicle interactive assistance mode is received, comprising:

[0158] In the case where the operation of the voice trigger key of the vehicle is received, it is determined that the trigger instruction triggering the vehicle interactive assistance mode is received.

[0159] The voice trigger key is a physical key or a touch key originally used to activate a voice control system in the vehicle. In the embodiments of the present application, the voice trigger key of the vehicle is reconfigured, and the function triggered by the voice trigger key is changed from activating the voice control system to activating and starting the vehicle interactive assistance mode or the vehicle interactive assistance function.

[0160] Based on the scheme of the embodiments, the voice trigger key of the vehicle is used as the first preset key of the trigger instruction triggering the vehicle interactive assistance mode. Since the driver is in a state of inconvenience to use voice input when using the vehicle interactive assistance mode, for example, the driver is a deaf-mute or other language-impaired person, the voice trigger key has low applicability to them. By using the voice trigger key of the vehicle as the first preset key, the voice trigger key can be reused, and a new key does not need to be reserved for the vehicle interactive assistance mode. By making relevant settings to the voice trigger key, resources can be fully utilized.

[0161] In other embodiments, the trigger instruction triggering the vehicle interactive assistance mode comprises:

[0162] Obtaining a first image obtained by photographing the driver seat;

[0163] In the case where the first body language action is identified from the first image, it is determined that the trigger instruction triggering the vehicle interactive assistance mode is received.

[0164] The manner of obtaining the first image obtained by photographing the driver seat is not limited. For example, a camera device such as a camera can be arranged inside the vehicle to have a camera range including a region of the driver seat (a region of the main driver seat of the vehicle) and a certain expansion space to be able to photograph the body language action of the driver. The camera is used to photograph to obtain an image of the region of the driver seat (the region where the main driver seat of the vehicle is located), thereby obtaining the first image.

[0165] The body language action refers to an action expressed by each part of the body that can convey information or instructions, which can include head actions (such as nodding and shaking), hand gestures (such as spreading fingers, making a fist, and raising the thumb), and facial expressions (such as winking and opening the mouth).

[0166] After obtaining the first image, image analysis can be performed on one or more frames of the first image to identify whether the first body language action exists in the first image. In a case where the first body language action is identified, it can be determined that the trigger instruction triggering the vehicle interactive assistance mode is received. The specific image analysis manner is not limited. For example, the first image can be analyzed and processed by a pre-trained image analysis model to output an image analysis result of whether the first body language action exists in the first image.

[0167] The first body language action is a pre-set body language action that can trigger the vehicle interactive assistance mode. It can be understood that the first body language action is a body language action that can be performed by the driver under the condition that the driving safety condition is met. The specific action type of the first body language action is not limited. In some examples, the first body language action can include a waving action, a fist gesture, a thumbs-up gesture, a preset number of blinking operations, a preset expression, and the like.

[0168] In some examples, the first body language action can also be a combination of two or more body language actions. This can avoid the possibility that the adopted body language action conflicts with the body language action set by other systems or applications when only one body language action is used. In addition, using two or more body language actions can improve the accuracy of receiving the trigger instruction.

[0169] In specific examples of the present application, the first body language action can be a combination of two body language actions, such as a combination of opening the mouth and thumbs-up, and the like. This can avoid the situation that the operation complexity is caused by too many body language actions when the number of body language actions is three or more. However, the present application is not limited thereto.

[0170] Based on this embodiment, the body language action of the driver on the driver's seat can be identified. In a case where the first body language action is identified, it can be determined that the trigger instruction triggering the vehicle interactive assistance mode is received. Therefore, a new button for the vehicle interactive assistance mode does not need to be reserved, and the related buttons of the vehicle do not need to be set. By image shooting and image recognition of the driver's seat, the trigger instruction triggering the vehicle interactive assistance mode can be identified.

[0171] The manner of obtaining the first interactive operation instruction of the driver is not limited, and some examples thereof are described below.

[0172] In some embodiments, the entering the in-vehicle interaction assistance mode in response to the trigger instruction includes: entering the in-vehicle interaction assistance mode in response to the trigger instruction, and displaying the plurality of first preset dialogue information through the vehicle display device.

[0173] At this time, the first interaction operation instruction of the driver is acquired, and the first interaction content information of the first interaction operation instruction is acquired based on the first correspondence relationship between the preset interaction operation instruction and the interaction content information, including: receiving and responding to the selection operation on the displayed first preset dialogue information, determining the first preset dialogue information selected by the selection operation as the first interaction content information, wherein the first interaction operation instruction includes a selection instruction corresponding to the selection operation on the displayed first preset dialogue information.

[0174] The first preset dialogue information is preset dialogue information, which can be dialogue information that can be used in a high-frequency scenario in a driving environment, such as "Please fasten the safety belt of the left rear passenger, the vehicle will start soon", "Please fasten the safety belt of the right rear passenger, the vehicle will start soon", "The current vehicle power / oil is sufficient, and it is expected to travel XX kilometers, and the whole journey is about XX minutes", "The current vehicle power / oil is insufficient, and it is expected to travel XX kilometers, I will go to XX gas station / charging station first", and other self-defined first preset dialogue information.

[0175] In some embodiments, the plurality of first preset dialogue information set in advance can be displayed through the vehicle display device.

[0176] In another example, displaying the plurality of first preset dialogue information through the vehicle display device includes:

[0177] Acquiring the first vehicle state information;

[0178] From the plurality of first preset dialogue information set in advance, one or more first preset dialogue information that meets the first vehicle state information is filtered out;

[0179] The filtered first preset dialogue information is displayed through the vehicle display device.

[0180] The first vehicle state information is information related to a vehicle state that can assist in determining which first preset dialogue to display. In some examples, the first vehicle state information can include seat occupancy information, a detection signal of a seat belt sensor, a detection signal of a door switch of the vehicle, etc. For example, if the first vehicle state information includes that the left rear seat is occupied by a passenger but the passenger has not fastened the seat belt (no detection signal of the seat belt sensor of the left rear seat is received), the selected first preset dialogue information can include "Please fasten the seat belt of the left rear passenger. The vehicle will start soon." For another example, if the first vehicle state information includes that the right side door is not completely closed (no closing signal of the door switch of the right side door is received), the selected first preset dialogue information can include "The door is not completely closed (right rear door). Please check and close it tightly." First preset dialogue information that is applicable in different vehicle states can all be selected as the first dialogue information, such as "The current vehicle power / fuel level is sufficient. It is estimated that the vehicle can travel XX kilometers, and the entire journey will take about XX minutes," "The current vehicle power / fuel level is insufficient. It is estimated that the vehicle can travel XX kilometers. I will go to XX gas station / charging station first," etc., but not limited thereto.

[0181] The manner of receiving the selection operation on the first preset dialogue information is not limited. In some examples, a control of a selection item can be provided for each displayed first preset dialogue information. The driver can click the control of the corresponding selection item to issue a selection operation on a certain first preset dialogue information displayed, and the first preset dialogue information corresponding to the selection item of the selection operation is selected as the first interactive content information selected by the selection operation.

[0182] In other examples, an image of the driver's seat can be captured, and image recognition can be performed on the obtained image. Based on the recognized body language action, it is determined that the received selection operation is, for example, when the driver is recognized to stretch out one finger, it is determined that the selection operation on the first selection item of the plurality of first preset dialogue information displayed is received, and when the driver is recognized to stretch out three fingers, it is determined that the selection operation on the third selection item of the plurality of first preset dialogue information displayed is received, but not limited thereto.

[0183] Based on this embodiment, after entering the vehicle interactive assistance mode, a plurality of first preset dialogue information can be displayed through the vehicle display device. Then, the driver can determine the corresponding first interactive content information by selecting the plurality of first preset dialogue information, which can simply and conveniently meet the interactive needs of the driver with language barriers such as the deaf and mute.

[0184] In other embodiments, referring to Figure 2 The above obtaining the first interactive operation instruction of the driver can include:

[0185] Step S201: Obtain the gaze direction of the driver.

[0186] The gaze direction of the driver refers to the specific position or angle at which the eyes of the driver focus. The specific way of obtaining the gaze direction of the driver is not limited, and in some examples, the gaze direction of the driver can be obtained by capturing the physiological characteristic changes of the eyeball through eye tracking and then calculating the line of sight direction. The specific way of obtaining the gaze direction of the driver through eye tracking can use any existing or future eye tracking method, and the embodiments of the present application do not limit this.

[0187] Step S202: In a case where it is determined that the driver gazes at the rearview mirror based on the gaze direction of the driver, it is determined that the first interaction operation instruction of the driver is received.

[0188] After obtaining the gaze direction of the driver, the object of the driver's gaze can be determined, such as whether the driver gazes at the rearview mirror. In a case where the driver gazes at the rearview mirror, it can be determined that the first interaction operation instruction of the driver is received. The way of determining whether the object of the driver's gaze is the rearview mirror is not limited, for example, the relative positional relationship between the driver's seat and the rearview mirror can be combined with the gaze direction to determine whether the object of the driver's gaze is the rearview mirror, but is not limited thereto.

[0189] In this embodiment, in a case where the driver gazes at the rearview mirror, it is considered that the driver has a demand for information interaction since the vehicle interactive assistance mode has been entered. That is, in a case where the vehicle interactive assistance mode is entered, when it is monitored that the driver gazes at the rearview mirror, it is considered that the driver has a demand for information interaction, and thus it is determined that the first interaction operation instruction of the driver is received.

[0190] Based on this embodiment, by monitoring the target gaze direction of the driver, in a case where it is determined that the driver gazes at the rearview mirror, it is determined that the first interaction operation instruction of the driver is received. Accordingly, the first interaction operation instruction can be monitored by tracking the target gaze direction of the driver, and the driver does not need to perform other processing operations. The first interaction operation instruction can be issued by adjusting the gaze direction, and the driving operation of the hands can not be affected. The way of issuing the first interaction operation instruction is simple and convenient.

[0191] In some embodiments, referring to FIG. 1, the step S202 of determining that the first interaction operation instruction of the driver is received in a case where it is determined that the driver gazes at the rearview mirror based on the gaze direction of the driver includes: Figure 3

[0192] ​Step S2021: In a case that the driver's gaze rearview mirror is determined based on the driver's gaze direction, and the second interaction operation instruction of the driver is received, it is determined that the first interaction operation instruction of the driver is received.

[0193] The second interaction operation instruction is an interaction operation instruction that can be combined with the driver's gaze direction in the vehicle interactive assistance mode, that is, the second interaction operation instruction can be combined with the driver's gaze direction to determine whether the first interaction operation instruction of the driver is received. It can be understood that the second interaction operation performed by the driver is an interaction operation that can be performed under the condition that the driving safety can be met, for example, an operation of clicking a button or selecting an option on a touch screen, a simple body language action, etc., but not limited thereto.

[0194] Accordingly, in a case that the driver's gaze rearview mirror is determined based on the driver's gaze direction, and the second interaction operation instruction of the driver is received, it is determined that the first interaction operation instruction of the driver is received, thereby reducing the possibility of false triggering caused by the need to gaze at the rearview mirror during driving, and facilitating further improvement of driving safety.

[0195] The manner of receiving the second interaction operation instruction is not limited, and some examples thereof are described below.

[0196] In some embodiments, in a case that the operation of the second preset button of the vehicle is received, it is determined that the second interaction operation instruction of the driver is received.

[0197] The second preset button can be a button different from the first preset button, and in some feasible implementation manners, the second preset button can also be set to be the same as the first preset button. In this case, when the operation of the first preset button of the vehicle is initially received, it can be considered that the trigger instruction for triggering the vehicle interactive assistance mode is received, thereby activating and entering the vehicle interactive assistance mode. When the operation of the first preset button of the vehicle is further received after the vehicle interactive assistance mode has been entered, since the vehicle interactive assistance mode is in the starting state at this time, it can be considered that the second interaction operation instruction is received, and then it is considered together with whether the driver's gaze rearview mirror is monitored to determine whether the first interaction operation instruction of the driver indicating that the driver has an interaction demand is received, but not limited thereto.

[0198] Accordingly, by configuring the second preset button, the driver can issue the second interaction operation instruction by operating the second preset button, and the manner of issuing the second interaction operation instruction is simple and convenient.

[0199] In some embodiments, the method further comprises:

[0200] obtaining a second image obtained by photographing the driver seat;

[0201] In a case where the second body language action is identified from the second image, it is determined that the second interaction operation instruction of the driver is received.

[0202] The second body language action is a body language action that can be used in cooperation with or in combination with whether the gaze of the driver is monitored to the rearview mirror to determine whether the first interaction operation instruction of the driver exists interaction demand is received, which can be the same as or different from the first body language action described above. In the embodiment of the application, the second body language action is set to be different from the first body language action.

[0203] Based on the scheme of the embodiment, the driver on the driver's seat can be identified by the body language action, and in a case where the second body language action is identified, it is determined that the second interaction operation instruction is received, so that a new button does not need to be reserved for receiving the second interaction operation instruction, and the related buttons of the vehicle do not need to be set. By image shooting and image recognition on the driver's seat, the second interaction operation instruction can be obtained, which can be a non-sensing trigger under the action of the driver observing the rearview mirror, which is fast and direct.

[0204] The manner of obtaining the first interaction content information of the first interaction operation instruction is not limited, and some examples are described below.

[0205] In some embodiments, based on the first correspondence relationship between the preset interaction operation instruction and the interaction content information, the first interaction content information of the first interaction operation instruction is obtained, including:

[0206] Based on the type of the rearview mirror to which the gaze of the driver is fixed, the second preset dialogue information corresponding to the type of the rearview mirror is obtained, wherein the first correspondence relationship between the preset interaction operation instruction and the interaction content information includes the correspondence relationship between the type of the rearview mirror and the second preset dialogue information.

[0207] The obtained second preset dialogue information is determined as the first interaction content information.

[0208] The type of the rearview mirror of the vehicle is not limited, for example, the type of the rearview mirror in some examples can include an in-vehicle rearview mirror, a left side rearview mirror of the vehicle, and a right side rearview mirror of the vehicle, etc. For different types of rearview mirrors, corresponding second preset dialogue information can be set respectively, so that after the type of the rearview mirror to which the gaze of the driver is fixed is determined, the corresponding second preset dialogue information can be determined accordingly.

[0209] The second preset dialogue information is pre-set dialogue information. In this embodiment, the dialogue information can be determined based on the state that the driver may see when looking at the rearview mirror, such as "The vehicle has not come to a complete stop, please wait until it comes to a complete stop before getting out of the car to avoid accidents" or "Temporary parking, please do not open the door to get out of the car", but it is not limited to these.

[0210] Based on this embodiment, the first interactive content information of the driver can be obtained by determining the type of rearview mirror that the driver is looking at. For the driver, he only needs to look at the corresponding rearview mirror to send the first interactive content information corresponding to that rearview mirror. The way to send the first interactive content information is simple and convenient.

[0211] In some specific examples, the rearview mirrors include the interior rearview mirror, the left-side rearview mirror, and the right-side rearview mirror. The second preset dialogue information corresponding to the interior rearview mirror, the left-side rearview mirror, and the right-side rearview mirror is different.

[0212] The associated states may differ depending on whether the driver's gaze is fixed on the interior rearview mirror, the left side rearview mirror, or the right side rearview mirror. This can lead to different corresponding verbal messages. For example, when the driver's gaze is on the interior rearview mirror, the corresponding second verbal message might be: "The vehicle has not come to a complete stop. Please wait until it stops completely before getting out to avoid accidents," or "Temporary parking. Do not open the door to get out." When the driver's gaze is on the right side rearview mirror, the corresponding second verbal message might be: "There is a vehicle approaching from behind. Please observe carefully before opening the door and open it slowly to ensure safety." When the driver's gaze is on the left side rearview mirror, the corresponding second verbal message might be: "There is a vehicle approaching from behind. Please exit from the right side door," and so on. However, these are not the only possible interpretations.

[0213] Based on this embodiment, different second preset dialogue messages are set for the rearview mirror inside the vehicle, the left rearview mirror, and the right rearview mirror. By looking at different rearview mirrors of the vehicle, the driver can match different second preset dialogue messages and trigger different first interactive content messages, thereby improving the information richness of in-vehicle interaction.

[0214] In other embodiments, obtaining the first interactive content information of the first interactive operation instruction based on the preset first correspondence between interactive operation instructions and interactive content information includes:

[0215] Obtain the second preset dialogue information corresponding to the second interactive operation instruction, and the first correspondence between the preset interactive operation instruction and the interactive content information, including the correspondence between the second interactive operation instruction and the second preset dialogue information;

[0216] The acquired second preset dialogue information is determined as the first interactive content information.

[0217] The second preset dialogue information is dialogue information preset in advance, and the content of the second preset dialogue information is not limited.

[0218] Based on the scheme of this embodiment, different second preset dialogue information can be set for different second interaction operation instructions, so that the driver can match different second preset dialogue information by executing different second interaction operations to issue different second interaction operation instructions, thereby triggering the issuance of different first interaction content information, and the information richness of the vehicle interaction is improved.

[0219] In some other examples, the first interaction content information of the first interaction operation instruction is obtained based on the preset first correspondence relationship between the interaction operation instruction and the interaction content information, and the method comprises:

[0220] Based on the type of the rearview mirror at which the driver's gaze is fixed and the second interaction operation instruction, the corresponding second preset dialogue information is obtained, and the preset first correspondence relationship between the interaction operation instruction and the interaction content information comprises a correspondence relationship between the type of the rearview mirror and the second preset dialogue information.

[0221] The obtained second preset dialogue information is determined as the first interaction content information.

[0222] Based on the scheme of this embodiment, different second preset dialogue information can be set for different combinations of the second interaction operation instruction and the type of the rearview mirror, and through the combination of the second interaction operation instruction and the type of the rearview mirror, the possibility of false triggering caused by the driver's need to fix his gaze on the rearview mirror during driving can be reduced, which is conducive to improving driving safety. At the same time, different second preset dialogue information can be matched through different combinations of the second interaction operation instruction and the type of the rearview mirror, thereby triggering the issuance of different first interaction content information, and the information richness of the vehicle interaction is improved.

[0223] In some other examples, the first interaction content information of the first interaction operation instruction is obtained based on the preset first correspondence relationship between the interaction operation instruction and the interaction content information, and the method comprises:

[0224] Second vehicle state information is obtained.

[0225] Based on the second vehicle state information and the type of the rearview mirror at which the driver's gaze is fixed, the corresponding second preset dialogue information is obtained, and the preset first correspondence relationship between the interaction operation instruction and the interaction content information comprises a correspondence relationship between the second vehicle state information and the type of the rearview mirror and the second preset dialogue information.

[0226] The obtained second preset dialogue information is determined as the first interaction content information.

[0227] The second vehicle state information is information related to a vehicle state that can assist in determining the content of information to be provided to the passenger in combination with the type of the rearview mirror, which can be state information related to the use or driving of the vehicle, and the second vehicle state information in some examples can include at least one of a vehicle speed and a vehicle gear.

[0228] The content of the driver interaction prompt can not be the same in the case of the vehicle being in different states, and therefore, the second vehicle state information can be combined with the type of the rearview mirror on which the driver's gaze is fixed to determine the second preset dialogue information corresponding thereto.

[0229] Taking an example in which the second vehicle state information includes a vehicle speed of the vehicle:

[0230] If the vehicle speed is 0, the vehicle can be in a state of waiting for a red light or temporary parking, and if the rearview mirror on which the driver's gaze is fixed at this time is an interior rearview mirror, the corresponding second preset dialogue information can include prompt information for temporary parking, such as "temporary parking, please do not open the door to get off the vehicle" and the like, and if the rearview mirror on which the driver's gaze is fixed at this time is a right rearview mirror, the corresponding second preset dialogue information can include prompt information for observing when opening the door, such as "there is a vehicle coming from behind, please observe before opening the door, slowly open the door, and ensure safety" and the like, and if the rearview mirror on which the driver's gaze is fixed at this time is a left rearview mirror, the corresponding second preset dialogue information can include prompt information for getting off the vehicle from the right door, such as "there is a vehicle coming from behind, please get off the vehicle from the right door" and the like.

[0231] If the vehicle speed is less than a preset speed threshold, it indicates that the vehicle is in a low-speed driving state, and at this time, the vehicle can be slowing down before parking, such as the vehicle reaching the destination but not yet parked in a safe area and still slowly moving to find a parking space, and if the rearview mirror on which the driver's gaze is fixed at this time is an interior rearview mirror, the corresponding second preset dialogue information can include prompt information for the vehicle not being parked stably, such as "the vehicle is not parked stably, please wait until it is parked stably before getting off the vehicle to avoid accidents" and the like, but not limited thereto.

[0232] Taking an example in which the second vehicle state information includes a vehicle gear of the vehicle, if the vehicle gear is P, it indicates that the vehicle is in a state of long-time parking or engine off, and at this time, the driver and / or passenger can get off the vehicle, and if the rearview mirror on which the driver's gaze is fixed at this time is a right rearview mirror, the corresponding second preset dialogue information can include prompt information for observing when opening the door, such as "there is a vehicle coming from behind, please observe before opening the door, slowly open the door, and ensure safety" and the like, and if the rearview mirror on which the driver's gaze is fixed at this time is a left rearview mirror, the corresponding second preset dialogue information can include prompt information for getting off the vehicle from the right door, such as "there is a vehicle coming from behind, please get off the vehicle from the right door" and the like, but not limited thereto.

[0233] For example, if the vehicle speed is less than or equal to the preset vehicle speed and the vehicle gear is not P, it indicates that the vehicle is in a low-speed driving state, and the vehicle may need to slow down and stop. If the rearview mirror that the driver is looking at is the interior rearview mirror, the corresponding second preset dialogue information can include a prompt that the vehicle is not parked, such as "The vehicle is not parked. Please wait until it is parked before getting off to avoid accidents", but not limited to this.

[0234] Based on the scheme of the embodiment, different second preset dialogue information can be set based on the combination of the second vehicle state information and the type of the rearview mirror. Accordingly, the second preset dialogue information obtained by matching can be adapted to the current second vehicle state information, the first interaction content information determined and output can be adapted to the current vehicle state, and the vehicle interaction experience can be further improved.

[0235] The manner of outputting the first interaction content information in step S103 is not limited, and some of the manners are exemplified below.

[0236] In some embodiments, outputting the first interaction content information includes voice broadcasting the first interaction content information.

[0237] Voice broadcasting refers to converting the text content to be output into audio and playing the audio through an audio playing device.

[0238] Accordingly, when outputting the first interaction content information, the first interaction content information can be output by voice broadcasting, so that passengers without hearing impairment can hear the voice and immediately know the first interaction content information expressed by the driver, which can meet the vehicle interaction needs between the driver with language impairment such as deaf-mutism and the passenger with normal hearing. The passengers without hearing impairment can include passengers with normal hearing without wearing a hearing aid, and passengers with normal hearing after wearing a hearing aid.

[0239] In some embodiments, outputting the first interaction content information includes displaying the first interaction content information on a display screen corresponding to the target passenger seat.

[0240] The display screen corresponding to the target passenger seat refers to a display screen arranged in the vehicle and capable of facilitating the passenger on the target passenger seat to view the display content. In general, the display screen corresponding to the target passenger seat can be arranged in front of the target passenger seat, i.e., in the position close to the head direction relative to the target passenger seat. For example, the target passenger seat is the right rear passenger seat in the vehicle, and the display screen corresponding to the target passenger seat can be arranged behind the front seat (e.g., the front passenger seat). For example, the target passenger seat is the left rear passenger seat in the vehicle, and the display screen corresponding to the target passenger seat can be arranged behind the driver seat. For example, the target passenger seat is the front passenger seat, and the display screen corresponding to the target passenger seat can be arranged in front of the front passenger seat, but is not limited thereto, as long as the passenger on the passenger seat can view the display content displayed on the display screen.

[0241] The target passenger seat is the seat where the target passenger to which the first interaction content information of the driver needs to be conveyed is located. The target passenger can be all the passengers in the vehicle, and in this case, the target passenger seat can be all the seats in the vehicle except the driver seat, or all the passenger seats in the vehicle. In other examples, the target passenger can be a specific passenger in the vehicle, and in this case, the target passenger seat can be the passenger seat where the specific passenger is located.

[0242] The manner of determining the target passenger or the target passenger seat is not limited, and in some examples, the seat distribution diagram of the vehicle can be displayed on the vehicle display device, and the seat selected by the driver on the seat distribution diagram can be determined as the target passenger seat. For example, after the passenger gets on the vehicle, the passenger has interacted with the driver through sign language, and after the passenger gets on the seat, the driver selects the seat on which the passenger is located on the seat distribution diagram to determine the seat as the target passenger seat. For example, the driver knows in advance that a passenger is a hearing-impaired person, and after the passenger gets on the seat, the driver selects the seat on which the passenger is located on the seat distribution diagram to determine the seat as the target passenger seat. In other examples, when the camera in the vehicle captures the passenger interacting through sign language, the seat on which the passenger is located can be determined as the target passenger seat, but is not limited thereto.

[0243] Accordingly, when the first interactive content information is output, the first interactive content information can be output by being displayed on the display screen corresponding to the target passenger seat, so that whether the passenger has normal hearing or has a hearing impairment, the passenger can learn the first interactive content information expressed by the driver in time through the content displayed on the display screen, and the vehicle interactive demand between the driver with a language impairment such as a deaf-mute and the passenger with normal hearing and / or a hearing impairment can be met.

[0244] In some embodiments, outputting the first interactive content information includes voice broadcasting the first interactive content information and displaying the first interactive content information on the display screen corresponding to the target passenger seat.

[0245] Accordingly, when the first interactive content information is output, the first interactive content information is output by being displayed on the display screen corresponding to the target passenger seat while being voice broadcast, so that the output by the dual output modes of voice and display screen display can meet the demand of different passengers and help improve the possibility of the first interactive content information of the driver reaching the passengers.

[0246] The manner in which the first interactive content information is displayed on the display screen corresponding to the target passenger seat is not limited.

[0247] In some embodiments, displaying the first interactive content information on the display screen corresponding to the target passenger seat includes:

[0248] Textually displaying the first interactive content information on the display screen corresponding to the target passenger seat.

[0249] According to this embodiment, the first interactive content information of the driver can be learned by the passenger by viewing the text content displayed on the display screen corresponding to the seat of the passenger, which is intuitive and convenient.

[0250] In some embodiments, displaying the first interactive content information on the display screen corresponding to the target passenger seat includes:

[0251] Obtaining a sign language video of the first interactive content information and displaying the sign language video on the display screen corresponding to the target passenger seat.

[0252] The manner in which the sign language video of the first interactive content information is obtained is not limited, and in some examples, the corresponding sign language video can be generated in real time based on the first interactive content information, for example, after the first interactive content information is punctuated, a sign language sub-video corresponding to each word after punctuation is obtained, and each sign language sub-video is spliced to obtain a sign language video corresponding to the first interactive content information.

[0253] In some examples, the number of pre-stored interaction content information can be limited, and therefore, for each interaction content information, a corresponding sign language video can be pre-recorded and stored, so that after obtaining the first interaction content information, the sign language video corresponding to the first interaction content information can be searched from the pre-stored sign language videos.

[0254] The manner of searching for the sign language video corresponding to the first interaction content information is not limited, and in some specific examples, a correspondence relationship among the interaction content identifier, the interaction content information, and the sign language video can be pre-stored, the first correspondence relationship between the preset interaction operation instruction and the interaction content information can be a correspondence relationship between the interaction operation instruction and the interaction content identifier, and therefore, after obtaining the first interaction operation instruction, the first interaction content identifier corresponding to the first interaction operation instruction can be obtained based on the first correspondence relationship, and the first interaction content information and the sign language video corresponding to the first interaction content identifier can be obtained based on the correspondence relationship among the interaction content identifier, the interaction content information, and the sign language video. In some other specific examples, the first correspondence relationship between the preset interaction operation instruction and the interaction content information further includes an association relationship between the interaction content information and the sign language video, such as a storage address of the sign language video, and therefore, after obtaining the first interaction content information, the sign language video corresponding to the first interaction content information can be searched based on the storage address of the associated sign language video. It can be understood that in the extraction embodiment, the sign language video corresponding to the first interaction content information can also be obtained in other manners, as long as the content expressed by the obtained sign language video is consistent with the first interaction content information.

[0255] Accordingly, by obtaining the sign language video of the first interaction content information and displaying the sign language video on the display screen corresponding to the target passenger seat, the passenger can know the first interaction content information of the driver by viewing the sign language video displayed on the display screen corresponding to the seat, especially for the passengers with hearing impairment and inconvenient to read text, such as children with hearing impairment, the elderly with hearing impairment, and the like, so that the first interaction content information expressed by the driver can be known through the sign language video displayed on the display screen without the need of text reading ability, the convenience of vehicle interaction is improved, and the interaction demand of the passengers with hearing impairment and inconvenient to read text can be met.

[0256] In some other examples, displaying the first interaction content information on the display screen corresponding to the target passenger seat includes: displaying the first interaction content information in text on the display screen corresponding to the target passenger seat, and obtaining the sign language video of the first interaction content information, and displaying the sign language video on the display screen corresponding to the target passenger seat.

[0257] Accordingly, by displaying the first interactive content information in text on the display screen corresponding to the target passenger seat, and also obtaining the sign language video of the first interactive content information and displaying it on the display screen corresponding to the target passenger seat, for passengers with the ability to read text, the first interactive content information expressed by the driver can be obtained through the displayed text content, and for passengers with hearing impairment, the first interactive content information expressed by the driver can be obtained through the sign language video displayed on the display screen, thereby improving the vehicle interactive needs of different passengers.

[0258] In some embodiments, outputting the first interactive content information further includes: based on the first interactive content information, controlling the target passenger seat to vibrate.

[0259] In some embodiments, outputting the first interactive content information further includes: based on the first interactive content information, controlling the target passenger seat to vibrate.

[0260] According to the above embodiment, when outputting the first interactive content information, the target passenger seat is also controlled to vibrate, so that the passenger sitting on the target passenger seat can be reminded by vibration, for example, to pay attention to the output voice information or the information displayed on the display screen, which can further improve the possibility of the first interactive content information of the driver reaching the passengers.

[0261] In some embodiments, outputting the first interactive content information further includes: based on the first interactive content information, controlling the atmosphere lamp of the target passenger seat to flash.

[0262] In some embodiments, outputting the first interactive content information further includes: based on the first interactive content information, controlling the atmosphere lamp of the target passenger seat to flash.

[0263] Accordingly, when outputting the first interactive content information, the atmosphere lamp of the target passenger seat is also controlled to flash, so that the passenger sitting on the target passenger seat can be reminded by the flashing of the atmosphere lamp, for example, to pay attention to the output voice information or the information displayed on the display screen, which can further improve the possibility of the first interactive content information of the driver reaching the passengers, and thereby improve the vehicle interactive performance.

[0264] In some embodiments, the outputting the first interaction content information further includes: based on the first interaction content information, controlling the target passenger seat to vibrate and controlling the atmosphere lamp of the target passenger seat to flash.

[0265] Accordingly, when the first interaction content information is output, the passenger sitting on the target passenger seat can be reminded twice through the seat vibration and the atmosphere lamp flashing, which can further improve the possibility of the driver delivering the first interaction content information to the passenger, and thus the vehicle interactive performance can be improved.

[0266] In some embodiments, with reference to Figure 4 The vehicle interactive method can further include:

[0267] Step S401: Obtain second interaction content information initiated by the passenger.

[0268] The second interaction content information is the interaction content information initiated or provided by the passenger on the passenger seat, which needs to be known by the driver.

[0269] The way of obtaining the second interaction content information is not limited, and in some examples, it can be the interaction content information input by the passenger through the display screen corresponding to the seat (for example, the display screen in front of the seat) or the interaction content information selected and determined by selecting the relevant selection item. When the interaction content information is received through the display screen corresponding to a certain passenger seat or the selection operation of a certain selection item, the received interaction content information or the interaction content information corresponding to the selection operation can be taken as the second interaction content information initiated by the passenger.

[0270] In other examples, the voice information in the vehicle can be collected, and the voice information can be audio-recognized and converted into text, and the obtained text content can be taken as the second interaction content information initiated by the passenger.

[0271] In other examples, the position area of the passenger seat can be imaged to obtain an image, and the obtained image can be sign language recognized, and the content expressed by the recognized sign language can be taken as the obtained second interaction content information initiated by the passenger.

[0272] It can be understood that in other embodiments, other ways can also be used to obtain the second interaction content information initiated by the passenger, as long as the information content expressed by the passenger on the passenger seat can be obtained.

[0273] Step S402: Display the second interaction content information through the vehicle display device.

[0274] After obtaining the second interaction content information, the obtained second interaction content information can be displayed through the vehicle-mounted display device, so that the driver can know the interaction content of the passenger. The type of the vehicle-mounted display device is not limited, and the vehicle-mounted display device in some examples may, for example, be a HUD (Head-Up Display, head-up display or head-up display), a center display screen, etc., but is not limited thereto,

[0275] Based on this embodiment, the second interaction content information initiated by the passenger seat can also be obtained and displayed through the vehicle-mounted display device, so that the second interaction content information of the passenger can be obtained and displayed through the vehicle-mounted display device, so that the driver can know the interaction content information expressed by the passenger by viewing the content displayed by the vehicle-mounted display device, thereby realizing the two-way interaction between the driver and the passenger, and helping to further improve the vehicle interaction performance.

[0276] Wherein, after obtaining the second interaction content information, the second interaction content information can be directly displayed through the vehicle-mounted display device. In the embodiment of the application, after obtaining the second interaction content information, the second interaction content information can be displayed on demand to improve driving safety.

[0277] Accordingly, in some embodiments, displaying the second interaction content information through the vehicle-mounted display device comprises:

[0278] Obtaining third vehicle state information;

[0279] In a case where it is determined based on the third vehicle state information that the vehicle is in a safe driving state, the second interaction content information is displayed through the vehicle-mounted display device.

[0280] The second vehicle state information is vehicle state-related information that can be used to determine whether the vehicle is in a safe driving state. It can be state information related to the use or driving of the vehicle. In some examples, the third vehicle state information can include at least one of the vehicle speed and the vehicle gear.

[0281] The manner of determining whether the vehicle is in a safe driving state based on the third vehicle state information is not limited. In some specific examples, the third vehicle state information includes the vehicle speed. At this time, it can be determined that the vehicle is in a safe driving state when the vehicle speed is less than or equal to a preset vehicle speed, or when the vehicle speed is less than or equal to the preset vehicle speed for a duration greater than or equal to a first preset duration. The preset vehicle speed and the first preset duration can be set in combination with the safety requirements of the driving scene. For example, the preset vehicle speed in some examples can include 30 kmh (kilometers per hour), and the first preset duration in some examples can include 5s (seconds), but is not limited thereto,

[0282] In some embodiments, the third vehicle state information comprises a vehicle gear, and the vehicle is determined to be in the safe driving state when the vehicle gear is a first preset gear. In some embodiments, the first preset gear comprises a P gear, an N gear, or the like.

[0283] In some embodiments, the third vehicle state information comprises a vehicle speed and a vehicle gear, and the vehicle is determined to be in the safe driving state when the vehicle speed is less than or equal to a preset vehicle speed, or when the vehicle speed is less than or equal to the preset vehicle speed for a duration greater than or equal to a first preset duration, and the vehicle gear is a first preset gear. In other embodiments, other third vehicle state information and other manners of determining whether the vehicle is in the safe driving state can be used.

[0284] Based on the above embodiment, the second interactive content information is displayed on the vehicle display device only when the vehicle is determined to be in the safe driving state based on the third vehicle state information after the second interactive content information is received, which can achieve two-way interaction between the driver and the passenger while improving driving safety.

[0285] In some embodiments, as shown in FIG. 5, the step S402 of displaying the second interactive content information on the vehicle display device can comprise: Figure 5

[0286] Step S501: Determine the urgency type of the second interactive content information.

[0287] The urgency type of the second interactive content information refers to the type of the urgency degree of processing the second interactive content information.

[0288] The manner of determining the urgency type of the second interactive content information is not limited. In some embodiments, when the display screen is used to provide the interactive content information to the passenger for the passenger to select the second interactive content information, the corresponding urgency type can be set for each second interactive content information, so that the urgency type of the second interactive content information can be directly obtained after the second interactive content information is obtained.

[0289] In some embodiments, the information category of the second interactive content information can be determined first, and the corresponding urgency type can be determined based on the information category. For example, a set of urgency categories is set, and the set of urgency categories includes a plurality of information categories. If the information category of the identified second interactive content information is in the set of urgency categories, i.e., the information category of the identified second interactive content information is an element of the set of urgency categories, it can be considered that the urgency type of the second interactive content information is urgent, otherwise, it can be considered that the urgency type of the second interactive content information is non-urgent. In some embodiments, the information category of the second interactive content information can be determined first, and the corresponding urgency type can be determined based on the information category. For example, a set of urgency categories is set, and the set of urgency categories includes a plurality of information categories. If the information category of the identified second interactive content information is in the set of urgency categories, i.e., the information category of the identified second interactive content information is an element of the set of urgency categories, it can be considered that the urgency type of the second interactive content information is urgent, otherwise, it can be considered that the urgency type of the second interactive content information is non-urgent.

[0290] It can be understood that, in some examples, a non-urgent category set can also be provided, the non-urgent category set includes a plurality of information categories, and the intersection between the urgent category set and the non-urgent category set is empty, that is, there is no intersection, if the identified information category of the second interaction content information is an element of the urgent category set, it can be considered that the urgent category of the second interaction content information is urgent, and if the identified information category of the second interaction content information is an element of the non-urgent category set, it can be considered that the urgent category of the second interaction content information is non-urgent. In this case, if the identified information category of the second interaction content information is neither an element of the urgent category set nor an element of the non-urgent category set, for example, in the case of identifying the information category of the second interaction content information by using the machine learning model, if the classification probability of the machine learning model for each set information category of the second interaction content information is less than the preset probability threshold, it can be considered that the information category of the second interaction content information is neither an element of the urgent category set nor an element of the non-urgent category set, that is, the information category cannot be identified. At this time, the second interaction content information can be directly displayed in real time through the vehicle display device to avoid the influence of not displaying and processing in time in the case that the second interaction content information is uncategorized and needs urgent processing. It can also be that after determining that the third vehicle state information is in a safe driving state, the second interaction content information is displayed through the vehicle display device to meet the demand of safe driving. The specific setting can be based on actual technical needs.

[0291] The manner of determining the information category of the second interaction content information is not limited, for example, the machine learning model obtained by training can be used to classify the second interaction content information to obtain the information category of the second interaction content information, but is not limited thereto. After obtaining the information category, the urgent type of the second interaction content information can be determined based on the information category.

[0292] The specific category of the information category belonging to the urgent type is not limited, for example, the information category belonging to the urgent type in some examples can be a physical discomfort category (such as needing to go to the bathroom, car sickness, etc.), a vehicle abnormality category (such as an abnormal sound at the bottom of the car, etc.), a trip change, etc., but is not limited thereto.

[0293] In other examples, the information urgent type classification model obtained by training can be directly used to process the second interaction content information to obtain the classification result that the urgent type of the second interaction content information is urgent or non-urgent.

[0294] Step S502: In the case that the urgent type of the second interaction content information is urgent, the second interaction content information is displayed in real time through the vehicle display device.

[0295] In a case where the emergency type of the second interaction content information is determined to be emergency, it is indicated that the second interaction content information has a high degree of emergency and needs to be processed in time, and thus the second interaction content information can be displayed in real time through the vehicle-mounted display device.

[0296] In a case where the emergency type of the second interaction content information is non-emergency, if it is determined based on the third vehicle state information that the vehicle is in a safe driving state, the second interaction content information is displayed through the vehicle-mounted display device.

[0297] In a case where the emergency type of the second interaction content information is determined to be non-emergency, it is indicated that the second interaction content information has a low degree of emergency and does not need to be processed in time, and thus the second interaction content information can be displayed through the vehicle-mounted display device in a case where it is determined based on the third vehicle state information that the vehicle is in a safe driving state, i.e., in a case where the safe driving requirement is met.

[0298] Based on the embodiment, after obtaining the second interaction content information of the passenger, the emergency type of the second interaction content information is first determined, in a case where the emergency type of the second interaction content information is emergency, the second interaction content information is displayed in real time through the vehicle-mounted display device to meet the emergency requirement of the passenger, and in a case where the emergency type of the second interaction content information is non-emergency, the second interaction content information is displayed through the vehicle-mounted display device in a case where it is determined based on the third vehicle state information that the vehicle is in a safe driving state, so that the two-way interaction between the driver and the passenger is realized in a case where it is determined that the vehicle is in a safe driving state.

[0299] In some embodiments, the vehicle-mounted interaction method further includes:

[0300] In a case where no interaction operation instruction is received within the second preset time length, the vehicle-mounted interaction assistance mode is exited.

[0301] The specific time length of the second preset time length is not limited, and can be a time length preset in the vehicle-mounted interaction assistance mode, or a time length set by the user. In some specific examples, the second preset time length can be determined by counting the interval time length between adjacent information interactions in a plurality of consecutive information interaction processes, for example, the maximum interval time length between any adjacent information interactions is taken as the second preset time length, or the plurality of interval time lengths are statistically calculated by averaging, weighted averaging, etc., to obtain the second preset time length, but are not limited thereto.

[0302] The second preset time period in which the interaction operation instruction is not received can refer to the first interaction operation instruction, the second interaction operation instruction, and the third interaction operation instruction. In the second preset time period in which the interaction operation instruction is not received, it is indicated that there is no demand for vehicle interaction or the possibility of the demand for vehicle interaction is low, and thus the vehicle interaction assistance mode can be exited, the occupation of resources can be reduced, and the driving safety can be further improved.

[0303] In some embodiments, the vehicle interaction method can further include:

[0304] The second interaction content information is displayed through the vehicle display device, and the vehicle interaction assistance mode is entered.

[0305] According to the embodiment, the second interaction content information is displayed through the vehicle display device, and the vehicle interaction assistance mode is entered, so that the feedback of the driver with a language barrier such as a deaf-mute person to the second interaction content information can be captured in time, and the vehicle interaction performance is further improved.

[0306] In some embodiments, referring to Figure 6 As shown in the above embodiment, the vehicle interaction method can further include:

[0307] Step S601: Obtain a third interaction operation instruction of the driver, and obtain third interaction content information of the third interaction operation instruction based on a second correspondence relationship between the interaction operation instruction and the interaction content information.

[0308] The third interaction operation instruction is an instruction for indicating the specific interaction content information of the driver of the driver seat in response to the second interaction content information of the passenger of the passenger seat.

[0309] Step S602: Output the third interaction content information.

[0310] After obtaining the third interaction content information, the third interaction content information can be output to inform the passenger on the passenger seat of the third interaction content information fed back by the driver. The specific implementation manner of outputting the third interaction content information can be the same as the manner of outputting the first interaction content information, which is not described herein.

[0311] According to the embodiment, after obtaining and displaying the second interaction content information, the third interaction content information of the driver is obtained based on the third interaction operation instruction of the driver, so that the feedback information of the driver to the second interaction content information of the passenger is obtained, the reply and feedback of the driver to the second interaction content information of the passenger are realized, and the vehicle interaction performance can be further improved.

[0312] The specific determination manner of the third interaction content information is not limited, and some examples are illustrated below.

[0313] In some embodiments, the method further includes:

[0314] Obtaining recommended dialogue information generated by performing intent understanding on the second interaction content information, and displaying the recommended dialogue information through the vehicle display device.

[0315] At this time, the third interaction operation instruction of the driver is obtained, including: obtaining a selection operation on the displayed recommended dialogue information, the third interaction operation instruction includes a selection instruction corresponding to the selection operation on the displayed recommended dialogue information, and the third interaction content information includes the recommended dialogue information selected by the selection operation on the displayed recommended dialogue information.

[0316] The manner of performing intent understanding on the second interaction content information and generating recommended dialogue information is not limited, for example, in some examples, an intent recognition model obtained by training can be used to perform intent understanding on the second interaction content information, and corresponding recommended dialogue information can be generated based on the recognized intent. The specific manner of training the intent recognition model and the specific model architecture of the intent recognition model are not limited in the embodiments of the present application. In other embodiments, other manners can also be used to obtain the recommended dialogue information of the second interaction content information, which is not limited in the embodiments of the present application.

[0317] Based on this embodiment, the corresponding recommended dialogue information can be generated and displayed after the second interaction content information is understood, in which case the driver can obtain the third interaction content information to be output by performing a selection operation on the recommended dialogue information, and the intelligent recommendation of the dialogue based on the second interaction content information can be performed, which greatly reduces the time length of the driver's information reply, further improves the vehicle interaction performance, and helps to improve the driving safety during the vehicle interaction process.

[0318] The manner of obtaining the third interaction operation instruction is not limited, for example, similar to the manner of obtaining the first interaction operation instruction and the second interaction operation instruction, the third interaction operation instruction can be obtained through the selection operation of the driver on the multiple selection items on the vehicle display device, or the third interaction operation instruction can be obtained through the body language action performed by the driver, and the embodiments of the present application do not limit the same.

[0319] Taking the third interaction operation instruction obtained through the body language action performed by the driver as an example, the third interaction operation instruction of the driver is obtained, including:

[0320] Obtaining a third image obtained by photographing the driver's seat;

[0321] In a case where the third body language action is identified from the third image, it is determined that the third interaction operation instruction of the driver is received.

[0322] The third body language action can be different from the first body language action and the second body language action, so as to strictly distinguish the first body language action, the second body language action and the third body language action. In other embodiments, the third body language action is the same as the first body language action and the second body language action, as long as it can be accurately distinguished that the first interaction operation instruction, the second interaction operation instruction or the third interaction operation instruction is triggered in different cases.

[0323] It can be understood that in other modes or systems in the vehicle, the first body language action, the second body language action and the third body language action pre-set in the embodiments of the present application can be set to be different from the body language actions of the other modes or systems, so as to avoid the conflict problem of triggering different instructions due to the same body language action.

[0324] The third body language action in some examples can include nodding, OK gesture, etc., and the corresponding third interaction content information can be, for example, “OK”, “I know”, “OK”. The third body language action in other examples can include shaking head or five-finger open gesture, etc., and the corresponding third interaction content information can be, for example, “not OK”, “I disagree”, “I don't understand your meaning”, etc., but is not limited thereto.

[0325] Based on the embodiment, the driver on the driver's seat can be identified by performing body language action, and in a case where the third body language action is identified, it is determined that the third interaction operation instruction of the driver is received, so that the driver can obtain the third interaction operation instruction of the driver by performing the related third body language action, by image shooting and image recognition on the driver's seat, which is helpful to further improve the driving safety.

[0326] In some embodiments, the method further comprises:

[0327] Obtaining operation suggestion information generated by performing semantic analysis on the second interaction content information;

[0328] Performing vehicle control operation based on the operation suggestion information.

[0329] The operation suggestion information is related suggestion information of an executable operation generated based on a semantic analysis of the second interaction content information. For example, in a case where the second interaction content information includes content related to car sickness, the generated operation suggestion information can include turning on a car sickness relief function of the vehicle, and the like. For another example, in a case where the second interaction content information includes content related to a desire to use a restroom, the generated operation suggestion information can include adding a location of a closest restroom to a navigation waypoint, and the like.

[0330] In some embodiments, the operation suggestion information is obtained, and the vehicle control operation is executed based on the operation suggestion information.

[0331] The generated operation suggestion information is displayed through a vehicle display device.

[0332] In a case where a confirmation operation is received for the displayed operation suggestion information, the vehicle control operation corresponding to the operation suggestion information is executed.

[0333] Accordingly, after the operation suggestion information is obtained, the operation suggestion information can be displayed, and in a case where a confirmation operation is received for the operation suggestion information, the corresponding vehicle control operation is executed, such as turning on a car sickness relief function of the vehicle, adding a location of a closest restroom to a navigation waypoint, and navigating, and the like.

[0334] In some specific examples, in a case where a confirmation operation is received for the operation suggestion information, a corresponding interface of the function of the displayed operation suggestion information can be jumped to, such as an interface related to a car sickness relief function, and the like, but is not limited thereto.

[0335] Based on the embodiments of the vehicle interaction method as described above, the following can be illustrated in combination with some specific examples thereof.

[0336] In the following specific examples, the entry into the vehicle interaction assistance mode is triggered by a voice trigger button, and the first interaction content information is determined based on a type of a rearview mirror on which the driver gazes and second vehicle state information.

[0337] Reference Figure 7As shown, the vehicle interaction method in the specific example monitors a trigger instruction for triggering the vehicle interaction assistance mode in step S701, and determines whether the trigger instruction is received in step S702, for example, whether the monitored operation instruction is the operation instruction of the voice trigger button. If the operation instruction of the voice trigger button is received, it is determined that the trigger instruction for triggering the vehicle interaction assistance mode is received, and step S703 is entered. If the trigger instruction is not received, step S701 is returned to continue monitoring the trigger instruction for triggering the vehicle interaction assistance mode.

[0338] In step S703, it is determined whether the current speed of the vehicle is 0 km / h (kilometers per hour). If not, step S704 is entered. If yes, step S707 is entered.

[0339] In step S704, it is determined whether the current speed of the vehicle is less than or equal to a preset speed (for example, 30 km / h), and the current gear of the vehicle is not P. If both conditions are met, step S705 is entered. If both conditions cannot be met at the same time, step S713 is entered to end the current processing process.

[0340] In step S705, the eye gaze position of the driver is detected to obtain the gaze direction of the driver's eyes, and step S706 is entered.

[0341] In step S706, it is determined whether the driver's eyes gaze at the inside rearview mirror based on the gaze direction of the driver's eyes. If yes, step S709 is entered. Otherwise, step S713 is entered to end the current processing process.

[0342] In step S707, the eye gaze position of the driver is detected to obtain the gaze direction of the driver's eyes, and step S708 is entered.

[0343] In step S708, the eye gaze position of the driver is determined based on the gaze direction of the driver's eyes, and the type of the rearview mirror gazed at by the eyes is determined. If the inside rearview mirror is gazed at, step S710 is entered. If the right rearview mirror is gazed at, step S711 is entered. If the left rearview mirror is gazed at, step S712 is entered.

[0344] In step S709, since the speed is not 0 and is less than or equal to the preset speed, and the gear is not P, the driver's eyes gaze at the inside rearview mirror. Therefore, the interaction content information is output during low-speed driving, and is related to the driver's gaze at the inside rearview mirror. At this time, "The vehicle is not stable, please wait until it is parked before getting off to avoid accidents" can be output, and then step S713 is entered to end the current processing process.

[0345] In step S710, since the vehicle speed is 0 and the driver's gaze is on the inside rearview mirror at this time, the interactive content information is output in the state that the vehicle is stopped running, and is related to the driver's gaze on the inside rearview mirror, and at this time, "temporary parking, please do not open the door to get off" can be output, and then step S713 is entered, and the current processing process is ended.

[0346] In step S711, since the vehicle speed is 0 and the driver's gaze is on the right rearview mirror at this time, the interactive content information is output in the state that the vehicle is stopped running, and is related to the driver's gaze on the right rearview mirror, and at this time, "there is a car behind, please pay attention to observation before opening the door, slowly open the door, and ensure safety" can be output, and then step S713 is entered, and the current processing process is ended.

[0347] In step S712, since the vehicle speed is 0 and the driver's gaze is on the left rearview mirror at this time, the interactive content information is output in the state that the vehicle is stopped running, and is related to the driver's gaze on the left rearview mirror, and at this time, "there is a car behind, please get off from the right door" can be output, and then step S713 is entered, and the current processing process is ended.

[0348] In the case that the passenger needs to provide the information communicated to the driver, with reference to Figure 8 In this specific example, in step S801, the second interactive content information initiated by the passenger is acquired, and step S802 is entered.

[0349] In step S802, the emergency type of the second interactive content information is determined, if it is an emergency type, step S805 is entered, if it is a non-emergency type, step S803 is entered.

[0350] In step S803, the third vehicle state information is acquired, and it is determined that it is in a safe driving state based on the third vehicle state information, if it is not in a safe driving state, step S804 is entered, if it is in a safe driving state, step S806 is entered.

[0351] In step S804, the acquired second interactive content information is stored to the notification center, and step S803 is returned, and the monitoring of whether it is in a safe driving state based on the third vehicle state information is continued.

[0352] In step S805, since the second interactive content information received at this time is of an emergency type, the display of the second interactive content information can be triggered in real time, and step S807 is entered.

[0353] In step S806, since the second interactive content information received at this time is of a non-emergency type, and the current has been in a safe driving state, the display of the second interactive content information can be triggered, and step S807 is entered.

[0354] In step S807, the second interaction content information is displayed through the vehicle display device, and the in-vehicle interaction assistance mode is activated.

[0355] It should be understood that, although each step in the flowchart involved in each of the above-described embodiments is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowchart involved in each of the above-described embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least some of the other steps or steps or stages in other steps.

[0356] Based on the same inventive concept, the embodiments of the present application also provide an in-vehicle interaction device for implementing the above-mentioned in-vehicle interaction method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more in-vehicle interaction device embodiments provided below can refer to the limitations of the in-vehicle interaction method described above, which will not be repeated here.

[0357] In one embodiment, an in-vehicle interaction device is provided, comprising: an assistance triggering module, an information acquisition module and an information output module, wherein:

[0358] The assistance triggering module is configured to receive a triggering instruction for triggering an in-vehicle interaction assistance mode, and enter the in-vehicle interaction assistance mode in response to the triggering instruction;

[0359] The information acquisition module is configured to acquire a first interaction operation instruction of a driver in the in-vehicle interaction assistance mode, and acquire first interaction content information of the first interaction operation instruction based on a first correspondence relationship between a preset interaction operation instruction and interaction content information;

[0360] The information output module is configured to output the first interaction content information.

[0361] In some embodiments, the assistance triggering module is configured to determine that the triggering instruction for triggering the in-vehicle interaction assistance mode is received in the case that an operation on a first preset button of the vehicle is received.

[0362] In some embodiments, the first preset button includes a voice trigger button; and the assistance triggering module is configured to determine that the triggering instruction for triggering the in-vehicle interaction assistance mode is received in the case that an operation on the voice trigger button of the vehicle is received.

[0363] In some embodiments, the auxiliary trigger module is configured to acquire a first image captured by a camera arranged at the driver seat; and determine that the trigger instruction for triggering the in-vehicle interactive assistance mode is received, in a case where a first body language action is recognized from the first image.

[0364] In some embodiments, the auxiliary trigger module is configured to enter the in-vehicle interactive assistance mode in response to the trigger instruction, and display a plurality of first preset dialogue information through the vehicle display device.

[0365] The information acquisition module is configured to receive and, in response to a selection operation on the displayed first preset dialogue information, determine the first preset dialogue information selected by the selection operation as the first interactive content information, wherein the first interactive operation instruction comprises a selection instruction corresponding to the selection operation on the displayed first preset dialogue information.

[0366] In some embodiments, the information acquisition module is configured to acquire first vehicle state information; filter one or more first preset dialogue information that is consistent with the first vehicle state information from a plurality of first preset dialogue information that is pre-set; and display the filtered first preset dialogue information through the vehicle display device.

[0367] In some embodiments, the information acquisition module is configured to acquire a gaze direction of the driver; and determine that the first interactive operation instruction of the driver is received, in a case where it is determined that the driver gazes at the rearview mirror based on the gaze direction of the driver.

[0368] In some embodiments, the information acquisition module is configured to determine that the first interactive operation instruction of the driver is received, in a case where it is determined that the driver gazes at the rearview mirror based on the gaze direction of the driver, and the second interactive operation instruction of the driver is received.

[0369] In some embodiments, the information acquisition module is configured to determine that the second interactive operation instruction of the driver is received, in a case where an operation on a second preset button of the vehicle is received.

[0370] In some embodiments, the information acquisition module is configured to acquire a second image captured by a camera arranged at the driver seat; and determine that the second interactive operation instruction of the driver is received, in a case where a second body language action is recognized from the second image.

[0371] In some embodiments, the information acquisition module is configured to acquire second preset dialogue information corresponding to a type of the rearview mirror based on the type of the rearview mirror that the driver gazes at, wherein the first correspondence relationship between the preset interactive operation instruction and the interactive content information comprises a correspondence relationship between the type of the rearview mirror and the second preset dialogue information; and determine the acquired second preset dialogue information as the first interactive content information.

[0372] In some embodiments, the information obtaining module is configured to obtain second preset dialogue information corresponding to the second interaction operation instruction, the first correspondence relationship between the preset interaction operation instruction and the interaction content information comprises a correspondence relationship between the second interaction operation instruction and the second preset dialogue information, and determine the obtained second preset dialogue information as the first interaction content information.

[0373] In some embodiments, the information obtaining module is configured to obtain second preset dialogue information corresponding to the second interaction operation instruction based on the type of the rearview mirror where the driver's gaze is fixed and the second interaction operation instruction, the first correspondence relationship between the preset interaction operation instruction and the interaction content information comprises a correspondence relationship between the type of the rearview mirror and the second interaction operation instruction and the second preset dialogue information, and determine the obtained second preset dialogue information as the first interaction content information.

[0374] In some embodiments, the information obtaining module is configured to obtain second vehicle state information, and obtain second preset dialogue information corresponding to the second vehicle state information based on the second vehicle state information and the type of the rearview mirror where the driver's gaze is fixed, the first correspondence relationship between the preset interaction operation instruction and the interaction content information comprises a correspondence relationship between the second vehicle state information and the type of the rearview mirror and the second preset dialogue information, and determine the obtained second preset dialogue information as the first interaction content information.

[0375] In some embodiments, the second vehicle state information comprises at least one of a vehicle speed and a vehicle gear.

[0376] In some embodiments, the information output module is configured to perform voice broadcast on the first interaction content information.

[0377] In some embodiments, the information output module is configured to display the first interaction content information on a display screen corresponding to the target passenger seat.

[0378] In some embodiments, the information output module is configured to display the first interaction content information in text on a display screen corresponding to the target passenger seat.

[0379] In some embodiments, the information output module is configured to obtain a sign language video of the first interaction content information and display the sign language video on a display screen corresponding to the target passenger seat.

[0380] In some embodiments, the information output module is configured to control the target passenger seat to vibrate based on the first interaction content information.

[0381] In some embodiments, the information output module is configured to control the ambient light of the target passenger seat to flash based on the first interaction content information.

[0382] In some embodiments, the device further comprises:

[0383] The display control module is configured to acquire the second interactive content information initiated by the passenger, and display the second interactive content information through the in-vehicle display device.

[0384] In some embodiments, the display control module is configured to acquire third vehicle state information, and display the second interactive content information through the in-vehicle display device in a case where it is determined based on the third vehicle state information that the vehicle is in a safe driving state.

[0385] In some embodiments, the display control module is configured to determine an urgency type of the second interactive content information, display the second interactive content information through the in-vehicle display device in real time in a case where the urgency type of the second interactive content information is urgent, and display the second interactive content information through the in-vehicle display device in a case where the urgency type of the second interactive content information is non-urgent and the vehicle is in a safe driving state based on the third vehicle state information.

[0386] In some embodiments, the third vehicle state information comprises a vehicle gear, and the display control module is configured to determine that the vehicle is in a safe driving state in a case where the vehicle gear is a first preset gear, the first preset gear comprising a P gear or an N gear.

[0387] In some embodiments, the third vehicle state information comprises a vehicle speed, and the display control module is configured to determine that the vehicle is in a safe driving state in a case where a duration in which the vehicle speed is less than or equal to a preset vehicle speed is greater than or equal to a first preset duration.

[0388] In some embodiments, the device further comprises:

[0389] The feedback output module is configured to acquire a third interactive operation instruction of the driver, acquire third interactive content information of the third interactive operation instruction based on a second correspondence relationship between the preset interactive operation instruction and the interactive content information, and output the third interactive content information.

[0390] In some embodiments, the feedback output module is configured to acquire recommended dialogue information generated by performing intent understanding on the second interactive content information, and display the recommended dialogue information through the in-vehicle display device, and acquire the third interactive operation instruction of the driver, including acquiring a selection operation on the displayed recommended dialogue information, the third interactive operation instruction comprising a selection instruction corresponding to the selection operation on the displayed recommended dialogue information, and the third interactive content information comprising the recommended dialogue information corresponding to the selection operation on the displayed recommended dialogue information.

[0391] In some embodiments, the feedback output module is configured to acquire a third image captured by a camera arranged at the driver's seat; and determine that the third interaction operation instruction of the driver is received, if a third body language action is recognized from the third image.

[0392] In some embodiments, the auxiliary trigger module is further configured to exit the in-vehicle interaction assistance mode, if the interaction operation instruction is not received within the second preset time length.

[0393] The modules in the in-vehicle interaction device described above can be implemented by software, hardware, or a combination thereof, in whole or in part. The modules described above can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the modules.

[0394] In one embodiment, an electronic device, which can be a terminal, is provided, and the computer device can be a terminal. The electronic device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The communication interface of the computer device is configured to perform wired or wireless communication with an external terminal. The wireless communication can be achieved by WIFI, mobile cellular network, NFC (Near Field Communication), or other technologies. The computer program is executed by the processor to implement an in-vehicle interaction method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball, or touchpad arranged on the shell of the computer device. The input device can also be an external keyboard, touchpad, or mouse, etc.

[0395] Those skilled in the art can understand that the structure of the electronic device described above is only part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0396] In some embodiments, an electronic device is provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the in-vehicle interaction method in any of the above embodiments.

[0397] In some embodiments, a computer readable storage medium is provided, having stored thereon a computer program, which, when executed by a processor, implements the steps of the in-vehicle interaction method of any of the above embodiments.

[0398] In one embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the in-vehicle interaction method of any of the above embodiments.

[0399] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium and can include the processes of the above-mentioned embodiments when executed. Any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of a non-volatile and volatile memory. The non-volatile memory can include a read-only memory (ROM), a magnetic tape, a floppy disk, a flash memory, an optical storage, a high-density embedded non-volatile memory, a resistive memory (ReRAM), a magnetoresistive random access memory (MRAM), a ferroelectric memory (FRAM), a phase change memory (PCM), a graphene memory, etc. The volatile memory can include a random access memory (RAM) or an external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, etc., and is not limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., and is not limited thereto.

[0400] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not contradict each other, they should be considered within the scope of the present disclosure.

[0401] The above embodiments only express several implementation manners of the present application, which are described in a more specific and detailed manner, but cannot be understood as a limitation on the patent scope of the present application. It should be noted that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, which all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A vehicle-mounted interaction method, characterized in that, The method includes: Receive a trigger command to activate the in-vehicle interactive assistance mode, and in response to the trigger command, enter the in-vehicle interactive assistance mode; In the vehicle-mounted interactive assistance mode, the driver's first interactive operation command is obtained, and based on the preset first correspondence between the interactive operation command and the interactive content information, the first interactive content information of the first interactive operation command is obtained. Output the information from the first interaction.

2. The method according to claim 1, characterized in that, The receiving of the trigger command to trigger the in-vehicle interactive assistance mode includes: Upon receiving an operation on the vehicle's first preset button, it is confirmed that a trigger command to activate the in-vehicle interactive assistance mode has been received.

3. The method according to claim 2, characterized in that, The first preset button includes a voice trigger button; The step of determining that a trigger command for triggering the in-vehicle interactive assistance mode has been received upon receiving an operation on a first preset button on the vehicle includes: Upon receiving an operation on the vehicle's voice-activated button, it is determined that the trigger command for triggering the in-vehicle interactive assistance mode has been received.

4. The method according to claim 1, characterized in that, The receiving of the trigger command to trigger the in-vehicle interactive assistance mode includes: Acquire the first image taken from the driver's seat; If a first body language action is identified from the first image, it is determined that a trigger command for triggering the in-vehicle interactive assistance mode has been received.

5. The method according to claim 1, characterized in that: The step of entering the vehicle interaction assistance mode in response to the trigger command includes: entering the vehicle interaction assistance mode in response to the trigger command, and displaying multiple first preset dialogue messages through the vehicle display device; The step of obtaining the driver's first interactive operation command and, based on a preset first correspondence between the interactive operation command and the interactive content information, obtaining the first interactive content information of the first interactive operation command includes: The system receives and responds to a selection operation on the displayed first preset dialogue information, and determines the first preset dialogue information selected by the selection operation as the first interactive content information, wherein the first interactive operation instruction includes a selection instruction corresponding to the selection operation on the displayed first preset dialogue information.

6. The method according to claim 5, characterized in that, The display of multiple first preset dialogue messages via the vehicle display device includes: Obtain the status information of the first vehicle; From a plurality of pre-set first preset dialogue messages, one or more first preset dialogue messages that match the first vehicle status information are selected; The selected first preset script information is displayed through the vehicle display device.

7. The method according to claim 1, characterized in that, The acquisition of the driver's first interactive operation command includes: Obtain the driver's gaze direction; If the driver's gaze is determined to be directed towards the rearview mirror based on the driver's gaze direction, the driver's first interactive operation command is confirmed to have been received.

8. The method according to claim 7, characterized in that, The step of determining that the driver's gaze is fixed on the rearview mirror based on the driver's gaze direction, includes: If it is determined that the driver's gaze is fixed on the rearview mirror based on the driver's gaze direction, and the driver's second interactive operation command is received, then the driver's first interactive operation command is determined to have been received.

9. The method according to claim 8, characterized in that, The method further includes any one of the following two: First item: Upon receiving an operation on the second preset button of the vehicle, it is determined that the driver's second interactive operation command has been received; Second item: Acquire a second image taken from the driver's seat; If a second body language action is identified from the second image, it is determined that a second interactive operation command from the driver has been received.

10. The method according to claim 7 or 8, characterized in that, The first interaction content information of the first interaction operation instruction is obtained based on a preset first correspondence between the interaction operation instruction and the interaction content information, including: Based on the type of rearview mirror that the driver is looking at, a second preset dialogue message corresponding to the type of rearview mirror is obtained. The first correspondence between the preset interactive operation instructions and interactive content information includes the correspondence between the type of rearview mirror and the second preset dialogue message. The acquired second preset dialogue information is determined as the first interactive content information.

11. The method according to claim 8 or 9, characterized in that, The first interaction content information of the first interaction operation instruction is obtained based on a preset first correspondence between the interaction operation instruction and the interaction content information, including at least one of the following two items: First item: Obtain second preset dialogue information corresponding to the second interactive operation instruction, wherein the first correspondence between the preset interactive operation instruction and the interactive content information includes the correspondence between the second interactive operation instruction and the second preset dialogue information; The acquired second preset dialogue information is determined as the first interactive content information; Second item: Based on the type of rearview mirror that the driver is looking at and the second interactive operation command, the corresponding second preset dialogue information is obtained. The first correspondence between the preset interactive operation command and the interactive content information includes the type of rearview mirror and the correspondence between the second interactive operation command and the second preset dialogue information. The acquired second preset dialogue information is determined as the first interactive content information.

12. The method according to any one of claims 7 to 9, characterized in that, The method of obtaining the first interactive content information of the first interactive operation instruction based on the preset correspondence between interactive operation instructions and interactive content information includes: Obtain the status information of the second vehicle; Based on the second vehicle status information and the type of rearview mirror that the driver is looking at, the corresponding second preset dialogue information is obtained. The first correspondence between the preset interactive operation instructions and interactive content information includes the correspondence between the second vehicle status information and the type of rearview mirror and the second preset dialogue information. The acquired second preset dialogue information is determined as the first interactive content information.

13. The method according to claim 12, characterized in that, include: The second vehicle status information includes at least one of vehicle speed and vehicle gear.

14. The method according to claim 1, characterized in that, The output of the first interactive content information includes at least one of the following: First item: The first interactive content information is read aloud via voice. Second item: The first interactive content information is displayed on the screen corresponding to the target passenger's seat.

15. The method according to claim 14, characterized in that, Displaying the first interactive content information on the display screen corresponding to the target passenger seat includes at least one of the following two items: First item: The first interactive content information is displayed as text on the screen corresponding to the target passenger's seat; Second item: The sign language video of the first interactive content information is obtained and displayed on the display screen corresponding to the target passenger's seat.

16. The method according to claim 14 or 15, characterized in that, The output of the first interactive content information further includes at least one of the following: First item: Based on the first interactive content information, control the target passenger seat to vibrate; Second item: Based on the first interactive content information, the ambient light of the target passenger seat is controlled to flash.

17. The method according to claim 1, characterized in that, The method further includes: Obtain information about the second interaction initiated by the passenger; The second interactive content information is displayed through the vehicle-mounted display device.

18. The method according to claim 17, characterized in that, The step of displaying the second interactive content information through an in-vehicle display device includes: Obtain the status information of the third vehicle; If the vehicle is determined to be in a safe driving state based on the third vehicle status information, the second interactive content information will be displayed through the in-vehicle display device.

19. The method according to claim 18, characterized in that, The step of displaying the second interactive content information through an in-vehicle display device includes: Determine the urgency type of the second interactive content information; When the urgency type of the second interactive content information is "urgent", the second interactive content information will be displayed in real time through the vehicle display device; If the emergency type of the second interactive content information is non-emergency, and if the vehicle is determined to be in a safe driving state based on the third vehicle status information, the second interactive content information will be displayed through the in-vehicle display device.

20. The method according to claim 18 or 19, characterized in that, The method further includes at least one of the following: First item: The third vehicle status information includes the vehicle gear position. When the vehicle gear position is a first preset gear, it is determined that the vehicle is in a safe driving state. The first preset gear includes P gear or N gear. Second item: The third vehicle status information includes vehicle speed. If the duration of the vehicle speed being less than or equal to a preset speed is greater than or equal to a first preset duration, it is determined that the vehicle is in a safe driving state.

21. The method according to claim 17, characterized in that, The method further includes: Obtain the driver's third interactive operation command, and based on the preset second correspondence between the interactive operation command and the interactive content information, obtain the third interactive content information of the third interactive operation command; Output the third interactive content information.

22. The method according to claim 21, characterized in that, The method further includes: The recommended dialogue information generated by understanding the intent of the second interactive content information is obtained, and the recommended dialogue information is displayed through the vehicle display device; The process of obtaining the driver's third interactive operation instruction includes: obtaining a selection operation on the displayed recommended dialogue information, wherein the third interactive operation instruction includes a selection instruction corresponding to the selection operation on the displayed recommended dialogue information, and the third interactive content information includes the recommended dialogue information selected by the selection operation on the displayed recommended dialogue information.

23. The method according to claim 21, characterized in that, The acquisition of the driver's third interactive operation command includes: Acquire a third image taken from the driver's seat; If a third body language action is identified from the third image, it is determined that a third interactive operation command from the driver has been received.

24. The method according to any one of claims 1 to 9, 14, 15, and 17 to 19, characterized in that, The method further includes: If no interactive operation command is received within the second preset time period, the vehicle interactive assistance mode will be exited.

25. The method according to any one of claims 17 to 19, characterized in that, The method further includes: Obtain operation suggestion information generated by semantic analysis of the second interaction content information; Vehicle control operations are performed based on the aforementioned operational suggestion information.

26. The method according to claim 25, characterized in that, Performing vehicle control operations based on the aforementioned operation suggestion information includes: The generated operation suggestion information is displayed on the vehicle display device; Upon receiving confirmation of the displayed operation suggestion information, the vehicle control operation corresponding to the operation suggestion information is executed.

27. An in-vehicle interactive device, characterized in that, The device includes: An auxiliary triggering module is used to receive a trigger command to trigger the in-vehicle interactive assistance mode, and in response to the trigger command, enter the in-vehicle interactive assistance mode; The information acquisition module is used to acquire the driver's first interactive operation command in the vehicle interactive assistance mode, and acquire the first interactive content information of the first interactive operation command based on the preset first correspondence between the interactive operation command and the interactive content information. The information output module is used to output the first interactive content information.

28. An electronic device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 26.

29. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 26.

30. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 26.

Images