Interaction method and device of vehicle terminal, electronic equipment, medium and vehicle

Abstract

The invention relates to an interaction method and device of a vehicle terminal, electronic equipment, a medium and a vehicle, and particularly relates to the technical field of intelligent vehicle control. Comprising the following steps: acquiring an in-vehicle audio signal and a user image sequence collected within a preset duration; performing audio detection on the in-vehicle audio signal based on an audio detection model to determine an audio confidence coefficient; and performing gesture detection on the user image sequence based on a gesture detection model, and determining a visual confidence coefficient, wherein the audio confidence coefficient represents the probability that the in-vehicle audio signal comprises a preset audio signal corresponding to a preset gesture, and the visual confidence coefficient represents the probability that the preset gesture is detected from the user image sequence; and issuing a preset control instruction corresponding to the preset gesture under the condition that the audio confidence coefficient and / or the visual confidence coefficient meet a preset condition. The embodiment of the invention is used for solving the problems that man-machine interaction of the vehicle terminal is not suitable for specific crowds and the accuracy is not high.

Description

Technical Field

[0001] The present disclosure relates to the field of intelligent automobile control technology, and in particular to an interaction method, device, electronic equipment, medium and vehicle of a vehicle terminal. Background Art

[0002] At present, the forms of human-computer interaction in the field of intelligent vehicle control include traditional physical buttons or touch interactions, as well as voice interactions. Physical buttons or touch interactions require the operation of specific buttons or touch areas. This interaction method brings inconvenience to the driver and affects safe driving. Voice interaction requires the user to say a set sentence, and does not take into account a specific group of people, which limits the scope of application to a certain extent. In addition, voice interaction is easily disturbed by environmental noise inside and outside the car, resulting in low accuracy in realizing vehicle control functions through voice interaction. Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides an interaction method, device, electronic device, medium and vehicle for a vehicle terminal, which improves the accuracy of human-computer interaction, has a wider audience range and stronger scene adaptability.

[0004] In order to achieve the above objectives, the technical solutions provided by the embodiments of the present disclosure are as follows:

[0005] In a first aspect, the present disclosure provides an interaction method of a vehicle terminal, comprising:

[0006] Acquire the in-vehicle audio signal and user image sequence collected within a preset time length;

[0007] Performing audio detection on the in-car audio signal based on the audio detection model to determine the audio confidence; and performing gesture detection on the user image sequence based on the gesture detection model to determine the visual confidence; wherein the audio confidence represents the probability that the in-car audio signal includes a preset audio signal corresponding to a preset gesture, and the visual confidence represents the probability of detecting the preset gesture from the user image sequence;

[0008] When the audio confidence and / or visual confidence meets the preset conditions, a preset control instruction corresponding to the preset gesture is issued.

[0009] As an optional implementation of the embodiment of the present disclosure, the preset condition includes any one of the following:

[0010] The audio confidence is greater than or equal to a first audio confidence threshold;

[0011] The visual confidence is greater than or equal to a first visual confidence threshold;

[0012] The audio confidence is greater than or equal to a second audio confidence threshold, and the visual confidence is greater than or equal to a second visual confidence threshold;

[0013] The first audio confidence threshold is greater than the second audio confidence threshold, and the first visual confidence threshold is greater than the second visual confidence threshold.

[0014] As an optional implementation of the embodiment of the present disclosure, before performing audio detection on the in-vehicle audio signal based on the audio detection model and determining the audio confidence, the method further includes:

[0015] Matching the in-car audio signal with the preset audio signal;

[0016] Perform audio detection on the in-car audio signal based on the audio detection model to determine the audio confidence, including:

[0017] When the in-car audio signal matches the preset audio signal, the in-car audio signal is input into the audio detection model to obtain the audio confidence output by the audio detection model. The audio detection model is a convergence model trained based on the sample audio signal.

[0018] As an optional implementation of the embodiment of the present disclosure, before performing audio detection on the in-vehicle audio signal based on the audio detection model and determining the audio confidence, the method further includes:

[0019] Performing filtering and signal enhancement processing on the in-vehicle audio signal to obtain a first audio signal;

[0020] Matching the first audio signal with a preset audio signal;

[0021] Perform audio detection on the in-car audio signal based on the audio detection model to determine the audio confidence, including:

[0022] When the first audio signal matches the preset audio signal, the first audio signal is input into an audio detection model to obtain an audio confidence level output by the audio detection model, where the audio detection model is a convergence model trained based on the sample audio signal.

[0023] As an optional implementation of the embodiment of the present disclosure, after matching the first audio signal with the preset audio signal, the method further includes:

[0024] In a case where the first audio signal matches the preset audio signal, extracting a second audio signal from the first audio signal, the second audio signal including an audio frame matching the preset audio signal;

[0025] Perform audio detection on the in-car audio signal based on the audio detection model to determine the audio confidence, including:

[0026] The second audio signal is input into the audio detection model to obtain the audio confidence output by the audio detection model.

[0027] As an optional implementation of the embodiment of the present disclosure, the gesture detection model includes: a finger key point detection sub-model and an action recognition sub-model;

[0028] Before performing gesture detection on the user image sequence based on the gesture detection model and determining the visual confidence, the method further includes:

[0029] Matching user image sequences with preset gesture images;

[0030] Perform gesture detection on the user image sequence based on the gesture detection model and determine the visual confidence, including:

[0031] When the user image sequence matches the preset gesture image, the user image sequence is input into the finger key point detection sub-model to obtain the finger key point sequence output by the finger key point detection sub-model, where the finger key point detection sub-model is a convergent model obtained by training based on the sample image sequence;

[0032] The finger key point sequence is input into the action recognition sub-model to obtain the visual confidence output by the action recognition sub-model. The action recognition sub-model is a convergent model trained based on the sample key point sequence.

[0033] As an optional implementation of the embodiment of the present disclosure, after matching the user image sequence with the preset gesture image, the method further includes:

[0034] When the user image sequence matches the preset gesture image, performing hand region detection on the user image sequence;

[0035] The user image sequence is cropped according to the detected hand area to obtain a gesture image sequence;

[0036] Perform gesture detection on the user image sequence based on the gesture detection model and determine the visual confidence, including:

[0037] Inputting the gesture image sequence into the finger key point detection sub-model to obtain the finger key point sequence corresponding to the gesture image sequence output by the finger key point detection sub-model;

[0038] The finger key point sequence is input into the action recognition sub-model to obtain the visual confidence output by the action recognition sub-model.

[0039] In a second aspect, the present disclosure provides an interactive device for a vehicle terminal, the device comprising:

[0040] An acquisition module, used to acquire in-vehicle audio signals and user image sequences collected within a preset time length;

[0041] An audio and video detection module, configured to perform audio detection on an in-car audio signal based on an audio detection model to determine an audio confidence; and to perform gesture detection on a user image sequence based on a gesture detection model to determine a visual confidence; wherein the audio detection model is configured to identify a preset audio signal corresponding to a preset gesture, the audio confidence indicates the probability that the in-car audio signal includes the preset audio signal, and the visual confidence indicates the probability that the preset gesture is detected from the user image sequence;

[0042] The processing module is used to issue a preset control instruction corresponding to a preset gesture when the audio confidence and / or visual confidence meets a preset condition.

[0043] As an optional implementation of the embodiment of the present disclosure, the preset condition includes any one of the following:

[0044] The audio confidence is greater than or equal to a first audio confidence threshold;

[0045] The visual confidence is greater than or equal to a first visual confidence threshold;

[0046] The audio confidence is greater than or equal to a second audio confidence threshold, and the visual confidence is greater than or equal to a second visual confidence threshold;

[0047] The first audio confidence threshold is greater than the second audio confidence threshold, and the first visual confidence threshold is greater than the second visual confidence threshold.

[0048] As an optional implementation of the embodiment of the present disclosure, the audio and video detection module is further used to: perform audio detection on the in-vehicle audio signal based on the audio detection model, and before determining the audio confidence, match the in-vehicle audio signal with a preset audio signal;

[0049] The audio and video detection module is specifically used to perform audio detection on the in-car audio signal based on the audio detection model to determine the audio confidence: when the in-car audio signal matches the preset audio signal, the in-car audio signal is input into the audio detection model to obtain the audio confidence output by the audio detection model. The audio detection model is a convergent model trained based on the sample audio signal.

[0050] As an optional implementation of the embodiment of the present disclosure, the audio and video detection module is further used to: perform audio detection on the in-vehicle audio signal based on the audio detection model, and before determining the audio confidence, filter and perform signal enhancement processing on the in-vehicle audio signal to obtain a first audio signal; match the first audio signal with a preset audio signal;

[0051] The audio and video detection module is specifically used for: in the process of performing audio detection on the in-vehicle audio signal based on the audio detection model and determining the audio confidence: when the first audio signal matches the preset audio signal, the first audio signal is input into the audio detection model to obtain the audio confidence output by the audio detection model, and the audio detection model is a convergent model obtained by training based on the sample audio signal.

[0052] As an optional implementation of the embodiment of the present disclosure, the audio and video detection module is further used to extract a second audio signal from the first audio signal after matching the first audio signal with the preset audio signal, if the first audio signal matches the preset audio signal, the second audio signal includes an audio frame matching the preset audio signal;

[0053] The audio and video detection module is specifically used to: in the process of performing audio detection on the in-vehicle audio signal based on the audio detection model to determine the audio confidence: input the second audio signal into the audio detection model to obtain the audio confidence output by the audio detection model.

[0054] As an optional implementation of the embodiment of the present disclosure, the gesture detection model includes: a finger key point detection sub-model and an action recognition sub-model;

[0055] The audio and video detection module is further used to match the user image sequence with a preset gesture image before performing gesture detection on the user image sequence based on the gesture detection model and determining the visual confidence;

[0056] The audio and video detection module is specifically used for: in the process of performing gesture detection on the user image sequence based on the gesture detection model and determining the visual confidence, when the user image sequence matches the preset gesture image, inputting the user image sequence into the finger key point detection sub-model to obtain the finger key point sequence output by the finger key point detection sub-model, and the finger key point detection sub-model is a convergent model obtained by training based on the sample image sequence;

[0057] The finger key point sequence is input into the action recognition sub-model to obtain the visual confidence output by the action recognition sub-model. The action recognition sub-model is a convergent model trained based on the sample key point sequence.

[0058] As an optional implementation of the embodiment of the present disclosure, the audio and video detection module is further used to perform hand area detection on the user image sequence after matching the user image sequence with the preset gesture image, if the user image sequence matches the preset gesture image;

[0059] The user image sequence is cropped according to the detected hand area to obtain a gesture image sequence;

[0060] The audio and video detection module is specifically used for: in the process of performing gesture detection on the user image sequence based on the gesture detection model and determining the visual confidence, the gesture image sequence is input into the finger key point detection sub-model to obtain the finger key point sequence corresponding to the gesture image sequence output by the finger key point detection sub-model; the finger key point sequence is input into the action recognition sub-model to obtain the visual confidence output by the action recognition sub-model.

[0061] In a third aspect, the present disclosure provides an electronic device comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the computer program implements the vehicle terminal interaction method as described in the first aspect or any optional embodiment thereof.

[0062] In a fourth aspect, the present disclosure provides a computer-readable storage medium, comprising: a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the interaction method of the vehicle terminal as described in the first aspect or any optional embodiment thereof is implemented.

[0063] In a fifth aspect, the present disclosure provides a vehicle, comprising: an interactive device of a vehicle terminal as described in the second aspect or any optional embodiment, or an electronic device as described in the third aspect.

[0064] In a fifth aspect, the present disclosure provides a computer program product, including: the computer program product includes a computer program, and when the computer program runs on a computer, the computer implements the interaction method of the vehicle terminal as described in the first aspect or any optional embodiment thereof.

[0065] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0066] The disclosed embodiment provides an interactive method, device, electronic device, medium and vehicle of a vehicle terminal, wherein the method first obtains an in-vehicle audio signal and a user image sequence collected within a preset time length, then performs audio detection on the in-vehicle audio signal based on an audio detection model, determines the audio confidence, so as to determine the probability that the in-vehicle audio signal includes a preset audio signal corresponding to a preset gesture, and performs gesture detection on the user image sequence based on a gesture detection model to determine the visual confidence, so as to determine the probability of detecting a preset gesture from the user image sequence, and further, when the audio confidence and visual confidence meet the preset conditions, a control instruction corresponding to the preset gesture is issued to perform the corresponding operation to realize the corresponding function. The disclosed embodiment detects the preset gesture by combining audio and video, which can eliminate the interference of the ambient noise inside and outside the vehicle on the audio detection, accurately detect whether the user makes a preset gesture, and realize the corresponding function in response to the detection of the preset gesture. The vehicle terminal interactive mode is novel, which improves the accuracy of human-computer interaction, and has a wider audience range and stronger scene adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0068] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0069] Figure 1 A schematic diagram of a flow chart of an interactive method of a vehicle terminal according to an embodiment of the present disclosure;

[0070] Figure 2 A schematic diagram of the process of audio and video detection provided by an embodiment of the present disclosure;

[0071] Figure 3 A schematic diagram of the structure of a vehicle trunk opening device provided in an embodiment of the present disclosure;

[0072] Figure 4 The present invention is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0073] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0074] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0075] In order to solve some or all of the technical problems existing in the related art, the embodiment of the present disclosure provides an interactive method, device, electronic device, medium and vehicle of a vehicle terminal, wherein the method first obtains the in-vehicle audio signal and user image sequence collected within a preset time length, then performs audio detection on the in-vehicle audio signal based on the audio detection model, determines the audio confidence, so as to determine the probability that the in-vehicle audio signal includes the preset audio signal corresponding to the preset gesture, and performs gesture detection on the user image sequence based on the gesture detection model to determine the visual confidence, so as to determine the probability of detecting the preset gesture from the user image sequence, and further, when the audio confidence and visual confidence meet the preset conditions, the control instruction corresponding to the preset gesture is issued to perform the corresponding operation to realize the corresponding function. The embodiment of the present disclosure detects the preset gesture by combining audio and video, which can eliminate the interference of the ambient noise inside and outside the vehicle on the audio detection, accurately detect whether the user makes the preset gesture, and realize the corresponding function in response to the detection of the preset gesture. The vehicle terminal interaction mode is novel, the accuracy of human-computer interaction is improved, and the audience range is wider and the scene adaptability is stronger.

[0076] An interactive method of a vehicle terminal provided in an embodiment of the present disclosure can be implemented by an interactive device of the vehicle terminal or an electronic device, and the electronic device includes but is not limited to a vehicle terminal, a server, a personal computer, a laptop computer, a tablet computer, a smart phone, etc. The operating system of the electronic device may include Android, a mobile operating system (iOS) developed by Apple, an operating system (Windows) developed by Microsoft Corporation of the United States, etc., and the embodiments of the present disclosure are not limited to this. The electronic device can be operated alone to implement the present disclosure, or it can be connected to a network and implement the present disclosure through interactive operations with other computer devices in the network. Among them, the network where the electronic device is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN) network, etc.

[0077] It should be noted that the protection scope of the interactive method of a vehicle terminal described in an embodiment of the present disclosure is not limited to the execution order of the steps listed in this embodiment, and all solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present disclosure are included in the protection scope of the present disclosure.

[0078] like Figure 1 As shown, Figure 1The flowchart of an interactive method of a vehicle terminal according to an embodiment of the present disclosure is shown in FIG. 1 . The method can be executed by an interactive device of the vehicle terminal, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 1 As shown, the method mainly includes the following steps S101 to S103:

[0079] S101: Acquire an in-vehicle audio signal and a user image sequence collected within a preset time length.

[0080] In some embodiments, the in-vehicle audio signal is acquired through a distributed sound collector, which may be a distributed microphone or a microphone array, and the distributed sound collector is deployed at different locations in the vehicle cabin, and the audio signals at different locations in the vehicle are synchronously collected through the distributed microphones.

[0081] In some embodiments, a user image sequence is acquired through an image collector, and the user image sequence includes user images corresponding to multiple preset time points within a preset time period. The image collector can be a camera-based passenger monitoring system, and the passenger monitoring system (Occupancy Monitoring System, OMS) can use the cabin camera to perceive the entire cabin area, detect the occupants and perform high-precision classification and biometric detection on them, including capturing and tracking postures and movements.

[0082] S102, performing audio detection on the in-vehicle audio signal based on the audio detection model to determine the audio confidence; and performing gesture detection on the user image sequence based on the gesture detection model to determine the visual confidence.

[0083] Among them, the audio detection model is used to identify the preset audio signal corresponding to the preset gesture, and is a converged model pre-trained based on the sample audio signal; the audio confidence represents the probability that the preset audio signal corresponding to the preset gesture is included in the in-vehicle audio signal; the finger key point detection sub-model is used to identify the finger key points in the image, and is a converged model pre-trained based on the sample image sequence; the action recognition sub-model is used to identify the action corresponding to the key point, and is a converged model pre-trained based on the sample key point sequence; the visual confidence represents the probability of detecting the preset gesture in the user image sequence.

[0084] In some embodiments, the preset gesture may be "snapping fingers", etc., where the user uses the middle finger of the left or right hand to quickly hit the muscle at the base of the thumb, and the snapping action will produce a "crackling" sound caused by the vibration of the sound cavity. Correspondingly, the preset audio signal corresponding to the preset gesture may be the crackling sound of snapping fingers. Due to the influence of factors such as the strength of snapping fingers, the crackling sound produced by snapping fingers also has different acoustic characteristics such as volume and timbre. In the embodiments of the present disclosure, there may be multiple preset audio signals.

[0085] The following will explain the audio detection process and gesture detection process respectively:

[0086] (1) Audio detection

[0087] In some embodiments, in the process of performing audio detection on an in-car audio signal based on an audio detection model to determine the audio confidence, it includes: matching the in-car audio signal with a preset audio signal, and when the in-car audio signal matches the preset audio signal, inputting the in-car audio signal into the audio detection model to obtain the audio confidence output by the audio detection model, wherein the audio detection model is a convergent model trained based on a sample audio signal.

[0088] Specifically, before performing audio detection on the in-car audio signal based on the audio detection model and determining the audio confidence, the in-car audio signal is first matched with a preset audio signal, that is, multiple preset audio signals are used as templates, and the in-car audio signal is used to perform similarity matching with them. If the in-car audio signal matches any one of the preset audio signals, it can be preliminarily determined that there is a preset audio signal in the in-car audio signal, indicating that it is preliminarily recognized that the user may make a snapping action.

[0089] Based on the above embodiment, audio detection is performed on the in-car audio signal based on the audio detection model to determine the audio confidence, including: inputting the in-car audio signal into the audio detection model to obtain the audio confidence output by the audio detection model, so as to further accurately identify the finger snapping action.

[0090] It should be noted that the audio detection model in the above embodiment is used for audio event detection (AudioEventDetect, AED), which can automatically detect non-voice events, such as the sound of snapping fingers. The training process of the audio detection model includes: first, obtaining a sample audio signal, annotating the sample audio signal, and annotating the type of the sample audio signal. For example, if the sample audio signal is a sound of snapping fingers, it is annotated as snapping fingers. Optionally, the sample audio signal may also include other audio signals other than the preset audio signal corresponding to the preset gesture as negative samples to improve the robustness of the model, such as the sound of closing a door, the sound of joint friction, the sound of applause, the sound of tongue clicking, etc.; then, the sample audio signal is input into a pre-trained model, and the pre-trained model predicts the type of the sample audio signal to obtain the prediction confidence output by the pre-trained model. The prediction confidence indicates the possibility of the type of the sample audio signal predicted by the pre-trained model; further, the pre-trained model is trained according to the prediction confidence and the annotation, as well as the preset loss function. When the preset loss function value is greater than the loss threshold, it indicates that the model has not converged. Then, the pre-trained model parameters are adjusted until the preset loss function value is less than or equal to the preset loss threshold, thereby obtaining a converged audio detection model.

[0091] In some embodiments, before performing audio detection on the in-vehicle audio signal based on the audio detection model to determine the audio confidence, it also includes: filtering and signal enhancement processing on the in-vehicle audio signal to obtain a first audio signal; matching the first audio signal with a preset audio signal; performing audio detection on the in-vehicle audio signal based on the audio detection model to determine the audio confidence, including: when the first audio signal matches the preset audio signal, inputting the first audio signal into the audio detection model to obtain the audio confidence output by the audio detection model.

[0092] Specifically, in the process of performing audio detection on the in-car audio signal and determining the audio confidence, on the basis of the aforementioned embodiment, after obtaining the in-car audio signal, the in-car audio signal is first filtered and signal enhanced to remove irrelevant audio signals such as noise signals and human voice signals in the in-car audio signal, and the effective audio signal is amplified and enhanced to obtain a first audio signal to facilitate subsequent accurate identification; further, in the process of performing audio detection through an audio detection model, the first audio signal is used as the input of the audio detection model to identify whether a preset audio signal exists in the first audio signal, and obtain the audio confidence output by the audio detection model.

[0093] In the above embodiment, before using the audio detection model for detection, the in-vehicle audio signal is firstly denoised and enhanced to eliminate noise interference, which is beneficial to improving the accuracy of audio detection.

[0094] In some embodiments, after matching the first audio signal with the preset audio signal, it also includes: when the first audio signal matches the preset audio signal, extracting a second audio signal from the first audio signal, the second audio signal includes an audio frame that matches the preset audio signal; performing audio detection on the in-vehicle audio signal based on an audio detection model to determine the audio confidence, including: inputting the second audio signal into the audio detection model to obtain the audio confidence output by the audio detection model.

[0095] Specifically, when the first audio signal matches the preset audio signal, it indicates that a preliminary determination is made that the first audio signal includes the preset audio signal, such as a preliminary identification of the sound of snapping fingers, and then the audio frame matching the first audio signal and the preset audio signal is determined to extract the second audio signal. The second audio signal is a further processing of the in-car audio signal, which more accurately locates the position where the in-car audio signal includes the preset audio signal. In the process of performing audio detection through the audio detection model, the extracted second audio signal is input into the audio detection model to obtain the audio confidence of the target audio signal output.

[0096] In some embodiments, in the process of extracting the second audio signal from the first audio signal, first, in the first audio signal, an initial audio frame and a terminal audio frame that match the preset audio signal are determined, and then audio frames in the range corresponding to the initial audio frame to the terminal audio frame are extracted to obtain the second audio signal. Optionally, after determining the initial audio frame and the terminal audio frame that match the preset audio signal, based on the preset duration, audio frames in the preset duration before the initial audio frame, from the initial audio frame to the terminal audio frame, and the preset duration after the terminal audio frame are extracted. It can be understood that the second audio signal includes all audio frames corresponding to the range from the audio frame corresponding to the preset duration before the initial audio frame to the audio frame corresponding to the preset duration after the terminal audio frame, thereby appropriately expanding the audio signal range, avoiding audio clipping and missing important information, and not affecting audio detection.

[0097] Exemplarily, the preset duration is Δts, the time corresponding to the initial audio frame is t0, and the time corresponding to the end audio frame is t1, then the audio frames within the period of [t0-Δt, t1+Δt] are extracted as the second audio frames.

[0098] In the above embodiment, a specific target audio signal is extracted from the collected in-vehicle audio signal, so as to use the audio detection model to more accurately detect whether the target audio signal is a preset audio signal corresponding to a preset gesture.

[0099] In some embodiments, by performing audio detection on the in-car audio signal to determine the audio confidence, a target audio signal corresponding to the audio confidence can also be obtained. The target audio signal can be the original in-car audio signal in the above embodiment, or the preliminarily processed first audio signal, or the further processed second audio signal. It can be understood that the target audio signal can represent the audio signal of the user snapping his fingers that is successfully detected from the in-car audio signal. It should be noted that the target audio signal may not exist in the in-car audio signal, that is, the sound of snapping fingers is not detected from the in-car audio signal.

[0100] (2) Gesture Detection

[0101] In some embodiments, before performing gesture detection on a user image sequence based on a gesture detection model to determine the visual confidence, it also includes: matching the user image sequence with a preset gesture image; performing gesture detection on the user image sequence based on the gesture detection model to determine the visual confidence, including: when the user image sequence matches the preset gesture image, inputting the user image sequence into a finger key point detection sub-model to obtain a finger key point sequence output by the finger key point detection sub-model, wherein the finger key point detection sub-model is a convergent model trained based on a sample image sequence; inputting the finger key points into an action recognition sub-model to obtain the visual confidence output by the action recognition sub-model, wherein the action recognition sub-model is a convergent model trained based on a sample key point sequence.

[0102] Specifically, before performing gesture detection on a user image sequence based on a gesture detection model and determining the visual confidence, the user image sequence is first matched with a preset gesture image, where the preset gesture image is an image corresponding to a preset gesture, such as a snapping finger image. The preset gesture image is used as a template to perform similarity matching with the user image sequence to preliminarily determine whether a preset gesture exists in the user image sequence. The number of preset gesture images may be multiple. If the user image sequence is similar to any one of the preset gesture images, it is determined that the user image sequence matches the preset gesture image. Furthermore, in the process of performing gesture detection through the gesture detection model, the user image sequence is specifically input into a finger key point detection sub-model to obtain a finger key point sequence output by the finger key point detection sub-model, and then the finger key point sequence is input into an action recognition sub-model to obtain a visual confidence output by the action recognition sub-model to more accurately identify whether a preset gesture exists in the user image sequence.

[0103] In the above embodiment, the finger key point detection submodel and the action recognition submodel are both pre-trained convergent models. The training process of the finger key point detection submodel includes: firstly obtaining a sample image sequence, which can be a hand image sequence, and marking the finger key points therein on the basis of the sample image sequence as the true value of the model training; then inputting the sample image sequence into the constructed initial key point detection model to obtain the predicted finger key point sequence output by the initial key point detection model, and then calculating the loss function value according to the predicted gesture key point sequence, the true value and the preset loss function to characterize the gap between the model prediction value and the true value, so that when the loss function value is greater than the loss threshold, the model parameters of the initial key point detection model are adjusted until the loss function value is less than or equal to the loss threshold, and a converged finger key point detection submodel is obtained.

[0104] The training process of the action recognition sub-model includes: obtaining a sample key point sequence, which can be a hand key point sequence, marking the gesture type based on the sample key point sequence as the true value for model training, and then inputting the sample key point sequence into the constructed initial action recognition sub-model, so that the initial action recognition sub-model predicts the gesture type of the sample key point sequence, and obtains the prediction confidence output by the initial action recognition sub-model. The prediction confidence indicates the possibility of what type of gesture the sample key point sequence is. The loss function value is calculated based on the prediction confidence, the true value and the preset loss function, so that when the loss function value is greater than the loss threshold, the model parameters of the initial key point detection model are adjusted until the loss function value is less than or equal to the loss threshold, and a converged action recognition sub-model is obtained.

[0105] In the above embodiment, after matching the user image sequence with the preset gesture image, it also includes: when the user image sequence matches the preset finger image, performing hand area detection on the user image sequence; cropping the user image sequence according to the detected finger area to obtain a gesture image sequence; performing gesture detection on the user image sequence based on the gesture detection model to determine the visual confidence, including: inputting the gesture image sequence into the finger key point detection sub-model to obtain a finger key point sequence corresponding to the gesture image sequence output by the finger key point detection sub-model.

[0106] Specifically, when the user image sequence matches the preset gesture image, it means that the user image sequence is preliminarily determined to include the preset gesture, and then the finger key point detection sub-model is used to detect the finger key point sequence. Before this process, the user image sequence is firstly identified and extracted for the region of interest. In the embodiment of the present disclosure, the region of interest is the hand region. Optionally, the user image sequence is firstly detected for the hand region. When the user image sequence includes the hand region, the corresponding hand region is cropped from the user image sequence to obtain a gesture image sequence, and then the gesture image sequence is input into the finger key point detection sub-model to obtain the finger key point sequence output by the finger key point detection sub-model.

[0107] The above-mentioned embodiment eliminates the interference of irrelevant elements in the user image sequence on gesture detection by identifying and extracting the gesture area of ​​the user image sequence. The gesture information contained in the gesture image sequence input into the finger key point detection submodel accounts for a larger proportion, which can improve the accuracy and efficiency of finger key point detection.

[0108] In some embodiments, by performing gesture detection on a user image sequence based on a finger key point detection submodel and an action recognition submodel and determining visual confidence, a corresponding target image sequence can also be obtained. The target image sequence can be the original user image sequence in the above embodiment, or it can be a gesture image sequence extracted by hand region recognition. It can be understood that the target image sequence represents an image sequence in which the user performs a finger snapping action. It should be noted that the user image sequence may not include the target image sequence, that is, the finger snapping action is not detected from the user image sequence.

[0109] The above embodiments can be combined arbitrarily. For example, Figure 2 As shown, Figure 2 A schematic diagram of a flow chart of audio and video detection provided in an embodiment of the present disclosure, in Figure 1 Based on the process shown, before executing step S102, the following steps S20a-S22a and S20b-22b may be included:

[0110] S20a, filtering and signal enhancement processing are performed on the in-vehicle audio signal to obtain an intermediate audio signal;

[0111] S21a, matching the intermediate audio signal with a preset audio signal.

[0112] S22a: When the intermediate audio signal matches the preset audio signal, extract the target audio signal from the intermediate audio signal.

[0113] Specifically, when performing audio detection on the in-car audio signal based on the audio detection model to determine the audio confidence, based on the above steps S20a to S22a, the target audio signal is input into the audio detection model to obtain the audio confidence output by the audio detection model.

[0114] In the process of executing the above steps S20a to S22a, the following steps may be executed simultaneously:

[0115] S20b, matching the user image sequence with the preset gesture image.

[0116] S21b: When the user image sequence matches the preset gesture image, perform hand region detection on the user image sequence.

[0117] S22b, cropping the user image sequence according to the detected hand area to obtain a gesture image sequence.

[0118] Specifically, when performing gesture detection on a user image sequence based on a gesture detection model to determine the visual confidence, based on the above steps S20b to 22b, the gesture image sequence is input into the finger key point detection sub-model to obtain a finger key point sequence corresponding to the gesture image sequence output by the finger key point detection sub-model; then, the finger key point sequence is input into the action recognition sub-model to obtain the visual confidence output by the action recognition sub-model.

[0119] It should be noted that Figure 2 The specific implementation of each step shown is the same as or similar to that of the aforementioned embodiment, and is not elaborated herein in the present disclosure.

[0120] like Figure 2 The steps shown combine the audio detection of the in-car audio signal with the gesture detection of the user image sequence, complementing each other's advantages and eliminating the interference of the environmental noise inside and outside the car on the detection results. It can accurately identify whether the user makes a preset gesture through the combination of audio and video. It is suitable for control scenarios where the user does not want or cannot interact with voice, and improves the human-computer interaction performance.

[0121] S103: When the audio confidence and / or the visual confidence meets a preset condition, a preset control instruction corresponding to the preset gesture is issued.

[0122] The preset control instructions corresponding to the preset gestures include but are not limited to: turning on / off the interior ambient light, playing / pausing multimedia, turning on / off the air conditioner, etc. The preset control instructions indicate which component or device to operate, and the corresponding vehicle terminal sends the preset control instructions to the corresponding component or device.

[0123] In some embodiments, the preset condition includes any one of the following: (A) the audio confidence is greater than or equal to a first audio confidence threshold; (B) the visual confidence is greater than or equal to a first visual confidence threshold; (C) the audio confidence is greater than or equal to a second audio confidence threshold, and the visual confidence is greater than or equal to a second visual confidence threshold. Wherein, the first audio confidence threshold is greater than the second audio confidence threshold, and the first visual confidence threshold is greater than the second visual confidence threshold.

[0124] The following describes the situation where the audio confidence and / or visual confidence meets any of the above preset conditions:

[0125] (A) The audio confidence is greater than or equal to a first audio confidence threshold.

[0126] In some embodiments, after executing step S102 to perform audio detection and gesture detection, a target audio signal is obtained but a target image sequence is not obtained, then the audio confidence corresponding to the target audio signal is compared with a first audio confidence threshold, where the first audio confidence threshold is a larger value to ensure that when gesture detection fails to identify that the user has snapped their fingers, it is still possible to accurately identify whether the user has snapped their fingers through the audio detection result; when the audio confidence is greater than or equal to the first audio confidence threshold, it indicates that the in-vehicle audio signal includes a preset audio signal corresponding to a preset gesture, and the user intends to implement a control function through a preset gesture, and therefore a preset control instruction corresponding to the preset gesture is issued.

[0127] Exemplarily, the preset gesture is snapping fingers, and the preset control instruction corresponding to the preset gesture is turning on the interior atmosphere light. When it is detected that the audio confidence is greater than or equal to the first audio confidence threshold, the vehicle terminal sends a control instruction to the interior atmosphere light to control the turning on of the interior atmosphere light.

[0128] (B) The visual confidence is greater than or equal to the first visual confidence threshold.

[0129] In some embodiments, after executing step S102 to perform audio detection and gesture detection, a target image sequence is obtained but a target audio signal is not obtained, then the visual confidence corresponding to the target image sequence is compared with a first visual confidence threshold, where the first visual confidence threshold is a larger value to ensure that when the sound of snapping fingers is not identified through audio detection, it is still possible to accurately identify whether the user has snapped their fingers through the gesture detection results; when the video confidence is greater than or equal to the first visual confidence threshold, it indicates that the user image sequence includes a preset gesture and the user intends to implement a control function through the preset gesture, and therefore a preset control instruction corresponding to the preset gesture is issued.

[0130] (C) The audio confidence is greater than or equal to a second audio confidence threshold, and the visual confidence is greater than or equal to a second visual confidence threshold.

[0131] In some embodiments, after executing step S102 to perform audio detection and gesture detection, if a target audio signal and a target image sequence are obtained, the audio confidence corresponding to the target audio signal is compared with the second audio confidence threshold, and the visual confidence corresponding to the target image sequence is compared with the second visual confidence threshold. The second audio confidence threshold and the second visual confidence threshold are both small values. It can be understood that when the audio detects a finger snapping sound and the gesture detects a finger snapping action, the judgment threshold is lowered to avoid misjudgment of the recognition result; when the audio confidence is greater than or equal to the second audio confidence threshold, and the visual confidence is greater than or equal to the second visual confidence threshold, it indicates that there is a high possibility of a finger snapping sound in the in-car audio signal and a high possibility of a finger snapping action in the user image sequence. After comprehensive detection and judgment by combining audio and vision, the detection of the preset gesture is more accurate, and the preset gesture control realizes the corresponding function more accurately.

[0132] The above-mentioned embodiment analyzes the audio detection and gesture detection results and makes judgments according to different situations, thereby realizing multi-dimensional detection of preset gestures, improving the accuracy of gesture detection, and thus improving the accuracy of gesture control.

[0133] In summary, the disclosed embodiment provides an interactive method for a vehicle terminal, which first obtains an in-vehicle audio signal and a user image sequence collected within a preset time length, then performs audio detection on the in-vehicle audio signal based on an audio detection model, determines the audio confidence, so as to determine the probability that the in-vehicle audio signal includes a preset audio signal corresponding to a preset gesture, and performs gesture detection on the user image sequence based on a gesture detection model to determine the visual confidence, so as to determine the probability of detecting a preset gesture from the user image sequence, and further, when the audio confidence and the visual confidence meet the preset conditions, a control instruction corresponding to the preset gesture is issued to perform the corresponding operation to implement the corresponding function. The disclosed embodiment detects the preset gesture by combining audio and video, which can eliminate the interference of the ambient noise inside and outside the vehicle on the audio detection, accurately detect whether the user makes a preset gesture, and implement the corresponding function in response to the detection of the preset gesture. The vehicle terminal interactive method is novel, which improves the accuracy of human-computer interaction, and has a wider audience range and stronger scene adaptability.

[0134] like Figure 3 As shown, Figure 3 A schematic diagram of a vehicle trunk opening device provided in an embodiment of the present disclosure, the device comprising:

[0135] In a second aspect, the present disclosure provides an interactive device for a vehicle terminal, the device comprising:

[0136] An acquisition module 301 is used to acquire an in-vehicle audio signal and a user image sequence collected within a preset time length;

[0137] The audio and video detection module 302 is used to perform audio detection on the in-car audio signal based on the audio detection model to determine the audio confidence; and perform gesture detection on the user image sequence based on the gesture detection model to determine the visual confidence; wherein the audio detection model is used to identify the preset audio signal corresponding to the preset gesture, the audio confidence indicates the probability that the in-car audio signal includes the preset audio signal, and the visual confidence indicates the probability of detecting the preset gesture from the user image sequence;

[0138] The processing module 303 is used to issue a preset control instruction corresponding to a preset gesture when the audio confidence level and / or the visual confidence level meets a preset condition.

[0139] As an optional implementation of the embodiment of the present disclosure, the preset condition includes any one of the following:

[0140] The audio confidence is greater than or equal to a first audio confidence threshold;

[0141] The visual confidence is greater than or equal to a first visual confidence threshold;

[0142] The audio confidence is greater than or equal to a second audio confidence threshold, and the visual confidence is greater than or equal to a second visual confidence threshold;

[0143] The first audio confidence threshold is greater than the second audio confidence threshold, and the first visual confidence threshold is greater than the second visual confidence threshold.

[0144] As an optional implementation of the embodiment of the present disclosure, the audio and video detection module 302 is further used to: perform audio detection on the in-vehicle audio signal based on the audio detection model, and before determining the audio confidence, match the in-vehicle audio signal with a preset audio signal;

[0145] The audio and video detection module 302 is specifically used to perform audio detection on the in-car audio signal based on the audio detection model to determine the audio confidence: when the in-car audio signal matches the preset audio signal, the in-car audio signal is input into the audio detection model to obtain the audio confidence output by the audio detection model, and the audio detection model is a convergent model trained based on the sample audio signal.

[0146] As an optional implementation of the embodiment of the present disclosure, the audio and video detection module 302 is further used to: perform audio detection on the in-vehicle audio signal based on the audio detection model, and before determining the audio confidence, filter and enhance the signal of the in-vehicle audio signal to obtain a first audio signal; match the first audio signal with a preset audio signal;

[0147] The audio and video detection module 302 is specifically used for: in the process of performing audio detection on the in-vehicle audio signal based on the audio detection model to determine the audio confidence: when the first audio signal matches the preset audio signal, the first audio signal is input into the audio detection model to obtain the audio confidence output by the audio detection model, and the audio detection model is a convergent model trained based on the sample audio signal.

[0148] As an optional implementation of the embodiment of the present disclosure, the audio and video detection module 302 is further configured to, after matching the first audio signal with the preset audio signal, extract a second audio signal from the first audio signal when the first audio signal matches the preset audio signal, wherein the second audio signal includes an audio frame matching the preset audio signal;

[0149] The audio and video detection module 302 is specifically used to: in the process of performing audio detection on the in-vehicle audio signal based on the audio detection model to determine the audio confidence: input the second audio signal into the audio detection model to obtain the audio confidence output by the audio detection model.

[0150] As an optional implementation of the embodiment of the present disclosure, the gesture detection model includes: a finger key point detection sub-model and an action recognition sub-model;

[0151] The audio and video detection module 302 is further used to perform gesture detection on the user image sequence based on the gesture detection model, and before determining the visual confidence, match the user image sequence with a preset gesture image;

[0152] The audio and video detection module 302 is specifically used to: in the process of performing gesture detection on the user image sequence based on the gesture detection model and determining the visual confidence, when the user image sequence matches the preset gesture image, input the user image sequence into the finger key point detection sub-model to obtain the finger key point sequence output by the finger key point detection sub-model, and the finger key point detection sub-model is a convergent model obtained by training based on the sample image sequence;

[0153] The finger key point sequence is input into the action recognition sub-model to obtain the visual confidence output by the action recognition sub-model. The action recognition sub-model is a convergent model trained based on the sample key point sequence.

[0154] As an optional implementation of the embodiment of the present disclosure, the audio and video detection module 302 is further used to perform hand area detection on the user image sequence after matching the user image sequence with the preset gesture image, if the user image sequence matches the preset gesture image;

[0155] The user image sequence is cropped according to the detected hand area to obtain a gesture image sequence;

[0156] The audio and video detection module 302 is specifically used for: in the process of performing gesture detection on the user image sequence based on the gesture detection model and determining the visual confidence, the gesture image sequence is input into the finger key point detection sub-model to obtain the finger key point sequence corresponding to the gesture image sequence output by the finger key point detection sub-model; the finger key point sequence is input into the action recognition sub-model to obtain the visual confidence output by the action recognition sub-model.

[0157] like Figure 4 As shown, Figure 4 This is a structural diagram of an electronic device described in an embodiment of the present disclosure. The embodiment of the present disclosure provides an electronic device, which includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the interactive method of the vehicle terminal in the above method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0158] An embodiment of the present disclosure provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, each process of the interaction method of the vehicle terminal in the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0159] The present disclosure provides a vehicle, which includes: an interactive device of the vehicle terminal as described above, or an electronic device as described above, and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0160] The computer readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0161] An embodiment of the present disclosure provides a computer program product, which stores a computer program. When the computer program is executed by a processor, each process of the interaction method of the vehicle terminal in the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0162] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0163] In several embodiments provided by the present disclosure, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0164] In the present disclosure, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0165] In the present disclosure, memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0166] In the present disclosure, computer-readable media include permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0167] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0168] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An interactive method for a vehicle terminal, characterized in that: include: Acquire the in-vehicle audio signal and user image sequence collected within a preset time length; Performing audio detection on the in-vehicle audio signal based on an audio detection model to determine audio confidence; and, performing gesture detection on the user image sequence based on gesture detection to determine visual confidence; wherein the audio detection model is used to identify a preset audio signal corresponding to a preset gesture, the audio confidence indicates the probability that the in-car audio signal includes the preset audio signal, and the visual confidence indicates the probability of detecting the preset gesture from the user image sequence; When the audio confidence and / or the visual confidence meets a preset condition, a preset control instruction corresponding to the preset gesture is issued.

2. The method according to claim 1, characterized in that The preset condition includes any one of the following: The audio confidence is greater than or equal to a first audio confidence threshold; The visual confidence is greater than or equal to a first visual confidence threshold; The audio confidence is greater than or equal to a second audio confidence threshold, and the visual confidence is greater than or equal to a second visual confidence threshold; The first audio confidence threshold is greater than the second audio confidence threshold, and the first visual confidence threshold is greater than the second visual confidence threshold.

3. The method according to claim 1, characterized in that Before performing audio detection on the in-vehicle audio signal based on the audio detection model and determining the audio confidence, the method further includes: Matching the in-car audio signal with a preset audio signal; The performing audio detection on the in-vehicle audio signal based on the audio detection model to determine the audio confidence level includes: When the in-car audio signal matches a preset audio signal, the in-car audio signal is input into the audio detection model to obtain the audio confidence output by the audio detection model, where the audio detection model is a convergent model trained based on a sample audio signal.

4. The method according to claim 1, characterized in that: Before performing audio detection on the in-vehicle audio signal based on the audio detection model and determining the audio confidence, the method further includes: Performing filtering and signal enhancement processing on the in-vehicle audio signal to obtain a first audio signal; Matching the first audio signal with a preset audio signal; performing audio detection on the in-vehicle audio signal based on an audio detection model to determine the audio confidence level includes: In the case where the first audio signal matches the preset audio signal, the first audio signal is input into the audio detection model to obtain the audio confidence output by the audio detection model, where the audio detection model is a convergent model trained based on a sample audio signal.

5. The method according to claim 4, characterized in that After matching the first audio signal with a preset audio signal, the method further includes: In a case where the first audio signal matches the preset audio signal, extracting a second audio signal from the first audio signal, wherein the second audio signal includes an audio frame matching the preset audio signal; The performing audio detection on the in-vehicle audio signal based on the audio detection model to determine the audio confidence level includes: The second audio signal is input into the audio detection model to obtain the audio confidence output by the audio detection model.

6. The method according to claim 1, characterized in that The gesture detection model includes: a finger key point detection sub-model and an action recognition sub-model; Before performing gesture detection on the user image sequence based on the gesture detection model and determining the visual confidence, the method further includes: Matching the user image sequence with a preset gesture image; The performing gesture detection on the user image sequence based on the gesture detection model to determine the visual confidence includes: In the case where the user image sequence matches the preset gesture image, the user image sequence is input into the finger key point detection sub-model to obtain a finger key point sequence output by the finger key point detection sub-model, wherein the finger key point detection sub-model is a convergent model obtained by training based on a sample image sequence; The finger key point sequence is input into the action recognition sub-model to obtain the visual confidence output by the action recognition sub-model, wherein the action recognition sub-model is a convergent model obtained by training based on the sample key point sequence.

7. The method according to claim 6, characterized in that After matching the user image sequence with the preset gesture image, the method further includes: When the user image sequence matches the preset gesture image, performing hand region detection on the user image sequence; Cropping the user image sequence according to the detected hand area to obtain a gesture image sequence; The performing gesture detection on the user image sequence based on the gesture detection model to determine the visual confidence includes: Inputting the gesture image sequence into a finger key point detection sub-model to obtain a finger key point sequence corresponding to the gesture image sequence output by the finger key point sub-detection model; The finger key point sequence is input into the action recognition sub-model to obtain the visual confidence output by the action recognition sub-model.

8. An interactive device for a vehicle terminal, characterized in that: include: An acquisition module, used to acquire in-vehicle audio signals and user image sequences collected within a preset time length; An audio and video detection module, used to perform audio detection on the in-car audio signal based on an audio detection model to determine audio confidence; and, performing gesture detection on the user image sequence based on a gesture detection model to determine a visual confidence; wherein the audio detection model is used to identify a preset audio signal corresponding to a preset gesture, the audio confidence indicates the probability that the in-car audio signal includes the preset audio signal, and the visual confidence indicates the probability of detecting the preset gesture from the user image sequence; The processing module is used to issue a preset control instruction corresponding to the preset gesture when the audio confidence and / or the visual confidence meets a preset condition.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the vehicle terminal interaction method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: include: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the interaction method of the vehicle terminal according to any one of claims 1 to 7 is implemented.

11. A vehicle, characterized in that: include: The interactive device of the vehicle terminal as claimed in claim 8, or the electronic device as claimed in claim 9.

Images