Vehicle multi-mode interaction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a vehicle multi-mode interaction method and device, electronic equipment and a storage medium. The method comprises the steps that a finger pointing area corresponding to a target user in a target vehicle is acquired; collecting a vehicle external image corresponding to the finger pointing area; determining a multi-scale image corresponding to the finger pointing area in the vehicle external image, and performing multi-scale splicing based on the multi-scale image to obtain a target spliced image; acquiring voice data corresponding to the target user; and inputting the target spliced image and the voice data into a preset multi-mode interaction model to obtain a reply text. According to the embodiment of the invention, by combining the gesture pointing and voice input of the user, more visual and natural interaction experience is realized, and the image and voice information is comprehensively analyzed, so that the accurate reply text conforming to the intention of the user is generated.

Description

Technical Field

[0001] The present invention relates to the field of vehicle control technology, and in particular to a vehicle multimodal interaction method, device, electronic equipment and storage medium. Background Art

[0002] With the rapid development of in-vehicle intelligent technology, in-vehicle interactive systems play an increasingly important role in the in-vehicle user experience.

[0003] At present, in order to solve the above technical problems, the relevant technologies generally use in-vehicle interaction methods. However, the existing in-vehicle interaction methods are mainly based on touch screens, physical buttons, etc. However, these interaction methods require users to divert their attention from driving tasks, which poses a safety hazard. In addition, the system does not combine the out-of-cabin environment, resulting in an inability to accurately understand the user's semantic intentions. Summary of the invention

[0004] The purpose of the present invention is to provide a vehicle multimodal interaction method to solve the problem of how to control the motor to improve the vehicle stability in the prior art.

[0005] In order to achieve the above object, the present invention discloses a vehicle multimodal interaction method, the method comprising:

[0006] Obtain the finger pointing area corresponding to the target user inside the target vehicle;

[0007] Acquiring an image of the vehicle exterior corresponding to the area pointed by the finger;

[0008] Determining a multi-scale image corresponding to the area pointed by the finger in the vehicle exterior image, and performing multi-scale stitching based on the multi-scale image to obtain a target stitching image;

[0009] Collecting voice data corresponding to the target user;

[0010] The target spliced ​​image and the voice data are input into a preset multimodal interaction model to obtain a reply text.

[0011] Optionally, obtaining the finger pointing area corresponding to the target user inside the target vehicle includes:

[0012] Perform hand detection inside the target vehicle to obtain three-dimensional hand key points;

[0013] Based on the three-dimensional hand key points, the three-dimensional spatial position coordinates corresponding to all the skeletal key points corresponding to the hand of the target user in a preset coordinate system are obtained;

[0014] The finger pointing area is determined according to the three-dimensional space position coordinates.

[0015] Optionally, the performing hand detection on the interior of the target vehicle to obtain three-dimensional hand key points includes:

[0016] Collecting an in-vehicle image corresponding to a target user inside the target vehicle through an in-vehicle camera, wherein the in-vehicle image includes a hand of the target user;

[0017] Inputting the in-vehicle image into a preset hand detection model to identify the hand area of ​​the target user;

[0018] Hand skeleton key points are detected in the hand area to obtain three-dimensional hand key points.

[0019] Optionally, the performing hand skeleton key point detection on the hand region to obtain three-dimensional hand key points includes:

[0020] Performing hand skeleton key point detection based on the hand area and a preset hand posture estimation model to obtain three-dimensional hand key points;

[0021] Among them, the preset hand posture estimation model includes a three-dimensional hand joint detection module, and the three-dimensional hand joint detection module is used to predict the three-dimensional hand key points and two-dimensional hand key points based on the in-vehicle image.

[0022] Optionally, the acquiring the three-dimensional spatial position coordinates corresponding to all the skeletal key points corresponding to the hand of the target user in a preset coordinate system based on the three-dimensional hand key points includes:

[0023] Determine the position coordinates corresponding to the preset root joint and the absolute depth of the preset root joint;

[0024] The three-dimensional spatial position coordinates corresponding to all the skeletal key points of the target user's hand in the preset coordinate system are obtained according to the position coordinates corresponding to the preset root joint, the absolute depth of the preset root joint and the three-dimensional hand key points.

[0025] Optionally, determining the finger pointing area according to the three-dimensional space position coordinates includes:

[0026] Determining whether a target finger corresponding to the target user is in an extended state;

[0027] If yes, then obtaining the distance between the target finger tip and the palm center point according to the three-dimensional spatial position coordinates;

[0028] Determining whether the angle between the fingertip of the target finger and the middle section of the target finger is within a preset range;

[0029] If so, it is determined to be a single target finger gesture, and the finger pointing area is determined according to the three-dimensional spatial position coordinates corresponding to the target finger.

[0030] Optionally, the finger pointing area includes a side window area and a front area, and the side window area includes a left window area and a right window area.

[0031] Optionally, the step of inputting the target spliced ​​image and the voice data into a preset multimodal interaction model to obtain a reply text comprises:

[0032] The target stitched image and the voice data are input into a preset multimodal interaction model so that the preset multimodal interaction model integrates the target stitched image and the voice data, identifies the user intent corresponding to the target user, and generates a reply text based on the user intent.

[0033] Optionally, after the step of inputting the target spliced ​​image and the voice data into a preset multimodal interaction model to obtain a reply text, the method includes:

[0034] Convert the reply text into a target voice.

[0035] The target voice is played through a vehicle-mounted playing device.

[0036] The present invention discloses a vehicle multi-modal interaction device, the device comprising:

[0037] An acquisition module, used to acquire the finger pointing area corresponding to the target user inside the target vehicle;

[0038] A first acquisition module, used to acquire an image of the vehicle exterior corresponding to the area pointed by the finger;

[0039] A stitching module, used for determining a multi-scale image corresponding to the area pointed by the finger in the vehicle external image, and performing multi-scale stitching based on the multi-scale image to obtain a target stitching image;

[0040] A second collection module is used to collect voice data corresponding to the target user;

[0041] The multimodal interaction module is used to input the target spliced ​​image and the voice data into a preset multimodal interaction model to obtain a reply text.

[0042] An embodiment of the present invention further discloses an electronic device, comprising at least one processor and a memory communicatively connected to the at least one processor;

[0043] The memory is used to store computer programs;

[0044] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0045] The embodiment of the present invention further discloses a computer-readable storage medium on which a computer program is stored. When executed by one or more processors, the processors are enabled to execute the method described in the embodiment of the present invention.

[0046] Beneficial effects of the present invention:

[0047] The embodiment of the present invention obtains the finger pointing area corresponding to the target user inside the target vehicle; collects the vehicle external image corresponding to the finger pointing area; determines the multi-scale image corresponding to the finger pointing area in the vehicle external image, and performs multi-scale stitching based on the multi-scale image to obtain the target stitching image; collects the voice data corresponding to the target user; and inputs the target stitching image and the voice data into a preset multimodal interaction model to obtain a reply text. That is, the embodiment of the present invention realizes a more intuitive and natural interactive experience by combining the user's gesture pointing and voice input. First, the user's finger pointing area is captured by the in-vehicle sensor, and then the real-time image of the area is obtained by the external camera of the vehicle, and multi-scale image processing and stitching are performed to generate a clear target stitching image. At the same time, the system also collects the user's voice data and inputs this information into a pre-trained multimodal interaction model. The model can comprehensively analyze the image and voice information to generate an accurate reply text that meets the user's intention. This interaction method not only improves the convenience and accuracy of vehicle operation, but also enhances the safety and user experience during driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of a vehicle multimodal interaction method provided in an embodiment of the present invention;

[0049] Figure 2 A flowchart of another vehicle multimodal interaction method provided in an embodiment of the present invention;

[0050] Figure 3 A schematic diagram of the structure of a vehicle multi-modal interaction device provided in an embodiment of the present invention;

[0051] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention;

[0052] Figure 5 A schematic diagram of a vehicle multi-modal interaction process provided by an embodiment of the present invention;

[0053] Figure 6 A schematic diagram of a pointing area generation process provided by an embodiment of the present invention;

[0054] Figure 7 A schematic diagram of an exemplary preset hand posture estimation model provided in an embodiment of the present invention;

[0055] Figure 8 A schematic diagram of an exemplary multi-scale stitching and multi-modal interaction provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0057] Reference Figure 1 , shows a flowchart of a vehicle multimodal interaction method provided in an embodiment of the present invention, which specifically includes the following steps:

[0058] It should be noted that the present application can be applied to an integrated vehicle multi-modal interaction system. The overall system principle is as follows: Figure 5 As shown, it includes an in-cabin camera for capturing images inside the car (including the user's hands), a microphone, a speech-to-text module, an out-cabin camera for capturing images of the area pointed to by the user, a GPT-4LMMs backend, and a text-to-speech synthesis module.

[0059] The in-cabin camera captures the in-cabin image, obtains the index finger pointing direction through hand detection and hand posture estimation, and then uses the out-of-cabin camera combined with the calculated pointing point direction to capture the image information of the corresponding area, and performs multi-scale cropping and splicing on the captured image to obtain a multi-scale spliced ​​world view; at the same time, the input speech is captured through the microphone, and the corresponding text is obtained through the speech recognition module; then the text and the multi-scale spliced ​​image are provided as input to LMMs for processing; the obtained reply text is then converted into reply speech through the language synthesis module, and played through the speaker to complete the response.

[0060] Step 101, obtaining the finger pointing area corresponding to the target user inside the target vehicle;

[0061] It should be noted that in the embodiment of the present application, first, gesture estimation is required to determine the pointing direction and area. The finger can be any finger of the target user. In order to facilitate the understanding of the present application, the index finger is used as the target finger for explanation in the following.

[0062] Specifically, the 3D spatial position coordinates of the index finger in the world coordinate system are obtained based on hand detection and hand 3D key point detection, the intersection of the line between the two key points of the index finger and the existing plane is calculated, and it is determined whether the intersection is in the plane area to determine whether the finger is pointing to the plane. Here, the three orientation areas are mainly the front of the vehicle, the left window, and the right window.

[0063] Step 102, collecting the vehicle exterior image corresponding to the area pointed by the finger;

[0064] It should be noted that, in the embodiment of the present application, after the finger pointing area is determined in step 101, the vehicle external image corresponding to the finger pointing area can be collected.

[0065] Specifically, the image outside the vehicle corresponding to the area pointed by the finger can be collected by a camera, and the image outside the vehicle corresponding to the area pointed by the finger can be collected and processed by a camera facing the outside world, that is, an external camera. Alternatively, the image outside the vehicle corresponding to the area pointed by the finger can be collected by an internal camera installed in the vehicle that can collect the image outside the vehicle.

[0066] Specifically, the image outside the vehicle pointed to by the index finger of the target user is captured. It should be noted that the image collected in this part is the image outside the vehicle, that is, the capture of the scene outside the cabin.

[0067] Step 103, determining a multi-scale image corresponding to the area pointed to by the finger in the vehicle exterior image, and performing multi-scale stitching based on the multi-scale image to obtain a target stitching image;

[0068] It should be noted that, in the embodiment of the present application, a multi-scale image related to the point pointed by the user's index finger is cropped from the image, and multi-scale stitching is performed.

[0069] Among them, multi-scale stitching can also be understood as multi-scale image cropping. Specifically, by cropping multi-scale images related to the point where the user's index finger is pointing from the image obtained from the world-facing camera, these images provide diverse information, which helps to more comprehensively understand the user's visual pointing focus.

[0070] For example, in the embodiment of the present application, cropping can be performed from two scales, and the scale size is selected as 1920*1080 and 1280*720, and then spliced ​​into one image as input, which can be referred to Figure 8 , Figure 8 It is a multi-scale splicing and multi-mode interaction graph.

[0071] Step 104, collecting voice data corresponding to the target user;

[0072] It should be noted that in the embodiment of the present application, a microphone is installed on the control panel or dashboard of the vehicle-mounted intelligent system to capture the user's voice input. The microphone is connected to the voice recognition module through the voice input interface of the vehicle-mounted intelligent system, and the captured voice signal is transmitted to the voice recognition module for processing.

[0073] It should be noted that, in the embodiment of the present application, the voice recognition module receives the voice signal transmitted by the microphone, recognizes and analyzes the user's voice input, and converts the voice signal into text.

[0074] The speech recognition module uses advanced speech recognition algorithms, such as OpenAI's Whisper model, to transcribe speech to text and provide text input for the multimodal large model. It can accurately recognize the user's voice commands and improve the system's interactive efficiency and accuracy.

[0075] Step 105: input the target spliced ​​image and the voice data into a preset multimodal interaction model to obtain a reply text.

[0076] It should be noted that in the embodiments of the present application, multimodal data can be processed by a preset multimodal interaction model to identify the user's true intention, wherein the preset multimodal interaction model can be GPT-4 and LMMs. LMMs expands the capabilities of large languages ​​and not only receives text but also accepts image data as input. By integrating multimodal information, it can realize the understanding of user intentions and output generation.

[0077] In specific implementation, the vehicle multimodal interaction system integrated in the present application interfaces with OpenAI's GPT-4LMM through OpenAI's Python API by uploading converted text and a multi-scale spliced ​​world view containing pointing points (for example, encoded in base 64 format), and GPT-4LMM returns its response as text.

[0078] Further, after step 105, that is, after the step of inputting the target spliced ​​image and the voice data into a preset multimodal interaction model to obtain a reply text, the method may include:

[0079] Convert the reply text into a target voice.

[0080] The target voice is played through a vehicle-mounted playing device.

[0081] It should be noted that in the embodiment of the present application, the text-to-speech synthesis module receives the reply text generated by the GPT-4LMMs backend and converts the text into a voice signal. For example, using the ElevenLabs Turbo model, the text information is converted into a natural and fluent voice through speech synthesis technology to facilitate users to understand the feedback information of the AI ​​through hearing, and the voice signal generated by the text-to-speech synthesis module can be played through the speaker to complete the user interaction process.

[0082] The embodiment of the present invention obtains the finger pointing area corresponding to the target user inside the target vehicle; collects the vehicle external image corresponding to the finger pointing area; determines the multi-scale image corresponding to the finger pointing area in the vehicle external image, and performs multi-scale stitching based on the multi-scale image to obtain the target stitching image; collects the voice data corresponding to the target user; and inputs the target stitching image and the voice data into a preset multimodal interaction model to obtain a reply text. That is, the embodiment of the present invention realizes a more intuitive and natural interactive experience by combining the user's gesture pointing and voice input. First, the user's finger pointing area is captured by the in-vehicle sensor, and then the real-time image of the area is obtained by the external camera of the vehicle, and multi-scale image processing and stitching are performed to generate a clear target stitching image. At the same time, the system also collects the user's voice data and inputs this information into a pre-trained multimodal interaction model. The model can comprehensively analyze the image and voice information to generate an accurate reply text that meets the user's intention. This interaction method not only improves the convenience and accuracy of vehicle operation, but also enhances the safety and user experience during driving.

[0083] Reference Figure 2 , shows a flowchart of a vehicle multimodal interaction method provided in an embodiment of the present invention, which specifically includes the following steps:

[0084] Step 201, performing hand detection on the interior of the target vehicle to obtain three-dimensional hand key points;

[0085] Furthermore, the hand detection is performed on the interior of the target vehicle to obtain three-dimensional hand key points including:

[0086] Collecting an in-vehicle image corresponding to a target user inside the target vehicle through an in-vehicle camera, wherein the in-vehicle image includes a hand of the target user;

[0087] Inputting the in-vehicle image into a preset hand detection model to identify the hand area of ​​the target user;

[0088] Hand skeleton key points are detected in the hand area to obtain three-dimensional hand key points.

[0089] Furthermore, the detecting of hand skeleton key points for the hand region to obtain three-dimensional hand key points includes:

[0090] Performing hand skeleton key point detection based on the hand area and a preset hand posture estimation model to obtain three-dimensional hand key points;

[0091] Among them, the preset hand posture estimation model includes a three-dimensional hand joint detection module, and the three-dimensional hand joint detection module is used to predict the three-dimensional hand key points and two-dimensional hand key points based on the in-vehicle image.

[0092] Step 202, based on the three-dimensional hand key points, obtain the three-dimensional spatial position coordinates corresponding to all the skeletal key points corresponding to the hand of the target user in a preset coordinate system;

[0093] Further, the acquiring of the three-dimensional spatial position coordinates corresponding to all the skeletal key points corresponding to the hand of the target user in a preset coordinate system based on the three-dimensional hand key points includes:

[0094] Determine the position coordinates corresponding to the preset root joint and the absolute depth of the preset root joint;

[0095] The three-dimensional spatial position coordinates corresponding to all the skeletal key points of the target user's hand in the preset coordinate system are obtained according to the position coordinates corresponding to the preset root joint, the absolute depth of the preset root joint and the three-dimensional hand key points.

[0096] Step 203: determine the finger pointing area according to the three-dimensional space position coordinates.

[0097] Further, determining the finger pointing area according to the three-dimensional space position coordinates includes:

[0098] Determining whether a target finger corresponding to the target user is in an extended state;

[0099] If yes, then obtaining the distance between the target finger tip and the palm center point according to the three-dimensional spatial position coordinates;

[0100] Determining whether the angle between the fingertip of the target finger and the middle section of the target finger is within a preset range;

[0101] If so, it is determined to be a single target finger gesture, and the finger pointing area is determined according to the three-dimensional spatial position coordinates corresponding to the target finger.

[0102] Furthermore, the finger pointing area includes a side window area and a front area, and the side window area includes a left window area and a right window area.

[0103] It should be noted that, in the embodiment of the present application, in the above steps 201-203, first, the overall solution flow of the index finger pointing direction and area judgment is as follows: Figure 6 As shown, the specific steps are as follows:

[0104] Hand detection: The in-cabin camera captures an RGB image of the interior of the car, including the target user's hands. Based on the RGB single image, the trained hand detection model yolov7-lite is used to perform hand detection.

[0105] Hand posture estimation: Train and use the Minimal-Hand hand posture estimation model to detect hand skeleton key points, and regress the 2D hand skeleton node coordinates and 3D hand skeleton node coordinates through all modules of the hand posture estimation model.

[0106] Minimal Hand consists of two main modules, namely the 3D hand joint detection module (DetNet) and the inverse kinematics module (IKNet). DetNet is a joint detection network that predicts the 2D and 3D positions of hand joints from a single RGB image. It includes three parts: feature extractor, 2D detector, and 3D detector. The overall structure is as follows: Figure 7 As shown in the figure; the input of the feature extractor is a single image, and the output is a feature block F; the 2D detector is a compressed 2-layer CNN, which takes the feature block F as input and outputs the heat map H corresponding to 21 joints, and H contains the joint prediction confidence map of 21 hand joints; the 3D detector regresses the 3D hand joint position from the confidence map H and the feature map F, inputs the feature block F and the confidence map H, outputs the direction vector D of the child node relative to the root node in the middle, and finally outputs the position L of the three-dimensional joint point of the hand; specifically, in the 3D detector, a 2-layer CNN is first used to estimate D, and the input is the heat map H and the feature map F, and then H, F, and D are connected and input into another 2-layer CNN to obtain the final L. In order to alleviate the ambiguity of the depth scale in the monocular environment, the predicted coordinates are relative to the root joint and are normalized by the length of the reference bone. The middle of the metacarpal phalange is selected as the root joint, and the bone from this joint to the wrist is defined as the reference bone length.

[0107] Since the predicted coordinates are relative to the root joint, to get the actual coordinates of all nodes, you need to know the actual coordinates of the root node, and then train and use another model RootNet. This model can obtain the absolute depth of the middle finger root joint, and then obtain the absolute depth of all hand bone nodes in the z direction in the camera coordinate system. Combined with the camera coordinate system to the world coordinate system, the 3D spatial position of all hand bone nodes in the real scene can be obtained.

[0108] Index finger pointing recognition: A single index finger pointing gesture refers to a gesture that only uses the index finger to point to any direction.

[0109] First, determine whether the length of the index finger is greater than that of other fingers to ensure that the index finger is extended; then determine the distance from the fingertip to the center of the palm; finally, determine the angle between the tip of the index finger and the middle of the index finger. If the angle is within a certain range (set threshold: greater than 50 degrees and less than 180 degrees), it is determined to be a single index finger pointing; overall, a comprehensive judgment is made through these steps to determine whether it is a single index finger pointing gesture.

[0110] Pointing direction and area: Based on gesture recognition, the 3D spatial position coordinates of the index finger in the world coordinate system are obtained, the intersection of the line between the two key points of the index finger and the existing plane is calculated, and it is determined whether the intersection is within the plane area to determine whether the finger is pointing to the plane (divided into three direction areas: left window, front, and right window).

[0111] Step 204, collecting the vehicle exterior image corresponding to the area pointed by the finger;

[0112] Step 205, determining a multi-scale image corresponding to the area pointed to by the finger in the vehicle exterior image, and performing multi-scale stitching based on the multi-scale image to obtain a target stitching image;

[0113] Step 206, collecting voice data corresponding to the target user;

[0114] Step 207: input the target spliced ​​image and the voice data into a preset multimodal interaction model to obtain a reply text.

[0115] It should be noted that the above steps refer to the previous discussion and will not be repeated here.

[0116] The embodiment of the present invention obtains the finger pointing area corresponding to the target user inside the target vehicle; collects the vehicle external image corresponding to the finger pointing area; determines the multi-scale image corresponding to the finger pointing area in the vehicle external image, and performs multi-scale stitching based on the multi-scale image to obtain the target stitching image; collects the voice data corresponding to the target user; and inputs the target stitching image and the voice data into a preset multimodal interaction model to obtain a reply text. That is, the embodiment of the present invention realizes a more intuitive and natural interactive experience by combining the user's gesture pointing and voice input. First, the user's finger pointing area is captured by the in-vehicle sensor, and then the real-time image of the area is obtained by the external camera of the vehicle, and multi-scale image processing and stitching are performed to generate a clear target stitching image. At the same time, the system also collects the user's voice data and inputs this information into a pre-trained multimodal interaction model. The model can comprehensively analyze the image and voice information to generate an accurate reply text that meets the user's intention. This interaction method not only improves the convenience and accuracy of vehicle operation, but also enhances the safety and user experience during driving.

[0117] Reference Figure 3 , shows a vehicle multimodal interaction device provided by an embodiment of the present invention, the device comprising:

[0118] An acquisition module 301 is used to acquire a finger pointing area corresponding to a target user inside a target vehicle;

[0119] A collection module 302, used to collect the vehicle external image corresponding to the area pointed by the finger;

[0120] A stitching module 303 is used to determine a multi-scale image corresponding to the finger pointing area in the vehicle external image, and perform multi-scale stitching based on the multi-scale image to obtain a target stitching image;

[0121] A collection module 304 is used to collect voice data corresponding to the target user;

[0122] The multimodal interaction module 305 is used to input the target spliced ​​image and the voice data into a preset multimodal interaction model to obtain a reply text.

[0123] The embodiment of the present invention obtains the finger pointing area corresponding to the target user inside the target vehicle; collects the vehicle external image corresponding to the finger pointing area; determines the multi-scale image corresponding to the finger pointing area in the vehicle external image, and performs multi-scale stitching based on the multi-scale image to obtain the target stitching image; collects the voice data corresponding to the target user; and inputs the target stitching image and the voice data into a preset multimodal interaction model to obtain a reply text. That is, the embodiment of the present invention realizes a more intuitive and natural interactive experience by combining the user's gesture pointing and voice input. First, the user's finger pointing area is captured by the in-vehicle sensor, and then the real-time image of the area is obtained by the external camera of the vehicle, and multi-scale image processing and stitching are performed to generate a clear target stitching image. At the same time, the system also collects the user's voice data and inputs this information into a pre-trained multimodal interaction model. The model can comprehensively analyze the image and voice information to generate an accurate reply text that meets the user's intention. This interaction method not only improves the convenience and accuracy of vehicle operation, but also enhances the safety and user experience during driving.

[0124] The embodiment of the present invention further provides an electronic device, such as Figure 4 As shown, it includes a processor 401, a device interface 402, a memory 403 and a bus 404;

[0125] Memory 403, used for storing computer programs;

[0126] The processor 401 is used to execute the program stored in the memory 403 to implement the following steps:

[0127] Obtain the finger pointing area corresponding to the target user inside the target vehicle;

[0128] Acquiring an image of the vehicle exterior corresponding to the area pointed by the finger;

[0129] Determining a multi-scale image corresponding to the area pointed by the finger in the vehicle exterior image, and performing multi-scale stitching based on the multi-scale image to obtain a target stitching image;

[0130] Collecting voice data corresponding to the target user;

[0131] The target spliced ​​image and the voice data are input into a preset multimodal interaction model to obtain a reply text.

[0132] The bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0133] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0134] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0135] The present invention also provides a storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the vehicle multimodal interaction method of the aforementioned embodiment.

[0136] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0137] The algorithm and display provided herein are not inherently related to any particular computer, virtual device or other equipment. According to the above description, it is obvious that the structure required for constructing this type of device is. In addition, the present invention is not directed to any specific programming language. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific language is to disclose the best mode of the present invention.

[0138] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0139] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: that the claimed invention requires more features than those expressly recited in each claim. Rather, as reflected in the claims below, inventive aspects lie in less than all of the features of the individual embodiments disclosed above. Therefore, the claims that follow the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present invention.

[0140] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0141] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in the sorting device according to the present invention. The present invention may also be implemented as a device or apparatus program for executing part or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0142] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0143] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, apparatuses and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0144] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0145] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0146] It should be noted that the various data-related processes in the embodiments of the present application are all carried out in compliance with the corresponding data protection laws and policies of the country where the device is located, and with the authorization given by the owner of the corresponding device.

Claims

1. A vehicle multimodal interaction method, characterized in that: The method comprises: Obtain the finger pointing area corresponding to the target user inside the target vehicle; Acquiring an image of the vehicle exterior corresponding to the area pointed by the finger; Determining a multi-scale image corresponding to the area pointed by the finger in the vehicle exterior image, and performing multi-scale stitching based on the multi-scale image to obtain a target stitching image; Collecting voice data corresponding to the target user; The target spliced ​​image and the voice data are input into a preset multimodal interaction model to obtain a reply text.

2. The method according to claim 1, characterized in that The step of obtaining the finger pointing area corresponding to the target user inside the target vehicle comprises: Perform hand detection inside the target vehicle to obtain three-dimensional hand key points; Based on the three-dimensional hand key points, the three-dimensional spatial position coordinates corresponding to all the skeletal key points corresponding to the hand of the target user in a preset coordinate system are obtained; The finger pointing area is determined according to the three-dimensional space position coordinates.

3. The method according to claim 2, characterized in that The hand detection is performed on the interior of the target vehicle to obtain three-dimensional hand key points including: Collecting an in-vehicle image corresponding to a target user inside the target vehicle through an in-vehicle camera, wherein the in-vehicle image includes a hand of the target user; Inputting the in-vehicle image into a preset hand detection model to identify the hand area of ​​the target user; Hand skeleton key points are detected in the hand area to obtain three-dimensional hand key points.

4. The method according to claim 3, characterized in that The detecting of the hand skeleton key points for the hand region to obtain the three-dimensional hand key points includes: Performing hand skeleton key point detection based on the hand area and a preset hand posture estimation model to obtain three-dimensional hand key points; Among them, the preset hand posture estimation model includes a three-dimensional hand joint detection module, and the three-dimensional hand joint detection module is used to predict the three-dimensional hand key points and two-dimensional hand key points based on the in-vehicle image.

5. The method according to claim 2, characterized in that: The step of acquiring the three-dimensional spatial position coordinates corresponding to all the skeletal key points corresponding to the hand of the target user in a preset coordinate system based on the three-dimensional hand key points includes: Determine the position coordinates corresponding to the preset root joint and the absolute depth of the preset root joint; The three-dimensional spatial position coordinates corresponding to all the skeletal key points of the target user's hand in the preset coordinate system are obtained according to the position coordinates corresponding to the preset root joint, the absolute depth of the preset root joint and the three-dimensional hand key points.

6. The method according to claim 2, characterized in that Determining the finger pointing area according to the three-dimensional space position coordinates includes: Determining whether a target finger corresponding to the target user is in an extended state; If yes, then obtaining the distance between the target finger tip and the palm center point according to the three-dimensional spatial position coordinates; Determining whether the angle between the fingertip of the target finger and the middle section of the target finger is within a preset range; If so, it is determined to be a single target finger gesture, and the finger pointing area is determined according to the three-dimensional spatial position coordinates corresponding to the target finger.

7. The method according to claim 6, characterized in that The finger pointing area includes a side window area and a front area, and the side window area includes a left window area and a right window area.

8. The method according to claim 1, characterized in that The step of inputting the target spliced ​​image and the voice data into a preset multimodal interaction model to obtain a reply text includes: The target stitched image and the voice data are input into a preset multimodal interaction model so that the preset multimodal interaction model integrates the target stitched image and the voice data, identifies the user intent corresponding to the target user, and generates a reply text based on the user intent.

9. The method according to claim 1, characterized in that: After the step of inputting the target spliced ​​image and the voice data into a preset multimodal interaction model to obtain a reply text, the method includes: Convert the reply text into a target voice. The target voice is played through a vehicle-mounted playing device.

10. A vehicle multimodal interaction device, characterized in that: The device comprises: An acquisition module, used to acquire the finger pointing area corresponding to the target user inside the target vehicle; A first acquisition module, used to acquire an image of the vehicle exterior corresponding to the area pointed by the finger; A stitching module, used for determining a multi-scale image corresponding to the area pointed by the finger in the vehicle external image, and performing multi-scale stitching based on the multi-scale image to obtain a target stitching image; A second collection module is used to collect voice data corresponding to the target user; The multimodal interaction module is used to input the target spliced ​​image and the voice data into a preset multimodal interaction model to obtain a reply text.

11. An electronic device, characterized in that: include: at least one processor; a memory communicatively coupled to the at least one processor; The memory has a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the vehicle multimodal interaction method according to any one of claims 1 to 9.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the vehicle multimodal interaction method according to any one of claims 1 to 9 is implemented.

Images