Operation intention recognition method, interaction method and related device

Abstract

The invention discloses an operation intention recognition method, an interaction method and a related device, and relates to the technical field of man-machine interaction, the operation intention recognition method comprises the following steps: obtaining multi-modal data of a target user, the multi-modal data comprising hand image data and voice data; according to the multi-modal data of the target user, obtaining a finger endpoint three-dimensional coordinate, a finger pointing vector radian, a target operation and an operation confidence coefficient; determining whether the target user has an operation intention or not according to the operation confidence; and if the target user has the operation intention, determining an operation object corresponding to the target operation according to the finger endpoint three-dimensional coordinate and the finger pointing vector radian so as to obtain an operation intention recognition result containing the target operation and the operation object corresponding to the target operation. According to the operation intention recognition method disclosed by the invention, the operation intention can be accurately recognized, so that the interaction accuracy can be improved, and the user experience can be improved.

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction technology, and in particular to an operation intention recognition method, an interaction method and related devices. Background Art

[0002] Human-computer interaction refers to the process of information exchange and interaction between humans and devices. In many application scenarios (such as smart cockpit scenarios), human-computer interaction is mostly voice interaction (such as turning on the air conditioner and closing the windows by voice).

[0003] It is understandable that when performing voice interaction, the user's voice needs to be recognized. However, due to many factors (such as environmental noise), the user's operation intention cannot be accurately recognized through voice recognition. It is understandable that the inability to accurately recognize the user's operation intention will lead to low interaction accuracy, which will seriously affect the user experience. Summary of the invention

[0004] In view of this, the present application provides an operation intention recognition method, an interaction method and related devices, which are used to solve the problem that the current solution cannot accurately recognize the user's operation intention, which leads to low interaction accuracy and affects the user experience. The technical solution is as follows:

[0005] The first aspect of the present application provides an operation intention recognition method, comprising:

[0006] Acquire multimodal data of a target user, wherein the multimodal data includes hand image data and voice data;

[0007] According to the multimodal data of the target user, the three-dimensional coordinates of the finger endpoints, the radian of the finger pointing vector, the target operation and the operation confidence are obtained;

[0008] Determining whether the target user has an operation intention according to the operation confidence;

[0009] If the target user has an operation intention, the operation object corresponding to the target operation is determined according to the three-dimensional coordinates of the finger endpoint and the radian of the finger pointing vector to obtain an operation intention recognition result including the target operation and the operation object corresponding to the target operation.

[0010] In a possible implementation manner, the hand image data includes a hand RGB image and a hand depth map corresponding to the hand RGB image.

[0011] In a possible implementation, obtaining hand image data of a target user includes:

[0012] Using an image acquisition device to acquire an RGB image of the target user's hand and a depth map corresponding to the RGB image;

[0013] Perform hand detection on the collected RGB image using a pre-trained first hand detection model to obtain position information of a hand area in the collected RGB image, wherein the first hand detection model is trained using a training image in which a hand area is marked;

[0014] Extracting the hand region from the acquired RGB image according to the position information of the hand region to obtain a hand RGB image;

[0015] According to the position information of the hand area, the hand area is extracted from the depth map corresponding to the collected RGB image to obtain a hand depth map corresponding to the hand RGB image.

[0016] In a possible implementation, the first hand detection model is trained using a training image with a hand region marked, and is assisted by a pre-trained second hand detection model, and the scale of the second hand detection model is larger than that of the first hand detection model;

[0017] The training goal of the first hand detection model is to make the hand area obtained by using the first hand detection model to perform hand detection on the training image close to the hand area obtained by using the second hand detection model to perform hand detection on the training image, and to make the hand area obtained by using the first hand detection model to perform hand detection on the training image close to the hand area annotated on the training image.

[0018] In a possible implementation, acquiring the three-dimensional coordinates of the finger endpoint, the radian of the finger pointing vector, the target operation, and the operation confidence according to the multimodal data of the target user includes:

[0019] Using the pre-trained multimodal information recognition model, information recognition is performed on the multimodal data of the target user to obtain the three-dimensional coordinates of the finger endpoints, the radian of the finger pointing vector, the target operation, and the operation confidence;

[0020] The multimodal information recognition model is trained using training multimodal data annotated with three-dimensional coordinates of finger endpoints, radians of finger pointing vectors, target operations, and operation confidence levels, and the training multimodal data includes training hand image data and training voice data.

[0021] In a possible implementation, the multimodal information recognition model obtained by pre-training is used to perform information recognition on the multimodal data of the target user to obtain the three-dimensional coordinates of the finger endpoint, the radian of the finger pointing vector, the target operation and the operation confidence, including:

[0022] Using the pre-trained multimodal information recognition model, extract features from the hand image data and voice data of the target user to obtain image features and voice features;

[0023] Using the multimodal information recognition model, the image feature and the speech feature are fused to obtain fused features;

[0024] The multimodal information recognition model is used to predict the three-dimensional coordinates of the finger endpoint, the curvature of the finger pointing vector, the target operation and the operation confidence based on the fused features.

[0025] In a possible implementation, the three-dimensional coordinates of the finger endpoint and the radian of the finger pointing vector are both data in a camera coordinate system;

[0026] The determining, according to the three-dimensional coordinates of the finger endpoint and the radian of the finger pointing vector, an operation object corresponding to the target operation includes:

[0027] Convert the three-dimensional coordinates of the finger endpoint in the camera coordinate system to the three-dimensional coordinates of the finger endpoint in the world coordinate system;

[0028] Convert the finger pointing vector radian in the camera coordinate system to the finger pointing direction vector in the world coordinate system;

[0029] According to the position information of each candidate operation object and the three-dimensional coordinates of the finger endpoint and the finger pointing direction vector in the world coordinate system, the operation object corresponding to the target operation is determined from the candidate operation objects.

[0030] In a possible implementation, determining the operation object corresponding to the target operation from the candidate operation objects according to the position information of the candidate operation objects and the three-dimensional coordinates of the finger endpoint and the finger pointing direction vector in the world coordinate system includes:

[0031] For each candidate operation object: determine whether a target ray intersects with the candidate operation object according to the position information of the candidate operation object, the normal vector of the plane where the candidate operation object is located, and the three-dimensional coordinates of the finger endpoint and the finger pointing direction vector in the world coordinate system, wherein the target ray is a ray with the three-dimensional coordinates of the finger endpoint in the world coordinate system as the ray starting point and the direction indicated by the finger pointing direction vector in the world coordinate system as the ray direction;

[0032] The candidate operation object intersecting with the target ray is determined as the operation object corresponding to the target operation.

[0033] A second aspect of the present application provides an interaction method, comprising:

[0034] Using any of the above operation intention recognition methods to perform operation intention recognition, and obtaining an operation intention recognition result including a target operation and an operation object corresponding to the target operation;

[0035] Controlling the operation object corresponding to the target operation to perform the target operation.

[0036] The third aspect of the present application provides an operation intention recognition device, comprising: a multimodal data acquisition module, a hand information and operation information acquisition module, an operation intention determination module and an operation object determination module;

[0037] The multimodal data acquisition module is used to acquire multimodal data of the target user, wherein the multimodal data includes hand image data and voice data;

[0038] The hand information and operation information acquisition module is used to acquire the three-dimensional coordinates of the finger endpoints, the radian of the finger pointing vector, the target operation and the operation confidence according to the multimodal data of the target user;

[0039] The operation intention determination module is used to determine whether the target user has an operation intention according to the operation confidence;

[0040] The operation object determination module is used to determine the operation object corresponding to the target operation according to the three-dimensional coordinates of the finger endpoint and the radian of the finger pointing vector when the target user has an operation intention, so as to obtain an operation intention recognition result including the target operation and the operation object corresponding to the target operation.

[0041] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0042] The memory is used to store computer programs;

[0043] The processor is used to execute the computer program so that the electronic device can implement the steps of any one of the above-mentioned operation intention recognition methods, or implement the steps of the above-mentioned interaction method.

[0044] In a fifth aspect, the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of any one of the above-mentioned operation intention recognition methods, or implement the steps of the above-mentioned interaction method.

[0045] In a sixth aspect, the present application provides a computer program product, comprising computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the steps of any one of the above-mentioned operation intention recognition methods, or implements the steps of the above-mentioned interaction method.

[0046] By means of the above technical scheme, the operation intention recognition method provided by the present application first obtains the multimodal data (including hand image data and voice data) of the target user, then obtains the three-dimensional coordinates of the finger endpoints, the radian of the finger pointing vector, the target operation and the operation confidence according to the multimodal data of the target user, and then determines whether the target user has the operation intention according to the operation confidence. If the target user has the operation intention, the operation object corresponding to the target operation is determined according to the three-dimensional coordinates of the finger endpoints and the radian of the finger pointing vector, so as to obtain the operation intention recognition result including the target operation and the operation object corresponding to the target operation. The operation intention recognition method provided by the present application is an operation intention recognition method based on multimodal data (i.e., hand image data and voice data). Since multimodal data can provide richer information and data of different modes can complement each other (for example, voice data is easily affected by environmental noise, while image data is not affected by environmental noise), therefore, the operation intention recognition by combining image data and voice data can obtain a more accurate operation intention, thereby improving the interaction accuracy, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0048] Figure 1 A schematic diagram of a hardware structure of a terminal provided in an embodiment of the present application;

[0049] Figure 2 A flowchart of a method for identifying operation intention provided in an embodiment of the present application;

[0050] Figure 3A schematic diagram of a process for obtaining a hand RGB image of a target user and a hand depth map corresponding to the hand RGB image provided in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of obtaining a first hand detection model by using a training image with hand regions marked, supplemented by a pre-trained large-scale second hand detection model, provided in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of a camera coordinate system and a world coordinate system provided in an embodiment of the present application;

[0053] Figure 6 A schematic diagram of the structure of an operation intention recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0055] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0056] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0057] The operation intention recognition method and interaction method provided in this application can be applied to terminals (such as car computers).

[0058] Figure 1 An optional hardware structure diagram of a terminal is shown.

[0059] refer to Figure 1As shown, the terminal may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 1 This is merely an example of a terminal and does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown in the figure, or a combination of certain components, or different components.

[0060] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the terminal. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0061] The display unit 140 may be used to display information input by a user or information provided to a user, various menus of the terminal, an interactive interface, file display, and / or playback of any multimedia file.

[0062] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0063] The processor 170 is the control center of the terminal. It uses various interfaces and lines to connect various parts of the entire terminal. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal and processes data, thereby controlling the terminal as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0064] Among them, the memory 120 can be used to store software codes related to the operation intention recognition method and the interaction method, the processor 170 can execute the software code in the memory 120, and can also schedule other units (such as the above-mentioned input unit 130 and the display unit 140) to realize corresponding functions.

[0065] The radio frequency unit 110 (optional) can be used to receive and send information or receive and send signals during a call. For example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the radio frequency unit 110 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LowNoiseAmplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General PacketRadio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0066] In the embodiment of the present application, the radio frequency unit 110 can send data to other devices, and can also receive data sent by other devices. It should be understood that the radio frequency unit 110 is optional, and can be replaced by other communication interfaces, such as a network port.

[0067] The terminal also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions.

[0068] The terminal also includes an external interface 180, which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal to communicate with other devices, or to connect a charger to charge the terminal.

[0069] Although not shown, the terminal may further include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described in detail here.

[0070] Next, the operation intention recognition method and interaction method provided by this application are introduced through the following embodiments.

[0071] See also Figure 2, shows a schematic flow chart of an operation intention recognition method provided in an embodiment of the present application, which may include:

[0072] Step S201: Acquire multimodal data of a target user.

[0073] The multimodal data of the target user includes hand image data and voice data of the target user.

[0074] When the target user interacts with a device, he can issue a voice command and point his hand at the device. For example, in the smart cockpit scenario, if the user wants to open the sunroof, he can issue a voice command "open" and point his hand at the sunroof. It should be noted that since the target user points his hand at the device, he only needs to say simple command words (such as open, close, etc.), that is, the user only needs to say the required operation without saying the corresponding operation object. Of course, this embodiment is not limited to this. In addition to the required operation, the voice command issued by the user may also include the operation object corresponding to the required operation. For example, the user can also say "open the sunroof" and point his hand at the sunroof.

[0075] When a user interacts with a device, he or she usually points at the device with his or her hands while issuing voice commands. In order to recognize the user's operating intention, the voice data of the target user and image data including the target user's hands can be collected simultaneously. For the collected image data including the target user's hands, the hand area can be extracted therefrom to obtain the target user's hand image data, thereby obtaining multimodal data including the target user's hand image data and voice data.

[0076] In one possible implementation, the hand image data of the target user may include an RGB image of the hand of the target user. In order to improve the operation intention recognition effect, in another possible implementation, the hand image data of the target user may include, in addition to the RGB image of the hand, a hand depth map corresponding to the RGB image of the hand.

[0077] Step S202: According to the multimodal data of the target user, the three-dimensional coordinates of the finger endpoints, the radian of the finger pointing vector, the target operation and the operation confidence are obtained.

[0078] This embodiment acquires the three-dimensional coordinates of the finger endpoints, the radian of the finger pointing vector, the target operation, and the operation confidence by analyzing and processing the multimodal data of the target user.

[0079] Among them, the three-dimensional coordinates of the finger endpoint and the radian of the finger pointing vector are data in the camera coordinate system. The three-dimensional coordinates of the finger endpoint in the camera coordinate system are the three-dimensional coordinates of the finger endpoint (i.e., fingertip) in the camera coordinate system. The radian of the finger pointing vector in the camera coordinate system represents the direction of the finger in the camera coordinate system, which is represented by the angle γ1 between the finger and the x-axis of the camera coordinate system, the angle γ2 between the finger and the y-axis of the camera coordinate system, and the angle γ3 between the finger and the z-axis of the camera coordinate system. That is, the radian of the finger pointing vector can be expressed as (γ1, γ2, γ3). The target operation is the operation that needs to be performed (such as "open"), and the operation confidence represents the possibility of an operation intention.

[0080] Step S203: Determine whether the target user has an operation intention based on the operation confidence.

[0081] Specifically, it is determined whether the operation confidence is greater than a preset confidence threshold (for example, 0.5). If the operation confidence is greater than the preset confidence threshold, it is determined that the target user has an operation intention. If the operation confidence is less than or equal to the preset confidence threshold, it is determined that the target user has no operation intention.

[0082] Step S204: If the target user has an operation intention, the operation object corresponding to the target operation is determined according to the three-dimensional coordinates of the finger endpoint and the radian of the finger pointing vector, and an operation intention recognition result including the target operation and the operation object corresponding to the target operation is obtained.

[0083] Specifically, based on the three-dimensional coordinates of the finger endpoint, the arc of the finger pointing vector, and the position information of several candidate operation objects (such as air conditioning, windows, sunroofs, etc. in the smart cockpit scene), the operation object corresponding to the target operation is determined from several candidate operation objects.

[0084] It should be noted that if it is determined based on the operation confidence that the target user has no operation intention, the acquisition and analysis of the multimodal data will continue.

[0085] The operation intention recognition method provided in the embodiment of the present application first obtains the multimodal data of the target user (including hand image data and voice data), then obtains the three-dimensional coordinates of the finger endpoints, the radian of the finger pointing vector, the target operation and the operation confidence according to the multimodal data of the target user, and then determines whether the target user has an operation intention according to the operation confidence. If the target user has an operation intention, the operation object corresponding to the target operation is determined according to the three-dimensional coordinates of the finger endpoints and the radian of the finger pointing vector, so as to obtain an operation intention recognition result including the target operation and the operation object corresponding to the target operation. The operation intention recognition method provided in the embodiment of the present application is an operation intention recognition method based on multimodal data (i.e., hand image data and voice data). Since multimodal data can provide richer information and data of different modes can complement each other (for example, voice data is easily affected by environmental noise, while image data is not affected by environmental noise), therefore, the operation intention recognition is performed by combining image data and voice data, so that a more accurate operation intention can be obtained, thereby improving the interaction accuracy, thereby improving the user experience.

[0086] The above embodiment mentioned that the multimodal data of the target user includes hand image data and voice data of the target user, and the hand image data of the target user may include a hand RGB image and a depth map corresponding to the hand RGB image.

[0087] In another embodiment of the present application, an implementation process of obtaining a hand RGB image of a target user and a hand depth map corresponding to the hand RGB image is introduced.

[0088] like Figure 3 As shown, the process of obtaining the target user's hand RGB image and the depth map corresponding to the hand RGB image may include:

[0089] Step S301: using an image acquisition device to acquire an RGB image of a target user's hand and a depth map corresponding to the RGB image.

[0090] An RGBD camera may be used to capture an RGB image containing the hands of the target user and a depth map corresponding to the RGB image. Of course, this embodiment is not limited to this. For example, an RGB image capture device and a depth map capture device disposed at the same position may also be used to capture an RGB image containing the hands of the target user and a corresponding depth map.

[0091] Step S302: Perform hand detection on the collected RGB image using the pre-trained first hand detection model to obtain position information of the hand area in the collected RGB image.

[0092] The first hand detection model is trained using training images with hand areas marked.

[0093] The first hand detection model in this embodiment can be a small-scale hand detection model (in some application scenarios, due to limited computing resources, usually only a small-scale hand detection model can be deployed). In a possible implementation, the first hand detection model can be trained using a training image with annotated hand areas, and assisted by a pre-trained large-scale second hand detection model. The training goal of the first hand detection model is to make the hand area obtained by hand detection of the training image using the first hand detection model approach the hand area obtained by hand detection of the training image using the second hand detection model, and to make the hand area obtained by hand detection of the training image using the first hand detection model approach the hand area annotated in the training image. Assisted by the large-scale second hand detection model, the performance of the first hand detection model trained according to the above goals is close to the performance of the large-scale second hand detection model.

[0094] Specifically, Figure 4 As shown, the process of using the training image with the hand area marked and the large-scale second hand detection model obtained by pre-training to train the first hand detection model may include:

[0095] Step a1-a-1, use the second hand detection model to extract features from the training image to obtain first image features corresponding to the training image, and map the first image features corresponding to the training image to a high-dimensional space to obtain second image features corresponding to the training image.

[0096] The training image is converted into a vector to obtain a vector representation of the training image, and the vector representation of the training image is input into the image feature extraction module of the second hand detection model for feature extraction to obtain a first image feature corresponding to the training image. After obtaining the first image feature corresponding to the training image, the first image feature corresponding to the training image is input into the feature mapping module of the second hand detection model. The feature mapping module maps the first image feature corresponding to the training image to a high-dimensional space to obtain a second image feature corresponding to the training image. It should be noted that the first image feature corresponding to the training image is a high-dimensional feature. By mapping the first image feature corresponding to the training image to a high-dimensional space, a higher-dimensional image feature can be obtained.

[0097] Step a1-a-2: using the second hand detection model and taking the second image features corresponding to the training image as a basis, predict the probability that each pixel in the training image belongs to the hand, and obtain a first probability distribution.

[0098] The second image feature corresponding to the training image is input into the probability prediction module of the second hand detection model. The probability prediction module of the second hand detection model predicts the probability that each pixel in the training image belongs to the hand according to the second image feature corresponding to the training image to obtain a first probability distribution.

[0099] Step a1-b-1, use the first hand detection model to extract features from the training image to obtain third image features corresponding to the training image, and map the third image features corresponding to the training image to a high-dimensional space to obtain fourth image features corresponding to the training image.

[0100] The vector representation of the training image is input into the image feature extraction module of the first hand detection model for feature extraction to obtain a third image feature corresponding to the training image. After obtaining the third image feature corresponding to the training image, the third image feature corresponding to the training image is mapped to a high-dimensional space to obtain a fourth image feature corresponding to the training image.

[0101] Step a1-b-2: using the first hand detection model, based on the third image feature corresponding to the training image, predict the probability that each pixel in the training image belongs to the hand, and obtain a second probability distribution; and using the first hand detection model, based on the fourth image feature corresponding to the training image, predict the probability that each pixel in the training image belongs to the hand, and obtain a third probability distribution.

[0102] The third image feature corresponding to the training image is input into the probability prediction module of the first hand detection model. The probability prediction module of the first hand detection model predicts the probability that each pixel in the training image belongs to the hand based on the third image feature corresponding to the training image to obtain a second probability distribution. The fourth image feature corresponding to the training image is also input into the probability prediction module of the first hand detection model. The probability prediction module of the first hand detection model predicts the probability that each pixel in the training image belongs to the hand based on the fourth image feature corresponding to the training image to obtain a third probability distribution.

[0103] Step a2: determine a first prediction loss based on the first probability distribution and the third probability distribution, and determine a second prediction loss based on the second probability distribution and the hand area annotated in the training image.

[0104] Both the first prediction loss and the second prediction loss may adopt cross entropy loss. The calculation method of the cross entropy loss is prior art and will not be described in detail in this embodiment.

[0105] Step a3: Fuse the first prediction loss with the second prediction loss to obtain a fused loss.

[0106] If the first prediction loss is represented as L1 and the second prediction loss is represented as L2, the first prediction loss L1 and the second prediction loss L2 can be fused as shown in the following formula to obtain the fused loss, that is, the final loss L final :

[0107] L final =λL1+(1-λ)L2(1)

[0108] Step a4: update the parameters of the first hand detection model according to the fusion loss.

[0109] The first hand detection model is iteratively trained for multiple times in the above manner until a training end condition is met (for example, a set number of training iterations is reached, the model converges, etc.).

[0110] Through the above process, a first hand detection model with better performance can be obtained. Then, the trained first hand detection model can be used to perform hand detection on an RGB image containing the hand of the target user acquired by an image acquisition device. Specifically, the RGB image containing the hand of the target user is represented as a vector to obtain a vector representation of the RGB image. The vector representation of the RGB image is input into a feature extraction module of the first hand detection model for feature extraction to obtain image features corresponding to the RGB image. The image features corresponding to the RGB image are input into a probability prediction module of the first hand detection model to obtain the probability that each pixel in the RGB image belongs to the hand. The pixel points in the RGB image belonging to the hand are determined according to the probability that each pixel in the RGB image belongs to the hand. The position information of the hand area is determined according to the pixel points in the RGB image belonging to the hand.

[0111] It should be noted that the parameters of the first hand detection model obtained through training are usually float type data. In order to reduce memory usage and speed up the inference speed, the first hand detection model can be quantized to 8 bits, that is, the parameters of the first hand detection model are converted from float type to int8 type. Then, when inputting data into the first hand detection model, the input data is first quantized to 8 bits, that is, the input data is converted to int8 type before input. After obtaining the output data of the first hand detection model, the output data is restored to float type data.

[0112] In a possible implementation, the input data may be quantized to 8 bits as shown in equations (2) and (3):

[0113] (2)

[0114] (3)

[0115] Among them, V x Represents input data, V x ′ Represents the quantization result of the input data, V max and V min They represent the maximum and minimum values ​​of the input data respectively, S represents the scaling factor, S>>1 represents a right shift, S is generally 256, and S shifted right by one bit is 128.

[0116] After obtaining the output data of the first hand detection model, the output data can be restored to float type data according to the following equations (4) and (5):

[0117] (4)

[0118] (5)

[0119] Among them, V y ′ Indicates output data, V y Indicates the restored float type data.

[0120] Step S303: extracting the hand region from the collected RGB image according to the position information of the hand region to obtain a hand RGB image.

[0121] Once the position information of the hand region in the RGB image is known, the hand region can be extracted from the RGB image according to the position information of the hand region to obtain the hand RGB image.

[0122] Step S304: extracting the hand area from the depth map corresponding to the collected RGB image according to the position information of the hand area, and obtaining a hand depth map corresponding to the hand RGB image.

[0123] The position of the hand area in the depth map corresponding to the RGB image is the same as the position information of the hand area in the RGB image. Therefore, after obtaining the position information of the hand area in the RGB image, the hand area can be extracted from the depth map corresponding to the RGB image according to the position information of the hand area in the RGB image, thereby obtaining the hand depth map corresponding to the hand RGB image.

[0124] In another embodiment of the present application, the specific implementation process of "Step S202: obtaining the three-dimensional coordinates of the finger endpoint, the radian of the finger pointing vector, the target operation and the operation confidence according to the multimodal data of the target user" in the above embodiment is introduced.

[0125] In one possible implementation, the process of obtaining the three-dimensional coordinates of the finger endpoints, the curvature of the finger pointing vector, the target operation, and the operation confidence based on the multimodal data of the target user may include: using a pre-trained multimodal information recognition model to perform information recognition on the multimodal data of the target user to obtain the three-dimensional coordinates of the finger endpoints, the curvature of the finger pointing vector, the target operation, and the operation confidence.

[0126] Among them, the multimodal information recognition model is trained using training multimodal data labeled with three-dimensional coordinates of finger endpoints, curvature of finger pointing vectors, target operations and operation confidences. The training multimodal data includes training hand image data and training voice data. The training goal of the multimodal information recognition model is to make the three-dimensional coordinates of finger endpoints, curvature of finger pointing vectors, target operations and operation confidences obtained by the multimodal information recognition model through information recognition of the training multimodal data close to the three-dimensional coordinates of finger endpoints, curvature of finger pointing vectors, target operations and operation confidences labeled in the training multimodal data.

[0127] Specifically, the process of using a pre-trained multimodal information recognition model to perform information recognition on the target user's multimodal data to obtain the three-dimensional coordinates of the finger endpoints, the arc of the finger pointing vector, the target operation, and the operation confidence may include: step b1, using the pre-trained multimodal information recognition model to extract features from the target user's hand image data and the target user's voice data, respectively, to obtain image features and voice features.

[0128] The multimodal information recognition model may include an image feature extraction module and a speech feature extraction module. The image feature extraction module of the multimodal information recognition model may be used to extract features from the target user's hand image data to obtain image features. The speech feature extraction module of the multimodal information recognition model may be used to extract features from the target user's speech data to obtain speech features.

[0129] Step b2: using a multimodal information recognition model, fuse the image features with the speech features to obtain fused features.

[0130] The multimodal information recognition model may further include a feature fusion module, which may be used to fuse image features with speech features to obtain fused features.

[0131] When the feature fusion module of the multimodal information recognition model fuses the image features with the speech features, it can first use the feature fusion method based on the self-attention mechanism to perform feature fusion processing on the image features to obtain the first fused image features, and use the feature fusion method based on the self-attention mechanism to perform feature fusion processing on the speech features to obtain the first fused speech features, and then use the feature fusion method based on the cross-attention mechanism to integrate the first fused speech features into the first fused image features to obtain the second fused image features integrated with the speech features, and use the feature fusion method based on the cross-attention mechanism to integrate the first fused image features into the first fused speech features to obtain the image features integrated with the speech features. The second fused speech features are obtained, and then the feature fusion method based on the self-attention mechanism is used to further fuse the second fused image features fused with speech features, and on this basis, the image features of the historical image data (assuming that the current acquired image data is the nth frame, and the historical image data is the image data before the nth frame) are integrated to obtain the third fused image features, and the feature fusion method based on the self-attention mechanism is used to further fuse the second fused speech features fused with image features, and on this basis, the speech features of the historical speech data are integrated to obtain the third fused speech features, and finally the third fused image features are spliced ​​with the third fused speech features to obtain the final feature fusion result.

[0132] Step b3: Utilize the multimodal information recognition model and predict the three-dimensional coordinates of the finger endpoint, the curvature of the finger pointing vector, the target operation, and the operation confidence based on the fused features.

[0133] The multimodal information recognition model may also include an information prediction module, which may be used to predict the three-dimensional coordinates of the finger endpoint, the curvature of the finger pointing vector, the target operation, and the operation confidence level based on the fused features.

[0134] In another embodiment of the present application, the implementation process of "determining the operation object corresponding to the target operation according to the three-dimensional coordinates of the finger endpoint and the radian of the finger pointing vector" in step S204 in the above embodiment is introduced.

[0135] The process of determining the operation object corresponding to the target operation according to the three-dimensional coordinates of the finger endpoint and the arc of the finger pointing vector may include:

[0136] Step c1: convert the three-dimensional coordinates of the finger endpoint in the camera coordinate system into the three-dimensional coordinates of the finger endpoint in the world coordinate system.

[0137] See also Figure 5, shows a schematic diagram of the camera coordinate system and the world coordinate system, assuming that the angles between the camera coordinate system and the world coordinate system in the x, y, and z directions are θ, β, and α respectively, and the distances between the origin of the camera coordinate system and the origin of the world coordinate system in the x, y, and z directions are , the three-dimensional coordinates of the finger endpoint in the camera coordinate system can be converted to the three-dimensional coordinates of the finger endpoint in the world coordinate system as follows:

[0138] (6)

[0139] Among them, (x c ,y c ,z c ) represents the three-dimensional coordinates of the finger endpoint in the camera coordinate system, (x w ,y w ,z w ) represents the three-dimensional coordinates of the finger endpoint in the world coordinate system.

[0140] Step c2: convert the finger pointing vector radian in the camera coordinate system into the finger pointing direction vector in the world coordinate system.

[0141] If the finger pointing vector in the camera coordinate system is expressed as (γ1,γ2,γ3), then the finger pointing direction vector in the world coordinate system is .

[0142] Step c3: Determine the operation object corresponding to the target operation from the candidate operation objects according to the position information of each candidate operation object and the three-dimensional coordinates of the finger endpoint and the finger pointing direction vector in the world coordinate system.

[0143] Specifically, according to the position information of each candidate operation object and the three-dimensional coordinates of the finger endpoint and the finger pointing direction vector in the world coordinate system, the process of determining the operation object corresponding to the target operation from each candidate operation object may include:

[0144] Step c31: For each candidate operation object: determine whether the target ray intersects with the candidate operation object based on the position information of the candidate operation object, the normal vector of the plane where the candidate operation object is located, and the three-dimensional coordinates of the finger endpoint and the finger pointing direction vector in the world coordinate system.

[0145] The target ray is the three-dimensional coordinates of the finger endpoint in the world coordinate system (x w ,y w ,z w ) is the starting point of the ray, and the finger pointing direction vector in the world coordinate system The direction indicated is the ray direction.

[0146] Assume that the plane equation of a candidate operation object is ax+by+cz+d=0, then the plane normal vector =(a,b,c), P is any point on the plane, calculate:

[0147] (7)

[0148] Where F represents the three-dimensional coordinates of the finger endpoint in the world coordinate system (x w ,y w ,z w ), P represents the coordinates of any point on the candidate operation object (coordinates in the world coordinate system), The finger pointing direction vector.

[0149] If t>=0, it is determined that the target ray intersects with the candidate operation object.

[0150] Through the above process, the intersection status of the target ray and each candidate operation object can be obtained.

[0151] Step c32: Determine the candidate operation object that intersects with the target ray as the operation object corresponding to the target operation.

[0152] The candidate operation object intersecting with the target ray is the object pointed to by the hand of the target user, that is, the operation object corresponding to the target operation.

[0153] Based on the operation intention recognition method provided in the above embodiment, an embodiment of the present application also provides an interaction method, which includes: using the operation intention recognition method provided in the above embodiment to perform operation intention recognition, and obtaining an operation intention recognition result including a target operation and an operation object corresponding to the target operation; controlling the operation object corresponding to the target operation to perform the target operation.

[0154] The interactive method provided in the embodiment of the present application can accurately determine the operation intention by combining the hand image data and voice data of the target user, thereby reducing the false awakening rate and improving the interaction accuracy. In addition, in the smart cockpit scenario, the operation intention recognition can be carried out in conjunction with the hand depth map to realize the mutual control of the front and back rows (people sitting in the back row control the devices in the front row, and people sitting in the front row control the devices in the back row).

[0155] The above mainly introduces the operation intention recognition method provided in the embodiment of the present application, and the device corresponding to the above operation intention recognition method will be introduced below.

[0156] See also Figure 6 , Figure 6A structural schematic diagram of an operation intention recognition device provided in an embodiment of the present application, the operation intention recognition device may include: a multimodal data acquisition module 601, a hand information and operation information acquisition module 602, an operation intention determination module 603 and an operation object determination module 604.

[0157] The multimodal data acquisition module 601 is used to acquire the multimodal data of the target user, where the multimodal data of the target user includes hand image data and voice data.

[0158] The hand information and operation information acquisition module 602 is used to acquire the three-dimensional coordinates of the finger endpoints, the radian of the finger pointing vector, the target operation and the operation confidence according to the multimodal data of the target user.

[0159] The operation intention determination module 603 is used to determine whether the target user has an operation intention according to the operation confidence.

[0160] The operation object determination module 604 is used to determine the operation object corresponding to the target operation according to the three-dimensional coordinates of the finger endpoint and the radian of the finger pointing vector when the target user has an operation intention, so as to obtain an operation intention recognition result including the target operation and the operation object corresponding to the target operation.

[0161] In a possible implementation, the hand image data includes a hand RGB image and a hand depth map corresponding to the hand RGB image.

[0162] In a possible implementation, when acquiring the hand image data of the target user, the multimodal data acquisition module 601 is specifically used to:

[0163] Using an image acquisition device to acquire an RGB image of a target user's hand and a depth map corresponding to the RGB image;

[0164] Perform hand detection on the collected RGB image using a pre-trained first hand detection model to obtain position information of a hand region in the collected RGB image, wherein the first hand detection model is trained using a training image in which a hand region is marked;

[0165] According to the position information of the hand area, the hand area is extracted from the collected RGB image to obtain the hand RGB image;

[0166] According to the position information of the hand area, the hand area is extracted from the depth map corresponding to the collected RGB image, and a hand depth map corresponding to the hand RGB image is obtained.

[0167] In a possible implementation, the first hand detection model is trained using a training image with a hand region marked, and is supplemented by a pre-trained second hand detection model, and the scale of the second hand detection model is larger than that of the first hand detection model;

[0168] The training goal of the first hand detection model is to make the hand area obtained by performing hand detection on the training image using the first hand detection model close to the hand area obtained by performing hand detection on the training image using the second hand detection model, and to make the hand area obtained by performing hand detection on the training image using the first hand detection model close to the hand area annotated on the training image.

[0169] In a possible implementation, the hand information and operation information acquisition module 602 is specifically used to obtain the three-dimensional coordinates of the finger endpoint, the radian of the finger pointing vector, the target operation, and the operation confidence according to the multimodal data of the target user:

[0170] Using the pre-trained multimodal information recognition model, the multimodal data of the target user is recognized to obtain the three-dimensional coordinates of the finger endpoint, the arc of the finger pointing vector, the target operation, and the operation confidence;

[0171] Among them, the multimodal information recognition model is trained using training multimodal data marked with three-dimensional coordinates of finger endpoints, radians of finger pointing vectors, target operations, and operation confidence levels, and the training multimodal data includes training hand image data and training voice data.

[0172] In a possible implementation, the hand information and operation information acquisition module 602 uses a pre-trained multimodal information recognition model to perform information recognition on the multimodal data of the target user to obtain the three-dimensional coordinates of the finger endpoint, the arc of the finger pointing vector, the target operation, and the operation confidence, specifically for:

[0173] Using the pre-trained multimodal information recognition model, features are extracted from the hand image data and voice data of the target user to obtain image features and voice features;

[0174] Using the multimodal information recognition model, image features and speech features are fused to obtain fused features;

[0175] The multimodal information recognition model is used to predict the three-dimensional coordinates of the finger endpoint, the curvature of the finger pointing vector, the target operation and the operation confidence based on the fused features.

[0176] The three-dimensional coordinates of the finger endpoints and the radian of the finger pointing vector obtained by the hand information and operation information obtaining module 602 are both data in the camera coordinate system.

[0177] In a possible implementation, when the operation object determining module 604 determines the operation object corresponding to the target operation according to the three-dimensional coordinates of the finger endpoint and the radian of the finger pointing vector, it is specifically used to:

[0178] Convert the three-dimensional coordinates of the finger endpoint in the camera coordinate system to the three-dimensional coordinates of the finger endpoint in the world coordinate system;

[0179] Convert the finger pointing vector radian in the camera coordinate system to the finger pointing direction vector in the world coordinate system;

[0180] According to the position information of each candidate operation object and the three-dimensional coordinates of the finger endpoint and the finger pointing direction vector in the world coordinate system, the operation object corresponding to the target operation is determined from each candidate operation object.

[0181] In a possible implementation, the operation object determination module 604 is specifically used to determine the operation object corresponding to the target operation from each candidate operation object according to the position information of each candidate operation object and the three-dimensional coordinates of the finger endpoint and the finger pointing direction vector in the world coordinate system:

[0182] For each candidate operation object: determine whether a target ray intersects with the candidate operation object according to the position information of the candidate operation object, the normal vector of the plane where the candidate operation object is located, and the three-dimensional coordinates of the finger endpoint and the finger pointing direction vector in the world coordinate system, wherein the target ray is a ray with the three-dimensional coordinates of the finger endpoint in the world coordinate system as the ray starting point and the direction indicated by the finger pointing direction vector in the world coordinate system as the ray direction;

[0183] The candidate operation object intersecting with the target ray is determined as the operation object corresponding to the target operation.

[0184] The operation intention recognition device provided in the embodiment of the present application determines the operation intention based on the multimodal data (i.e., hand image data and voice data) of the target user when performing operation intention recognition. Since multimodal data can provide richer information and data of different modalities can complement each other, the operation intention recognition is performed by combining image data and voice data to obtain more accurate operation intentions, thereby reducing the false wake-up rate and improving the interaction accuracy, thereby improving the user experience.

[0185] An embodiment of the present application also provides an electronic device, which may include: at least one processor, at least one communication interface, at least one memory and at least one communication bus.

[0186] In the embodiment of the present application, the number of the processor, the communication interface, the memory, and the communication bus is at least one, and the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0187] The processor may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application;

[0188] The memory may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0189] The memory stores a program, and the processor can call the program stored in the memory, and the program is used to implement the steps of the operation intention recognition method or the steps of the interaction method provided in the above embodiment.

[0190] An embodiment of the present application also provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the operation intention recognition method or the steps of the interaction method provided in the above embodiment.

[0191] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the steps of the operation intention recognition method or the steps of the interaction method provided in the above embodiment.

[0192] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0193] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0194] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0195] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A method for identifying operation intention, characterized in that: include: Acquire multimodal data of a target user, wherein the multimodal data includes hand image data and voice data; According to the multimodal data of the target user, the three-dimensional coordinates of the finger endpoints, the radian of the finger pointing vector, the target operation and the operation confidence are obtained; Determining whether the target user has an operation intention according to the operation confidence; If the target user has an operation intention, the operation object corresponding to the target operation is determined according to the three-dimensional coordinates of the finger endpoint and the radian of the finger pointing vector to obtain an operation intention recognition result including the target operation and the operation object corresponding to the target operation.

2. The operation intention recognition method according to claim 1, characterized in that: The hand image data includes a hand RGB image and a hand depth map corresponding to the hand RGB image.

3. The operation intention recognition method according to claim 2, characterized in that: Obtain the target user's hand image data, including: Using an image acquisition device to acquire an RGB image of the target user's hand and a depth map corresponding to the RGB image; Perform hand detection on the collected RGB image using a pre-trained first hand detection model to obtain position information of a hand area in the collected RGB image, wherein the first hand detection model is trained using a training image in which a hand area is marked; Extracting the hand region from the acquired RGB image according to the position information of the hand region to obtain a hand RGB image; According to the position information of the hand area, the hand area is extracted from the depth map corresponding to the collected RGB image to obtain a hand depth map corresponding to the hand RGB image.

4. The operation intention recognition method according to claim 3, characterized in that: The first hand detection model is trained using a training image with a hand area marked, and is assisted by a pre-trained second hand detection model, wherein the scale of the second hand detection model is greater than that of the first hand detection model; The training goal of the first hand detection model is to make the hand area obtained by using the first hand detection model to perform hand detection on the training image close to the hand area obtained by using the second hand detection model to perform hand detection on the training image, and to make the hand area obtained by using the first hand detection model to perform hand detection on the training image close to the hand area annotated on the training image.

5. The operation intention recognition method according to claim 1, characterized in that: The acquiring, according to the multimodal data of the target user, the three-dimensional coordinates of the finger endpoint, the radian of the finger pointing vector, the target operation and the operation confidence comprises: Using the pre-trained multimodal information recognition model, information recognition is performed on the multimodal data of the target user to obtain the three-dimensional coordinates of the finger endpoints, the radian of the finger pointing vector, the target operation, and the operation confidence; The multimodal information recognition model is trained using training multimodal data annotated with three-dimensional coordinates of finger endpoints, radians of finger pointing vectors, target operations, and operation confidence levels, and the training multimodal data includes training hand image data and training voice data.

6. The operation intention recognition method according to claim 5, characterized in that: The multimodal information recognition model obtained by pre-training is used to perform information recognition on the multimodal data of the target user to obtain the three-dimensional coordinates of the finger endpoint, the radian of the finger pointing vector, the target operation and the operation confidence, including: Using the pre-trained multimodal information recognition model, extract features from the hand image data and voice data of the target user to obtain image features and voice features; Using the multimodal information recognition model, the image feature and the speech feature are fused to obtain fused features; The multimodal information recognition model is used to predict the three-dimensional coordinates of the finger endpoint, the curvature of the finger pointing vector, the target operation and the operation confidence based on the fused features.

7. The operation intention recognition method according to claim 1, characterized in that: The three-dimensional coordinates of the finger endpoint and the radian of the finger pointing vector are both data in the camera coordinate system; The determining, according to the three-dimensional coordinates of the finger endpoint and the radian of the finger pointing vector, an operation object corresponding to the target operation includes: Convert the three-dimensional coordinates of the finger endpoint in the camera coordinate system to the three-dimensional coordinates of the finger endpoint in the world coordinate system; Convert the finger pointing vector radian in the camera coordinate system to the finger pointing direction vector in the world coordinate system; According to the position information of each candidate operation object and the three-dimensional coordinates of the finger endpoint and the finger pointing direction vector in the world coordinate system, the operation object corresponding to the target operation is determined from the candidate operation objects.

8. The method for identifying operation intention according to claim 7, characterized in that: The step of determining the operation object corresponding to the target operation from the candidate operation objects according to the position information of the candidate operation objects and the three-dimensional coordinates of the finger endpoint and the finger pointing direction vector in the world coordinate system includes: For each candidate operation object: determine whether a target ray intersects with the candidate operation object according to the position information of the candidate operation object, the normal vector of the plane where the candidate operation object is located, and the three-dimensional coordinates of the finger endpoint and the finger pointing direction vector in the world coordinate system, wherein the target ray is a ray with the three-dimensional coordinates of the finger endpoint in the world coordinate system as the ray starting point and the direction indicated by the finger pointing direction vector in the world coordinate system as the ray direction; The candidate operation object intersecting with the target ray is determined as the operation object corresponding to the target operation.

9. An interactive method, characterized in that: include: Using the operation intention recognition method according to any one of claims 1 to 8 to perform operation intention recognition, and obtain an operation intention recognition result including a target operation and an operation object corresponding to the target operation; Controlling the operation object corresponding to the target operation to perform the target operation.

10. An operation intention recognition device, characterized in that: include: Multimodal data acquisition module, hand information and operation information acquisition module, operation intention determination module and operation object determination module; The multimodal data acquisition module is used to acquire multimodal data of the target user, wherein the multimodal data includes hand image data and voice data; The hand information and operation information acquisition module is used to acquire the three-dimensional coordinates of the finger endpoints, the radian of the finger pointing vector, the target operation and the operation confidence according to the multimodal data of the target user; The operation intention determination module is used to determine whether the target user has an operation intention according to the operation confidence; The operation object determination module is used to determine the operation object corresponding to the target operation according to the three-dimensional coordinates of the finger endpoint and the radian of the finger pointing vector when the target user has an operation intention, so as to obtain an operation intention recognition result including the target operation and the operation object corresponding to the target operation.

11. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the steps of the operation intention recognition method as described in any one of claims 1 to 8, or implement the steps of the interaction method as described in claim 9.

12. A computer storage medium, characterized in that: The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the operation intention recognition method as described in any one of claims 1 to 8, or implement the steps of the interaction method as described in claim 9.

13. A computer program product, characterized in that It includes computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the steps of the operation intention recognition method as described in any one of claims 1 to 8, or implement the steps of the interaction method as described in claim 9.

Images