Voice and gesture multi-mode fusion unmanned platform control method and system

Abstract

The invention discloses a voice and gesture multi-mode fusion unmanned platform control method and system, and belongs to the field of human-computer interaction. The method comprises the following steps: S1, acquiring three-dimensional position information of key points of a hand; s2, identifying a gesture action; s3, classifying the gesture instruction: if the gesture instruction is a voice wake-up instruction, executing S4 to S7; if the instruction is an unmanned platform control instruction, executing S7; if the instruction is other instruction, continuing to execute S1; s4, collecting voice information; s5, converting the voice information into text information; s6, recognizing instruction operation in the text information by using a voice instruction recognition module; and S7, updating an unmanned platform instruction slot, and sending an instruction to the unmanned platform. The unmanned platform is controlled through voice and gesture multi-mode fusion, the control convenience, flexibility and accuracy of the unmanned platform are improved, and therefore more natural and visual control experience is provided for a user.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of human-computer interaction, and particularly relates to a voice and gesture multi-modal fusion unmanned platform control method and system. BACKGROUND

[0002] With the rapid development of unmanned platform technology, the application of unmanned driving, unmanned aerial vehicles, and automated logistics is gradually popularized. Voice and gesture, as two natural ways of human-computer interaction, have important significance in improving user experience, operation convenience, and efficiency. Traditional remote control methods and manual control methods often have problems such as complicated operation, delayed response, and heavy user burden. Especially when operating in complex environments, users may need to be distracted by multiple control devices or operation methods, which limits the widespread application of unmanned platforms. Voice and gesture control methods can fully utilize human natural behavior, provide more intuitive and convenient interaction methods, and greatly improve the control flexibility of unmanned platforms and user operation experience.

[0003] However, in the prior art, although voice control and gesture control each have certain advantages, they also have certain limitations. For example, voice control is easily disturbed in noisy environments, reducing accuracy; and gesture control may be affected by spatial limitations or unclear user posture in long-distance or complex environments. Therefore, how to effectively integrate voice and gesture interaction methods, fully utilize their respective advantages, and improve the control accuracy and stability of unmanned platforms remains a difficult problem in current technology. SUMMARY

[0004] The present application proposes a voice and gesture multi-modal fusion unmanned platform control method and system, which effectively improves the convenience of unmanned platform control and aims to solve the limitations of single control methods in the prior art. By integrating voice recognition and gesture recognition technologies, the present application can achieve more convenient and efficient control of unmanned platforms.

[0005] A voice and gesture multi-modal fusion unmanned platform control method, comprising the following steps: S1, collecting hand key point three-dimensional position and speed information; S2, recognizing gesture actions; S3, classifying gesture instructions according to gesture actions: if the gesture action is a voice wake-up instruction, executing S4-S7; if the gesture action is an unmanned platform operation instruction, executing S7; if the gesture action is other instructions, continuing to execute S1; S4, collecting voice information; S5, converting voice information into text information; S6, using a voice instruction recognition module to recognize operation instructions in the text information; S7 updates the command slot of the unmanned platform and sends operation commands to the unmanned platform.

[0006] Optionally, step S1 is performed using a gesture acquisition module, which can acquire the three-dimensional spatial coordinates of each skeletal joint of the hand and the movement speed of the palm. Each skeletal joint includes the metacarpophalangeal joints, proximal phalangeal joints, middle phalangeal joints, and fingertips of each finger. The three-dimensional coordinates of the relevant skeletal joints of each finger are as follows: thumb (… ),index finger( ), middle finger ( ), ring finger ( ), little finger ( ),in: , , and These are the three-dimensional coordinates of the metacarpophalangeal joint of the thumb, the proximal phalanx joint, the middle phalanx joint, and the fingertip. , , and These are the three-dimensional coordinates of the metacarpophalangeal joint of the index finger, the proximal phalanx joint, the middle phalanx joint, and the fingertip. , , and These are the three-dimensional coordinates of the metacarpophalangeal joint of the middle finger, the proximal phalanx joint, the middle phalanx joint, and the fingertip. , , and These are the three-dimensional coordinates of the metacarpophalangeal joint, proximal phalanx joint, middle phalanx joint, and fingertip of the ring finger, respectively. , , and These are the three-dimensional coordinates of the metacarpophalangeal joint of the little finger, the proximal phalanx joint, the middle phalanx joint, and the fingertip; while the movement speed of the palm includes the lateral movement speed of the palmar skeletal points along the horizontal plane. The longitudinal movement speed of the palm bones along the horizontal plane and the speed of movement of the palm bones along the height direction The horizontal plane is defined as the left-right direction, the vertical plane is defined as the front-back direction, and the height direction is defined as the up-down direction.

[0007] Optionally, the gestures include: pinching with two fingers, pinching with three fingers, keeping hands still, raising hands while spreading them out, lowering hands while spreading them out, raising hands while spreading them out to the left, raising hands while spreading them out to the right, raising hands while spreading them out to the front, and raising hands while spreading them out to the back, or all of these gestures.

[0008] Optionally, the step S2 of recognizing the gesture action comprises the following sub-steps: S21, if and , is a geometric length tolerance, the gesture action is recognized as a two-finger pinch, otherwise, go to next step; S22, if and , the gesture action is recognized as a three-finger pinch, otherwise, go to next step; S23, if and , , , , , , the gesture action is recognized as a hand-spread gesture, wherein is an angle maximum tolerance; in the formula, the symbol represents an angle, represents an angle formed by as a vertex and and as two sides; S24, if none of the above is met, the gesture action is recognized as other gestures.

[0009] Optionally, the sub-step S23 comprises the following sub-steps: if , , , it is recognized as a hand-spread static gesture; if , , , it is recognized as a hand-spread left swing gesture; if , , , it is recognized as a hand-spread right swing gesture; if , , , it is recognized as a hand-spread up swing gesture; if , , , it is recognized as a hand-spread down swing gesture; if , , , it is recognized as a hand-spread forward swing gesture; if , , , it is recognized as a hand-spread backward swing gesture; wherein, is the speed tolerance.

[0010] Optionally, in the step S3, the gesture action of the voice wake-up instruction includes double-finger pinch and three-finger pinch; the gesture action of the unmanned platform operation instruction includes hand spreading and keeping still, hand spreading and moving up, hand spreading and moving down, hand spreading and moving left, hand spreading and moving right, hand spreading and moving forward, and hand spreading and moving backward; and the gesture action of the other instruction is other gesture.

[0011] Optionally, in the step S7, the unmanned platform operation instruction slot information includes {“on_moving”: yes / false; “moving_direction”: ahead / back / left / right / up / down / null}, wherein “on_moving” is used to indicate whether the unmanned platform is moving, “yes” indicates “yes”, and “false” indicates “no”; “moving_direction” indicates the flight direction of the unmanned platform, which can be selected from “ahead”, “back”, “left”, “right”, “up”, “down”, and “keep”, respectively indicating “flying forward”, “flying backward”, “flying left”, “flying right”, “flying up”, “flying down”, and “keeping state”.

[0012] Optionally, the correspondence between the gesture operation instruction and the unmanned platform operation instruction is that hand spreading and keeping still corresponds to stopping flying, hand spreading and moving up corresponds to flying upward, hand spreading and moving down corresponds to flying downward, hand spreading and moving left corresponds to flying leftward, hand spreading and moving right corresponds to flying rightward, hand spreading and moving forward corresponds to flying forward, and hand spreading and moving backward corresponds to flying backward.

[0013] Optionally, the voice instruction recognition module recognizes the voice instruction by constructing a question and answer module and then using a general large language model.

[0014] In addition, the present application also provides a voice and gesture multi-mode fusion unmanned platform control system, which comprises a microprocessor and a memory connected with each other, and the microprocessor is programmed or configured to execute the steps of the voice and gesture multi-mode fusion unmanned platform control method.

[0015] Compared with the prior art, the technical scheme of the present application has the following beneficial effects: I. Compared with the traditional voice control method of the unmanned platform, the present application does not need to train the voice recognition system according to the voice input instruction, but uses an open-source general voice recognition tool combined with a general large language model to recognize the input voice control instruction of the unmanned platform.

[0016] Compared with the voice or gesture single-mode unmanned platform slot method, the unmanned platform control method based on voice and gesture multi-modal fusion is constructed, and the convenience of unmanned platform control is greatly improved. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the drawings shown.

[0018] Figure 1 is a basic flowchart of the method of the embodiment of the present application; Figure 2 is a three-dimensional position diagram of the hand joint in the embodiment of the present application; Figure 3 is a three-finger pinch gesture collection diagram in the embodiment of the present application; Figure 4 is an interface interaction state diagram after the wake-up voice instruction in the embodiment of the present application; Figure 5 is a state diagram of the unmanned platform control instruction slot after inputting the "fly forward" voice in the embodiment of the present application; Figure 6 is a state diagram of the unmanned platform control instruction slot after making a hand-down gesture in the embodiment of the present application. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0020] The present application designs a voice and gesture multi-modal fusion unmanned platform control method, as shown in Figure 1 The steps and technical principles are as follows: Step S1, collecting three-dimensional position information and speed information of hand key points by using a gesture collection module; The gesture collection module can collect three-dimensional spatial position coordinates of each skeletal joint of the hand and the moving speed of the palm center, wherein each skeletal joint includes each finger metacarpal joint, proximal phalanx joint, middle phalanx joint and fingertip point. The three-dimensional position coordinates of each finger related skeletal joint are as follows: ), index finger ( ), middle finger ( ), ring finger (ring finger) ), little finger (little finger) , wherein: , , and are the three-dimensional position coordinates of the metacarpophalangeal joint, proximal phalanx joint, middle phalanx joint and fingertip point of the thumb, respectively; , , and are the three-dimensional position coordinates of the metacarpophalangeal joint, proximal phalanx joint, middle phalanx joint and fingertip point of the index finger, respectively; , , and are the three-dimensional position coordinates of the metacarpophalangeal joint, proximal phalanx joint, middle phalanx joint and fingertip point of the middle finger, respectively; , , and are the three-dimensional position coordinates of the metacarpophalangeal joint, proximal phalanx joint, middle phalanx joint and fingertip point of the ring finger, respectively; , , and are the three-dimensional position coordinates of the metacarpophalangeal joint, proximal phalanx joint, middle phalanx joint and fingertip point of the little finger, respectively; and the movement speed of the palm includes the movement speed of the palm skeleton point along the horizontal plane in the horizontal direction , the movement speed of the palm skeleton point along the horizontal plane in the longitudinal direction , and the movement speed of the palm skeleton point along the height direction .

[0021] A three-dimensional space coordinate system is established for describing the three-dimensional positions of the hand key points and the movement speed of the palm, which is the basis for realizing gesture action recognition. The coordinate system direction of the three-dimensional space coordinate system is defined as follows: Horizontal direction (X-axis): defined as left-right direction, corresponding to the horizontal movement speed of the palm skeleton point along the horizontal plane; Horizontal direction (Y-axis): defined as front-back direction, corresponding to the horizontal movement speed of the palm skeleton point along the horizontal plane; Height direction (Z-axis): defined as up-down direction, corresponding to the movement speed of the palm skeleton point along the height direction.

[0022] In the examples of the present application, the Leap Motion Controller 2 sensor is used to collect the three-dimensional position and speed information of each bone joint of the hand in real time, mainly including the metacarpal joint, proximal bone, middle metacarpal bone and distal bone of each finger joint, and the three-dimensional position coordinates are respectively ),index finger( ), middle finger ( ), ring finger ( ), little finger ( The movement velocity component of the palm skeleton is... , , .

[0023] A three-dimensional diagram of the joints of the hand is shown below. Figure 2 As shown. Figure 2 It shows the positional definitions of the joints of the hand bones in this three-dimensional coordinate system, including the joint coordinates of the thumb, index finger, middle finger, ring finger, and little finger.

[0024] Step S2: Use the gesture recognition module to recognize gesture actions; In this embodiment, the main hand gestures include the following: pinching with two fingers, pinching with three fingers, keeping hands still, raising hands while spreading them out, lowering hands while spreading them out, raising hands while spreading them out to the left, raising hands while spreading them out to the right, raising hands while spreading them out to the front, and raising hands while spreading them out to the back.

[0025] The gesture recognition module in step S2 specifically includes the following sub-steps S21 to S24: S21, if and , If the geometric length tolerance is met, it is identified as a two-finger pinch; otherwise, proceed to the next step. S22, if and , If the geometric length tolerance is met, it is identified as a three-finger pinch; otherwise, proceed to the next step. S23, if , , , , , , , If so, it is recognized as a shrug gesture, and further includes the following sub-steps: S231, if , , If so, it will be recognized as a still, open-hand gesture; S232, if , , If so, it is recognized as a leftward waving gesture; S233, if , , If so, it will be recognized as a shrug with the hands outstretched to the right. S234, if , , If so, it will be recognized as a gesture of spreading one's hands on a shrug; S235, if , , If so, it will be recognized as a shrug or downward gesture; S236, if , , If so, it is recognized as a shrug or outstretched hand gesture. S237, if , , If so, it will be recognized as a shrug or wave. S24. If none of the above applies, then it is identified as another gesture.

[0026] Step S3: Classify the gesture commands according to the gesture actions: If it is a voice wake-up command, execute S4 to S7; if it is a gesture control command for the unmanned platform, execute S7; if it is another command, continue to execute S1.

[0027] Step S4: Collect voice information using the voice acquisition module.

[0028] Step S5: Use the speech-to-text conversion module to convert speech information into text information.

[0029] Step S6: Use the voice command recognition module to recognize the command operation in the text information; The voice command recognition module constructs a question-and-answer module and then uses a general-purpose large language model to recognize voice commands. General-purpose large language models include DeepSeek, GPT-4, Gemini, Claude, LLaMA, and domestic models such as ERNIE, Qwen, and Xunfei Xinghuo.

[0030] Step S7: Update the command slot of the unmanned platform and send the operation command to the unmanned platform.

[0031] The correspondence between gesture operation commands and unmanned platform control commands is as follows: spreading your hand and keeping it still corresponds to stopping flight; spreading your hand and raising it corresponds to flying upwards; spreading your hand and lowering it corresponds to flying downwards; spreading your hand and raising it to the left corresponds to flying to the left; spreading your hand and raising it to the right corresponds to flying to the right; spreading your hand and raising it forward corresponds to flying forwards; and spreading your hand and raising it backwards corresponds to flying backwards.

[0032] In this embodiment, after making a three-finger pinch gesture, the hand key point positions collected by the gesture acquisition module are as follows: Figure 3 As shown. After activating voice commands with a three-finger pinch gesture, the interface interaction state is as follows. Figure 4 As shown, you are prompted to enter a voice command.

[0033] In this embodiment, the Python speech_recognition library is used to drive the microphone for audio recording, and the voice control command "fly forward" is input.

[0034] In this embodiment, the recorded audio file is converted into a text file speech_to_tex using Alibaba's open-source speech model SenseVoice, where speech_to_tex = "flying forward".

[0035] In this embodiment, the constructed voice command recognition and question-and-answer module is as follows: Question 1: "I'm controlling a drone. Based on my current <state list>, and my <instructions>, analyze my intentions and derive the <target state list> I want. Below is a JSON-formatted <state list>: {"on_moving": yes; "moving_direction": ahead}; The "on_moving" option indicates whether the drone is moving and can be selected from "yes" or "false". The "moving_direction" option can be selected from "ahead", "back", "left", "right", "up", "down", and "keep", which respectively represent "flying forward", "flying backward", "flying left", "flying right", "flying up", "flying down", and "keeping the state".

[0036] Question 2: "My control command is "+speech_to_tex+" Please return the list of states I want in JSON format according to my control command. Questions 1 and 2 are progressively input into the Tongyi Qianwen Large Language Model. Based on the voice input "fly forward," the system updates the unmanned platform control command slot to {"on_moving": yes; "moving_direction": ahead}, as shown below. Figure 5 As shown.

[0037] In this embodiment, gesture control of the unmanned platform continues. After making a downward shrug gesture, the unmanned platform control command slot is updated to {"on_moving": yes; "moving_direction": down}, and the unmanned platform's flight state changes to downward flight, as shown below. Figure 6 As shown.

[0038] The present invention further provides a voice and gesture multi-modal fusion unmanned platform control system, including a microprocessor and a memory interconnected thereto, wherein the microprocessor is programmed or configured to execute the steps of the voice and gesture multi-modal fusion unmanned platform control method.

[0039] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. All equivalent designs made under the inventive concept of the present invention using the contents of the present invention specification and drawings, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A method for controlling an unmanned platform using multi-modal fusion of voice and gesture, characterized in that, Includes the following steps: S1, collects three-dimensional position and velocity information of key points on the hand; S2, recognizes hand gestures; S3, classify the gesture commands according to the gesture: if the gesture is a voice wake-up command, then execute S4 to S7; if the gesture is an unmanned platform operation command, then execute S7; if the gesture is another command, then continue to execute S1. S4, collects voice information; S5 converts voice information into text information; S6 uses a voice command recognition module to recognize operation commands in text information; S7 updates the command slot of the unmanned platform and sends operation commands to the unmanned platform.

2. The unmanned platform control method based on voice and gesture multi-modal fusion according to claim 1, characterized in that, Step S1 is performed using a gesture acquisition module, which can acquire the three-dimensional spatial coordinates of each skeletal joint of the hand and the movement speed of the palm. Each skeletal joint includes the metacarpophalangeal joints, proximal phalangeal joints, middle phalangeal joints, and fingertips. The three-dimensional coordinates of the relevant skeletal joints of each finger are as follows: thumb (… ),index finger( ), middle finger ( ), ring finger ( ), little finger ( ),in: , , and These are the three-dimensional coordinates of the metacarpophalangeal joint of the thumb, the proximal phalanx joint, the middle phalanx joint, and the fingertip. , , and These are the three-dimensional coordinates of the metacarpophalangeal joint of the index finger, the proximal phalanx joint, the middle phalanx joint, and the fingertip. , , and These are the three-dimensional coordinates of the metacarpophalangeal joint of the middle finger, the proximal phalanx joint, the middle phalanx joint, and the fingertip. , , and These are the three-dimensional coordinates of the metacarpophalangeal joint, proximal phalanx joint, middle phalanx joint, and fingertip of the ring finger, respectively. , , and These are the three-dimensional coordinates of the metacarpophalangeal joint of the little finger, the proximal phalanx joint, the middle phalanx joint, and the fingertip; while the movement speed of the palm includes the lateral movement speed of the palmar skeletal points along the horizontal plane. The longitudinal movement speed of the palm bones along the horizontal plane and the speed of movement of the palm bones along the height direction The horizontal plane is defined as the left-right direction, the vertical plane is defined as the front-back direction, and the height direction is defined as the up-down direction.

3. The unmanned platform control method based on voice and gesture multi-modal fusion according to claim 2, characterized in that, The gestures include: pinching with two fingers, pinching with three fingers, keeping hands still, raising hands while spreading them out, lowering hands while spreading them out, raising hands while spreading them out to the left, raising hands while spreading them out to the right, raising hands while spreading them out to the front, and raising hands while spreading them out to the back, or all of these gestures.

4. The unmanned platform control method based on voice and gesture multi-modal fusion according to claim 3, characterized in that, Step S2, which identifies hand gestures, includes the following sub-steps: S21, if and , If the geometric length tolerance is met, the gesture will be recognized as a two-finger pinch; otherwise, proceed to the next step. S22, if and If the gesture is recognized as a three-finger pinch, proceed to the next step; otherwise, proceed to the next step. S23, if, and , , , , , The gesture will be recognized as a shrug. The maximum tolerance for angle; The symbols in the formula are... Indicates angle, Indicates As the vertex, with and The angle formed by the two sides; S24. If none of the above conditions are met, the gesture will be identified as another gesture.

5. The unmanned platform control method based on voice and gesture multi-modal fusion according to claim 4, characterized in that, Sub-step S23 includes the following sub-steps: if , , If so, it will be recognized as a still, open-hand gesture; if , , If so, it is recognized as a leftward waving gesture; if , , If so, it will be recognized as a shrug with the hands outstretched to the right. if , , If so, it will be recognized as a gesture of spreading one's hands on a shrug; if , , If so, it will be recognized as a shrug or downward gesture; if , , If so, it is recognized as a shrug or outstretched hand gesture. if , , If so, it will be recognized as a shrug or wave. in, For speed tolerance.

6. The unmanned platform control method based on voice and gesture multi-modal fusion according to claim 1, characterized in that, In step S3, the hand gestures for voice wake-up commands include pinching with two fingers and pinching with three fingers; the hand gestures for unmanned platform operation commands include shrugging hands still, shrugging hands up, shrugging hands down, shrugging hands left, shrugging hands right, shrugging hands forward, and shrugging hands backward; the hand gestures for other commands are other gestures.

7. The unmanned platform control method based on voice and gesture multi-modal fusion according to claim 1, characterized in that, In step S7, the unmanned platform control command slot information includes {"on_moving": yes / false; "moving_direction": ahead / back / left / right / up / down / null}, where "on_moving" indicates whether the unmanned platform is moving, "yes" means "yes" and "false" means "no"; "moving_direction" indicates the flight direction of the unmanned platform, which can be selected from "ahead", "back", "left", "right", "up", "down" and "keep", which respectively represent "fly forward", "fly backward", "fly left", "fly right", "fly up", "fly down" and "keep state".

8. The unmanned platform control method based on voice and gesture multi-modal fusion according to claim 1, characterized in that, The correspondence between gesture operation commands and unmanned platform control commands is as follows: spreading your hand and keeping it still corresponds to stopping flight; spreading your hand and raising it corresponds to flying upwards; spreading your hand and lowering it corresponds to flying downwards; spreading your hand and raising it to the left corresponds to flying to the left; spreading your hand and raising it to the right corresponds to flying to the right; spreading your hand and raising it forward corresponds to flying forwards; and spreading your hand and raising it backwards corresponds to flying backwards.

9. The unmanned platform control method based on voice and gesture multi-modal fusion according to claim 1, characterized in that, The voice command recognition module constructs a question-and-answer module and then uses a general large language model to recognize voice commands.

10. A voice and gesture multi-modal fusion unmanned platform control system, comprising a microprocessor and a memory interconnected thereon, characterized in that, The microprocessor is programmed or configured to perform the steps of the voice and gesture multimodal fusion unmanned platform control method according to any one of claims 1 to 9.

Images