A human-swarm interaction method and system based on augmented reality

Abstract

The application discloses a kind of person-cluster interaction method and system based on augmented reality, including the following steps: obtaining custom hand gesture, hand gesture is mapped to real unmanned aerial vehicle cluster to generate the hand operation instruction of real cluster, based on hand operation instruction, guide real unmanned aerial vehicle cluster to complete task according to hand operation instruction, or: obtain the interactive operation instruction of real cluster, based on interactive operation instruction, guide real unmanned aerial vehicle cluster to complete task according to interactive operation instruction, or: obtain the area operation instruction of real cluster, based on area operation instruction, guide real unmanned aerial vehicle cluster to gather in the area corresponding to area operation instruction to complete task;The interaction method and system can provide more natural, various interaction modes by fusing physical world and virtual world.

Description

Technical Field

[0001] This invention relates to the field of augmented reality technology, and in particular to a human-crowd interaction method and system based on augmented reality. Background Technology

[0002] Augmented Reality (AR) technology is a technology that combines the real world with virtual information based on real-time computer computing and multi-sensor fusion. This technology simulates and reproduces human senses such as sight, hearing, smell, and touch, overlaying virtual information onto real information to provide an experience that transcends the real world. HoloLens, developed by Microsoft in 2016, is a state-of-the-art head-mounted AR device that allows users to interact with their environment using holograms. Its functionality is more advanced than traditional AR devices, adding features such as 3D display, gaze design, and gesture design.

[0003] Human-to-machine interaction (HMI) refers to the process of information exchange between humans and machines. The goal of HMI is to transform user behavior and states (input) into a representation that machines can understand and manipulate through appropriate metaphors, and to transform machine behavior and states (output) into a representation that humans can understand and manipulate, and then feed this back to humans. Traditional HMI methods are relatively simple, relying on machine language and using intermediate devices such as keyboards and mice for interaction. Summary of the Invention

[0004] Based on the technical problems existing in the background technology, this invention proposes a human-crowd interaction method and system based on augmented reality, which integrates the physical world and the virtual world and can provide more natural and diverse interaction methods.

[0005] The present invention proposes a human-cluster interaction method based on augmented reality, comprising the following steps:

[0006] A custom hand gesture is obtained, and the hand gesture is mapped to a real drone swarm to generate hand operation commands for the real swarm. Based on the hand operation commands, the real drone swarm is guided to complete the task according to the hand operation commands. The hand gesture is responded to by detecting and tracking the joints of both hands through a HoloLens2 device, and determining the hand gesture based on the joints; or:

[0007] Obtain interactive operation commands from the real drone cluster, and guide the real drone cluster to complete tasks according to the interactive operation commands. The interactive operation commands are responded to by adding a virtual drone cluster in the real scene and obtaining the pose state of the virtual drone cluster, and generating interactive operation commands through the interaction between the real drone cluster and the virtual drone cluster; or:

[0008] The system obtains the regional operation instructions of the real drone cluster and guides the real drone cluster to gather in the area corresponding to the regional operation instructions to complete the task. The regional operation instructions are generated by changing the parameter information in the working environment of the real drone cluster and conveying the importance of each location in the real environment to the real drone cluster.

[0009] Furthermore, after acquiring a custom hand pose, the hand pose is mapped to a real drone swarm to generate hand operation commands for the real swarm. Based on the hand operation commands, the real drone swarm is guided to complete the task according to the hand operation commands. The hand pose response is achieved by detecting and tracking the joints of both hands through the HoloLens2 device. The hand pose determination based on the joints specifically includes:

[0010] (A1) Use the HoloLens2 augmented reality device to perform frame-by-frame detection of hand images within the field of view;

[0011] (A2) Extract the three-dimensional coordinates of the hand skeleton nodes and the number of hands N from the hand image in (A1) and map them into the rendered digital hand model;

[0012] (A3) Calculate the geometric features of the gesture based on the three-dimensional coordinates of the hand skeleton nodes, and determine whether the number of hands N is greater than 2. If yes, proceed to (A4); otherwise, proceed to (A5).

[0013] (A4) The gesture is currently invalid;

[0014] (A5) If the number of hands N equals 2, then the two-handed gesture recognition algorithm is used to recognize the two-handed gestures according to the gesture geometric features to obtain the gesture recognition action; if the number of hands is less than 2 and the detected hand is the right hand, then it is an invalid gesture; otherwise, the one-handed gesture recognition algorithm is used to recognize the one-handed gesture according to the gesture geometric features to obtain the gesture recognition action.

[0015] (A6) Define special gestures based on gesture geometry features;

[0016] (A7) Select the virtual target drone object by tapping in the air, bind elements to key parts of the finger joints, and trigger hand operation commands when the elements are activated.

[0017] Furthermore, the gesture geometry features in (A3) include the fingertip coordinates P of each finger. tip The coordinates P of the middle joint of each finger. middle Palm coordinates P palm Palm forward vector L forward finger bending angle threshold θ curl The threshold angle of finger extension θ straightThe angle θ between the normal line pointing upwards from the palm and the forward ray from the camera. palm，camera The threshold angle θ between the palm-up normal and the forward ray of the camera when the palm faces the camera. facing The threshold θ for palm opening flat The distance d between the tips of the index and middle fingers of the same hand index，middle .

[0018] Furthermore, in step (A6), specifically:

[0019] If all fingers are extended and the palm faces the camera, the gesture recognition action is "palm facing inwards", and the real drone operation command is "take off".

[0020] If all fingers are extended and the back of the hand faces the camera, the gesture recognition action is "palm facing outwards", and the actual drone operation command is "land".

[0021] If the thumb is extended straight up, the remaining fingers are bent towards the palm, the palm faces the camera, and the detected hand is the left hand, then the gesture recognition action is "raise a thumb", and the actual drone operation command is "turn left".

[0022] If the thumb is extended straight up, the remaining fingers are bent towards the palm, the palm faces the camera, and the detected hands are the left and right hands, then the gesture recognition action is "raise two thumbs", and the real drone operation command is "turn right".

[0023] If the index and middle fingers are straight, the remaining fingers are bent, the distance between the fingertips of the index and middle fingers is greater than 4 centimeters, and the detected hand is the left hand, then the gesture recognition action is "a victory gesture", and the real drone operation command is "up".

[0024] If the index and middle fingers are straight, the remaining fingers are bent, the distance between the fingertips of the index and middle fingers is greater than 4 centimeters, and the detected hands are the left and right hands, then the gesture recognition action is "two victory gestures", and the real drone operation command is "down".

[0025] If the index finger is straight and the remaining fingers are bent, and the detected hand is the left hand, then the gesture recognition action is "single finger", and the actual drone operation command is "forward".

[0026] If the index finger is straight and the remaining fingers are bent, and the detected hands are the left and right hands, then the gesture recognition action is "two fingers", and the actual drone operation command is "backward".

[0027] Further, in step (A7), the virtual target drone object is selected by tapping in the air. Specifically, the virtual target drone is aimed at with a ray, and the ray trajectory is displayed in the virtual scene. If the index finger and thumb briefly touch and then separate while the ray is aimed at the virtual drone, the drone is selected and highlighted.

[0028] Furthermore, in obtaining interactive operation instructions from the real drone cluster, and guiding the real drone cluster to complete tasks according to the interactive operation instructions, the interactive operation instructions specifically include the following: (The interactive operation instructions are provided in the context of adding a virtual drone cluster in the real scene and obtaining the pose state of the virtual drone cluster.)

[0029] (B1) Create and add virtual drone models to the set scene, and create animations for the virtual drone swarm;

[0030] (B2) Design a UI interface to access the pose status of the virtual drone cluster. The pose status includes drone coordinates, drone attitude, drone trajectory, head-up display, flight screen, and environmental information.

[0031] (B3) Draw the desired path by grabbing and dragging the virtual drone cluster, and record and save the key points of the desired path;

[0032] (B4) Based on the pose state and key points of the desired path, the real UAV cluster and the virtual UAV cluster interact to generate interactive operation instructions, guiding the real UAV cluster to complete the task according to the interactive operation instructions.

[0033] Furthermore, in obtaining the regional operation instructions of the real drone cluster, and guiding the real drone cluster to gather in the area corresponding to the regional operation instructions to complete the task, the regional operation instructions are generated by changing the parameter information in the working environment of the real drone cluster to convey the importance of each location in the real environment to the real drone cluster. Specifically, this includes:

[0034] (C1) Create a two-dimensional scene map based on the real-world environment. The working domain of the real drone swarm on the two-dimensional scene map is... The density function φ(q, t) represents the importance of q∈D at time t, where R represents the real number field, D represents the two-dimensional scene map of the actual drone operation, and q represents any point on the two-dimensional scene map.

[0035] (C2) Click on the area of ​​interest on the 2D scene map to increase the density function value of the clicked area;

[0036] (C3) Use a Gaussian mixture model with fixed weights and shapes to transform the clicked area into a real-time density function on a two-dimensional scene map. Update the control rate of the real drone swarm according to the real-time density function, and use the control rate as an area operation command to guide the real drone swarm to gather in the specified area to complete the task.

[0037] Furthermore, the density function φ(q,t) and the control law The formula is as follows:

[0038]

[0039]

[0040]

[0041]

[0042]

[0043]

[0044]

[0045]

[0046] Where ε0 represents a positive constant, μ i (t) represents the center position of the i-th Gaussian function. μ i The derivative of (t), r i (t) represents the click position, τ i，1 τ represents the time at which the i-th Gaussian function is added. i，2 Let H(p,t) represent the time when the i-th Gaussian function leaves, and let V represent the cost function. i (p) represents the Vino unit managed by the i-th virtual drone, m i and c i p is the mass and centroid of the i-th Vino unit. i ∈D represents the position of the i-th drone, R represents the number of fingers tapped by the user, q represents any point on the 2D scene map, p represents the actual position of the drone, and l i (t) represents the weight of the i-th Gaussian function, T represents the transpose of the matrix, n represents the number of drones in the actual cluster, K is the scaling term introduced to compensate for modeling errors, and α is used to adjust l i (t) is a parameter of the rate of change. Let c represent the partial derivative of p. Let c denote the partial derivative of c with respect to t, and I denote the identity matrix.

[0047] A human-to-crowd interaction system based on augmented reality includes a first acquisition and execution module, a second acquisition and execution module, and a third acquisition and execution module;

[0048] The first acquisition and execution module is used to acquire a custom hand posture, map the hand posture to a real drone cluster to generate hand operation instructions for the real cluster, and guide the real drone cluster to complete the task according to the hand operation instructions. The hand posture is responded to by detecting and tracking the joints of both hands through the HoloLens2 device, and determining the hand posture based on the joints.

[0049] The second execution module is used to obtain the interactive operation instructions of the real cluster, and guide the real drone cluster to complete the task according to the interactive operation instructions. The interactive operation instructions are in response to adding a virtual drone cluster in the real scene and obtaining the pose state of the virtual drone cluster. The interactive operation instructions are generated through the interaction between the real drone cluster and the virtual drone cluster.

[0050] The third acquisition and execution module is used to acquire the regional operation instructions of the real cluster, and guide the real UAV cluster to gather in the area corresponding to the regional operation instructions to complete the task. The regional operation instructions are generated by changing the parameter information in the working environment of the real UAV cluster and conveying the importance of each location in the real environment to the real UAV cluster.

[0051] A computer-readable storage medium storing a plurality of programs for being invoked by a processor and executed as described above in the human-cluster interaction method.

[0052] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0053] The advantages of the augmented reality-based human-swarm interaction method and system provided by this invention are as follows: By adding virtual swarm objects to the real environment, this invention connects a real drone swarm that cannot be directly operated with a virtual drone swarm that can be directly operated, achieving a linkage between virtual information and the real world, providing users with a self-immersive interaction. Simultaneously, the gestures designed for operating the virtual swarm robots are easy to understand and implement, conforming to user habits and making the interaction more natural. Therefore, the interaction method proposed in this application combines the real world and virtual information, providing diverse, efficient, and flexible interaction methods. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the process of the present invention;

[0055] Figure 2 This is a schematic diagram of the structure of the present invention;

[0056] Figure 3 A rendering of a hand joint model. Detailed Implementation

[0057] The technical solution of the present invention will now be described in detail through specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0058] like Figures 1 to 3 As shown, the present invention proposes a human-cluster interaction method based on augmented reality, which includes the following steps:

[0059] (A) Obtain a custom hand gesture, map the hand gesture to a real drone swarm to generate hand operation commands for the real swarm, and guide the real drone swarm to complete the task according to the hand operation commands. The hand gesture is responded to by detecting and tracking the joints of both hands through a HoloLens2 device, and determining the hand gesture based on the joints; or:

[0060] (B) Obtain interactive operation instructions from the real drone cluster, and guide the real drone cluster to complete tasks according to the interactive operation instructions. The interactive operation instructions are responded to by adding a virtual drone cluster in the real scene and obtaining the pose state of the virtual drone cluster, and generating interactive operation instructions through the interaction between the real drone cluster and the virtual drone cluster; or:

[0061] (C) Obtain the regional operation instructions of the real cluster, and guide the real UAV cluster to gather in the area corresponding to the regional operation instructions to complete the task. The regional operation instructions are generated by changing the parameter information in the working environment of the real UAV cluster and conveying the importance of each location in the real environment to the real UAV cluster.

[0062] Three human-swarm gesture interaction methods are provided through (A) to (C). By adding virtual swarm objects to the real environment, a real drone swarm that cannot be directly operated is linked to a virtual drone swarm that can be directly operated, realizing the linkage between virtual information and the real world and providing users with a self-immersive interaction. At the same time, the gestures designed for operating the virtual swarm robot are easy to understand and implement, conform to user habits, and make the interaction more natural. Therefore, the interaction method proposed in this application combines the real world and virtual information, while providing diverse, efficient, and flexible interaction methods.

[0063] The following sections will provide detailed explanations of the three interaction methods (A), (B), and (C).

[0064] Specifically, (A) includes:

[0065] (A1) Use the HoloLens2 augmented reality device to perform frame-by-frame detection of hand images within the field of view;

[0066] HoloLens 2 acquires environmental image information, specifically using the fourth-generation Kinect sensor and employing ToF (Time of Flight) technology to construct real-world environmental data of the user's location. Kinect uses a forward-facing signal generator for depth sensing and a downward-tilted, more precise signal generator for gesture tracking.

[0067] (A2) Extract the three-dimensional coordinates of the hand skeleton nodes and the number of hands N from the hand image in (A1) and map them into the rendered digital hand model;

[0068] Render a digital hand model of both hands, representing each hand with 26 joints, specifically as follows: Figure 3 As shown, small squares, a three-axis coordinate system, and small balls are used to represent joints, palms, and fingertips, respectively, with a focus on the pose of the index fingertip and palm.

[0069] (A3) Calculate the geometric features of the gesture based on the three-dimensional coordinates of the hand skeleton nodes, and determine whether the number of hands N is greater than 2. If yes, proceed to (A4); otherwise, proceed to (A5).

[0070] (A4) The gesture is currently invalid;

[0071] (A5) If the number of hands N equals 2, then the two-handed gesture recognition algorithm is used to recognize the two-handed gestures according to the gesture geometric features to obtain the gesture recognition action; if the number of hands is less than 2 and the detected hand is the right hand, then it is an invalid gesture; otherwise, the one-handed gesture recognition algorithm is used to recognize the one-handed gesture according to the gesture geometric features to obtain the gesture recognition action.

[0072] (A6) Define special gestures based on gesture geometry features;

[0073] Although HoloLens2 comes pre-loaded with gestures such as clicks and air taps, these gestures cannot be naturally bound to the operation commands of the virtual target drone swarm. Therefore, this embodiment defines a variety of gestures for interaction with the swarm. The main idea is to first request joint data from the device through the IMixedRealityHand interface, then perform logical judgment on the joint pose, and a gesture is generated when a certain logic is satisfied.

[0074] The specific geometric features of gestures include: the fingertip coordinates P of each finger. tip The coordinates P of the middle joint of each finger. middle Palm coordinates P palm Palm forward vector L forward finger bending angle threshold θ curl The threshold angle θ for finger extension straight The angle θ between the normal line pointing upwards from the palm and the forward ray from the camera. palm，camera The threshold angle θ between the palm-up normal and the forward ray of the camera when the palm faces the camera. facing The threshold θ for palm opening flat The distance d between the tips of the index and middle fingers of the same hand index，middle The following parameters are calculated based on the geometric features of the gesture:

[0075] The pointing vector L of each finger:

[0076] L. = P. middle -P. tip (1)

[0077] Vector L from the palm to the tip of the index finger palm，index :

[0078] L palm，index =P index.tip -T palm (2)

[0079] The vector L from the tip of the index finger to the tip of the ring finger ring，index :

[0080] L ring，index =P ring.tip -Pindex.tip (3)

[0081] The cross product of equations (2) and (3) yields the reference vector L. cross :

[0082] L cross =L palm，index ×L ring，index (4)

[0083] If the detected hand is the right hand, then the result of equation (4) remains unchanged; if it is the left hand, then equation (4) is multiplied by -1. Reference vector L cross With the forward vector L of the palm forward The included angle is θ:

[0084]

[0085] Based on the aforementioned geometric feature information of the gestures, logical judgments are made to define special gestures, specifically including:

[0086] (A6-1) If all fingers are extended and the palm faces the camera, the gesture recognition action is "palm facing inward," and the actual drone operation command is "take off." The specific formula that is satisfied is:

[0087]

[0088] (A6-2) If all fingers are extended and the back of the hand faces the camera, the gesture recognition action is "palm facing outwards," and the actual drone operation command is "land." The specific formula that is satisfied is:

[0089]

[0090] (A6-3) If the thumb is extended straight up, the remaining fingers are bent towards the palm, the palm faces the camera, and the detected hand is the left hand, then the gesture recognition action is "raise a thumb," and the actual drone operation command is "turn left." The specific formula is as follows:

[0091]

[0092] (A6-4) If the thumb is straight up and the remaining fingers are bent towards the palm, the palm faces the camera, and the detected hands are the left and right hands, then the gesture recognition action is "raise two thumbs", and the real drone operation command is "turn right". The specific formula is the same as (8).

[0093] (A6-5) If the index and middle fingers are straight, the remaining fingers are bent, the distance between the fingertips of the index and middle fingers is greater than 4 cm, and the detected hand is the left hand, then the gesture recognition action is "a victory sign," and the actual drone operation command is "up." The specific formula is as follows:

[0094]

[0095] (A6-6) If the index and middle fingers are straight, the remaining fingers are bent, the distance between the fingertips of the index and middle fingers is greater than 4 cm, and the detected hands are the left and right hands, then the gesture recognition action is "two victory gestures", the real drone operation command is "down", and the specific formula is the same as (9).

[0096] (A6-7) If the index finger is straight, the remaining fingers are bent, the back of the hand faces the camera, and the detected hand is the left hand, then the gesture recognition action is "single finger," and the actual drone operation command is "forward." The specific formula is as follows:

[0097]

[0098] (A6-8) If the index finger is straight, the remaining fingers are bent, the back of the hand is facing the camera, and the detected hands are the left and right hands, then the gesture recognition action is "two fingers", the real drone operation command is "backward", and the specific formula is the same as (10).

[0099] This embodiment designs a variety of simple and highly accurate gestures ((A6-1) to (A6-8)) to map gestures to complex swarm operation instructions, thereby reducing the difficulty of interaction. When HoloLens2 detects any valid gesture set in (A6-1) to (A6-8), it triggers hand operation instructions to guide the real drone swarm to complete the task according to the interaction operation instructions.

[0100] (A7) Select the virtual target drone object by tapping in the air, bind elements to key parts of the finger joints, and when the elements are activated, trigger hand operation commands to guide the real drone swarm to complete the task according to the hand operation commands.

[0101] In the Sydney scene, virtual target drones are selected using a remote tapping technique. The user aims at the drone with a ray, and the ray's trajectory is displayed in the scene. If the index finger and thumb briefly touch and then separate while aiming at the drone, the drone is selected and highlighted. Simultaneously, elements are bound to key points on the hand joints. When these elements are activated, operation commands are triggered, including tapping the palm to open a user interface displaying real drone swarm information.

[0102] This embodiment maps custom gestures to hand operation commands of a real drone swarm, primarily through a HandConstraint resolver. This resolver includes four events: OnHandActivate, OnHandDeactivate, OnFirstHandDetected, and OnLastHandLost. These events are triggered when all gestures are satisfied, when the gesture condition changes from satisfied to unsatisfied, when the first gesture is detected, and when all detected gestures are lost, respectively. By defining the triggering event as sending the corresponding operation command code, the mapping from gestures to operation commands can be achieved.

[0103] Specifically, (B) includes:

[0104] (B1) Create and add virtual drone models to the set scene, and create animations for the virtual drone swarm, including rotor rotation and drone flight animations, to enhance the realism of the interaction.

[0105] (B2) Design a UI interface to access the pose status of the virtual drone cluster. The pose status includes drone coordinates, drone attitude, drone trajectory, head-up display, flight screen, and environmental information.

[0106] A virtual drone is a representation of a real drone in a scenario. Changes in the pose and other information of the real drone are reflected in the virtual drone, allowing users to intuitively and naturally grasp the status of the real drone by directly observing its behavior. The UI includes various functions such as preset trajectories, mode selection, trajectory playback, and adding devices, and transmits flight images in real time to monitor the cluster's pose, enabling users to conveniently obtain the status of all targets in real time.

[0107] (B3) Draw the desired path by grabbing and dragging the virtual drone cluster, and record and save the key points of the desired path;

[0108] Specifically, the process involves using air taps to select a virtual target drone, dragging the selected drone to draw a trajectory in the air, displaying the trajectory in a bright color in the virtual scene, highlighting key points in the trajectory, and saving the result.

[0109] (B4) Based on the pose state and key points of the desired path, the real UAV cluster and the virtual UAV cluster interact to generate interactive operation instructions, guiding the real UAV cluster to complete the task according to the interactive operation instructions.

[0110] By using steps (B1) to (B4), the real drone interacts directly with the virtual drone object, and then transmits and updates the information to the real cluster, thus achieving indirect interaction with the real cluster. First, virtual objects are added to the real scene. Then, through close-range clicking and dragging or long-range ray detection selection, the virtual objects are placed in the desired position or the desired trajectory is drawn. Finally, the key position data is recorded and sent to the real cluster.

[0111] Regarding (C): By changing the density function φ(q,t) on the 2D scene map in real time, information about the importance of a region is conveyed to the real cluster, guiding the cluster to quickly gather in the designated area. Specifically, this includes:

[0112] (C1) Create a two-dimensional scene map based on the real-world environment. The working domain of the real drone swarm on the two-dimensional scene map is... The density function φ(q, t) represents the importance of q∈D at time t, where R represents the number of fingers tapped by the user, D represents the two-dimensional scene map of the actual drone operation, and q represents any point on the two-dimensional scene map.

[0113] (C2) Click on the area of ​​interest on the 2D scene map to increase the density function value of the clicked area;

[0114] Users can increase the density function value of an area by clicking on it on a 2D scene map. The longer the click time, the higher the density function value, and the more important the area. In other words, users can change the density function by tapping on the area of ​​interest on the 2D scene map with their fingers.

[0115] (C3) Use a Gaussian mixture model with fixed weights and shapes to transform the clicked area into a real-time density function on a two-dimensional scene map. Update the control rate of the real drone swarm according to the real-time density function, and use the control rate as an area operation command to guide the real drone swarm to gather in the specified area to complete the task.

[0116] Gaussian Mixture Models (GMMs) approximate density functions of arbitrary shapes using a linear combination of k Gaussian functions, achieving the transformation from intuitive regions to numerical functions. To simplify calculations, the weights and shapes of the Gaussian functions are fixed, resulting in the specific formula for the density function:

[0117]

[0118] Where ε0 is a very small positive constant, ensuring that the density function φ(q,t) > 0 when there is no input, ∑ is the covariance, which determines the shape of the Gaussian function, R represents the number of fingers tapped by the user, and μ i (t) is the center of the Gaussian function, q represents any point on the two-dimensional scene map, p represents the position of the actual drone, and li (t) represents the weight of the i-th Gaussian function, and T represents the transpose of the matrix. μ i The derivative of (t).

[0119] To make the center of the Gaussian function asymptotically close to the tapping position r i (t), where K is a proportional term introduced to compensate for modeling errors, and specifically satisfies the following formula:

[0120]

[0121] l i (t) The weight of the i-th Gaussian function determines the weight of each Gaussian function. The longer the user clicks, the greater the weight should be. Therefore, the specific formula to be satisfied is:

[0122]

[0123] Where, τ i，1 τ represents the time at which the i-th Gaussian function is added. i，2 This represents the time when the i-th Gaussian function leaves, and α is used to adjust l. i (t) is a parameter of the rate of change, which determines l i The rate of change of (t).

[0124] To optimize the coverage performance of the density function by a real drone swarm, a position cost function H(p, t) is introduced, with the specific formula as follows, where the two-dimensional working domain... It is a convex region, p i ∈D represents the position of the i-th drone, V i (p) is the Vino unit managed by the i-th UAV, m i and c i The mass and centroid of the i-th Vino unit are:

[0125]

[0126]

[0127]

[0128] When H(p,t) is minimized using the gradient descent method, I represents the identity matrix and the control rate of a real UAV swarm. for:

[0129]

[0130] When the user changes the density function by clicking, the Venn diagram of the real drone swarm changes, and the control law shown in equation (17) changes. This will guide the cluster to approach the user's region of interest in the most optimal way.

[0131] This embodiment uses Socket communication, with the HoloLens device acting as the client and the devices controlling the real cluster acting as the server. Once their IP addresses and port numbers are set and the connection is successful, bidirectional data transmission can begin. The specific process is as follows: The server program binds a socket to a specified address and port, and uses this socket to wait for and listen for client connection requests. The client program sends a connection request to the server program at the address and port bound to the server. The server accepts the connection request and obtains a new socket. The server communicates with the client by reading and writing to the new socket.

[0132] In this embodiment, after obtaining hand operation commands, interactive operation commands, or area operation commands, the corresponding manipulation commands are transmitted to the real cluster via a socket interface to control the real cluster to complete the task according to the corresponding operation commands. The specific transmission process of the manipulation commands is as follows: establishing communication between the augmented reality interactive device and the drone control terminal, both of which are located in the same local area network; creating a socket object for listening to connections and binding it to the address and port of the real drone cluster control terminal, with the real drone control terminal acting as the server and the HoloLens2 augmented reality device acting as the client; HoloLens2 creating a socket to connect with the real drone cluster control terminal and sending command data; the control terminal receiving and processing the operation commands and sending the response data to the HoloLens2 terminal.

[0133] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A human-crowd interaction method based on augmented reality, characterized in that, Includes the following steps: A custom hand gesture is obtained and mapped to a real drone swarm to generate hand operation commands for the real swarm. Based on the hand operation commands, the real drone swarm is guided to complete the task according to the hand operation commands. The hand gesture is responded to by detecting and tracking the joints of both hands through a HoloLens2 device, and determining the hand gesture based on the joints. Specifically, it includes: (A1) Use the HoloLens2 augmented reality device to perform frame-by-frame detection of hand images within the field of view; (A2) Extract the three-dimensional coordinates of the hand skeleton nodes and the number of hands N from the hand image in (A1) and map them into the rendered digital hand model; (A3) Calculate the geometric features of the gesture based on the three-dimensional coordinates of the hand skeleton nodes, and determine whether the number of hands N is greater than 2. If yes, proceed to (A4); otherwise, proceed to (A5). (A4) The current gesture is invalid; (A5) If the number of hands N equals 2, then the two-handed gesture recognition algorithm is used to recognize the two-handed gestures according to the gesture geometric features to obtain the gesture recognition action; if the number of hands is less than 2 and the detected hand is the right hand, then it is an invalid gesture; otherwise, the one-handed gesture recognition algorithm is used to recognize the one-handed gesture according to the gesture geometric features to obtain the gesture recognition action. (A6) Define special gestures based on gesture geometry features; (A7) Select the virtual target drone object by tapping in the air, bind elements to key parts of the finger joints, and trigger hand operation commands when the elements are activated; The gesture geometry features in (A3) include the fingertip coordinates of each finger, the mid-joint coordinates of each finger, the palm coordinates, the palm forward vector, the finger bending angle threshold, the finger straightening angle threshold, the angle between the palm-up normal and the camera forward ray, the angle threshold between the palm-up normal and the camera forward ray when the palm faces the camera, the palm opening threshold, and the distance between the fingertips of the index and middle fingers of the same hand.

2. The augmented reality-based human-cluster interaction method according to claim 1, characterized in that, In step (A6), specifically: If all fingers are extended and the palm faces the camera, the gesture recognition action is "palm facing inwards", and the real drone operation command is "take off". If all fingers are extended and the back of the hand faces the camera, the gesture recognition action is "palm facing outwards," and the actual drone operation command is "land." If the thumb is extended straight up, the remaining fingers are bent towards the palm, the palm faces the camera, and the detected hand is the left hand, then the gesture recognition action is "raise a thumb", and the actual drone operation command is "turn left". If the thumb is extended straight up, the remaining fingers are bent towards the palm, the palm faces the camera, and the detected hands are the left and right hands, then the gesture recognition action is "raise two thumbs", and the real drone operation command is "turn right". If the index and middle fingers are straight, the remaining fingers are bent, the distance between the fingertips of the index and middle fingers is greater than 4 centimeters, and the detected hand is the left hand, then the gesture recognition action is "a victory gesture", and the real drone operation command is "up". If the index and middle fingers are straight, the remaining fingers are bent, the distance between the fingertips of the index and middle fingers is greater than 4 centimeters, and the detected hands are the left and right hands, then the gesture recognition action is "two victory gestures", and the real drone operation command is "down". If the index finger is straight and the remaining fingers are bent, and the detected hand is the left hand, then the gesture recognition action is "single finger", and the actual drone operation command is "forward". If the index finger is straight and the remaining fingers are bent, and the detected hands are the left and right hands, then the gesture recognition action is "two fingers", and the actual drone operation command is "backward".

3. The augmented reality-based human-cluster interaction method according to claim 1, characterized in that, Step (A7) Select the virtual target drone object using the air tap method. Specifically, use the ray to aim at the virtual target drone. The ray trajectory is displayed in the virtual scene. If the index finger and thumb briefly touch and then separate while the ray is aiming at the virtual drone, the drone is selected. The selected virtual drone is highlighted.

4. The augmented reality-based human-cluster interaction method according to claim 1, characterized in that, Also includes: The system acquires interactive operation commands from a real drone cluster, guides the real drone cluster to complete tasks according to these commands, and the interactive operation commands are generated through the interaction between the real and virtual drone clusters. Specifically, the interactive operation commands include: (B1) Create and add virtual drone models to the set scene, and create animations for the virtual drone swarm; (B2) Design a UI interface to access the pose status of the virtual drone cluster. The pose status includes drone coordinates, drone attitude, drone trajectory, head-up display, flight screen, and environmental information. (B3) Draw the desired path by grabbing and dragging the virtual drone cluster, and record and save the key points of the desired path; (B4) Based on the pose state and key points of the desired path, the real UAV cluster and the virtual UAV cluster interact to generate interactive operation instructions, guiding the real UAV cluster to complete the task according to the interactive operation instructions.

5. The augmented reality-based human-cluster interaction method according to claim 1, characterized in that, Also includes: Obtain the regional operation instructions of the real drone cluster, and guide the real drone cluster to gather in the area corresponding to the regional operation instructions to complete the task. The regional operation instructions are generated by changing the parameter information in the working environment of the real drone cluster to convey the importance of each location in the real environment to the real drone cluster. Specifically, this includes: (C1) Create a two-dimensional scene map based on the real-world environment. The working domain of the real drone swarm on the two-dimensional scene map is... Density function Indicates in Time point The importance of which, Represents the real number field. A two-dimensional scene map representing the actual operation of a drone. Represents any point on a two-dimensional scene map; (C2) Click on the area of ​​interest on the 2D scene map to increase the density function value of the clicked area; (C3) Use a Gaussian mixture model with fixed weights and shapes to transform the clicked area into a real-time density function on a two-dimensional scene map. Update the control rate of the real drone swarm according to the real-time density function, and use the control rate as an area operation command to guide the real drone swarm to gather in the specified area to complete the task.

6. The augmented reality-based human-cluster interaction method according to claim 5, characterized in that, Density function Control rate The formula is as follows: in, Represents positive numbers. Indicates the first The center position of a Gaussian function express The derivative, Indicates the clicked location. Indicates the first The time when the Gaussian function is added. Indicates the first The time when a Gaussian function leaves, Represents the cost function, Indicates the first The Vino unit is operated by a real drone. and It is the first The mass and centroid of each Vino unit Indicates the first The location of a real drone This indicates the number of fingers the user tapped. Represents any point on a two-dimensional scene map. Indicates the actual location of the drone. Indicates the first The weights of a Gaussian function, To represent the transpose of a matrix, The table shows the number of drones in the actual cluster. This is a proportional term introduced to compensate for modeling errors. For use in regulation A parameter of the rate of change, Represents the identity matrix.

7. An interactive system based on augmented reality-driven human-cluster interaction method according to claim 1, characterized in that, It includes a first acquisition and execution module, a second acquisition and execution module, and a third acquisition and execution module; The first acquisition and execution module is used to acquire a custom hand posture, map the hand posture to a real drone cluster to generate hand operation instructions for the real cluster, and guide the real drone cluster to complete the task according to the hand operation instructions. The hand posture is responded to by detecting and tracking the joints of both hands through the HoloLens2 device, and determining the hand posture based on the joints. The second execution module is used to obtain the interactive operation instructions of the real cluster, and guide the real drone cluster to complete the task according to the interactive operation instructions. The interactive operation instructions are in response to adding a virtual drone cluster in the real scene and obtaining the pose state of the virtual drone cluster. The interactive operation instructions are generated through the interaction between the real drone cluster and the virtual drone cluster. The third acquisition and execution module is used to acquire the regional operation instructions of the real cluster, and guide the real UAV cluster to gather in the area corresponding to the regional operation instructions to complete the task. The regional operation instructions are generated by changing the parameter information in the working environment of the real UAV cluster and conveying the importance of each location in the real environment to the real UAV cluster.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of programs for being invoked by a processor and executed as described in any one of claims 1 to 6.

Images