Mixed reality device and drone cluster interaction method, device, equipment and medium

Abstract

The present application discloses a method, device, equipment and medium for interaction between a mixed reality device and a drone cluster, and relates to the field of human-computer interaction technology. The method comprises: receiving a startup instruction of an operator through an open source engine, and starting a mixed reality scene in the open source engine; receiving a startup simulation node instruction of an operator through an operating system, and receiving a drone cluster unlocking instruction of an operator to control the drone cluster to switch to an unlocked state, realizing the interaction between the operator and the mixed reality scene through the mixed reality device, and generating a target point position; controlling the movement of the drone cluster according to the target point position and the target instruction, and rendering the movement path of the drone cluster in the open source engine; judging whether the drone cluster has reached the target point, and if not, continuously sending the target instruction until the drone cluster reaches the target point. The present application realizes data transmission between the mixed reality device and the drone cluster, and improves the interaction efficiency between the operator and the drone cluster.

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction technology, and in particular to a method, device, equipment and medium for interaction between a mixed reality device and a drone cluster. Background Art

[0002] With the rapid development of intelligent technology and the large-scale use of unmanned equipment, the working and operating areas of drone swarms are gradually overlapping with the activity space of people. When dealing with complex tasks such as navigation, drone swarms need to process a large amount of different types of sensor data, which makes it difficult for non-expert operators to operate accurately and fail to give full play to the advantages of drone swarms. At the same time, many traditional drone swarm control and communication methods rely on 2D display ground stations, which require operators to constantly pause work and check data visualization. A drone swarm refers to a collection of multiple drones (Unmanned Aerial Vehicle, UAV) that are under unified command to perform a task together.

[0003] Due to the difficulty in controlling and managing drone swarms, conventional interaction methods cannot efficiently interact with drone swarms. Summary of the invention

[0004] The purpose of the present application is to provide a method, device, equipment and medium for interaction between a mixed reality device and a drone cluster, which realizes data transmission between the mixed reality device and the drone cluster, effectively solves the problems of high operator load, difficult cluster control, and non-intuitive data display, and significantly improves the interaction efficiency between the operator and the drone cluster. It solves the problem that the existing interaction method in the prior art is unable to intuitively understand the scene by the operator, which affects the subsequent interaction efficiency between the operator and the drone cluster.

[0005] To achieve the above objectives, this application provides the following solutions.

[0006] In a first aspect, the present application provides a method for interaction between a mixed reality device and a drone cluster, comprising the following steps.

[0007] The operator's startup instruction is received through the open source engine, and the mixed reality scene is started in the open source engine; the operator's startup simulation node instruction is received through the operating system.

[0008] The operator's drone cluster unlocking instruction is received through a local computer or a mixed reality device to control the drone cluster to switch to an unlocked state; the open source engine is set on the local computer.

[0009] The operator can interact with the mixed reality scene through the mixed reality device to generate the target point position.

[0010] The movement of the drone cluster is controlled according to the target point position and the operator's target instruction, so that the drone cluster completes the action corresponding to the target instruction, and the movement path of the drone cluster is rendered in real time in the open source engine.

[0011] It is judged whether the drone cluster has reached the target point. If the drone cluster has not reached the target point, the target command will continue to be sent, otherwise the target command will be stopped.

[0012] In a second aspect, the present application provides a mixed reality device and a drone cluster interaction device, comprising the following modules.

[0013] The startup module is used to: receive the operator's startup instruction through the open source engine, and start the mixed reality scene in the open source engine; receive the operator's startup simulation node instruction through the operating system.

[0014] The unlocking module is used to: receive the operator's drone cluster unlocking instruction through a local computer or a mixed reality device to control the drone cluster to switch to an unlocked state; the open source engine is set on the local computer.

[0015] The target point position generation module is used to: realize the interaction between the operator and the mixed reality scene through the mixed reality device and generate the target point position.

[0016] The drone movement control module is used to control the movement of the drone cluster according to the target point position and the operator's target command, so that the drone cluster completes the action corresponding to the target command, and renders the movement path of the drone cluster in real time in the open source engine.

[0017] The target point arrival judgment module is used to judge whether the drone cluster has reached the target point. If the drone cluster has not reached the target point, the target command will continue to be sent, otherwise the target command will be stopped.

[0018] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned method for interaction between a mixed reality device and a drone cluster.

[0019] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for interaction between a mixed reality device and a drone cluster.

[0020] According to the specific embodiments provided by the present application, the present application discloses the following technical effects: the present application provides a method, device, equipment and medium for interaction between a mixed reality device and a drone cluster, receives the operator's startup instruction through an open source engine, and starts a mixed reality scene in the open source engine; receives the operator's startup simulation node instruction through the operating system, receives the operator's drone cluster unlocking instruction through the local computer or the mixed reality device, controls the drone cluster to switch to the unlocked state, realizes the interaction between the operator and the mixed reality scene through the mixed reality device, and generates the target point position; controls the drone cluster to move according to the target point position and the operator's target instruction, so that the drone cluster completes the action corresponding to the target instruction, and renders the moving path of the drone cluster in real time in the open source engine; determines whether the drone cluster reaches the target point, and if not, continues to send the target instruction until the drone cluster reaches the target point. The present application realizes data transmission between the mixed reality device and the drone cluster, effectively solves the problems of high operator load, difficult cluster control, and non-intuitive data display, significantly improves the interaction efficiency between the operator and the drone cluster, and solves the problem that the existing interaction method in the prior art cannot intuitively understand the scene, which affects the interaction efficiency of the subsequent operator and the drone cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 A flowchart of a method for interaction between a mixed reality device and a drone cluster provided in one embodiment of the present application.

[0023] Figure 2 A schematic diagram of a detailed process of interaction between a mixed reality device and a drone cluster provided in one embodiment of the present application.

[0024] Figure 3 A schematic diagram of a target point preview generation process provided in an embodiment of the present application.

[0025] Figure 4 A schematic diagram of an original formation consisting of five drones provided in an embodiment of the present application.

[0026] Figure 5 A schematic diagram of functional modules of a mixed reality device and a drone cluster interaction apparatus provided in one embodiment of the present application.

[0027] Figure 6A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0030] In an exemplary embodiment, Figure 1 As shown, a method for interaction between a mixed reality device and a drone cluster is provided, and the method includes the following steps 201 to 205.

[0031] Step 201: receiving a start instruction from an operator through an open source engine, and starting a mixed reality scene in the open source engine; and receiving a start simulation node instruction from an operator through an operating system.

[0032] Step 202: Receive the operator's drone cluster unlocking instruction through a local computer or a mixed reality device to control the drone cluster to switch to an unlocked state; the open source engine is set on the local computer.

[0033] Step 203: The operator interacts with the mixed reality scene through the mixed reality device to generate a target point position.

[0034] Step 204: Control the movement of the drone cluster according to the target point position and the operator's target command, so that the drone cluster completes the action corresponding to the target command, and renders the movement path of the drone cluster in real time in the open source engine.

[0035] Step 205: Determine whether the drone cluster has reached the target point. If the drone cluster has not reached the target point, continue to send the target command, otherwise stop sending the target command.

[0036] By implementing the above steps 201 to 205, compared with the traditional 2D ground station drone cluster control, the drone cluster control with the help of mixed reality equipment can make the operator feel personally in the environment, enhance the operator's understanding of the business scenario and human-machine collaboration ability, reduce the difficulty of drone cluster control and management, and enable non-expert operators to accurately and conveniently control the drone cluster. With the help of efficient human-computer interaction technology, the operator operates the virtual object in the virtual environment, and the real robot in the real physical environment executes the corresponding instructions, which allows the operator to get rid of the limitations of the physical environment, send instructions to the robot and perform operations in different physical environments, and expand the operator's operating range. At the same time, the interaction method can also be used for robot algorithm training, observation of state information, parameter optimization and other tasks.

[0037] In another exemplary embodiment of the present application, before the step of receiving the operator's drone cluster unlocking instruction through the local computer or mixed reality device, the mixed reality device and drone cluster interaction method also includes: determining whether the operating system is connected to the open source engine, if the connection between the operating system and the open source engine fails, waiting for the operating system to be connected to the open source engine; the data connection between the open source engine and the operating system is developed using the ROSBridge middle layer.

[0038] like Figure 2 As shown, the specific process of interaction between mixed reality devices, open source engines, and operating systems is introduced below.

[0039] Step 101: When the operator wants to control the drone cluster, jump to step 102 and step 103 at the same time; Step 102: Start the mixed reality scene in the open source engine, step 103: Start the simulation node in the operating system; Step 104: Wait for the operating system to connect to the open source engine; Step 105: Then enter the judgment process to judge whether the operating system is connected to the open source engine. If the connection fails, jump to step 104 and continue waiting for the operating system to connect to the open source engine.

[0040] In this embodiment, ROSbridge provides a WebSocket interface so that operators can access the ROS system through a web browser, mobile device or other non-ROS device. It allows communication between different platforms, such as communication between ROS and programming languages ​​such as Javascript, Python, and Java. With the help of ROSbridge, operators can realize functions such as remote control and data acquisition of robots or drones. At the same time, it also supports dynamic configuration and debugging of ROS systems, such as modifying ROS parameters, viewing ROS nodes, publishing ROS messages, etc., making development and testing more convenient. ROSbridge can be integrated with other ROS tools and libraries, such as RViz, ROSbag, etc. As a very important component in the ROS ecosystem, ROSbridge brings a more flexible and convenient way to the development and application of ROS systems, broadening the use and availability of robotics technology.

[0041] After waiting for the operating system to connect to the open source engine, step 106: ROSbridge opens the port subscription topic, at which time the data between the open source engine and the operating system can be communicated in real time. Step 107: User unlock command, the operator (user) can send the drone cluster unlock command on the local computer where the open source engine is set.

[0042] Before the step of receiving the operator's drone cluster unlocking instruction through the local computer or the mixed reality device, the mixed reality device and drone cluster interaction method also includes: receiving the operator's remote debugging start instruction through the mixed reality device, and controlling the mixed reality device to start the remote debugging function; judging whether the mixed reality device is connected to the open source engine, if the mixed reality device fails to connect to the open source engine, waiting for the mixed reality device to connect to the open source engine; if the mixed reality device is successfully connected to the open source engine, displaying the mixed reality scene picture in the open source engine in real time through the mixed reality device; the operator interacts with the mixed reality scene through the mixed reality interaction method, and receives the operator's drone cluster unlocking instruction or target instruction to perform mobile control operations on the drone cluster.

[0043] Step 108: Unlock the drone. In this embodiment, the unlock command of the drone cluster is sent to the operating system through ROSBridge to unlock the drone. Step 109: Determine whether the drone is unlocked. If so, jump to step 110, if not, jump to step 107. Step 110: The user issues a cmd command. Step 111: After receiving the cmd command, the drone cluster moves accordingly. Step 112: Render the moving path in real time in the open source engine. Step 113: Determine whether the target point has been reached. If so, jump to step 119, if not, jump to step 110. Step 114: Start remote debugging. Step 115: Wait for the mixed reality device to connect to the open source engine. Step 116: Determine whether the mixed reality device is connected to the open source engine. If so, jump to step 117, if not, return to step 115. Step 117: The mixed reality device interacts with the open source engine in real time. Step 118: The user operates the drone cluster with the help of the mixed reality device. Step 119: End.

[0044] In this embodiment, the step of "unlocking the drone cluster" can be specifically performed as follows: after the operator interacts with the "unlock" button in the hand menu, the unlock function in its mounting script is triggered, and the desired position information is released to the drone through the unlock function, providing the drone with a target point. Due to the OFFBOARD (external control mode) state switching logic of the Px4 flight control, the state of Px4 cannot be switched if the target point is not provided to Px4 in advance. Then set the client to request the drone to enter the "OFFBOARD" mode, and finally issue a "true" command to set the client drone to unlock.

[0045] In this embodiment, the steps for the drone cluster to take off can be specifically executed as follows: after the operator interacts with the "Take Off" button in the hand menu, the takeoff function in its mounting script is triggered. The takeoff function first obtains the coordinates of the current drone in the open source engine, thereby calculating the hovering position after takeoff, and uses the transformation matrix to convert the hovering position in the world coordinate system of the open source engine into the hovering position in the drone coordinate system, and then sends the converted hovering position to the drone cluster, and the converted hovering position can be sent in a loop.

[0046] After confirming that the drone cluster is unlocked, use the remote debugging function to operate the drone cluster with the help of mixed reality devices. After receiving the instructions, the drone cluster will move accordingly and render the movement path in real time in the open source engine.

[0047] Specifically: the operator starts the remote debugging function of the mixed reality device, and then enters the judgment process to determine whether the mixed reality device is connected to the open source engine. If the connection fails, the operator continues to wait for the mixed reality device to connect to the open source engine. If the connection is successful, the mixed reality device will display the picture in the open source engine in real time. At this time, the operator uses mixed reality interaction methods such as voice and gestures to interact with the mixed reality scene and operate the drone cluster.

[0048] Mixed reality interaction methods include gesture, voice and gaze three-channel mixed interaction technology. This application is based on the mixed reality device development platform and realizes the landing / unlocking, take-off and landing, fixed-point flight and formation flight of drone clusters. The following introduces the implementation principles and processes of each function.

[0049] In another exemplary embodiment of the present application, the above step 203 specifically includes the following steps 401 to 403.

[0050] Step 401: Collect the operator's hand depth map and the operator's eye movement data.

[0051] Step 402: Identify the operator's gestures according to the hand depth map.

[0052] Step 403: Determine the target point position according to the eye movement data and the gesture action through the mixed reality device.

[0053] Among them, step 402 can be replaced by the following steps 501~502.

[0054] Step 501: Identify the position of the operator's hand in the hand depth map according to the hand depth map.

[0055] Step 502: Using a random forest algorithm, the operator's hand is classified according to its position in the hand depth map to obtain the operator's gesture position; the operator's gesture position is used to fit the virtual hand in the mixed reality device to locate the operator's hand position in space to determine the gesture operation.

[0056] The gesture operation includes a pinching gesture; the gesture is represented by the thumb and index finger being placed together and then separated. When the operator performs the pinching gesture, a dotted line extends from the place where the fingers of the virtual hand are placed together in the mixed reality scene, and the dotted line extends infinitely to the far end until it collides with an object in the mixed reality scene. After the collision, the collided object is recorded as an active state; the active state indicates that the collided object is in a movable state.

[0057] Gesture interaction is implemented using a visual recognition scheme, and its main principle is as follows: First, a binocular camera mounted on a mixed reality device is used to collect a hand depth map. Then, the position of the hand in the image is detected in the depth map, and a bounding box of a fixed size is generated. The bounding box is a rectangular box that can completely contain the gesture area. Next, the random forest algorithm is used for classification to estimate the rough position of the hand in the bounding box. The position information is used for subsequent model hand fitting, fitting the simulated hand in the mixed reality device with the position information, and then accurately positioning the operator's hand position in space.

[0058] Random forest algorithm: First, the concepts of decision tree and ensemble learning need to be introduced. A decision tree is a common tree-structured machine learning algorithm and the main component of the random forest model. Ensemble learning is to calculate data using multiple machine learning models and integrate the results for output. Compared with a single estimator, ensemble learning has better generality and robustness, and can be conveniently parallelized to speed up the operation.

[0059] Random forest is a commonly used ensemble learning algorithm. Random forest uses the Bagging combination method to generate a forest from multiple decision trees. Different decision trees are used as weak classifiers, combined into the entire forest, and become a strong classifier. When constructing a random forest model, first different decision trees need to be constructed, and then the results estimated by different decision trees are used for voting to obtain the final result. The construction method of each specific tree is as follows: 1) Determine the size of the training set as N, and randomly and with replacement draw N sample images from the dataset as the training set for each tree; 2) Determine the feature dimension d of each sample image, specify a d << D, and randomly select d feature subsets from the sample features, and select the optimal one during the splitting of the decision tree; where D is the set feature dimension; 3) Do not set a pruning process for the tree. Its specific calculation process is that first, samples are taken from the sample data, and decision tree models are established respectively. Set the predicted value of each decision tree model as , then the predicted value of the entire random forest can be obtained by taking the average, which can be expressed as the following formula.

[0060] (1).

[0061] Among them, is the number of decision trees in the random forest, is the predicted value of the entire random forest, is the size of the training set, is the predicted value of each decision tree model, is the input sample image. After obtaining different results, each sample image is voted according to the classification result, and finally the classification can be determined. The loss function is expressed as the following formula.

[0062] (2).

[0063] in, For the total loss, are model parameters, For the The features of the sample images, For the The labels of sample images. According to the given classification model , obtain the patrol data of each classification model, and then use the residual The situation where the correct classification is greater than the wrong classification is found, as shown in the following formula.

[0064] (3).

[0065] in, is an indicator function, which takes the value 1 when the condition is met, otherwise it takes the value 0; is a classification model, which Classify and label the categories as or other categories ; is the calculation constant. Then the extrapolation error is calculated.

[0066] (4).

[0067] in; Indicated in The probability of the case being less than 0 is the probability of the model predicting an error. Finally, as the generalization error increases, the decision tree converges to the following formula.

[0068] (5).

[0069] in, Representation parameters The model predicts the probability of Represents a random variable and The joint probability distribution of is a parameter dependent Model.

[0070] Finally, solving the accurate position of the hand is abstracted into an optimization problem, which is accurately expressed by constructing an energy function and multiple constraint penalty terms. The LM algorithm is used to solve the optimization problem and obtain the final gesture position (27 hand nodes).

[0071] The LM algorithm is an iterative algorithm for finding the extreme value of a function. The algorithm combines the characteristics of the Newton method and the gradient method.

[0072] The recursive formula for Newton's method to find extreme values ​​is shown below.

[0073] (6).

[0074] in, Indicates the point of the current iteration, represents the point after iteration, is a function exist The Hessian matrix at , that is, the square matrix of all second-order partial derivatives; is the inverse matrix of the Hessian matrix; 𝐺 is the function exist The gradient vector at , that is, the vector of all first-order partial derivatives.

[0075] The gradient method ball extreme value recursion formula is shown as follows.

[0076] (7).

[0077] in, represents the iteration step size, express exist The gradient vector at , which contains the partial derivative of the function in each dimension.

[0078] The recursive formula of the LM algorithm is shown below.

[0079] (8).

[0080] It can be seen that this formula is in the Gauss-Newton formula Add an adjustment factor ,in is the step length, is the identity matrix, and a smaller one is used when the gradient drops too fast , making the whole formula close to the Gauss-Newton method, and using a larger , making the whole formula close to the gradient method. Finally, the 27 key nodes obtained are constructed with corresponding spatial geometric constraints to achieve specific gesture recognition.

[0081] In step 205, it is determined whether the drone cluster has reached the target point. If the drone cluster has not reached the target point, the command continues to be sent. If the drone cluster has reached the target point, the command is stopped and the process ends.

[0082] It is judged whether the drone cluster is unlocked. If the drone is not unlocked, it will continue to wait for the operator's unlocking command. If the drone is already unlocked, the operator can send cmd target commands such as take-off, fixed-point flight, and landing. These target commands are sent to the drone operating system through ROSbridge. The drone cluster receives these target commands and moves accordingly. The coordinates and posture of the drone cluster are also sent to the open source engine in real time through ROSbridge to render the movement path, and then enter the judgment process to judge whether the drone cluster has reached the target point. If the drone cluster has not reached the target point, it will continue to send target commands. If the drone cluster reaches the target point, it will stop sending target commands and end the target point arrival judgment.

[0083] In this embodiment, the steps of fixed-point flight of the drone cluster can be specifically executed as follows: after the operator uses the voice command "generate target point" or uses gestures to interact with the "target point tracking switch" button, a target point preview (blue cube) will be generated at the operator's eye gaze (a specified distance in front of the operator). The preview supports gesture interaction, including movement and rotation along the y-axis (the y-axis is the vertical upward axis in the open source engine coordinate system). After the operator moves the preview to a specific position, he uses the voice command "send target point" or uses gestures to interact with the "send" button. At this time, the system automatically exits the path point mode, the target point preview becomes non-interactive, and the target point position and posture after the coordinate system transformation are sent cyclically.

[0084] In another exemplary embodiment of the present application, in step 204, the movement of the drone cluster is controlled according to the target point position and the operator's target instruction, specifically including: using a leader-follower algorithm to control the movement of the drone cluster according to the target point position and the operator's target instruction.

[0085] The leader-follower algorithm is a multi-agent system control algorithm used to achieve collaborative motion control between multiple agents. It is widely used in the fields of robotics, intelligent transportation systems, etc. This application adopts this method to perform formation flight control of drone clusters.

[0086] The pilot-follower algorithm is used for formation flight control of drone swarms. The basic idea of ​​the algorithm is to divide the agents into two categories: pilots and followers. The pilot is responsible for planning the trajectory, while the followers measure the distance between themselves and the pilot. and angle difference , to achieve the movement of following the navigator. The navigator and the follower exchange data through wireless communication to achieve coordinated movement. Since the odometer carried by the drone used in this application establishes a local coordinate system based on its own initial position, for the drone cluster, it is necessary to convert the local coordinates of each drone in the drone cluster into the global coordinates of ROS through coordinate transformation, so that each drone uses a unified coordinate system. Its navigator motion model in 2D space is shown in the following formula.

[0087] (9).

[0088] in, is the horizontal coordinate of the navigator’s position, is the ordinate of the navigator’s position, ( , ) constitutes the position coordinates of the navigator, and They are and The derivative of Indicates that the navigator is x The speed in the axis direction, Indicates that the navigator is y The speed in the axis direction, Indicates the direction of the pilot; for The derivative of represents the angular velocity of the leader; represents the linear velocity of the leader, represents the angular velocity of the leader. The follower motion model is shown in the following equation.

[0089] (10).

[0090] in, For the The horizontal coordinate of the position of the follower, For the The vertical coordinate of the position of the follower, ( , ) constitutes The position coordinates of the followers, and They are and The derivative of and Indicates The position coordinates of the followers, Indicates The direction of each follower; for The derivative of Indicates The angular velocity of the follower; Indicates The linear velocity of the follower, Indicates The angular velocity of the follower.

[0091] For the leader motion model and the follower motion model, the proportional control strategy is expressed as shown in the following formula.

[0092] (11).

[0093] in, and are the control coefficients of linear velocity and angular velocity respectively. However, this method only considers the error between the leader and the follower, but not the error between followers. Therefore, this embodiment also introduces data exchange between different followers. We consider a system including N For a drone swarm consisting of 100 drones, each drone can be modeled as a first-order integrator model based on its kinematic model, which can be described as follows.

[0094] (12).

[0095] Among them, input It's a drone speed, It's a drone The target state is that all UAVs reach a given formation, and the formation controller is shown in the following formula.

[0096] (13).

[0097] in, The adjacency matrix W = [ ]∈RNXN element, indicating that only when >0: UAV With drones to communicate; Indicates drone The formation offset, Indicates drone The formation offset is determined by the desired formation configuration.

[0098] The adjacency matrix represents the communication topology within the drone cluster. For the sake of simplicity, this application assumes that the communication capabilities between drones in the same cluster are the same. At the same time, the adjacency matrix of the current drone cluster is determined by the formation of the current drone cluster. This application constructs the communication topology, such as Figure 4 As shown, a V-shaped formation consisting of 5 UAVs is shown, and the V-shaped formation includes a leader, follower 1, follower 2, follower 3 and follower 4. The communication topology of the formation composed of the UAV cluster is shown in the following formula.

[0099] (14).

[0100] in, The communication topology of the formation composed of a swarm of drones is shown in Figure 1. The communication topology of the formation composed of a swarm of drones shows that Follower 1 and Follower 2 can only receive data from the leader, Follower 3 receives data from Follower 1, and Follower 4 receives data from Follower 2. Data is transmitted bidirectionally in the formation.

[0101] like Figure 3As shown, the target point preview generation process specifically includes: Step 301: When the operator wants to generate the navigation target point of the drone cluster (the target point is the 3D space target point that the navigator in the drone cluster is expected to fly to). Step 302: First, interact through a preset voice command or a virtual menu. The voice interaction implementation scheme is: the operator interacts with the drone by speaking a preset voice command (such as "generate a target point"). Step 303: After the operator finishes speaking the preset voice command, a target point preview will be generated at the operator's eye gaze point in the mixed reality scene. The target point is represented as a translucent blue cube in the mixed reality scene. Step 304: Then the operator performs a "pinch" gesture, which is represented by the thumb and index finger fitting together and then separating. Step 305: When the operator performs this gesture, a dotted line will extend from the finger joint of the virtual hand in the mixed reality scene. The operator can move the hand or rotate the wrist to make the dotted line collide with the target point (blue cube). Step 306: Determine whether the dotted line and the target point collide. If no collision occurs, jump to step 304, the virtual hand dotted line disappears and waits for the operator's next operation. If no collision occurs, jump to step 307. Step 307: The brightness of the blue cube of the target point preview increases, prompting the operator that the target point is now in a movable state. Step 308: Then the operator moves the target point to the expected position by moving the hand or turning the wrist again, and the 6DOF movement of the cube can be achieved. Step 309: After the target point moves to the expected position, the operator releases the thumb and index finger to end the "pinch" gesture. Step 310: At this time, the target point cube becomes immovable, and the brightness is reduced to the initial brightness, prompting the operator that the target point movement operation has been completed, and the target point position and posture after the coordinate system transformation are sent cyclically. Step 311: End the target point preview generation process.

[0102] Quad-rotor drones cannot independently complete long-term complex tasks due to their limited carrying capacity and endurance. One effective way to solve this problem is to implement drone swarms. However, in addition to the difficulty of interaction during task execution, the difficulty of drone swarms also focuses on the centralized control and unified management of multiple drones.

[0103] This application uses mixed reality devices to interact with drone clusters. Mixed reality devices can fix the virtual world and the real world together, realizing data docking between the virtual world and the real world. Operators can interact with the virtual world in real time to affect the real world. Similarly, changes in the real world will also affect the display of the virtual world in real time, bringing the experiencer an "immersive feeling" of seamless transition between the virtual world and the real world. This application uses mixed reality devices to interact with drone clusters. Compared with traditional ground station platform button control, non-expert operators can also quickly get started in a short time, lowering the threshold for drone cluster control. At the same time, the operator's operation and the response of the drone cluster are intuitive, which is convenient for efficient task execution.

[0104] This application has the following beneficial effects.

[0105] (1) The method for interaction between a mixed reality device and a drone cluster proposed in this application can display the position and various sensor parameters of a drone cluster in real time in a virtual scene, allowing the operator to be immersed in the scene and better observe the status of the drone cluster and the interaction between the drone cluster and the scene. Compared with the traditional 2D ground station, the operator can obtain various data in the scene more directly and conveniently, thereby enhancing the understanding of the scene. At the same time, with the help of the hybrid device, the operator can also capture scene detail data that the traditional ground station cannot display, thereby achieving accurate insight into the scene.

[0106] (2) The method of interaction between the mixed reality device and the drone cluster proposed in this application realizes the real-time dynamic display of the scene. The real-time data transmission between the mixed reality device and the drone cluster is used to realize the real-time dynamic display of the scene. The operator's operating instructions in the virtual space can be transmitted to the drone cluster in reality in real time, and the drone cluster can obtain real-time data feedback, which facilitates the rapid deployment and implementation of various algorithms, plans, and tactics.

[0107] (3) The mixed reality device and drone cluster interaction method proposed in this application improves the efficiency of human-computer interaction. Various training tasks of drone clusters are achieved through human-computer interaction (gestures, eye movements, voice control, etc.). Compared with the traditional ground station platform button control, the control methods such as gestures, eye movements, and voice are more direct and convenient. Non-expert operators can also quickly get started in a short time, which reduces the threshold for drone cluster control. At the same time, various training tasks are achieved by combining virtual scenes. The operator's operation and the response of the drone cluster are intuitive, which facilitates efficient execution of tasks.

[0108] Based on the same inventive concept, the embodiment of the present application also provides a mixed reality device and drone cluster interaction device for implementing the mixed reality device and drone cluster interaction method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more mixed reality device and drone cluster interaction device embodiments provided below can refer to the above limitations on the mixed reality device and drone cluster interaction method, which will not be repeated here.

[0109] In an exemplary embodiment, Figure 5 As shown, a mixed reality device and drone cluster interaction device is provided, including the following modules.

[0110] The startup module T1 is used to: receive the operator's startup instruction through the open source engine, and start the mixed reality scene in the open source engine; receive the operator's startup simulation node instruction through the operating system.

[0111] The unlocking module T2 is used to: receive the operator's drone cluster unlocking instruction through a local computer or a mixed reality device to control the drone cluster to switch to an unlocked state; the open source engine is set on the local computer.

[0112] The target point position generation module T3 is used to: realize the interaction between the operator and the mixed reality scene through the mixed reality device and generate the target point position.

[0113] The drone movement control module T4 is used to control the movement of the drone cluster according to the target point position and the operator's target command, so that the drone cluster completes the action corresponding to the target command, and renders the movement path of the drone cluster in real time in the open source engine.

[0114] The target point arrival judgment module T5 is used to judge whether the drone cluster has reached the target point. If the drone cluster has not reached the target point, the target command continues to be sent, otherwise the target command is stopped.

[0115] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 6As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the interaction data between the mixed reality device and the drone cluster. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for interacting with a mixed reality device and a drone cluster is implemented.

[0116] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0117] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0118] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0119] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0120] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0121] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0122] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0123] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for interaction between a mixed reality device and a drone cluster, characterized in that: The method for interacting the mixed reality device with the drone cluster includes: Receive the operator's start-up instruction through the open source engine, and start the mixed reality scene in the open source engine; receive the operator's start-up simulation node instruction through the operating system; Receiving the operator's drone cluster unlocking instruction through a local computer or a mixed reality device to control the drone cluster to switch to an unlocked state; the open source engine is set on the local computer; The operator is enabled to interact with the mixed reality scene through the mixed reality device to generate the target point position, specifically including: collecting the operator's hand depth map and the operator's eye movement data; recognizing the operator's gesture action according to the hand depth map; and determining the target point position according to the eye movement data and the gesture action through the mixed reality device; Control the movement of the drone cluster according to the target point position and the operator's target instruction, so that the drone cluster completes the action corresponding to the target instruction, and renders the movement path of the drone cluster in real time in the open source engine; Control the movement of the drone cluster according to the target point position and the operator's target instruction, specifically including: Using the pilot-follow algorithm to control the movement of the drone cluster according to the target point position and the operator's target instruction It is judged whether the drone cluster has reached the target point. If the drone cluster has not reached the target point, the target command will continue to be sent, otherwise the target command will be stopped.

2. The method for interaction between a mixed reality device and a drone cluster according to claim 1, characterized in that: Before receiving the operator's drone cluster unlocking instruction step through the local computer or the mixed reality device, the mixed reality device and drone cluster interaction method further includes: Determine whether the operating system is connected to the open source engine. If the operating system fails to connect to the open source engine, wait for the operating system to connect to the open source engine. The data connection between the open source engine and the operating system is developed using the ROSBridge middle layer.

3. The method for interaction between a mixed reality device and a drone cluster according to claim 1, characterized in that: Before receiving the operator's drone cluster unlocking instruction step through the local computer or the mixed reality device, the mixed reality device and drone cluster interaction method further includes: Receiving the operator's remote debugging start command through the mixed reality device, and controlling the mixed reality device to start the remote debugging function; Determine whether the mixed reality device is connected to the open source engine. If the mixed reality device fails to connect to the open source engine, wait for the mixed reality device to connect to the open source engine. If the mixed reality device is successfully connected to the open source engine, the mixed reality scene image in the open source engine will be displayed in real time through the mixed reality device; the operator interacts with the mixed reality scene through the mixed reality interaction method, and receives the operator's drone cluster unlocking instructions or target instructions to perform mobile control operations on the drone cluster.

4. The method for interaction between a mixed reality device and a drone cluster according to claim 3, characterized in that: The mixed reality interaction method includes three-channel mixed interaction technology of gesture, voice and gaze.

5. The method for interaction between a mixed reality device and a drone cluster according to claim 1, characterized in that: Recognizing the gesture of the operator according to the hand depth map specifically includes: Identifying a position of the operator's hand in the hand depth map according to the hand depth map; The random forest algorithm is used to classify the operator's hand according to its position in the hand depth map to obtain the operator's gesture position; the operator's gesture position is used to fit the virtual hand in the mixed reality device to locate the operator's hand position in space to determine the gesture operation.

6. A mixed reality device and drone cluster interaction device based on the mixed reality device and drone cluster interaction method according to claim 1, characterized in that: The mixed reality device and the drone cluster interaction device include: The startup module is used to: receive the startup instruction of the operator through the open source engine, and start the mixed reality scene in the open source engine; receive the startup simulation node instruction of the operator through the operating system; The unlocking module is used to: receive the operator's drone cluster unlocking instruction through a local computer or a mixed reality device to control the drone cluster to switch to an unlocked state; the open source engine is set on the local computer; A target point position generation module is used to: enable the operator to interact with the mixed reality scene through the mixed reality device and generate the target point position; The UAV movement control module is used to: control the movement of the UAV cluster according to the target point position and the operator's target command, so that the UAV cluster completes the action corresponding to the target command, and renders the movement path of the UAV cluster in real time in the open source engine; The target point arrival judgment module is used to judge whether the drone cluster has reached the target point. If the drone cluster has not reached the target point, the target command will continue to be sent, otherwise the target command will be stopped.

7. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for interaction between a mixed reality device and a drone cluster as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for interaction between a mixed reality device and a drone cluster described in any one of claims 1 to 5 is implemented.

Images