A communication method and device combining gesture recognition and Bluetooth communication headset

Abstract

The present application provides a communication method and device combining gesture recognition and Bluetooth communication headsets, the method comprising: determining the topological structure of the ad hoc network according to the on-site environmental conditions of the rescue scene; using portable ad hoc network equipment and relay equipment to build a wireless transmission channel for information within the rescue area to connect the execution end and the command end according to the topological structure; the execution end includes a smart headset and a smart glove; the smart headset is connected to the smart glove via Bluetooth; the smart glove integrates a convolutional neural network algorithm to capture and analyze the user's hand movement information; the smart headset is connected to the smart glove via Bluetooth, receives the hand movement information, and sends it to the command end. The present application realizes efficient gesture interaction and command transmission of rescue personnel in complex rescue scenarios through a combination of lightweight intelligent interactive equipment.

Description

Technical Field

[0001] The present application belongs to the field of disaster relief communications, and in particular, relates to a communication method and device combining gesture recognition and a Bluetooth communication headset. Background Art

[0002] As a natural way of communication for humans, gesture communication plays an important role in our daily life and work. Gestures not only help us express our thoughts and emotions, but also enhance communication effectiveness and understanding, and thus play an important role in communication. Gesture recognition, as an important topic in computer science, aims to identify the meaning of human gestures through algorithms, convert gestures into command actions, and enable users to operate and control devices through simple gestures.

[0003] IMU (Inertial Measurement Unit) gesture interaction technology further improves the accuracy and stability of gesture recognition. This technology uses sensors to capture hand motion data and analyzes this data to achieve gesture recognition and interaction. IMU gesture interaction can more accurately, stably and reliably obtain the meaning of user gestures, making gesture interaction more feasible and efficient in a variety of complex environments.

[0004] However, in certain specific rescue scenarios, such as bushfire rescue, direct use of voice interaction may be limited by environmental factors, such as thick smoke, noise, etc., resulting in communication obstruction. In this case, rescuers need to use gesture interaction as an auxiliary means to convey information. There are two main application scenarios for existing gesture recognition technology: one is optical-based gesture recognition technology, and the other is to use data gloves to obtain gesture movements and then transmit them back to the helmet system for command transmission. However, both technologies have obvious defects: the former has poor recognition effect in dark scenes, and the latter increases the equipment load and operation difficulty of rescuers due to the high complexity of the system. Summary of the invention

[0005] The purpose of this application is to overcome the problems existing in the above-mentioned prior art and to provide a communication method and device combining gesture recognition and Bluetooth communication headset.

[0006] The present application provides a communication method combining gesture recognition and a Bluetooth communication headset, including:

[0007] Determine the topology of the ad hoc network based on the on-site environmental conditions of the rescue scenario;

[0008] According to the topological structure, a portable ad hoc network device and a relay device are used to build a wireless transmission channel for information within the rescue area to connect the execution end and the command end;

[0009] The execution end includes a smart headset and smart gloves;

[0010] The smart headset is connected to the smart gloves via Bluetooth;

[0011] The smart glove integrates a convolutional neural network algorithm to capture and analyze the user's hand movement information;

[0012] The smart headset is connected to the smart gloves via Bluetooth, receives the hand movement information, and sends it to the command end.

[0013] Optionally, the topology of the ad hoc network includes: a star structure or a tree structure.

[0014] Optionally, the relay device includes a drone.

[0015] Optionally, the command end displays in real time a real-life map of the rescue area, connection status of each headset, and location information of the rescue team members.

[0016] Optionally, the smart glove integrates an IMU module, a core board and structural parts;

[0017] The core board uses a convolutional neural network algorithm to accurately capture and analyze gestures.

[0018] The present application also provides a communication device combining gesture recognition and a Bluetooth communication headset, comprising:

[0019] The topology module determines the topology of the ad hoc network according to the on-site environmental conditions of the rescue scenario;

[0020] A connection module, which uses a portable ad hoc network device and a relay device to build a wireless transmission channel for information within the rescue area to connect the execution end and the command end according to the topological structure;

[0021] The execution end includes a smart headset and smart gloves;

[0022] The smart headset is connected to the smart gloves via Bluetooth;

[0023] The smart glove integrates a convolutional neural network algorithm to capture and analyze the user's hand movement information;

[0024] The smart headset is connected to the smart gloves via Bluetooth, receives the hand movement information, and sends it to the command end.

[0025] Optionally, the topology of the ad hoc network includes: a star structure or a tree structure.

[0026] Optionally, the relay device includes a drone.

[0027] Optionally, the command end displays in real time a real-life map of the rescue area, connection status of each headset, and location information of the rescue team members.

[0028] Optionally, the smart glove integrates an IMU module, a core board and structural parts;

[0029] The core board uses a convolutional neural network algorithm to accurately capture and analyze gestures.

[0030] The beneficial effects of this application are:

[0031] The present application provides a communication method combining gesture recognition and Bluetooth communication headsets, including: determining the topological structure of the ad hoc network according to the on-site environmental conditions of the rescue scene; using portable ad hoc network equipment and relay equipment to build a wireless transmission channel for information within the rescue area to connect the execution end and the command end according to the topological structure; the execution end includes a smart headset and a smart glove; the smart headset is connected to the smart glove via Bluetooth; the smart glove integrates a convolutional neural network algorithm to capture and analyze the user's hand movement information; the smart headset is connected to the smart glove via Bluetooth, receives the hand movement information, and sends it to the command end. The present application realizes efficient gesture interaction and command transmission of rescue personnel in complex rescue scenarios through a combination of lightweight intelligent interactive equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a communication flow diagram of the combination of gesture recognition and Bluetooth communication headset in this application;

[0033] Figure 2 It is a schematic diagram of the communication system architecture combining gesture recognition and Bluetooth communication headset in this application. DETAILED DESCRIPTION

[0034] The present application is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present application and implement it.

[0035] In response to the problems of difficulty in gesture recognition in jungle fire rescue scenarios and the increased burden and operation difficulty of integrated helmet displays on rescuers, this application proposes a communication method that combines gesture recognition and Bluetooth communication headsets. By predefining simple gesture models and integrating inertial navigation recognition algorithms on the glove side, smart gloves and smart headsets are directly connected to simplify complex interaction scenarios, allowing rescuers to travel light without affecting the interaction effect.

[0036] To achieve the above objectives, the present invention provides a lightweight intelligent interactive equipment combination, the main technologies used include self-organizing network construction technology, Bluetooth connection technology, wireless communication technology, gesture recognition technology, convolutional neural network and other technologies. The equipment used includes smart gloves, smart headphones, portable self-organizing network equipment, background server 2 (system server), etc. Based on the overall architecture and composition logic of product design, each module is organically combined to form a complete product.

[0037] Please refer to Figure 1 and Figure 2 As shown, the present application provides a communication method combining gesture recognition and a Bluetooth communication headset, the steps of which include:

[0038] S101. Determine the topology of the ad hoc network according to the on-site environmental conditions of the rescue scene.

[0039] Conduct a detailed network environment analysis1 for the bushfire rescue scenario. The analysis includes but is not limited to the following aspects:

[0040] 1. Determine the specific scope of the rescue area, including length, width and height, which will help evaluate the required communication range and signal coverage strength.

[0041] 2. Investigate the terrain features of the rescue area, such as mountains, plains, waters, etc. Different terrains have different effects on signal propagation and attenuation.

[0042] 3. Identify obstacles in the rescue area, such as trees, buildings, rocks, etc., which may block or interfere with the transmission of wireless signals.

[0043] 4. Analyze possible signal interference sources in the rescue area, such as other wireless communication devices, electronic devices or natural factors (such as lightning), which may affect the stability and reliability of the ad hoc network.

[0044] After completing the analysis of network environment 1, design a suitable ad hoc network topology structure according to the analysis results, including ad hoc network devices and wireless access points.

[0045] The topology of an ad hoc network determines the connection mode and communication path between nodes in the network, and has an important impact on the performance of the network. Common ad hoc network topologies include star, tree, and hybrid.

[0046] Among them, all nodes in the star topology are directly connected to the central node to form a star structure. This structure is simple and easy to manage, but the failure of the central node may cause the entire network to fail. In the tree topology, the nodes are organized in a hierarchical structure to form a tree structure. This structure has strong scalability and flexibility, but it has strong path dependence, and the failure of some nodes may affect the communication of their child nodes.

[0047] Combining the advantages of star and tree topologies, the network structure can be flexibly designed according to specific needs. This structure is usually more complex, but can adapt to complex rescue scenarios and diverse communication needs.

[0048] In this application, a tree or hybrid topology is selected to ensure comprehensive and stable signal coverage in view of the characteristics of the jungle fire rescue scene. In the specific design, factors such as the deployment location of the node, communication distance, and power control need to be considered to optimize network performance.

[0049] After completing the topology design, conduct preliminary simulation verification or field testing to evaluate the actual performance and stability of the network. Based on the test results, make necessary adjustments and optimizations to the topology to ensure that it can meet the communication needs in the rescue scenario.

[0050] S102, using a portable ad hoc network device and a relay device according to the topology structure to build a wireless transmission channel for information within the rescue area to connect the execution end and the command end.

[0051] First, deploy portable ad hoc network devices at key locations in the rescue area (such as commanding heights, open areas, etc.). As the core nodes of the network, these devices are responsible for data forwarding and routing to ensure the effective flow of information. Base station equipment must have good mobility and stability so that it can be quickly deployed and work continuously in complex and changing rescue environments. At the same time, interoperability between base station equipment should be ensured to form a complete communication network.

[0052] According to the specific terrain, obstacle distribution and communication needs of the rescue area, an appropriate number of relay devices (such as drones, mobile relay stations, etc.) are configured. These relay devices can automatically find the best communication path, expand the communication range and improve the communication quality.

[0053] When drones are used as relay devices, they can take advantage of their high altitude to avoid ground obstacles and achieve long-distance signal transmission. Mobile relay stations can be flexibly deployed as needed to ensure signal coverage without blind spots.

[0054] Use portable ad hoc network equipment and relay equipment to build a wireless transmission channel covering the entire rescue area. Ensure the stability and reliability of the signal during transmission by optimizing the network configuration. During the construction process, factors such as signal attenuation and interference need to be considered, and corresponding measures (such as adjusting the device position, increasing antenna gain, etc.) should be taken to improve signal quality.

[0055] After the self-organizing network is built, the network performance test is carried out. The test content includes key indicators such as signal strength, transmission delay, packet loss rate, etc., to comprehensively evaluate the performance of the network.

[0056] According to the test results, the network is optimized and adjusted. For example, relay equipment is added to weak signal areas, and equipment parameters are adjusted to improve transmission quality. Through continuous optimization, reliable transmission of information within the rescue area is ensured.

[0057] After the network performance test and optimization are completed, actual communication verification is carried out. By simulating the communication needs in the rescue scenario, the communication effect between the smart headset and smart gloves and the command end is tested. The problems found are debugged and repaired in a timely manner to ensure the stability and reliability of the entire communication system. At the same time, user feedback is collected to provide a basis for subsequent improvements and optimizations.

[0058] S103: The execution end includes a smart headset and smart gloves.

[0059] First, the rescuer equipment 4 on the execution end includes smart headphones and smart gloves. The IMU (inertial measurement unit) module is integrated into the smart gloves. The IMU module is the core component of the gesture recognition system, responsible for capturing the motion data of the hand, including acceleration, angular velocity, magnetic field and other information. These data are the key to achieving accurate gesture recognition.

[0060] Select a suitable location inside the glove to install the core board. The core board integrates a convolutional neural network algorithm, which is a key part of the gesture recognition algorithm. The algorithm can deeply analyze the hand motion data captured by the IMU module, so as to accurately identify the user's gestures.

[0061] The IMU module, core board and other necessary structural parts are fixed to the flexible gloves through professional sewing technology. This step ensures the comfort and durability of the gloves, while ensuring the stable operation of the IMU module and core board. The sewn gloves should have a good wearing experience and be easy for rescuers to use in various complex environments.

[0062] Next, make sure that the smart headset has Bluetooth communication function and is compatible with the smart gloves. Through Bluetooth connection technology, the headset can stably receive gesture recognition results from the smart gloves. This step is the key to gesture recognition and communication, which realizes the seamless connection between gesture recognition results and the command end.

[0063] After the smart headset is configured, perform a functional test. The test content includes but is not limited to the headset's receiving function, sending function, and stability of the Bluetooth connection. By sending preset gesture commands and observing the headset's response, you can verify whether the headset can accurately receive and send gesture recognition results. This step is an important part of ensuring that the headset works stably and reliably in rescue scenarios.

[0064] S104: The smart headset is connected to the smart glove via Bluetooth.

[0065] In this application, the connection between the smart headset and the smart gloves is achieved through Bluetooth technology. Bluetooth technology is an ideal choice for connecting smart wearable devices due to its low power consumption, high stability and wide compatibility. The specific implementation process is as follows:

[0066] Rescuers first need to pair the smart headset with the smart gloves via Bluetooth. This step is usually performed when the device is used for the first time to ensure that the two can establish a stable communication connection. During the pairing process, the smart headset searches for nearby Bluetooth devices and identifies the Bluetooth signal of the smart gloves. The user confirms by pressing the operation button on the headset or gloves to complete the pairing process.

[0067] Once paired successfully, the smart headset can receive data signals from the smart gloves. These data signals contain hand movement information captured and analyzed by the IMU module on the glove side.

[0068] The convolutional neural network algorithm integrated in the glove will process the hand motion data, extract gesture features, and send the gesture recognition results to the smart headset via Bluetooth.

[0069] After receiving the gesture recognition result, the smart headset will confirm the result. If the result is confirmed to be correct, the rescuer can send a command through the confirmation button on the headset.

[0070] The command is quickly sent to the command end device 3, including a terminal and / or a laptop computer, via Bluetooth technology, so that the command end personnel can quickly make decisions.

[0071] A comprehensive rescue management system is deployed at the command end, which integrates multiple functional modules to support the comprehensive management and command of rescue missions. Specific functional modules include:

[0072] Rescue scene real-life map management: used to import and display real-life maps of rescue areas, providing commanders with intuitive geographic information.

[0073] Smart headset management: used to monitor and manage the smart headset devices of all rescuers, including the device's connection status, location information, etc.

[0074] Communication between the command end and the headset: Realize wireless communication between the command end and the smart headset to ensure the timely transmission of instructions and information.

[0075] Sending instructions: Supports commanders to send instructions to rescue teams through the system, including support requests, mission completion, etc.

[0076] On the command end, pair the smart gloves, smart headphones and rescue management system to ensure that each rescuer's gloves and headphones can be stably connected to the command end system. The specific steps are as follows:

[0077] The command system searches for nearby smart headsets and smart gloves through Bluetooth technology.

[0078] Rescuers confirm pairing using the operating buttons on the headset or gloves to ensure a stable connection between the devices.

[0079] The connection status of each device is displayed in the rescue management system to ensure that the commander can understand the device connection status at any time.

[0080] The connection status and location of each headset are monitored in real time in the rescue management system. The specific implementation method is as follows:

[0081] The Bluetooth connection status information updates the headset's connection status in real time, ensuring that the commander can always know whether the device is online.

[0082] Combined with positioning technology (such as GPS or Beidou system), the real-time location information of the rescue team members is obtained and displayed in the system.

[0083] Import the real-life map of the rescue area into the command system and display it on the big screen. At the same time, display the connection status of each headset and the location of the rescue team members to provide an intuitive basis for command decision-making. The specific implementation method is as follows:

[0084] Import real-life map files (such as JPEG, PNG, etc.) of the rescue area through the system interface to ensure the accuracy and completeness of the map information.

[0085] The imported real-life map is displayed on a large screen to provide commanders with intuitive geographic information support.

[0086] The connection status of each headset and the location information of the rescue team members are marked on the map, and they are distinguished and displayed through different icons or colors.

[0087] S105. The smart glove integrates a convolutional neural network algorithm to capture and analyze the user's hand movement information.

[0088] Smart gloves are mainly composed of IMU (Inertial Measurement Unit) modules, core boards, structural parts and other components. These components are carefully designed and sewn onto the flexible gloves to ensure the comfort and wearability of the gloves. The IMU module is responsible for capturing the raw data of hand movements, such as acceleration, angular velocity, etc., which are the basis for gesture recognition.

[0089] The core board is integrated with a convolutional neural network algorithm. This algorithm can accurately capture and analyze the user's hand movement information through deep learning and analysis of hand movement data. Specifically, the convolutional neural network can automatically extract key features in hand movement, such as the degree of finger bending, the rotation angle of the palm, etc., and then convert these features into recognizable gesture commands.

[0090] When rescuers wear smart gloves and make specific gestures, the IMU module captures hand movement data in real time and transmits the data to the convolutional neural network algorithm on the core board for processing. The algorithm analyzes and compares the data to identify the corresponding gesture instructions, such as "request support." The recognition results are then fed back to the smart headset for further confirmation and sending by the rescuers.

[0091] The rescue team members started to carry out the rescue mission as planned and always maintained communication with the command personnel. During the rescue process, they needed to flexibly adjust the strategy according to the on-site situation to ensure the smooth progress of the rescue operation. The combination of smart gloves and smart headphones provides them with an efficient and reliable way to transmit information, allowing them to maintain effective communication even in complex and changing rescue environments.

[0092] During the rescue process, special circumstances may occur that make voice interaction impossible, such as noise interference and smoke obstruction at the fire scene. At this time, the rescue team can use gesture command interaction as a compensation method. They recognize gesture commands from the trained gesture set, such as "request support".

[0093] When the rescuer makes a specific gesture, the smart glove will immediately capture the hand movement data through the IMU module and analyze it using the convolutional neural network algorithm on the core board. The algorithm will quickly recognize the gesture command and feed back the analysis results to the smart headset worn.

[0094] After the headset receives the gesture analysis result, it will immediately notify the rescuer through a pop-up window or voice prompt. After the rescuer confirms that it is correct, he can press the confirmation button on the headset to send the command to the command end via Bluetooth. After receiving the command, the command end will immediately process it and take corresponding countermeasures, such as dispatching a support team.

[0095] S106. The smart headset is connected to the smart glove via Bluetooth, receives the hand movement information, and sends it to the command end.

[0096] The smart headset establishes a stable connection with the smart gloves through Bluetooth technology, realizing real-time data transmission. Specifically, when the smart gloves integrate convolutional neural network algorithms to capture the rescuer's hand movement information, this information is immediately processed and converted into recognizable instructions or signals. Subsequently, the smart headset receives these signals from the gloves through Bluetooth communication, ensuring the accuracy and timeliness of the information.

[0097] After receiving the hand movement information, the smart headset will further process the information to confirm the correctness of the gesture recognition results. If the gesture command is correctly recognized and the rescuer verifies it through the confirmation button on the headset, the smart headset will quickly send these commands to the command end. This process not only improves the efficiency and reliability of communication, but also ensures the accuracy of the commands, providing a strong guarantee for the success of the rescue operation.

[0098] When the command end receives the gesture command from the smart headset, it will immediately respond according to the command content. Take the command "Request support" as an example. When the command end receives the "Request support" message from team member No. 1, it will quickly initiate a notification command, such as "Team member No. 1 needs support." This command will not only be displayed in a pop-up window on the command end screen, but will also be synchronized to the devices of other team members and related collaborators through the system to ensure instant information sharing and collaborative response.

[0099] After receiving the support order, other team members and relevant collaborators will quickly take corresponding support measures, such as adjusting positions, preparing rescue equipment, etc., to support the corresponding team members to perform the corresponding rescue actions. This efficient command response and support mechanism greatly improves the overall efficiency and success rate of the rescue operation.

[0100] After the rescue mission is successful, both the team members and the command end personnel can initiate a confirmation message of mission completion. If initiated by a team member, they will send it to the command end through a preset gesture command and smart headset. Specifically, the team members will make a gesture of mission completion, and the smart gloves will recognize the gesture command and send it to the smart headset, which will then send it to the command end personnel. If initiated by the command end, the system will directly send a team-recovery command to all members.

[0101] When the rescue mission is completed, the team members will return to the base according to the instructions. During the return process, they can disconnect the connection between the devices to save power and reduce the burden on the equipment. If a team member fails to return on time due to special reasons, the command end can search based on the positioning information of the device to ensure that each team member can return safely until the mission is successfully completed.

[0102] The present application also provides a communication device combining gesture recognition and a Bluetooth communication headset, comprising:

[0103] The topology module determines the topology of the ad hoc network according to the on-site environmental conditions of the rescue scenario;

[0104] A connection module, which uses a portable ad hoc network device and a relay device to build a wireless transmission channel for information within the rescue area to connect the execution end and the command end according to the topological structure;

[0105] The execution end includes a smart headset and smart gloves;

[0106] The smart headset is connected to the smart gloves via Bluetooth;

[0107] The smart glove integrates a convolutional neural network algorithm to capture and analyze the user's hand movement information;

[0108] The smart headset is connected to the smart gloves via Bluetooth, receives the hand movement information, and sends it to the command end.

[0109] Furthermore, the topology of the ad hoc network includes: a star structure or a tree structure.

[0110] Furthermore, the relay device includes a drone.

[0111] Furthermore, the command end displays in real time a real-life map of the rescue area, the connection status of each headset, and location information of the rescue team members.

[0112] Furthermore, the smart glove integrates an IMU module, a core board and structural parts;

[0113] The core board uses a convolutional neural network algorithm to accurately capture and analyze gestures.

[0114] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and its core ideas of this application. The above is only the preferred implementation method of this application. It should be pointed out that due to the limitations of textual expression and the objective existence of infinite specific structures, ordinary technicians in this technical field can make some improvements, modifications or changes without departing from the principles of this application, and combine the above technical features in an appropriate manner; the improvements, modifications, changes or combinations, or the direct application of the concept and technical solution of the invention to other occasions without improvement, should be regarded as the scope of protection of this application.

Claims

1. A communication method combining gesture recognition and Bluetooth communication headset, characterized in that: include: Determine the topology of the ad hoc network based on the on-site environmental conditions of the rescue scenario; According to the topological structure, a portable ad hoc network device and a relay device are used to build a wireless transmission channel for information within the rescue area to connect the execution end and the command end; The execution end includes a smart headset and smart gloves; The smart headset is connected to the smart gloves via Bluetooth; The smart glove integrates a convolutional neural network algorithm to capture and analyze the user's hand movement information; The smart headset is connected to the smart gloves via Bluetooth, receives the hand movement information, and sends it to the command end.

2. According to claim 1, a communication method combining gesture recognition and Bluetooth communication headset, characterized in that: The topology of the ad hoc network includes: a star structure or a tree structure.

3. According to claim 1, a communication method combining gesture recognition and Bluetooth communication headset, characterized in that: The relay device includes a drone.

4. According to claim 1, a communication method combining gesture recognition and Bluetooth communication headset, characterized in that: The command terminal displays in real time a real-life map of the rescue area, the connection status of each headset, and the location information of the rescue team members.

5. According to claim 1, a communication method combining gesture recognition and Bluetooth communication headset, characterized in that: The smart glove integrates an IMU module, a core board and structural parts; The core board uses a convolutional neural network algorithm to accurately capture and analyze gestures.

6. A communication device combining gesture recognition and Bluetooth communication headset, characterized in that: include: The topology module determines the topology of the ad hoc network according to the on-site environmental conditions of the rescue scenario; A connection module, which uses a portable ad hoc network device and a relay device to build a wireless transmission channel for information within the rescue area to connect the execution end and the command end according to the topological structure; The execution end includes a smart headset and smart gloves; The smart headset is connected to the smart gloves via Bluetooth; The smart glove integrates a convolutional neural network algorithm to capture and analyze the user's hand movement information; The smart headset is connected to the smart gloves via Bluetooth, receives the hand movement information, and sends it to the command end.

7. A communication device combining gesture recognition and Bluetooth communication headset according to claim 6, characterized in that: The topology of the ad hoc network includes: a star structure or a tree structure.

8. The communication device combining gesture recognition and Bluetooth communication headset according to claim 6, characterized in that: The relay device includes a drone.

9. A communication device combining gesture recognition and a Bluetooth communication headset according to claim 6, characterized in that: The command terminal displays in real time a real-life map of the rescue area, the connection status of each headset, and the location information of the rescue team members.

10. A communication device combining gesture recognition and a Bluetooth communication headset according to claim 6, characterized in that: The smart glove integrates an IMU module, a core board and structural parts; The core board uses a convolutional neural network algorithm to accurately capture and analyze gestures.

Images