An artificial intelligence image recognition method based on edge computing

Abstract

The application discloses an artificial intelligence image recognition method based on edge computing, which comprises the following steps: acquiring the continuous gestures and facial image streams of rescuers in real time through an edge computing terminal, forming time sequence data packets after key frame extraction; analyzing gesture trajectory intentions based on a convolutional neural network and a long short-term memory network, simultaneously analyzing expression instruction modes based on a Transformer network, and generating composite task intentions through multi-modal fusion; synchronously, a UAV cluster collects environmental data and returns the data to the edge terminal, and a 3D semantic map of an identification hotspot, an obstacle and a dangerous area is constructed; performing safety verification according to the analyzed intentions in combination with the semantic map, and generating control instructions; after receiving the instructions, the UAV cluster autonomously and cooperatively executes the task through a swarm intelligence algorithm, and returns state data in real time. The application realizes accurate understanding and rapid response to complex rescue task intentions, significantly reduces interaction delay, and improves the accuracy, adaptability and operation safety of human-machine cooperation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence image recognition technology, and in particular to an artificial intelligence image recognition method based on edge computing. Background Technology

[0002] Currently, AI-based image recognition technology has been widely applied in the field of human-computer interaction. In complex scenarios such as emergency rescue, existing technologies mainly control equipment by recognizing discrete, static gestures or fixed commands. However, these methods struggle to effectively understand the continuous, natural, dynamic intent flows of rescuers, such as complex commands combining trajectory gestures and facial expression changes. Traditional centralized processing architectures typically transmit raw video data back to the cloud for recognition, making real-time response difficult in environments with weak networks and high latency. Furthermore, most recognition models focus only on a single modality (such as gestures or facial expressions), lacking collaborative analysis and joint reasoning of multimodal continuous action flows. This results in insufficient accuracy and robustness in intent parsing in complex environments, failing to meet the stringent requirements of immediate and accurate interaction in rescue missions.

[0003] To further improve response speed, some studies have attempted to deploy recognition models on edge devices. However, existing edge AI image recognition solutions mostly focus on isolated perception for single machines and single tasks, lacking deep collaboration with the autonomous environmental perception capabilities of unmanned swarms. In swarm control scenarios, simple command recognition cannot understand the physical context of the command, such as whether there are obstacles or hazards in the target area. This leads to the generated commands potentially conflicting with the real-time environmental state, requiring additional complex online verification and adjustments, which in turn increases system latency and uncertainty. Summary of the Invention

[0004] This application provides an edge computing-based artificial intelligence image recognition method that tightly couples continuous multimodal intent understanding, real-time environmental semantic perception, and cluster autonomous control to achieve the goal of generating tasks that can be safely executed.

[0005] This application provides an artificial intelligence image recognition method based on edge computing, including: S1 acquires continuous hand gestures and facial image streams of rescue commanders through a multimodal camera, extracts keyframes and timestamps them at the on-site edge computing terminal, and forms a standardized time-series image data packet; where the edge computing terminal specifically refers to a mobile computing terminal located at the rescue site that is responsible for core computing and coordination; S2, based on convolutional neural networks and LSTM networks, continuously recognizes the temporal gesture images of rescue commanders and analyzes their trajectories and intentions; at the same time, based on the Transformer network, it performs expression analysis on their temporal facial images, analyzes their command modality, and outputs the composite task intent through multimodal fusion. S3, while performing image recognition and analysis, the onboard sensors of the drone cluster simultaneously collect environmental images and point cloud data, transmit them back to the same on-site edge computing terminal and aggregate the data from each drone, and run the semantic mapping module to convert the collected data into a 3D semantic map, marking thermal imaging hotspots, obstacles and dangerous areas. S4, based on the spatial intent and task modality matching association rule base parsed from image recognition, maps the image trajectory to geographical regions and UAV sub-clusters, and maps the image modality to task type and parameters. After security verification in conjunction with the semantic map, it generates UAV control commands. S5, based on the received control commands, the drone swarm autonomously assigns roles and coordination strategies through swarm intelligence algorithms, executes tasks, and transmits status images and environmental data back to the edge terminal in real time.

[0006] Preferably, in step S2, parsing the trajectory and intent, and parsing the command modality, specifically includes: receiving standardized temporal image data packets, and parsing the gesture image sequence and facial image sequence in parallel; for the gesture image sequence, extracting the spatial features of each frame image through a convolutional neural network to obtain a gesture feature vector sequence, and inputting this sequence into a long short-term memory network for temporal modeling to output the classification result of the gesture trajectory intent and its geometric parameters; for the facial image sequence, firstly extracting the features of each frame image through a convolutional neural network and combining them with position encoding to form a facial feature sequence, and inputting this sequence into a Transformer encoder model for temporal dependency modeling to output the corresponding facial expression command modality; fusing the classification result of the gesture trajectory intent, the geometric parameters, and the facial expression command modality, and generating a structured composite task intent description through a multimodal fusion engine.

[0007] Preferably, the association rule base includes: a first association rule in the association rule base that maps the parsed gesture trajectory to a specific geographical area and dynamically allocates a suitable drone sub-cluster based on the drone's location and battery level; and a second association rule that maps the parsed facial expression modality and gesture type to a specific task type and task parameters.

[0008] Preferably, the autonomous allocation of roles and collaborative strategies through swarm intelligence algorithms specifically includes: the UAV sub-swarm receiving and parsing the control commands, extracting the task area, task type, and flight parameters; the UAV sub-swarm initiating a collaborative mechanism based on distributed swarm intelligence algorithms, exchanging status information through inter-UAV communication, and autonomously allocating roles and decomposing tasks; each UAV generating a local flight trajectory based on the assigned sub-tasks and flight parameters using an onboard trajectory planner; during flight, each UAV sensing the environment in real time based on onboard sensors and dynamically adjusting its flight path to avoid obstacles using a real-time obstacle avoidance algorithm; each UAV encapsulating its own status information and local sensing data into a status report, transmitting it back to the on-site edge computing terminal via a self-organizing communication network; and triggering a cluster collaborative emergency response mechanism when an abnormal status is detected, performing task reassignment or issuing a reset command from the edge computing terminal.

[0009] Preferably, the semantic mapping module converts the collected data into a 3D semantic map, and step S3 further includes: S31, based on the preliminary intent of the continuous actions parsed in S2, introduces a semantic map as the environmental context to compensate and calibrate the deviations caused by visual recognition, and generates high-level task instructions consistent with the environmental state. S32, the edge computing terminal receives the calibrated high-level task instructions and parses them into serialized and structured executable subtasks according to different task types; based on the real-time status images and pose data transmitted back by the UAV cluster, it dynamically matches the UAV with the best capabilities and status for each subtask through a distributed decision algorithm, forming a collaborative task execution scheme starting from visual instructions. S33: Each UAV, based on its assigned sub-task, integrates real-time perception from its own onboard visual sensors with a global semantic map to perform local path planning and dynamic obstacle avoidance; at the same time, it shares key visual information and location through inter-UAV communication, runs swarm intelligence algorithms to maintain sub-task formation and avoid collisions, and autonomously completes collaborative tasks triggered by visual commands.

[0010] Preferably, the compensation and calibration of deviations generated by visual recognition includes: receiving spatial trajectory parameters identified based on gesture image sequences and facial expression command modalities identified based on facial image sequences; accessing a semantic map that is constructed and maintained in real time; wherein, the units of the semantic map are labeled with semantic categories and confidence levels; extracting traversability constraint information of the region corresponding to the spatial trajectory parameters from the semantic map; performing geometric transformation on the original region defined by the spatial trajectory parameters based on the traversability constraint information through optimization processing, so that the transformed region, while maintaining the original geometric topology, coincides with the traversable region in the semantic map to the greatest extent and minimizes the intersection area with obstacle regions; and outputting calibrated spatial coordinate parameters.

[0011] Preferably, the formation of a collaborative task execution scheme starting with visual instructions includes: receiving high-level visual task instructions calibrated by environmental context, the instructions including task type, geospatial parameters, and dynamic priority; calling a pre-set task template knowledge base, matching the corresponding task decomposition template according to the task type, and substituting the geospatial parameters into the template to generate a structured subtask graph; acquiring real-time status information transmitted back by each UAV in the UAV cluster, including real-time pose, remaining battery power, task load, and onboard sensor capabilities; dynamically selecting the execution UAV for each subtask based on the resource requirements of each subtask in the subtask graph and the real-time status information of each UAV through a distributed decision algorithm; and generating a task-UAV mapping table based on the result of the dynamic selection; wherein the mapping table and the subtask graph together constitute the collaborative task execution scheme.

[0012] Preferably, the compensation and calibration of deviations caused by visual recognition further includes: S311 integrates visible light, infrared thermal imaging, and millimeter-wave radar sensors at the edge of the UAV swarm mission to perform time-synchronized acquisition and parallel preprocess the raw data of each modality to generate a standardized multimodal visual data packet with timestamp alignment. S312 dynamically matches the current working mode scenario of the drone cluster based on the image quality index in the multimodal visual data packet calculated in real time. S313 utilizes a cross-modal fusion network to extract features from each modality and adjust the visual recognition deviation value. Based on the adjusted deviation value, the edge computing terminal calculates the intention deviation and key inflection points of rescue personnel in different working modal scenarios. S314: Confirm the deviation values ​​of intent deviation and key inflection points in different modalities, and adjust the advanced task instructions accordingly.

[0013] Preferably, the multimodal visual data packet includes: multi-source sensing data collected and preprocessed at the same time stamp of the multimodal visual data packet, the core components of which include: a visible light RGB image matrix of standard resolution, an infrared thermal imaging data matrix calibrated and segmented by temperature, a radar point cloud cluster set characterizing human dynamics, and spatial pose information of each sensor.

[0014] Preferably, the dynamic matching of the current working modal scenario of the UAV cluster specifically includes: parallel calculation of the corresponding quality assessment index for each modality data in the multimodal visual data packet; determining the probability distribution of the current environment category based on the auxiliary features extracted from the data packet through a scene classification model; calculating the comprehensive score of each modality at the current moment according to the quality assessment index and the probability distribution; and dynamically selecting the dominant modality for intent recognition based on the comprehensive score.

[0015] One or more technical solutions provided in this application have at least the following technical effects or advantages: By efficiently processing continuous visual commands from command personnel at the edge and combining this with real-time environmental perception of the drone swarm, a precise understanding and rapid response to complex and dynamic rescue mission intentions are achieved. This method significantly reduces human-machine interaction latency, improves the accuracy of command parsing and environmental adaptability, and ensures efficient and safe operation of the drone swarm in complex environments through a cloud-based collaborative autonomous task allocation and execution mechanism. Ultimately, it constructs an efficient and robust human-machine collaborative intelligent rescue closed loop.

[0016] By deeply integrating real-time semantic maps into the continuous visual intent understanding process, a visual command calibration mechanism with enhanced environmental awareness was constructed, significantly improving the accuracy, security, and physical feasibility of visual intent recognition in complex environments. Furthermore, through visual command-based collaborative task parsing and dynamic resource allocation, efficient and autonomous transformation from high-level intents to multi-machine collaborative execution schemes was achieved, optimizing the overall cluster performance.

[0017] By introducing a multimodal data adaptive fusion and dynamic quality assessment mechanism, the robustness and accuracy of continuous visual intent recognition for rescuers in complex and harsh environments are significantly enhanced. The method intelligently selects the optimal perception modality based on the real-time environment and effectively compensates for recognition biases of a single modality through cross-modal feature fusion, thus stably reconstructing accurate user intent even under strong interference conditions. Ultimately, this method ensures highly reliable generation of UAV swarm control commands, significantly improving the adaptability of human-machine collaboration and the overall mission success rate in complex rescue scenarios. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating an artificial intelligence image recognition method based on edge computing, according to an embodiment of the present invention. Detailed Implementation

[0019] To facilitate understanding of the present invention, a more complete description of this application will be given below with reference to the accompanying drawings, which illustrate preferred embodiments of the invention. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to enable a more thorough and complete understanding of the disclosure of the present invention.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention; the terminology used herein includes any and all combinations of one or more of the associated listed items.

[0021] Example 1: Figure 1 This is a flowchart illustrating an artificial intelligence image recognition method based on edge computing according to an embodiment of the present invention.

[0022] like Figure 1 As shown, an artificial intelligence image recognition method based on edge computing includes the following steps: S1 acquires continuous hand gestures and facial image streams of rescue commanders through a multimodal camera, extracts keyframes and timestamps them on the on-site edge computing terminal, and forms standardized time-series image data packets.

[0023] Among them, edge computing terminals specifically refer to mobile computing terminals located at the rescue site that are responsible for core computing and coordination (e.g., ruggedized industrial tablet PCs, vehicle-mounted edge servers, or portable high-performance computing units).

[0024] Keyframe extraction focuses on extracting key state frames from the hand gestures and facial expressions of rescue commanders. Specifically, for gesture streams, it extracts the start and end points of the gesture trajectory, points of significant directional change (inflection points), and frames maintaining specific postures. For example, in a complete sequence from raising a hand, making a circle, to lowering the arm, only a few frames are extracted, such as the start of raising the hand, the apex of the circle, and the lowering of the arm, rather than the entire 30 frames per second image. For facial expression streams, it extracts frames where facial expressions change significantly, such as representative moments during the process of moving from calm to opening the mouth (issuing commands), from normal to frowning (expressing urgency), and continuous nodding.

[0025] S2, based on convolutional neural networks and LSTM networks, continuously recognizes the temporal gesture images of rescue commanders and analyzes their trajectories and intentions; at the same time, it analyzes the expressions of their temporal facial images based on Transformer networks, analyzes their command modalities, and outputs the composite task intentions through multimodal fusion.

[0026] Specifically, the standardized temporal image data packets generated in step S1 are received, and the recognition and parsing of gesture and facial images are performed in parallel. For the gesture image sequence, each frame of the image is first processed by a pre-trained lightweight convolutional neural network to extract spatial features, resulting in feature vectors representing key information about the hand. These feature vector sequences arranged in chronological order are then input into a long short-term memory network to model the continuous dynamic changes of the gesture. The network ultimately outputs a classification result of the macroscopic intent of the gesture trajectory (e.g., drawing a circle) and its corresponding geometric parameters (e.g., the center and radius of the circle).

[0027] For facial image sequences, each frame's features are extracted via a convolutional backbone network and combined with positional encoding to form a sequence embedding. This embedding sequence is then processed by a Transformer encoder-based model, which captures long-range dependencies between expression frames through its built-in multi-head self-attention mechanism. The model uses the contextual representation corresponding to the special markers at the beginning of the sequence to classify the corresponding command modality (e.g., confirmation or urgency). The intention category and geometric parameters output by the gesture recognition network are concatenated with the command modality output by the facial expression recognition network, and the concatenated joint feature vector is fed into a multi-modal fusion engine composed of a multilayer perceptron for decision-making and inference, thereby generating a structured composite task intent description, which is typically specified in the form of a tuple of (action type, spatial parameters, modality modifier).

[0028] S3, while performing image recognition and analysis, the onboard sensors of the drone cluster simultaneously collect environmental images and point cloud data, transmit them back to the same on-site edge computing terminal, and aggregate the data from each drone. The semantic mapping module then converts the collected data into a 3D semantic map, identifying thermal imaging hotspots, obstacles, and dangerous areas.

[0029] Specifically, while image recognition analyzes the instructions from rescue command personnel, the onboard sensors of each unit in the UAV swarm simultaneously collect ambient visible light images, lidar point clouds, and thermal imaging data. This raw data, accompanied by precise pose estimation and timestamps, is transmitted back to the on-site edge computing terminal in real time. The terminal first performs strict timestamp-based alignment on all heterogeneous data streams to ensure that all sensor observations are on the same time reference. Through a pre-calibrated sensor extrinsic parameter matrix, the observation data from different sensor coordinate systems are uniformly transformed to the global world coordinate system, achieving spatial reference normalization.

[0030] After achieving spatiotemporal synchronization, two types of image analysis are performed. For visible light images, a lightweight semantic segmentation network based on a MobileNetV2 encoder and a DeepLabV3+ decoder is used. This network extracts multi-scale features through the encoder and recovers spatial details through upsampling and skip connections in the decoder, ultimately outputting the semantic category of each pixel, such as walkable ground, building debris, and vegetation. For thermal imaging images, an adaptive threshold segmentation algorithm based on local window statistics is used to identify anomalous hotspot regions with temperatures significantly higher than the surrounding environment. A complete thermal anomaly region mask is then formed through morphological operations and connected component analysis.

[0031] The pixel-level semantic labels obtained from the above two-dimensional image analysis, along with the thermal anomaly region mask, are back-projected into three-dimensional space according to the inverse transformation relationship of the camera imaging model. This projection is then associated and registered with the corresponding LiDAR point cloud data. Through nearest neighbor search and projection association, each three-dimensional point cloud point is assigned its corresponding two-dimensional semantic label and attributes (such as whether it is a thermal anomaly point), thereby generating an enhanced three-dimensional point cloud containing rich semantic information. Each point includes geometric coordinates, semantic category, and confidence level.

[0032] The 3D space is discretized into a voxel grid, and a probability distribution regarding its semantic state is maintained for each voxel. When new enhanced 3D semantic point cloud data is input, the semantic category probability of each observed voxel is iteratively updated and fused using a Bayesian update rule based on the sensor's observation model. By continuously integrating observation data from multiple UAVs and multiple time scales, the most likely semantic labels and their probabilities for each voxel in the map are continuously optimized, ultimately outputting a 3D voxel map containing key semantic annotations such as suspected casualties (thermal imaging hotspots), structural obstacles, and fire / high-temperature hazard zones.

[0033] S4, based on the spatial intent and task modality matching association rule base parsed from image recognition, maps the image trajectory to geographical regions and UAV sub-clusters, and maps the image modality to task types and parameters. After combining with semantic map for security verification, it generates UAV control commands.

[0034] Among them, the first association rule in the association rule base is (spatial-resource mapping), which maps the parsed gesture trajectories (such as circles and polygons) to specific geographical areas and dynamically allocates appropriate drone sub-clusters based on the drone's location and battery level. The second association rule is (modal-task mapping), which maps the parsed facial expression modalities (confirmation, emergency) and gesture types (area delineation, pointing) to specific task types (full coverage scan, detailed exploration, material delivery) and task parameters (flight altitude, scan density, speed).

[0035] Specifically, based on the parsed spatial intent and task modality, combined with a real-time semantic map, executable control commands are generated through rule mapping, security verification, and instruction encoding. The rule inference engine receives a composite task intent tuple containing action type, spatial parameters, and modality modifiers. First, it applies a first association rule for space-resource mapping, converting the geometric parameters parsed from the gesture (such as center coordinates and radius) into specific geographical constraint areas. Then, based on the task requirements within the area, the real-time pose of each drone, and the remaining battery status, it dynamically calculates and selects the most suitable drone sub-cluster through an optimized resource allocation algorithm. Next, it applies a second association rule for modality-task mapping, parsing the combination of action type and facial expression modality into a specific task type (e.g., full-coverage scan) and its corresponding set of task parameters (including flight altitude, scan density, and cruise speed) through a preset condition mapping table.

[0036] The generated mission area and flight parameters are submitted to the safety verification area. This area queries a real-time 3D semantic map to perform collision detection and feasibility analysis on the mission area and preset flight path, checking for voxels semantically marked as obstacles or dangerous areas. If a conflict is detected, a local path replanning subroutine is triggered, dynamically adjusting the flight path or mission area boundaries based on semantic information to avoid risks. After verification and correction, the final target UAV sub-cluster list, mission type, safe mission area, and adjusted mission parameters are serialized and encoded according to a predefined lightweight binary communication protocol format, and encapsulated into a structured control command frame. This command frame is then reliably distributed or multicast to each UAV in the target UAV sub-cluster via the communication area of ​​the edge computing terminal, through time-division multiple access or a dedicated cluster communication link, completing the generation and issuance of commands.

[0037] S5, based on the received control commands, the drone swarm autonomously assigns roles and coordination strategies through swarm intelligence algorithms, executes tasks, and transmits status images and environmental data back to the edge terminal in real time.

[0038] Specifically, the drone sub-swarm receives and parses structured control commands from the edge computing terminal, extracting the specific task area, task type, and flight parameters. The swarm then initiates an autonomous collaborative mechanism based on a distributed swarm intelligence algorithm. Each drone exchanges its own status and location information with neighboring drones through local communication, and assigns roles and decomposes tasks according to preset collaborative strategies (such as dividing responsibility areas based on geographical proximity principles). For example, in a regional search task, each drone is assigned a dedicated search zone.

[0039] After determining their respective sub-tasks, each UAV, based on its assigned mission objective (such as covering a specific sub-area) and the flight altitude, speed, and other parameters specified in the instructions, invokes its onboard trajectory planner to generate a smooth flight trajectory from its current position to the mission start point. This planner employs a model predictive control method, using rolling optimization to calculate the optimal flight trajectory over a future period, balancing trajectory tracking accuracy with flight energy consumption.

[0040] During the execution of the planned trajectory, the UAV continuously uses onboard vision and lidar sensors to perceive its surroundings. When unexpected dynamic obstacles (such as birds or moving vehicles) or static obstacles (such as newly added trees) are detected on the planned path, the onboard processor immediately triggers a real-time obstacle avoidance algorithm. This algorithm calculates the direction of the virtual resultant force acting on the UAV based on real-time sensor data. This resultant force is a combination of an attractive force pointing towards the next waypoint and a repulsive force moving away from the obstacle. The UAV adjusts its flight speed and heading angle in real time accordingly to achieve localized obstacle avoidance.

[0041] Each drone periodically encapsulates its own status information (including 3D coordinates, flight speed, attitude angle, and remaining battery power) and compressed local key perception data (such as snapshots of detected abnormal targets and their geographic coordinates) into a status report data packet. This data packet is then relayed back to the field edge computing terminal via a self-organizing wireless communication network built within the cluster, through a multi-hop relay mechanism.

[0042] If a drone detects an anomaly during a mission, such as a battery level below a safe threshold, sensor malfunction, or prolonged entrapment, it will broadcast an alarm to other drones in the cluster and the edge computing terminal. This alarm will trigger the cluster's collaborative emergency response mechanism, potentially initiating a distributed mission renegotiation process, allowing the remaining drones to dynamically take over the mission of the faulty unit; or, after assessing the global status, the edge computing terminal may directly issue mission termination, return-to-home, or new reset commands to the relevant drones.

[0043] The technical solutions described in the embodiments of this application have at least the following technical effects or advantages: By efficiently processing continuous visual commands from command personnel at the edge and combining this with real-time environmental perception of the drone swarm, a precise understanding and rapid response to complex and dynamic rescue mission intentions are achieved. This method significantly reduces human-machine interaction latency, improves the accuracy of command parsing and environmental adaptability, and ensures efficient and safe operation of the drone swarm in complex environments through a cloud-based collaborative autonomous task allocation and execution mechanism. Ultimately, it constructs an efficient and robust human-machine collaborative intelligent rescue closed loop.

[0044] Example 2: In Example 1, the intentions of rescue commanders were analyzed through continuous visual recognition, and UAV control commands were generated based on static rule mapping and environmental semantic maps, achieving basic human-machine collaboration. However, this method has significant limitations when dealing with the high dynamism and strong interference of complex rescue scenes: its visual recognition process is relatively independent, lacking the ability to actively correct for distortions in intention expression caused by occlusion, changes in lighting, or non-standard user actions (i.e., deviations between the recognized spatial trajectory and the real physical environment), which may generate unrealistic or even dangerous commands; at the same time, its linear logic of recognition-mapping-execution and fixed task-resource mapping rules lack dynamic optimization and autonomous adaptation mechanisms based on real-time status and global efficiency when facing sudden obstacles, changes in multi-machine collaboration requirements, or task conflicts.

[0045] In some embodiments, the semantic mapping region is run to convert the collected data into a 3D semantic map. Step S3 further includes: S31, based on the preliminary intent of the continuous actions parsed in S2, introduces a semantic map as the environmental context to compensate for and calibrate the deviations caused by visual recognition, and generates high-level task instructions consistent with the environmental state.

[0046] Specifically, the system receives spatial trajectory parameters (e.g., a set of points describing the boundary of an area) identified by rescuers based on gesture image sequences, as well as facial expression command modalities identified based on facial image sequences, as determined in step S2. The calibration process is then initiated, accessing a semantic map constructed and maintained in real-time by the environmental perception module. This map is organized by voxels, with each unit labeled with a semantic category (e.g., passable ground, building obstacles, danger zone) and its corresponding confidence level.

[0047] The calibration operation extracts accessibility constraint information from the semantic map, corresponding to the regions with initially identified spatial parameters, i.e., the distribution of accessible and obstacle regions. An optimization algorithm automatically searches for a geometric transformation (e.g., translation, scaling, rotation), which is applied to the original region defined by the initially identified point set. The optimization goal is to maximize the overlap between the transformed region and the accessible regions in the semantic map, while minimizing the intersection area with obstacle regions, while preserving the basic geometric topology (e.g., closure, approximate shape) of the original region. The algorithm iteratively adjusts the transformation parameters, balancing region fit, obstacle avoidance, and shape change, until an optimal balance is found, thus outputting a set of calibrated, precise spatial coordinate parameters.

[0048] The system integrates identified facial expression modalities (such as "urgent" and "confirm") with the inherent danger level markers of the target area in the semantic map. A built-in weighted scoring and normalization function then transforms these two information sources into a quantified, dynamic task priority value between 0 and 1.

[0049] The calibrated spatial parameters, the task type inherited from the preliminary identification results, and the calculated dynamic priority values ​​are encapsulated into a structured high-level vision task instruction tuple. This instruction tuple, as the output of this step, ensures that the spatial definition of the high-level task objective is consistent with the safety and feasibility of the real physical environment, and provides clear instructions containing explicit spatial orientation and execution priority for subsequent steps.

[0050] S32, the edge computing terminal receives the calibrated high-level task instructions and parses them into serialized and structured executable subtasks according to different task types; based on the real-time status images and pose data transmitted back by the UAV cluster, it dynamically matches the UAV with the best capabilities and status for each subtask through a distributed decision algorithm, forming a collaborative task execution scheme starting from visual instructions.

[0051] Specifically, the edge computing terminal receives high-level vision task instructions calibrated by the environmental context. These instructions explicitly include the task type, a precise set of geospatial parameters, and dynamic priorities. The structured task parser within the edge computing terminal then activates. This parser invokes a pre-built task template knowledge base, matching the corresponding standardized task decomposition template based on the task type in the instruction. The template defines a series of atomic subtasks required to complete this type of task, along with the execution logic and temporal dependencies between these subtasks. The parser substitutes the specific spatial parameters from the instruction into the template, instantiating and generating a structured subtask relationship graph with a clear execution sequence and parameters.

[0052] After generating an executable subtask graph, each UAV periodically broadcasts its real-time state vector to the edge computing terminal. This vector integrates its real-time pose derived from visual odometry and inertial navigation data, remaining battery power, current mission load, and the capability set defined by its onboard sensor configuration. When a subtask is issued, it includes a declaration of its resource requirements for execution, including the expected sensor type, estimated operational range, and energy consumption.

[0053] Each participating drone calculates a comprehensive execution cost based on its real-time status and sub-task requirements. This cost evaluation function comprehensively considers multiple factors, including the drone's distance to the mission starting point, its remaining energy level, and the degree to which its sensor capabilities match the mission requirements. Each drone reports its calculated execution cost to the edge computing terminal, which selects the winning drone for each sub-task according to the criterion of minimizing cost.

[0054] This task allocation process is performed simultaneously on multiple subtasks that can be allocated in parallel. Ultimately, a complete task-drone mapping table is generated, which clearly defines the assignee of each subtask and its planned start sequence. This mapping table, together with the aforementioned structured subtask graph, constitutes a complete collaborative task execution scheme rooted in initial visual commands. This scheme is ultimately broadcast to the entire drone swarm via an edge server.

[0055] S33: Each UAV, based on its assigned sub-task, integrates real-time perception from its own onboard visual sensors with a global semantic map to perform local path planning and dynamic obstacle avoidance; at the same time, it shares key visual information and location through inter-UAV communication, runs swarm intelligence algorithms to maintain sub-task formation and avoid collisions, and autonomously completes collaborative tasks triggered by visual commands.

[0056] The technical solutions described in the embodiments of this application have at least the following technical effects or advantages: By deeply integrating real-time semantic maps into the continuous visual intent understanding process, a visual command calibration mechanism with enhanced environmental awareness was constructed, significantly improving the accuracy, security, and physical feasibility of visual intent recognition in complex environments. Furthermore, through visual command-based collaborative task parsing and dynamic resource allocation, efficient and autonomous transformation from high-level intents to multi-machine collaborative execution schemes was achieved, optimizing the overall cluster performance.

[0057] Example 3: By introducing an environmental semantic map for online calibration of visual recognition intent and constructing a collaborative task planning mechanism, the system's adaptability and task completion capabilities in dynamic environments are significantly improved. However, the core perceptual input of this method still relies on a single visible light visual channel. When faced with extreme interference conditions common in real rescue scenarios, such as severe degradation of optical images due to dense smoke, overexposure or information loss due to strong or dim light, and partial occlusion preventing the capture of key actions, the single visible light recognition process has inherent vulnerabilities. Its recognition accuracy will drop sharply, potentially leading to the failure of the entire collaborative control link, and failing to guarantee the robustness and continuity of intent understanding under any adverse conditions.

[0058] In some embodiments, step S31, which compensates for and calibrates the deviation generated by visual recognition, further includes: The S311 integrates visible light, infrared thermal imaging, and millimeter-wave radar sensors at the edge of the UAV swarm mission to perform time-synchronized data acquisition and parallel preprocess the raw data of each modality to generate a standardized multimodal visual data packet with timestamp alignment.

[0059] The multimodal vision data package contains multi-source sensing data collected and preprocessed at the same timestamp. Its core components include: a visible light RGB image matrix with standard resolution, an infrared thermal imaging data matrix that has been calibrated and segmented by temperature, a radar point cloud cluster set that characterizes human dynamics, and spatial pose information of each sensor.

[0060] S312 dynamically matches the current working mode scenario of the drone cluster based on the image quality index in the multimodal visual data packet calculated in real time.

[0061] Specifically, based on real-time acquired multimodal visual data packets, for visible light images, sharpness is quantified by calculating the variance of the response values ​​after Laplacian processing, and the signal-to-noise ratio is evaluated by calculating the logarithm of the ratio of the mean pixel value of a preset target region to the standard deviation of pixels in the background region. For thermal imaging data, the stability of the temperature gradient is measured by calculating the absolute value of the average temperature difference between the same region of interest in adjacent frames. For radar point cloud data, the richness of information is measured by statistically analyzing the number of 3D point clouds contained in the effective dynamic point cloud clusters obtained after clustering.

[0062] A lightweight scene classification model is run using auxiliary features extracted from data packets (average brightness of visible light images, overall ambient temperature of thermal imaging data, and motion state information provided by the inertial measurement unit). This model calculates the probability distribution of the current environment belonging to each preset category (e.g., normal lighting, strong backlight, dense smog, darkness, etc.) using a normalized exponential function.

[0063] The weighted scores of each modality under the current environment category probability are summed to calculate the comprehensive score of each modality at the current time. The modality with the highest comprehensive score is selected as the dominant modality for intent recognition in the current frame. The actual switching of the dominant modality is only performed when the comprehensive score of a certain modality is higher than that of the current dominant modality for several consecutive processing frames and exceeds a certain threshold range.

[0064] S313 utilizes a cross-modal fusion network to extract features from each modality and adjust the visual recognition deviation value. Based on the adjusted deviation value, the edge computing terminal calculates the intention deviation and key inflection points of rescue personnel in different working modal scenarios.

[0065] Specifically, the preprocessed and modality-selected multimodal visual data packets are input into three parallel lightweight neural network encoders for processing. Visible light images are input into an encoder optimized based on the MobileNetV3 architecture. This network extracts high-dimensional feature vectors describing the spatial location of skeletal joints and facial expression states through a series of depthwise separable convolutional and attention computation layers. Infrared thermal imaging data are input into a custom convolutional neural network with fewer layers. This network focuses on learning and outputting feature vectors representing the contour shape and stability of the human body's heat source region. Millimeter-wave radar point cloud clusters are input into a network based on the PointNet++ architecture. Processed through a multilayer perceptron and symmetric aggregation functions, it outputs feature vectors representing motion trajectories and micro-motion patterns. The feature vectors output by each encoder are linearly transformed to a unified feature dimension space through a dedicated fully connected projection layer.

[0066] The modal feature vectors, after being unified in dimension, are input into a lightweight Transformer encoding layer. This layer uses the dominant modality feature determined in step S312 as the core query, and the combined features of the remaining auxiliary modalities as the key and value. By calculating and normalizing the similarity between the query and the key, a set of attention weight distributions is generated. This weight distribution quantifies the compensation and enhancement strength of each auxiliary modality feature for the current dominant modality feature. This strength value is defined internally by the system as a visual recognition bias value, which objectively reflects the potential defects of single dominant modality recognition under current environmental interference and the degree of correction required. The weighted summed auxiliary features are added to the original dominant modality feature, and then processed by layer normalization to finally generate an enhanced feature representation that integrates multi-source information.

[0067] The fused feature sequences generated over a continuous time period are sequentially input into a bidirectional long short-term memory (LSTM) network for temporal modeling. This network, through its internal gating units, learns and memorizes long-range dependencies in the action sequence, analyzing the dynamic evolution of continuous actions. When processing the sequence, the network automatically identifies several moments when the action trend fundamentally changes; these points are defined as key inflection points (e.g., the instant a gesture trajectory abruptly changes from horizontal movement to vertical downward pressure). The final hidden state of the Bi-LSTM, containing complete contextual information of the entire action sequence, is fed into a multi-task prediction head. This prediction head simultaneously performs classification and regression tasks, outputting the predicted task type and calibrated spatial trajectory geometry parameters. The trajectory predicted by the network is compared with the trajectory initially identified based on a single dominant modality; the difference is quantified as intent bias, reflecting the improvement in recognition results before and after fusion correction. Finally, a multi-modal fusion-corrected intent instruction tuple containing a clear task type and precise spatial parameters is output.

[0068] S314: Confirm the deviation values ​​of intent deviation and key inflection points in different modalities, and adjust the advanced task instructions accordingly.

[0069] The technical solutions described in the embodiments of this application have at least the following technical effects or advantages: By introducing a multimodal data adaptive fusion and dynamic quality assessment mechanism, the robustness and accuracy of continuous visual intent recognition for rescuers in complex and harsh environments are significantly enhanced. The method intelligently selects the optimal perception modality based on the real-time environment and effectively compensates for recognition biases of a single modality through cross-modal feature fusion, thus stably reconstructing accurate user intent even under strong interference conditions. Ultimately, this method ensures highly reliable generation of UAV swarm control commands, significantly improving the adaptability of human-machine collaboration and the overall mission success rate in complex rescue scenarios.

[0070] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An artificial intelligence image recognition method based on edge computing, characterized in that, include: S1 acquires continuous hand gestures and facial image streams of rescue commanders through a multimodal camera, extracts keyframes and timestamps them at the on-site edge computing terminal, and forms a standardized time-series image data packet; where the edge computing terminal specifically refers to a mobile computing terminal located at the rescue site that is responsible for core computing and coordination; S2, based on convolutional neural networks and LSTM networks, continuously recognizes the temporal gesture images of rescue commanders and analyzes their trajectories and intentions; at the same time, based on the Transformer network, it performs expression analysis on their temporal facial images, analyzes their command modality, and outputs the composite task intent through multimodal fusion. S3, while performing image recognition and analysis, the onboard sensors of the drone cluster simultaneously collect environmental images and point cloud data, transmit them back to the same on-site edge computing terminal and aggregate the data from each drone, and run the semantic mapping module to convert the collected data into a 3D semantic map, marking thermal imaging hotspots, obstacles and dangerous areas. S4, based on the spatial intent and task modality matching association rule base parsed from image recognition, maps the image trajectory to geographical regions and UAV sub-clusters, and maps the image modality to task type and parameters. After security verification in conjunction with the semantic map, it generates UAV control commands. S5, based on the received control commands, the drone swarm autonomously assigns roles and coordination strategies through swarm intelligence algorithms, executes tasks, and transmits status images and environmental data back to the edge terminal in real time.

2. The artificial intelligence image recognition method based on edge computing as described in claim 1, characterized in that, In step S2, parsing the trajectory and intent, and parsing the command modality, specifically includes: receiving standardized temporal image data packets, and parsing the gesture image sequence and facial image sequence in parallel; for the gesture image sequence, extracting the spatial features of each frame image through a convolutional neural network to obtain a gesture feature vector sequence, and inputting this sequence into a long short-term memory network for temporal modeling to output the classification result of the gesture trajectory intent and its geometric parameters; for the facial image sequence, firstly, extracting the features of each frame image through a convolutional neural network and combining them with position encoding to form a facial feature sequence, and inputting this sequence into a Transformer encoder model for temporal dependency modeling to output the corresponding facial expression command modality; fusing the classification result of the gesture trajectory intent, the geometric parameters, and the facial expression command modality, and generating a structured composite task intent description through a multimodal fusion engine.

3. The artificial intelligence image recognition method based on edge computing as described in claim 1, characterized in that, The association rule base includes: the first association rule in the association rule base maps the parsed gesture trajectory to a specific geographical area, and dynamically allocates a suitable drone sub-cluster based on the drone's location and battery level; the second association rule maps the parsed facial expression modality and gesture type to a specific task type and task parameters.

4. The artificial intelligence image recognition method based on edge computing as described in claim 1, characterized in that, The autonomous allocation of roles and collaborative strategies through swarm intelligence algorithms specifically includes: the UAV sub-swarm receiving and parsing the control commands, extracting the task area, task type, and flight parameters; the UAV sub-swarm initiating a collaborative mechanism based on distributed swarm intelligence algorithms, exchanging status information through inter-UAV communication, and autonomously allocating roles and decomposing tasks; each UAV generating a local flight trajectory based on the assigned sub-tasks and flight parameters using an onboard trajectory planner; during flight, each UAV sensing the environment in real time based on onboard sensors and dynamically adjusting its flight path to avoid obstacles using a real-time obstacle avoidance algorithm; each UAV encapsulating its own status information and local sensing data into a status report, transmitting it back to the on-site edge computing terminal via a self-organizing communication network; and triggering a cluster collaborative emergency response mechanism when an abnormal status is detected, performing task reassignment or issuing a reset command from the edge computing terminal.

5. The artificial intelligence image recognition method based on edge computing as described in claim 1, characterized in that, The semantic mapping module converts the collected data into a 3D semantic map. Step S3 also includes: S31, based on the preliminary intent of the continuous actions parsed in S2, introduces a semantic map as the environmental context to compensate and calibrate the deviations caused by visual recognition, and generates high-level task instructions consistent with the environmental state. S32, the edge computing terminal receives the calibrated high-level task instructions and parses them into serialized and structured executable subtasks according to different task types; based on the real-time status images and pose data transmitted back by the UAV cluster, it dynamically matches the UAV with the best capabilities and status for each subtask through a distributed decision algorithm, forming a collaborative task execution scheme starting from visual instructions. S33: Each UAV, based on its assigned sub-task, integrates real-time perception from its own onboard visual sensors with a global semantic map to perform local path planning and dynamic obstacle avoidance; at the same time, it shares key visual information and location through inter-UAV communication, runs swarm intelligence algorithms to maintain sub-task formation and avoid collisions, and autonomously completes collaborative tasks triggered by visual commands.

6. The artificial intelligence image recognition method based on edge computing as described in claim 5, characterized in that, The compensation and calibration of deviations generated by visual recognition includes: receiving spatial trajectory parameters recognized based on gesture image sequences and facial expression command modalities recognized based on facial image sequences; accessing a semantic map that is constructed and maintained in real time; wherein, the units of the semantic map are labeled with semantic categories and confidence levels; extracting traversability constraint information of the region corresponding to the spatial trajectory parameters from the semantic map; performing geometric transformation on the original region defined by the spatial trajectory parameters based on the traversability constraint information through optimization processing, so that the transformed region, while maintaining the original geometric topology, coincides with the traversable region in the semantic map to the greatest extent and minimizes the intersection area with obstacle regions; and outputting calibrated spatial coordinate parameters.

7. The artificial intelligence image recognition method based on edge computing as described in claim 5, characterized in that, The formation of a collaborative task execution scheme starting with visual instructions includes: receiving high-level visual task instructions calibrated by environmental context, the instructions including task type, geospatial parameters, and dynamic priority; calling a pre-set task template knowledge base, matching the corresponding task decomposition template according to the task type, and substituting the geospatial parameters into the template to generate a structured subtask graph; acquiring real-time status information transmitted back by each UAV in the UAV cluster, including real-time pose, remaining battery power, task load, and onboard sensor capabilities; dynamically selecting the execution UAV for each subtask based on the resource requirements of each subtask in the subtask graph and the real-time status information of each UAV through a distributed decision algorithm; and generating a task-UAV mapping table based on the dynamic selection result; wherein, the mapping table and the subtask graph together constitute the collaborative task execution scheme.

8. The artificial intelligence image recognition method based on edge computing as described in claim 6, characterized in that, The compensation and calibration of deviations caused by visual recognition also includes: S311 integrates visible light, infrared thermal imaging, and millimeter-wave radar sensors at the edge of the UAV swarm mission to perform time-synchronized data acquisition and preprocess the raw data of each modality to generate a standardized multimodal visual data packet with timestamp alignment. S312 dynamically matches the current working mode scenario of the drone cluster based on the image quality index in the multimodal visual data packet calculated in real time. S313 utilizes a cross-modal fusion network to extract features from each modality and adjust the visual recognition deviation value. Based on the adjusted deviation value, the edge computing terminal calculates the intention deviation and key inflection points of rescue personnel in different working modal scenarios. S314: Confirm the deviation values ​​of intent deviation and key inflection points in different modalities, and adjust the advanced task instructions accordingly.

9. The artificial intelligence image recognition method based on edge computing as described in claim 8, characterized in that, The multimodal visual data package includes: multi-source sensing data collected and preprocessed at the same time stamp of the multimodal visual data package. Its core components include: a visible light RGB image matrix with standard resolution, an infrared thermal imaging data matrix that has been calibrated and segmented by temperature, a radar point cloud cluster set representing human dynamics, and spatial pose information of each sensor.

10. The artificial intelligence image recognition method based on edge computing as described in claim 8, characterized in that, The dynamic matching of the current working modal scenario of the drone cluster specifically includes: parallel calculation of the corresponding quality assessment index for each modality in the multimodal visual data packet; determining the probability distribution of the current environment category based on the auxiliary features extracted from the data packet through a scene classification model; calculating the comprehensive score of each modality at the current moment based on the quality assessment index and the probability distribution; and dynamically selecting the dominant modality for intent recognition based on the comprehensive score.

Images