Method and system for constructing multi-modal feature library of unmanned aerial vehicle

Abstract

This application provides a method and system for constructing a multimodal feature library for unmanned aerial vehicles (UAVs), relating to the field of data processing technology. The method includes: acquiring image sequences from dual-modal sensors above a power transmission channel; in response to an occlusion event caused by a first UAV object entering the shadow area of ​​a tower crossarm, acquiring a first continuous trajectory vector of a second UAV object traveling straight; when the first UAV object flies out of the occlusion area and becomes visible again, acquiring a first modal detection box set and a first infrared hotspot set of the first UAV object; calculating the geometric overlap probability based on the first modal detection box set and the first infrared hotspot set; generating an association cost matrix based on the geometric overlap probability and the first continuous trajectory vector; performing an instance binding operation on the first UAV object to generate a bound feature vector; and aggregating the bound feature vector into the UAV multimodal feature library. This application enables the binding of cross-modal instance relationships of re-visible targets, thereby forming multimodal features corresponding to actual UAV objects for inclusion in the library.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and system for constructing a multimodal feature library for unmanned aerial vehicles (UAVs). Background Technology

[0002] In the processes of power transmission channel inspection, external damage prevention monitoring, and low-altitude target monitoring, it is typically necessary to use sensors installed above or around the power transmission channel to continuously acquire images of unmanned aerial vehicle (UAV) targets. Based on the target's position, motion state, and image features in the image sequence, the UAV objects are detected, tracked, and their features are recorded. When multiple UAV objects exist in the monitoring scenario, it is usually also necessary to maintain the distinction between different UAV objects and to keep the features of the corresponding targets in different image frames or different perception modalities consistent with the same physical object.

[0003] In the above processing, target association typically relies on the continuous visibility state of the target in the image, changes in position between adjacent frames, and the correspondence between target responses under different modalities. However, in power transmission channel scenarios, tower crossarms, conductors, and their shadow areas may cause short-term target occlusion; especially after two small multi-rotor drones meet, if one drone enters the shadow area of ​​the tower crossarm and becomes temporarily invisible, while the other drone continues to travel straight, the re-visible drone target may simultaneously involve visible light detection results, infrared hotspot responses, and the trajectory continuation relationship after the meeting, making it difficult to directly determine the correspondence between object instances, modal detection results, and original identity identifiers. Summary of the Invention

[0004] This application provides a method and system for constructing a multimodal feature library for unmanned aerial vehicles (UAVs) to solve the problem of how to determine the correspondence between the visible light detection results, infrared hotspot results and the original object instance after the obscured UAV becomes visible again, and form binding features that can be used to construct a multimodal feature library for UAVs, when one UAV enters the shadow area of ​​the crossarm of the tower and causes short-term occlusion while the other UAV continues to travel straight above the power transmission channel.

[0005] The first aspect of this application provides a method for constructing a multimodal feature library for unmanned aerial vehicles (UAVs), including: Acquire an image sequence from a dual-mode sensor above a power transmission channel. The image sequence includes a first UAV object and a second UAV object after they intersect, and the first UAV object and the second UAV object are not equal. In response to the first UAV object entering the shadow area of ​​the pole crossarm and causing an occlusion event, the first continuous trajectory vector of the second UAV object moving straight is obtained; When the first drone object flies out of the obstruction area and becomes visible again, obtain the first modal detection box set and the first infrared hot spot set of the first drone object; The geometric overlap probability is calculated based on the first modal detection box set and the first infrared hot spot set, and an association cost matrix is ​​generated based on the geometric overlap probability and the first continuous trajectory vector. Based on the association cost matrix, an instance binding operation is performed on the first UAV object to generate a binding feature vector, and the binding feature vector is aggregated into the UAV multimodal feature library.

[0006] Optionally, in one possible implementation of the first aspect, acquiring the image sequence from the dual-modal sensor above the power transmission channel includes: The high-frequency jitter signal sequence of the gimbal of the dual-modal sensor is obtained, and a jitter displacement compensation matrix is ​​generated by performing a frequency domain integral transform on the high-frequency jitter signal sequence of the gimbal. Perform a dot product operation between the jitter displacement compensation matrix and the original pixel coordinate matrix of the image sequence to generate a stable image sequence with pixel-level alignment; The relative motion vectors of the first UAV object and the second UAV object at the moment of intersection are extracted from the stable image sequence; Based on the relative motion vector, the first UAV object and the second UAV object are assigned unequal identity indexes.

[0007] Optionally, in one possible implementation of the first aspect, generating the association cost matrix based on the geometric overlap probability and the first continuous trajectory vector includes: Extract the historical appearance feature vector of the first UAV object before the occlusion event occurs, and calculate the cosine similarity between the historical appearance feature vector and the current appearance feature vector corresponding to the first modality detection box set; When the cosine similarity is lower than a preset similarity threshold, an uncertainty suppression flag is added to the time smoothing constraint term corresponding to the first continuous trajectory vector, and the numerical feature of the time smoothing constraint term is modified to a zero vector sequence. Based on the uncertainty suppression flag, the time continuity cost component in the association cost matrix is ​​set to a zero vector, while the geometric cost component and appearance cost component are retained.

[0008] Optionally, in one possible implementation of the first aspect, the step of adding an uncertainty suppression flag to the time smoothing constraint term corresponding to the first continuous trajectory vector and modifying the numerical characteristics of the time smoothing constraint term to a zero vector sequence includes: Calculate the joint confidence score of the first UAV object based on the geometric cost component and the appearance cost component; When the joint confidence score is in the first interval, the feature data stream corresponding to the first UAV object is marked as reserved, and the feature data stream is allowed to enter the next processing node through the transmission channel; If the joint confidence score is in the second interval, the feature data stream corresponding to the first UAV object is marked as pending verification, and a pause forwarding operation is performed on the feature data stream. When the joint confidence score is in the third interval, the feature data stream corresponding to the first UAV object is marked as discarded and removed from the current processing queue.

[0009] Optionally, in one possible implementation of the first aspect, after marking the feature data stream corresponding to the first UAV object as pending verification and performing a pause forwarding operation on the feature data stream, the process includes: In response to the pending verification status, the first historical feature sequence of the first UAV object within a preset time window before the rendezvous is obtained; Obtain the second historical feature sequence of the second UAV object within a preset time window before the rendezvous; The current feature vector of the first modality detection box set is used to perform distance metric calculations with the first historical feature sequence and the second historical feature sequence respectively to generate a first distance matrix and a second distance matrix; When the trace of the first distance matrix is ​​less than the trace of the second distance matrix, it is determined that the first drone object is not associated with the trajectory of the second drone object.

[0010] Optionally, in one possible implementation of the first aspect, after determining that the first UAV object is not associated with the trajectory of the second UAV object when the trace of the first distance matrix is ​​less than the trace of the second distance matrix, the following steps are included: Perform a forward polynomial extrapolation on the location coordinates of the second UAV object that were lost during the occlusion event to generate a set of extrapolated location points; Search for the second modal detection box of the second UAV object within a preset neighborhood of the set of inferred location points; If the second modal detection box is found, the second modal detection box is re-associated and bound to the old identity of the second drone object.

[0011] Optionally, in one possible implementation of the first aspect, calculating the geometric overlap probability based on the first modal detection box set and the first infrared hotspot set includes: The coordinates of the four vertices of the first modal detection box set are extracted to form a first coordinate matrix, and the coordinates of the vertices of the smallest bounding rectangle of the first infrared hot spot set are extracted to form a second coordinate matrix. Calculate the intersection area and the union area of ​​the first coordinate matrix and the second coordinate matrix, where both the intersection area and the union area are positive real numbers; Divide the intersection area value by the union area value to generate the scalar value of the geometric overlap probability; When the scalar value is greater than the preset overlap threshold, it is determined that the first modal detection box set and the first infrared hot spot set belong to the same physical entity.

[0012] Optionally, in one possible implementation of the first aspect, the step of performing distance metric calculations on the current feature vector of the first modality detection box set with the first historical feature sequence and the second historical feature sequence respectively to generate a first distance matrix and a second distance matrix includes: The current feature vector is mapped to a preset dimension space to generate a current high-dimensional vector, and the first historical feature sequence is mapped to a preset dimension space to generate a first historical high-dimensional matrix; Calculate the dot product between the current high-dimensional vector and the transpose of the first historical high-dimensional matrix to generate the first distance matrix; Obtain the minimum element value in the first distance matrix, and perform a comparison operation between the minimum element value and a preset distance threshold to generate a reverse verification result.

[0013] Optionally, in one possible implementation of the first aspect, after marking the feature data stream corresponding to the first UAV object as discarded and removing the feature data stream from the current processing queue when the joint confidence score is in the third interval, the process includes: Extract the error fusion feature vector of the first UAV object marked as abandoned; An interception and blocking operation is performed on the erroneous fusion feature vector to prevent it from being written into the storage array of the UAV multimodal feature library; The erroneous fusion feature vector is transmitted to the active learning data pool for isolated storage.

[0014] Optionally, in one possible implementation of the first aspect, after performing the re-association and binding operation between the second modal detection box and the old identity of the second UAV object if the second modal detection box is found, the process includes: Extract the pure multimodal feature vector of the second UAV object, wherein the pure multimodal feature vector includes visible light channel features and infrared channel features; Perform a channel concatenation operation on the visible light channel features and the infrared channel features to generate a cross-modal joint feature vector; The cross-modal joint feature vector is written into the UAV multimodal feature library and stored as independent prototype data.

[0015] Optionally, in one possible implementation of the first aspect, the step of performing an instance binding operation on the first UAV object based on the association cost matrix to generate a binding feature vector, and aggregating the binding feature vector into the UAV multimodal feature library, includes: Extract the modality candidate pair corresponding to the minimum cost element from the correlation cost matrix. The modality candidate pair includes the target box in the first modality detection box set and the target hot spot in the first infrared hot spot set. The visible light pixel features of the target bounding box and the infrared radiation features of the target hot spot are concatenated to generate the binding feature vector; In response to the generation event of the bound feature vector, the comparison distance between the bound feature vector and the existing prototype feature vector in the UAV multimodal feature library is calculated; If the comparison distance is greater than the preset entry threshold, the bound feature vector is written into the UAV multimodal feature library as a new prototype.

[0016] A second aspect of this application provides a system for constructing a multimodal feature library for unmanned aerial vehicles (UAVs), comprising: The acquisition module is used to acquire image sequences from dual-mode sensors above the power transmission channel. The image sequences include a first UAV object and a second UAV object after intersection, and the first UAV object and the second UAV object are not equal. The response module, in response to the occlusion event caused by the first UAV object entering the shadow area of ​​the pole crossarm, obtains the first continuous trajectory vector of the second UAV object moving straight. The detection module is used to acquire the first modal detection box set and the first infrared hot spot set of the first drone object when the first drone object flies out of the occlusion area and becomes visible again; The calculation module calculates the geometric overlap probability based on the first modal detection box set and the first infrared hot spot set, and generates an association cost matrix based on the geometric overlap probability and the first continuous trajectory vector. The aggregation module is used to perform an instance binding operation on the first UAV object according to the association cost matrix to generate a binding feature vector, and aggregate the binding feature vector into the UAV multimodal feature library.

[0017] The method and system for constructing a multimodal feature library for unmanned aerial vehicles (UAVs) provided in this application have the following beneficial effects: This application acquires image sequences from dual-modal sensors above a power transmission channel. When a first UAV object enters the shadow area of ​​a tower crossarm and causes an occlusion event, it combines the first continuous trajectory vector formed by the second UAV object after intersecting and moving straight, as well as the first modal detection box set and the first infrared hotspot set obtained after the first UAV object becomes visible again. It then calculates the geometric overlap probability and generates an association cost matrix, and performs instance binding operations on the first UAV object based on the association cost matrix. Therefore, it can incorporate the cross-modal detection results, thermal radiation response, and motion relationship after intersection of the re-emerging target after occlusion into the same association judgment process. This allows the re-emerging first UAV object to form a bound feature vector with its corresponding modal features, thus providing a basis for writing feature data corresponding to the actual UAV object into the multimodal feature library and reducing the risk of mixed inclusion of features from different UAV objects due to intersection occlusion. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the method for constructing a multimodal feature library for unmanned aerial vehicles provided in this application embodiment; Figure 2 This is an application scenario diagram of the UAV multimodal feature library construction method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the UAV multimodal feature library construction system provided in the embodiments of this application; Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0020] The technical solutions of this application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0021] See Figure 1 This is a flowchart of the method for constructing a multimodal feature library for unmanned aerial vehicles provided in this application embodiment. Figure 1The execution entity of the method shown can be a software and / or hardware device. The execution entity of this application can include, but is not limited to, at least one of the following: user equipment, network equipment, etc. User equipment can include, but is not limited to, computers, smartphones, personal digital assistants (PDAs), and the aforementioned electronic devices. Network equipment can include, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of computers or network servers. Cloud computing is a type of distributed computing, consisting of a super virtual computer composed of a group of loosely coupled computers. This embodiment does not impose any limitations on this.

[0022] In scenarios where two drones meet above a power transmission channel, and one drone briefly becomes obscured by entering the shadow of a tower crossarm while the other continues moving straight, conventional methods of target association based on a single image location or trajectory continuity can easily lead to confusion between the newly visible drone and the drone continuing its forward movement. The drone multimodal feature library construction method provided in this application can be applied to scenarios such as... Figure 2 In the scenario shown, there is a first drone object 102 and a second drone object 104 above the power transmission channel. After the first drone object 102 passes, it enters the shadow area 106 of the tower crossarm, while the second drone object 104 continues to move straight. The dual-modal sensor 108 is used to collect visible light and infrared images containing the two drone objects.

[0023] The dual-modal sensor 108 transmits the acquired dual-modal image sequence to the image processing device 110. When the first UAV object 102 becomes visible again, the image processing device 110 acquires the corresponding set of visible light detection boxes and infrared hotspots, and combines this with the continuous trajectory formed by the straight movement of the second UAV object 104 to generate an association cost matrix for cross-modal instance binding. The image processing device 110 generates a binding feature vector for the first UAV object 102 based on the association cost matrix, and writes the binding feature vector into the UAV multimodal feature library 112, ensuring that the cross-modal features after occlusion recovery correspond to the actual UAV object.

[0024] Step S1: Obtain the image sequence of the dual-mode sensor above the power transmission channel. The image sequence includes the first UAV object and the second UAV object after intersection. The first UAV object and the second UAV object are not equal.

[0025] It should be noted that during low-altitude monitoring of power transmission channels, conventional processing can maintain UAV target tracking based on the proximity relationships and inter-frame motion relationships in continuous images. When the target remains visible and there are no close encounters in the scene, this processing can form a basic trajectory. However, near the crossarm of the tower, two small multi-rotor UAVs may encounter each other above the power transmission channel. The first UAV may briefly become invisible after entering the shadow area of ​​the crossarm, while the second UAV continues to move straight. At this time, the visible light target bounding box, infrared hotspot, and motion trajectory are likely to approach each other in the local space. To avoid subsequent confusion between the reappearing first UAV and the second UAV, this step first acquires a continuous image sequence from the dual-modal sensor and maintains an unequal relationship between the two UAVs in the image sequence.

[0026] Understandably, the dual-modal sensor includes a visible light imaging channel and an infrared imaging channel. The visible light imaging channel is used to acquire the drone's shape, outline, texture, and boundaries, while the infrared imaging channel is used to acquire the thermal radiation response generated by the drone's motors, battery, or fuselage. The image sequence is formed according to the acquisition time, and each frame can include a visible light image, an infrared image, a frame number, an acquisition time, gimbal attitude status, and image coordinate information. Visible light images and infrared images can be paired based on the same trigger signal or timestamp. Frames that cannot form a valid pair can be marked as modally asynchronous frames and will not be included in subsequent geometric overlap calculations.

[0027] The first drone object refers to a small multi-rotor drone that flies into the shadow area of ​​a pole crossarm and is briefly obscured, while the second drone object refers to a small multi-rotor drone that continues to travel straight after the initial encounter. The two are not equal because they are maintained separately in physical entity, identity index, historical feature cache, and trajectory cache. Independent data records are established for each to ensure that the candidate features of the first drone object after occlusion recovery do not directly use the straight-line trajectory of the second drone object.

[0028] In some embodiments, step S1 includes steps S11 to S14.

[0029] Step S11: Obtain the high-frequency jitter signal sequence of the gimbal from the dual-modal sensor, and perform frequency domain integral transformation on the high-frequency jitter signal sequence to generate a jitter displacement compensation matrix. The jitter displacement compensation matrix has a dimension of two rows and two columns.

[0030] Specifically, the high-frequency jitter signal sequence of the gimbal can be provided by the gimbal attitude sensor, gyroscope, accelerometer, or gimbal control feedback unit. This signal sequence is read based on the image frame timestamp and is used to record minute perturbations in the sensor's line of sight in the horizontal and vertical directions. The input to the frequency domain integral transform is the jitter signal within the corresponding time period of the current image frame, and the output is the horizontal compensation amount, vertical compensation amount, and cross-compensation amount that can be applied to the image plane coordinates. The generated two-row, two-column jitter displacement compensation matrix can record the horizontal coordinate correction relationship, the vertical coordinate correction relationship, and the cross-direction correction relationship, respectively. If the jitter signal of a certain frame is missing or exceeds the effective range of the sensor, the compensation matrix of the adjacent effective frame is used, or the frame is marked as an abnormal frame that does not participate in the generation of the stable image sequence.

[0031] It should be noted that step S11 converts the gimbal jitter signal into a pixel compensation matrix, so that subsequent image coordinate correction has a clear input and compensation object, thereby reducing the impact of gimbal jitter on the drone's motion judgment.

[0032] Step S12: Perform a dot product operation between the jitter displacement compensation matrix and the original pixel coordinate matrix of the image sequence to generate a stable image sequence with pixel-level alignment.

[0033] Specifically, the original pixel coordinate matrix can record the horizontal and vertical coordinates at pixel points or the center point of the target region. After applying the jitter displacement compensation matrix to the original pixel coordinate matrix, the compensated horizontal and vertical coordinates are obtained, and a stable image frame is reconstructed based on the compensated coordinates. To ensure that visible light and infrared images can be used together, the infrared image coordinates can be transformed to the visible light image coordinate system according to the dual-modal sensor calibration parameters or pre-established pixel mapping relationships, or the coordinates of both modes can be transformed to a unified stable coordinate system. If the registration error exceeds the allowable level in the configuration file, the cross-modal candidate relationship of the current frame enters the pending verification state, and the binding feature vector is not directly generated.

[0034] It should be noted that step S12 compensates and unifies the original pixel coordinates, so that the fixed background such as tower crossarms, conductors and insulators remain relatively stable in continuous frames, and enables the visible light detection frame and infrared hot spot to be judged in subsequent overlap under the same coordinate reference.

[0035] Step S13: Extract the relative motion vectors of the first UAV object and the second UAV object at the moment of intersection based on the stable image sequence.

[0036] Specifically, the moment of intersection can be determined based on the change in the distance between the centers of the two UAV detection frames and the overlap of their circumscribed regions in consecutive frames. Starting from a valid frame where the two targets begin to approach each other, if the distance between the centers of the two targets decreases frame by frame for several consecutive frames and then begins to increase in subsequent frames, the time period with the smallest distance is determined as the moment of intersection. If the circumscribed regions of the two targets overlap in the same frame, the time period from the start of the overlap to the completion of separation can also be determined as the moment of intersection. If both conditions occur simultaneously, the earlier-triggered intersection state is used as the starting point for recording. The relative motion vector can be formed by the changes in position, direction of motion, and direction of separation of the two UAVs before and after the intersection, used to express the relative proximity, offset, and separation relationships of the two UAV objects.

[0037] It should be noted that step S13, by limiting the recognition conditions at the moment of intersection, makes the extraction of relative motion vectors have a clear time range, thus avoiding misjudging ordinary proximity or detection box jitter as intersection.

[0038] Step S14: Assign unequal identity indexes to the first UAV object and the second UAV object based on the relative motion vector.

[0039] Specifically, the identification index is used to record object affiliation and can be associated with image frames, trajectory cache, candidate detection boxes, infrared hotspots, and feature vectors. The separation direction of two objects after intersection is determined based on their relative motion vectors. A first identification index is assigned to the UAV object entering the shadow area of ​​the tower crossarm, and a second identification index is assigned to the UAV object continuing straight. If the detection boxes briefly overlap during intersection, the historical trajectory and relative motion relationship before intersection are retained, and identities are not merged due to single-frame overlap. If the separation direction cannot be confirmed, both objects can be marked as pending verification, and the data entry operation is paused until a stable trajectory is provided in a subsequent frame.

[0040] It should be noted that step S14 separates the two physical entities through the identity index, so that subsequent occlusion events, straight trajectories, detection box sets, and infrared hotspot sets can all enter the processing chain according to object affiliation.

[0041] Preferably, step S1 involves acquiring bimodal images, compensating for gimbal jitter, unifying bimodal coordinates, identifying intersection moments, and assigning different identity indexes to form basic data with temporal continuity, coordinate consistency, and object independence. This provides an implementable data entry point for subsequent occlusion response, trajectory reference, cross-modal association, and feature storage.

[0042] Step S2: In response to the occlusion event caused by the first UAV object entering the shadow area of ​​the tower crossarm, obtain the first continuous trajectory vector of the second UAV object moving straight.

[0043] It should be noted that when the first UAV object flies into the shadow area of ​​the tower crossarm, the visible light detection frame may be segmented by the crossarm edge or disappear briefly, and the infrared hotspot may be obscured by the tower structure or mixed with the background thermal response. In this case, if only the positional continuity between adjacent frames is considered, the straight-line trajectory of the second UAV object may be mistakenly identified as the recovered trajectory of the first UAV object. Therefore, after identifying the occlusion event of the first UAV object, this step separately obtains the first continuous straight-line trajectory vector of the second UAV object as a reference for subsequent determination of whether the first UAV object has mistakenly connected to the trajectory of the second UAV object.

[0044] Specifically, the shadow area of ​​the tower crossarm can be pre-defined as a polygonal or rectangular region in a stable image coordinate system, or it can be updated based on the fixed background area of ​​the tower crossarm, the illumination direction, and historical image statistics. The triggering conditions for an occlusion event can include: the center or main area of ​​the first UAV object detection frame entering the shadow area of ​​the tower crossarm; the first UAV object failing to form a complete visible light detection frame in consecutive valid frames; interruption of the infrared hotspot response or truncation of the occluded area; and overlap between the outline of the first UAV object and the edge of the crossarm. These conditions can be combined for judgment. An occlusion event is determined when the detection frame enters the shadow area and exhibits continuous loss or hotspot interruption; if it only enters the shadow area but the target remains continuously clear, it is first marked as an occlusion risk state; if the target disappears but does not enter the shadow area, it is marked as a normal loss state and the occlusion recovery link in this step is not initiated.

[0045] The first continuous trajectory vector is a sequence of continuous motion position coordinates of the second UAV object after it intersects with the first UAV object. Sampling can begin from the first valid detection frame of the second UAV object after the intersection and continue until the first UAV object becomes visible again, the configured frame number is reached, or the second UAV object is temporarily lost. The first continuous trajectory vector can include frame number, timestamp, center coordinates, detection box size, direction of motion, identification index, and validity marker. If the detection result of a frame is invalid, that point is skipped and a missing marker is recorded; if consecutive missing frames exceed the configured threshold, the generation of the first continuous trajectory vector is stopped to avoid using unreliable trajectories.

[0046] The straight-line trend can be determined by the change in direction of continuous position points. If the second UAV object maintains the same direction setting in consecutive valid frames and does not enter the shadow area of ​​the tower crossarm, its trajectory is recorded as a straight-line trajectory. If the direction change exceeds the preset direction setting or there is a significant reversal, the trajectory is marked as a non-straight-line state and is prohibited from participating in the time smoothing constraint as the first continuous trajectory vector.

[0047] Preferably, step S2 clarifies the occlusion event triggering conditions, trajectory sampling start and end points, and the first continuous trajectory vector field, so that the straight trajectory of the second UAV object can be used as a time reference in the subsequent cost matrix, and avoids the first UAV object being incorrectly connected to the trajectory of the second UAV object when the occlusion is restored.

[0048] Step S3: When the first UAV object flies out of the occlusion area and becomes visible again, obtain the first modal detection box set and the first infrared hot spot set of the first UAV object.

[0049] It should be noted that when the first UAV object flies out of the shadow area of ​​the pole crossarm, the image may simultaneously contain the UAV body boundary, the edge of the crossarm structure, the background of the conductor, and other heat source responses. If a single visible light detection box or a single infrared hotspot is directly selected, it is easy to mistake the background edge or the second UAV object for the first UAV object. Therefore, in this step, when the first UAV object becomes visible again, it is not directly uniquely bound, but rather the first modal detection box set and the first infrared hotspot set are obtained, so that subsequent unified filtering can be performed through geometric overlap probability and correlation cost matrix.

[0050] Specifically, the first modal detection box set can be derived from visible light target detection results. Candidate boxes entering this set should meet the following conditions: matching the re-visibility time window, the exit direction of the occlusion area, and the identity state of the first UAV object. Candidate boxes can record their boundary position, center position, width, height, detection state, frame number, and candidate number. If the visible light channel only detects a local contour in the current frame, the candidate box is marked as partially visible and awaits supplementation in adjacent frames. If a candidate box is significantly inconsistent with the exit direction of the occlusion area, it is marked as a low-priority candidate or excluded.

[0051] The first set of infrared hotspots can be derived from the thermal response region extraction results in infrared images. First, thermal response regions are extracted from the infrared frames corresponding to the re-visible time window. Then, candidate hotspots are selected based on the hotspot's bounding rectangle, center, thermal response range, and its relationship with the exit position of the occlusion area. Hotspots entering the set should have their outline, minimum bounding rectangle, center, response state, frame number, and candidate number recorded. If the hotspot area is significantly smaller than the effective target level or significantly larger than the background heat source level, it can be marked as an abnormal hotspot and will not directly participate in the selection of effective candidate pairs. If the infrared hotspot set is empty, the first UAV object remains in a modally incomplete state, and cross-modal binding feature vectors are not generated temporarily; if the first modal detection box set is empty, geometric overlap probability calculation is not performed.

[0052] In this embodiment, both the first modal detection box set and the first infrared hotspot set can contain multiple candidates. Instead of directly determining a unique target in this step, the candidate number and frame number are retained, allowing for the subsequent combination of candidate boxes and candidate hotspots one by one, and the state of each candidate pair is recorded in the association cost matrix.

[0053] Preferably, step S3 makes the cross-modal input after the first UAV object becomes visible again clearer by defining the source of candidate boxes and candidate hotspots, entry set conditions, anomaly marking and empty set processing, providing an implementable data foundation for subsequent geometric overlap and cost matrix calculation.

[0054] Step S4: Calculate the geometric overlap probability based on the first modal detection box set and the first infrared hot spot set, and generate the association cost matrix based on the geometric overlap probability and the first continuous trajectory vector.

[0055] It should be noted that after the first UAV object becomes visible again, multiple candidate correspondences may form between the first modal detection box set and the first infrared hotspot set. Relying solely on visible light detection confidence may overlook the infrared thermal response, and relying solely on hotspots may lead to the misselection of background heat sources. To ensure that different candidate pairs can be compared uniformly, this step first calculates the geometric overlap probability between visible light candidate boxes and infrared candidate hotspots, and then combines this with the first continuous trajectory vector to form an association cost matrix. The rows of this matrix can correspond to candidate boxes in the first modal detection box set, and the columns can correspond to candidate hotspots in the first infrared hotspot set. Each element corresponds to a candidate pair consisting of a candidate box and a candidate hotspot. Each element can record geometric cost components, appearance cost components, temporal continuity cost components, and candidate state fields.

[0056] In some embodiments, step S4 includes steps S41 to S47.

[0057] Step S41: Extract the coordinates of the four vertices of the first modal detection box set to form a first coordinate matrix, and extract the coordinates of the vertices of the smallest bounding rectangle of the first infrared hot spot set to form a second coordinate matrix; Specifically, for each candidate box in the first modal detection box set, the coordinates of the four vertices are extracted in the order of top left, top right, bottom right, and bottom left to form a first coordinate matrix. For each candidate hot spot in the first infrared hot spot set, the minimum bounding rectangle is first determined based on the thermal response region, and then the coordinates are extracted in the same vertex order to form a second coordinate matrix. Before extracting the second coordinate matrix, the coordinates of the bounding rectangle of the infrared hot spot are transformed to a unified stable coordinate system according to the dual-modal sensor calibration parameters or pixel mapping relationship. If a hot spot cannot complete the coordinate transformation, the corresponding candidate pair is marked as having invalid coordinates.

[0058] It should be noted that step S41, by unifying the vertex order and coordinate reference, enables the visible light candidate boxes and infrared hot spots to have comparable geometric representations.

[0059] Step S42: Calculate the intersection area and union area of ​​the first coordinate matrix and the second coordinate matrix. Both the intersection area and the union area are positive real numbers.

[0060] Specifically, in a unified and stable coordinate system, the regions covered by the first and second coordinate matrices are compared to obtain the intersection area of ​​their overlapping regions and the union area corresponding to their common coverage. If the candidate box area or the area of ​​the hotspot's circumscribed rectangle is invalid, the corresponding candidate pair is marked as having an area anomaly and is not included in the effective overlap probability calculation. If two regions do not overlap, the intersection area can be recorded as the minimum effective level, and the candidate pair will subsequently be in a state of insufficient spatial overlap.

[0061] It should be noted that step S42 calculates the intersection area and union area of ​​the valid candidate regions, enabling cross-modal spatial relationships to be transformed into comparable area data.

[0062] Step S43: Divide the intersection area value by the union area value to generate a scalar value for the geometric overlap probability.

[0063] Specifically, the geometric overlap probability is used to represent the degree of spatial overlap between the visible light candidate box and the outer region of the infrared hotspot in the same candidate pair. A scalar value is generated for each valid candidate pair, and this value is written into the geometric field of the corresponding element in the association cost matrix. A higher scalar value indicates that the two modal candidate regions are closer in a unified coordinate system; a lower scalar value indicates that they may originate from different objects or background interference. If a candidate pair has been marked as having invalid coordinates or an abnormal area, this field is recorded as invalid and it does not participate in the minimum cost selection.

[0064] It should be noted that step S43, by writing the spatial overlap relationship into the candidate pair matrix elements, gives the subsequent geometric cost components a clear source.

[0065] Step S44: When the scalar value is greater than the preset overlap threshold, it is determined that the first modal detection box set and the first infrared hot spot set belong to the same physical entity.

[0066] Specifically, the preset overlap threshold can be pre-set based on the installation deviation between the visible light channel and the infrared channel, the difference in field of view, the image registration error, the size of the UAV target, and the monitoring distance of the power transmission channel. In a preferred embodiment, the preset overlap threshold can be 0.35 to 0.55, preferably 0.45; when the geometric overlap probability is greater than 0.45, it is determined that the corresponding visible light candidate box and the infrared hot spot candidate region have the same physical entity candidate relationship.

[0067] When the scalar value of the geometric overlap probability is greater than the preset overlap threshold, it is determined that the corresponding candidate box and candidate hotspot have the same physical entity candidate relationship, and the corresponding geometric cost component is reduced; when the scalar value is equal to the threshold, it can enter the pending verification state according to the conservative strategy; when the scalar value is less than the threshold, the candidate pair is marked as having insufficient spatial overlap, and the corresponding geometric cost component is increased or the candidate pair is excluded.

[0068] It should be noted that step S44 enables the geometric overlap judgment to be specifically executed by clarifying the setting basis, level and boundary processing method of the preset overlap threshold.

[0069] Step S45: Extract the historical appearance feature vector of the first UAV object before the occlusion event occurs, and calculate the cosine similarity between the historical appearance feature vector and the current appearance feature vector corresponding to the first modal detection box set.

[0070] Specifically, the historical appearance feature vector can be extracted from consecutive valid frames before the first UAV object enters the occlusion area, and includes at least the body contour features, rotor area features, visible light grayscale distribution features, and texture features. The feature components are arranged in a fixed order and normalized to form the historical appearance feature vector before occlusion. If multiple valid historical frames exist, they can be arranged in chronological order to form a historical appearance feature sequence, or stable frames or average representative features can be selected as the historical appearance feature vector.

[0071] The current appearance feature vector can be extracted from the image regions corresponding to each candidate box in the first modality detection box set. The order of feature components and the normalization method are consistent with the historical appearance feature vectors. After calculating the cosine similarity, the similarity result is written into the appearance field of the corresponding element of the association cost matrix. If the current candidate box is partially visible, the appearance field can be marked as incomplete, and its priority for direct entry into the database can be reduced.

[0072] It should be noted that step S45 provides an executable computational basis for the appearance cost component by clarifying the composition, source, and arrangement of historical and current appearance features.

[0073] Step S46: When the cosine similarity is lower than the preset similarity threshold, add an uncertainty suppression flag to the time smoothing constraint term corresponding to the first continuous trajectory vector, and modify the numerical feature of the time smoothing constraint term to a zero vector sequence.

[0074] Specifically, a preset similarity threshold is used to determine whether the current appearance features are consistent with the historical appearance features before occlusion. In a preferred embodiment, the preset similarity threshold can be between 0.60 and 0.80, preferably 0.70; when the cosine similarity is lower than 0.70, an uncertainty suppression flag is added to the time smoothing constraint term corresponding to the first continuous trajectory vector.

[0075] The temporal smoothing constraint term can be formed by the motion direction, adjacent position changes, and inter-frame continuous states in the first continuous trajectory vector, used to describe the temporal reference relationship between the second UAV object's straight trajectory and the current candidate. Since the first continuous trajectory vector originates from the second UAV object, continuing to use this temporal reference when the current appearance is inconsistent with the historical appearance of the first UAV object may lead to the first UAV object being incorrectly connected to the second UAV object's trajectory. Therefore, an uncertainty suppression flag is written into the candidate pair state field, and the temporal continuity component corresponding to the candidate pair is set to an invalid zero vector sequence. The length of the zero vector sequence can be consistent with the number of trajectory sampling points or the matrix element recording format corresponding to the candidate pair, used to indicate that this component does not participate in the current candidate pair cost selection.

[0076] It should be noted that step S46 ensures that unreliable temporal continuity will not dominate the binding of the first UAV object by clearly defining the preset similarity threshold, the composition of the time smoothing constraint, and the suppression of identifier writing.

[0077] In some embodiments, after adding an uncertainty suppression flag to the time smoothing constraint term corresponding to the first continuous trajectory vector and modifying the numerical characteristics of the time smoothing constraint term to a zero vector sequence in step S46, steps S461 to S464 are included.

[0078] Step S461: Calculate the joint confidence score of the first UAV object based on the geometric cost component and the appearance cost component.

[0079] Specifically, the input to the joint confidence score includes a geometric cost component and an appearance cost component. The geometric cost component reflects whether the visible light candidate box and the infrared hotspot satisfy the same physical entity candidate relationship, and the appearance cost component reflects whether the current candidate box continues the appearance before the first UAV object occluded it. The system can use a tiered combination rule to form the joint confidence score and can normalize the joint confidence score to between zero and one. In a preferred embodiment, the first interval can be greater than or equal to 0.75, the second interval can be greater than or equal to 0.45 and less than 0.75, and the third interval can be less than 0.45. When the geometric overlap reaches the same entity candidate condition and the appearance similarity reaches the similarity condition, the joint confidence score enters the first interval, and the feature data stream is marked as reserved; when only one of the two conditions is met, or both are near the boundary, the joint confidence score enters the second interval, and the feature data stream is marked as pending verification; when neither condition is met, the joint confidence score enters the third interval, and the feature data stream is marked as discarded. If a candidate bounding box or hotspot has invalid coordinates, abnormal area, or partially visible markers, the confidence level can be lowered by one level or the system can directly enter a pending verification state. In other embodiments, the boundaries of the first, second, and third intervals can be adjusted based on the monitoring equipment resolution, historical operating data, and power transmission channel scenario configuration.

[0080] It should be noted that step S461 forms a joint confidence score by combining different levels, so that geometric and appearance criteria can jointly determine the direction of candidate data flow.

[0081] Step S462: When the joint confidence score is in the first interval, mark the feature data stream corresponding to the first UAV object as reserved and allow the feature data stream to enter the next processing node through the transmission channel.

[0082] Specifically, the first interval represents a high-confidence candidate relationship. Candidate pairs entering this interval must at least meet the following conditions: valid coordinates, valid area, geometric overlap meeting a threshold condition, appearance similarity meeting a threshold condition, and not be marked as discarded or prohibited from writing. The feature data stream can include candidate box features, hotspot features, geometric overlap results, appearance similarity results, identity index, acquisition time, candidate pair number, and status field. After being marked as reserved, the data stream can enter subsequent instance binding nodes.

[0083] It should be noted that step S462 enables high-confidence candidates to continue participating in binding by limiting the retention status conditions, while keeping the status field traceable.

[0084] Step S463: If the joint confidence score is in the second interval, mark the feature data stream corresponding to the first UAV object as pending verification and perform a pause forwarding operation on the feature data stream; Specifically, the second interval represents the uncertainty of candidate relationships. Data streams entering this interval can be those that meet geometric conditions but have unstable appearances, or those that are similar in appearance but whose geometric overlap is affected by registration errors or occlusion edges. This data stream is stored in a temporary cache, and forwarding to the instance-bound node is paused. Boundary values ​​can be uniformly classified into the second interval for more conservative processing. In the pending verification state, the data stream retains candidate boxes, hotspots, current feature vectors, state fields, and joint confidence levels for subsequent historical feature comparisons.

[0085] It should be noted that step S463, by pausing the forwarding of the confidence data stream, prevents uncertain candidates from directly entering the database entry link before their attribution is confirmed.

[0086] In some embodiments, after marking the feature data stream corresponding to the first UAV object as pending verification and performing a pause forwarding operation on the feature data stream in step S463, steps S4631 to S4634 are included.

[0087] Step S4631: In response to the pending verification state, obtain the first historical feature sequence of the first UAV object within a preset time window before the rendezvous; Specifically, a preset time window is used to cover a segment of image where the first UAV object remains continuously visible before the encounter. This time window can be set to three levels: short window, medium window, and long window. Short windows can be used for close-range, high-frame-rate devices, while medium or long windows can be used for long-range or low-frame-rate devices. In a preferred embodiment, the preset time window can be 1 to 3 seconds before the first UAV object encounters, preferably 2 seconds; when the sensor frame rate is low or the UAV's movement speed is slow, the time window can be appropriately extended. The system reads the appearance features, infrared hotspot features, position features, and movement direction within this time window from the historical cache corresponding to the first UAV object's identity index, forming a first historical feature sequence in chronological order. If there are insufficient valid historical frames, the data stream to be verified remains paused or enters the manual review queue.

[0088] It should be noted that step S4631 enables the candidate to be checked to be compared with the state of the first UAV object before occlusion by specifying the source of the preset time window and the historical sequence field.

[0089] Step S4632: Obtain the second historical feature sequence of the second UAV object within a preset time window before the rendezvous.

[0090] Specifically, appearance features, infrared hotspot features, location features, and direction of movement are read from the historical cache corresponding to the second UAV object's identity index within the same preset time window to form a second historical feature sequence. The first and second historical feature sequences use the same field order and normalization caliber to ensure that subsequent distance measurements are comparable. If the second historical feature sequence is missing, only the first historical feature sequence is verified, but the second UAV object is not excluded based on the missing second sequence.

[0091] It should be noted that step S4632 enables the identification of whether the current candidate is closer to the object that continues to move straight by establishing a second drone object history reference.

[0092] Step S4633: Perform distance metric calculations between the current feature vector of the first modality detection box set and the first historical feature sequence and the second historical feature sequence, respectively, to generate a first distance matrix and a second distance matrix.

[0093] Specifically, the current feature vector originates from the candidate bounding box to be verified and may include appearance components, positional components, and necessary modal state components. The current feature vector is then compared with the first and second historical feature sequences to generate a first distance matrix and a second distance matrix. The first distance matrix represents the difference between the current candidate and the historical features of the first UAV object, and the second distance matrix represents the difference between the current candidate and the historical features of the second UAV object. Rows in the matrix can correspond to the current candidate to be verified, and columns can correspond to historical features arranged in chronological order. If trace extraction is required, diagonal comparison elements corresponding one-to-one with the current candidate and historical representative features can be selected from the matrix to form a traceable comparison matrix. If the number of candidates and the number of historical representative features are inconsistent, representative features can be selected in chronological order or invalid placeholders can be added, and invalid placeholders can be excluded from the trace comparison.

[0094] It should be noted that step S4633 clarifies the meaning of the matrix rows and columns and the traceable processing method, enabling the distance matrix to support subsequent trajectory misassociation judgment.

[0095] In some embodiments, step S4633 includes steps A1 to A3.

[0096] Step A1: Map the current feature vector to a preset dimension space to generate the current high-dimensional vector, and map the first historical feature sequence to a preset dimension space to generate the first historical high-dimensional matrix; Specifically, the preset dimension space can be consistent with the dimension of the prototype feature vector in the UAV multimodal feature library, or it can be uniformly set by the system configuration file. In a preferred embodiment, the preset dimension space can be 128-dimensional, 256-dimensional, or 512-dimensional, preferably 256-dimensional. During mapping, the current feature vector and the first historical feature sequence are arranged in a fixed order of appearance component, infrared component, position component, and state component, and mapped to the same preset dimension space; missing components can use invalid placeholders and write missing markers, and after normalization processing, the current high-dimensional vector and the first historical high-dimensional matrix are generated. In other embodiments, the dimension of the preset dimension space can be adjusted according to the prototype dimension of the feature library, the sensor acquisition accuracy, and the processor computing power.

[0097] It should be noted that step A1, by unifying the dimensions and feature order, enables the current features to be compared with historical features under a consistent data caliber.

[0098] Step A2: Calculate the dot product of the current high-dimensional vector and the transpose of the first historical high-dimensional matrix to generate the first distance matrix.

[0099] Specifically, the dot product is used to obtain the proximity relationship between the current high-dimensional vector and each historical feature in the first historical high-dimensional matrix. The dot product result is converted to a distance representation and written into the first distance matrix, ensuring that the elements in the matrix uniformly represent the degree of difference. If a historical feature has a missing label, the corresponding matrix element is marked as invalid and not included in the minimum element value and trace comparison.

[0100] It should be noted that step A2 converts the dot product result into a distance matrix, enabling the relationship between the current candidate and the historical state of the first UAV object to be recorded in a structured manner.

[0101] Step A3: Obtain the minimum element value in the first distance matrix, and perform a comparison operation between the minimum element value and the preset distance threshold to generate a reverse verification result.

[0102] Specifically, a preset distance threshold is used to determine whether the current candidate may still belong to the first UAV object. Before performing the comparison, the elements in the first distance matrix can be normalized to between zero and one, so that the smaller the distance value, the closer the current candidate is to the historical features of the first UAV object. The preset distance threshold can be set to three levels: strict, medium, and lenient, and consistent with the feature library prototype dimension. In a preferred embodiment, the preset distance threshold can be 0.25 to 0.45, preferably 0.35. If the minimum element value in the first distance matrix is ​​less than 0.35, a passed reverse verification result is generated; if the minimum element value is equal to 0.35, it is processed as pending verification; if the minimum element value is greater than 0.35, a failed reverse verification result is generated, and the second distance matrix is ​​used for further judgment. In other embodiments, the preset distance threshold can be adjusted according to the feature normalization method, UAV target size, and sensor resolution, but it can be fixed in one processing task and updated synchronously when the feature library dimension is adjusted.

[0103] Preferably, step S4633 unifies the feature dimensions, generates a distance matrix, and clarifies the meaning of the matrix elements, so that the current candidate can be compared with the historical features of the first UAV object and the second UAV object respectively, providing a basis for lifting the pending verification status.

[0104] Step S4634: When the trace of the first distance matrix is ​​less than the trace of the second distance matrix, it is determined that the first UAV object is not associated with the trajectory of the second UAV object.

[0105] Specifically, the trace of the first distance matrix can be obtained by summing the diagonal comparison elements between the current candidate and the first historical representative feature, and the trace of the second distance matrix can be obtained by summing the diagonal comparison elements between the current candidate and the second historical representative feature. A smaller trace indicates that the overall difference between the current candidate and the corresponding historical object is smaller. When the trace of the first distance matrix is ​​smaller than the trace of the second distance matrix, it is determined that the current candidate is closer to the historical state of the first UAV object, and the first UAV object is not associated with the trajectory of the second UAV object. If the two traces are equal or the difference is at a critical level, the status remains pending verification, and no binding feature vector is generated; if the trace of the second distance matrix is ​​smaller, the trajectory recovery search of the second UAV object is triggered.

[0106] It should be noted that step S4634 enables the data stream to be checked to determine whether a second UAV object trajectory misassociation has occurred by clarifying the comparison object, critical processing and output action of the trajectory.

[0107] In some embodiments, step S4634 is followed by steps B1 to B3.

[0108] Step B1: Perform a forward polynomial extrapolation on the position coordinates of the second UAV object that were lost during the occlusion event to generate a set of extrapolated position points.

[0109] Specifically, the forward polynomial extrapolation uses the most recent consecutive valid position points of the second UAV object as input, and can preferentially use low-order polynomials to extrapolate the positions of several subsequent frames. Position points participating in the extrapolation should have a valid frame number, timestamp, and center coordinates. If there are insufficient valid position points, or if the position point orientation change no longer satisfies the straight-line state, the extrapolation stops, and the second UAV object is marked as temporarily lost. The set of extrapolated position points is only used to limit the subsequent search range and is not directly added to the database as actual detection boxes.

[0110] It should be noted that step B1, by specifying the inference input, order selection, and failure handling, makes the prediction of the second UAV object location feasible.

[0111] Step B2: Search for the second modal detection box of the second UAV object within the preset neighborhood of the set of inferred location points.

[0112] Specifically, the preset neighborhood range can be a rectangular or circular area centered on the inferred location point, or it can be formed by expanding outwards according to the size of the most recent effective detection frame of the second UAV object. In a preferred embodiment, the preset neighborhood range can be an area formed by expanding outwards from the inferred location point in both the horizontal and vertical directions by 1.5 to 3 times the width and height of the most recent effective detection frame of the second UAV object, preferably by 2 times. If multiple candidate detection frames exist within the neighborhood, the system prioritizes selecting the detection frame that matches the historical appearance, infrared hotspot, and direction of movement of the second UAV object; if no detection frame is found, it remains in a temporarily lost state and waits for subsequent frames to continue searching. In other embodiments, the expansion factor can be adjusted according to the UAV flight speed, sensor frame rate, and image resolution.

[0113] It should be noted that step B2 enables the second modality detection box search to be performed by specifying the shape, level, and multi-candidate selection rules of the preset neighborhood range.

[0114] Step B3: If a second modal detection box is found, re-associate and bind the second modal detection box with the old identity of the second UAV object.

[0115] Specifically, the old identity identifier is the identity identifier index assigned to the second UAV object before the rendezvous. If a second modal detection box is found, the frame number, coordinates, appearance features, infrared features, and search source of the detection box are written under the old identity identifier of the second UAV object, restoring its independent trajectory. If multiple similar candidates exist in the search results and cannot be distinguished, re-association and binding are not performed; instead, the temporarily lost state is retained.

[0116] It should be noted that step B3 restores the correspondence between the second modal detection box and the old identity identifier, so that the features of the second drone object will not be mixed into the data to be bound to the first drone object.

[0117] In some embodiments, step B3 is followed by steps B31 to B33.

[0118] Step B31: Extract the pure multimodal feature vector of the second UAV object. The pure multimodal feature vector includes visible light channel features and infrared channel features.

[0119] Specifically, visible light channel features are extracted from the visible light region corresponding to the re-associated second modality detection box, and infrared channel features are extracted from the corresponding infrared region. This feature vector can be organized with the visible light component first and the infrared component last, and the old identifier and timestamp are recorded as auxiliary fields. These auxiliary fields do not participate in distance comparison; they are only used for indexing and verification.

[0120] It should be noted that step B31 extracts the multimodal features of the re-associated objects, enabling the second UAV object to form an independent feature representation.

[0121] Step B32: Perform a channel concatenation operation on the visible light channel features and the infrared channel features to generate a cross-modal joint feature vector.

[0122] Specifically, before channel concatenation, the visible light channel features and infrared channel features are normalized and length-aligned, and then concatenated in a preset order to form a cross-modal joint feature vector. The preset order can be fixed in the feature library configuration file, for example, recording visible light features first, then infrared features, and finally adding auxiliary fields.

[0123] It should be noted that step B32, by unifying the channel cascading order, enables the cross-modal joint feature vector to have a stable structure, which facilitates subsequent storage and retrieval.

[0124] Step B33: Write the cross-modal joint feature vector into the UAV multimodal feature library and perform storage operations as independent prototype data.

[0125] Specifically, when writing the cross-modal joint feature vector, the old identity of the second UAV object, its re-association status, acquisition time, and detection box source are saved simultaneously. This independent prototype data does not participate in the database entry judgment of the first UAV object's bound features.

[0126] It should be noted that step B33 maintains the feature boundaries between the continuing straight-moving object and the occlusion recovery object by independently storing the cross-modal joint features of the second UAV object.

[0127] Preferably, step B3 restores the old identity of the second UAV object and stores its cross-modal features independently, so that the straight-moving object after the intersection will not be mistakenly incorporated into the feature chain of the first UAV object.

[0128] Preferably, step S4634 enables the candidate to be checked to exclude erroneous associations with the second UAV object by comparing the historical matrix and restoring the trajectory of the second UAV object.

[0129] Preferably, step S463 prevents Zhongzhixin candidates from entering the formal binding process before confirming the ownership of the object by pausing forwarding, comparing historical features, and investigating erroneous trajectory associations.

[0130] Step S464: When the joint confidence score is in the third interval, mark the feature data stream corresponding to the first UAV object as discarded and remove the feature data stream from the current processing queue.

[0131] Specifically, the third interval represents low-confidence candidate relationships. If geometric overlap does not meet the same entity candidate condition, appearance similarity does not meet the similarity condition, or if the candidate pair has invalid coordinates, abnormal area, or cannot be corrected, the corresponding feature data stream is marked as discarded. The current processing queue may include an associated candidate queue, an instance binding pre-caching queue, and a pre-entry temporary storage queue. After removal, the candidate pair will no longer participate in the minimum cost element selection, nor will it generate a binding feature vector.

[0132] It should be noted that step S464 removes obviously unreliable candidates from the formal binding link by discarding low-confidence data streams.

[0133] In some embodiments, step S464 is followed by steps S4641 to S4643.

[0134] Step S4641: Extract the error fusion feature vector of the first UAV object marked as discarded.

[0135] Specifically, the erroneously fused feature vector can originate from miscombined visible light candidate boxes and infrared hotspots, or from candidate data disturbed by background edges, second UAV objects, or abnormal thermal responses. When extracting this vector, the frame number, candidate number, occlusion event number, and reason for discarding are recorded simultaneously.

[0136] It should be noted that step S4641, by clarifying the error fusion feature vector, makes the subsequent interception and blocking have a specific target.

[0137] Step S4642: Perform an interception and blocking operation on the erroneously fused feature vector to prevent the erroneously fused feature vector from being written to the storage array of the UAV multimodal feature library.

[0138] Specifically, blocking operations can be achieved by using a write-prohibition flag, canceling the ingestion task, or locking the storage interface. During the ingestion verification phase, this write-prohibition flag is read, and the corresponding feature vector is prevented from entering the actual storage array.

[0139] It should be noted that step S4642 prevents the feature library from receiving mixed features generated by occlusion crossover by blocking the writing of erroneous fusion feature vectors.

[0140] Step S4643: Transfer the erroneous fusion feature vector to the active learning data pool for isolated storage operation.

[0141] Specifically, in this embodiment, the active learning data pool is used for isolated storage and offline verification, and is not a necessary part of the model training process in the current method. The isolated storage data may include erroneously fused feature vectors, candidate sources, occlusion states, reasons for discarding, and write-prohibited results. This data is not used in the current formal feature library retrieval.

[0142] It should be noted that step S4643 separates abnormal data from formal data by isolating and saving abnormal samples, and retains the basis for subsequent analysis.

[0143] Preferably, step S464 prevents unreliable features from entering the formal data entry process for the first UAV object by discarding, blocking, and isolating low-confidence candidates.

[0144] Preferably, step S46 enables candidate features in the occlusion recovery stage to enter the retention, verification, or discard path according to their confidence level by suppressing unreliable temporal continuity, dividing confidence intervals, and setting branch processing.

[0145] Step S47: Based on the uncertainty suppression identifier, set the time continuity cost component in the association cost matrix to a zero vector, and retain the geometric cost component and the appearance cost component.

[0146] Specifically, each valid element of the association cost matrix corresponds to a candidate bounding box and a candidate hotspot. Each element records the geometric cost component, appearance cost component, temporal continuity cost component, and a state field. The geometric cost component is derived from the geometric overlap probability, the appearance cost component from the appearance similarity before and after occlusion, and the temporal continuity cost component from the first continuous trajectory vector. If an uncertainty suppression flag exists in the state field of a candidate pair, only the temporal continuity cost component within that element is marked as an invalid zero vector; the geometric cost component and appearance cost component are retained. If the flag does not exist, the temporal continuity cost component can continue to participate in cost comparison.

[0147] Invalid candidate pairs, invalid coordinate candidate pairs, area anomaly candidate pairs, and discarded candidate pairs can be recorded as unselectable states in the matrix. Candidate pairs to be verified only participate in the minimum cost element extraction in step S5 after historical feature verification is completed and they are restored to a valid state. After the association cost matrix is ​​updated, it is output to step S5.

[0148] It should be noted that step S47, by specifying the composition of matrix elements, the method of setting components to zero, and the candidate states, enables the association cost matrix to serve as a direct input for subsequent instance binding.

[0149] Preferably, step S4 transforms multiple cross-modal candidate relationships after the first UAV object becomes visible again into a filterable association cost matrix through geometric overlap, appearance comparison, trajectory suppression, joint confidence branching, and matrix state management, providing sufficient basis for generating binding feature vectors.

[0150] Step S5: Perform instance binding operation on the first UAV object according to the association cost matrix to generate binding feature vector, and aggregate the binding feature vector into the UAV multimodal feature library.

[0151] It should be noted that after the association cost matrix is ​​generated, the candidate relationships in the matrix need to be transformed into cross-modal feature representations that can be written into the feature library. If all candidate boxes or all hotspots are directly written, it is easy to write the background, the second UAV object, or low-confidence candidates into the feature library; if only a single modality is saved, it cannot represent the multimodal state after the first UAV object becomes visible again. Therefore, this step performs instance binding based on the association cost matrix, generates bound feature vectors, and compares them with existing prototypes before adding them to the library.

[0152] In some embodiments, step S5 includes steps S51 to S54.

[0153] Step S51: Extract the modal candidate pair corresponding to the minimum cost element from the correlation cost matrix. The modal candidate pair includes the target box in the first modal detection box set and the target hot spot in the first infrared hot spot set.

[0154] Specifically, a valid element refers to a matrix element that is in a reserved state or has been restored to a valid state after verification, and has not been marked as having invalid coordinates, an abnormal area, insufficient spatial overlap, being discarded, or prohibited from being written to. The valid elements in the association cost matrix are traversed, and the element with the lowest cost is selected based on the retained geometric cost component, appearance cost component, and available temporal continuity cost component within the element. If multiple elements have the same cost or their differences are at a critical level, candidate pairs with a joint confidence level in the first interval and that have not triggered uncertainty suppression are prioritized; if they still cannot be distinguished, they remain in a pending verification state, and no binding feature vector is generated. If no valid element exists, the binding process for the current frame ends, and data is awaited from subsequent frames.

[0155] It should be noted that step S51 provides a clear basis for the selection of target boxes and target hotspots by limiting valid elements and parallel processing rules.

[0156] Step S52: Perform channel concatenation of the visible light pixel features of the target bounding box and the infrared radiation features of the target hot spot to generate a bound feature vector.

[0157] Specifically, the visible light pixel features of the target bounding box can include the aircraft outline, rotor region, texture distribution, and visible light grayscale features. The infrared radiation features of the target hotspot can include the hotspot outline, thermal response range, hotspot center, and thermal response intensity features. Before channel concatenation, the two types of features are normalized and length-aligned, and then combined into a bound feature vector according to the preset order in the configuration file. The bound feature vector can first record the visible light pixel features, then the infrared radiation features, and finally record the first UAV object identification index, acquisition time, candidate pair number, and association cost source as auxiliary fields. The auxiliary fields can be used for indexing and verification, but do not participate in the comparison distance calculation.

[0158] It should be noted that step S52 ensures that the bound feature vector has a stable structure by clarifying the channel cascading objects, order, and the purpose of auxiliary fields.

[0159] Step S53: In response to the generation event of the bound feature vector, calculate the comparison distance between the bound feature vector and the existing prototype feature vector in the UAV multimodal feature library.

[0160] Specifically, after the bound feature vector is generated, a pre-entry comparison process is triggered. During comparison, only the visible light pixel features and infrared radiation features in the bound feature vector can be used; auxiliary fields such as identification, timestamp, and candidate number are not involved in the distance calculation. If multiple existing prototypes exist in the feature library, the comparison distance between the bound feature vector and each existing prototype is calculated separately, and the distance corresponding to the closest prototype is used as the current comparison distance. If the feature library is empty, the currently bound feature vector is directly used as the first candidate prototype for entry into the database.

[0161] It should be noted that step S53 makes the pre-entry comparison have executable rules by limiting the comparison fields and the distance acquisition method when there are multiple prototypes.

[0162] Step S54: If the comparison distance is greater than the preset entry threshold, the bound feature vector is written into the UAV multimodal feature library as a new prototype.

[0163] Specifically, a preset entry threshold is used to determine whether there is sufficient difference between the bound feature vector and the existing prototype. During the pre-entry comparison, the comparison distance between the bound feature vector and the existing prototype feature vector can be normalized to between 0 and 1, so that a larger comparison distance indicates a greater difference. The preset entry threshold can be set to three levels: conservative, standard, and extended, depending on the prototype dimension of the feature library, the monitoring task type, and the sensor configuration. In a preferred embodiment, the preset entry threshold can be between 0.40 and 0.65, preferably 0.50. If the comparison distance is greater than 0.50, the system writes the bound feature vector as a new prototype into the UAV multimodal feature library and saves the target bounding box, target hotspot, geometric overlap probability, source of the association cost matrix, occlusion event status, and acquisition time. If the comparison distance is equal to or less than 0.50, the system does not add a new prototype but can update the occurrence time, observation count, or auxiliary records of the existing prototype. In other embodiments, the preset entry threshold can be adjusted according to the feature library capacity, prototype update frequency and sensor acquisition accuracy; in a single monitoring task, the threshold can be read from the feature library configuration file and can be updated synchronously when the feature vector dimension or channel cascading order changes.

[0164] It should be noted that step S54, by specifying the preset entry threshold, boundary processing, and writing fields, ensures that only cross-modal features that have completed instance binding and have added value enter the UAV multimodal feature library.

[0165] Preferably, step S5 selects effective modal candidate pairs from the association cost matrix, generates structurally stable binding feature vectors, performs pre-database comparison, and determines the new prototype based on a threshold. This ensures that the visible light and infrared features formed by the first UAV object after flying out of the pole crossarm shadow area can be written into the feature database as cross-modal features of the same physical entity. Thus, the newly added data in the feature database maintains a correspondence with the actual UAV objects and reduces the possibility of mixed features of different objects being entered into the database after two UAVs intersect and occlude.

[0166] See Figure 3 This is a schematic diagram of the structure of the UAV multimodal feature library construction system provided in this application embodiment, including: The acquisition module is used to acquire image sequences from dual-mode sensors above the power transmission channel. The image sequences include the first UAV object and the second UAV object after intersection, and the first UAV object and the second UAV object are not equal. The response module, in response to the occlusion event caused by the first UAV object entering the shadow area of ​​the tower crossarm, obtains the first continuous trajectory vector of the second UAV object moving straight. The detection module is used to acquire the first modal detection box set and the first infrared hotspot set of the first UAV object when the first UAV object flies out of the occlusion area and becomes visible again; The calculation module calculates the geometric overlap probability based on the first modal detection box set and the first infrared hot spot set, and generates an association cost matrix based on the geometric overlap probability and the first continuous trajectory vector. The aggregation module is used to perform instance binding operations on the first UAV object based on the association cost matrix to generate binding feature vectors, and then aggregate the binding feature vectors into the UAV multimodal feature library.

[0167] Figure 3 The system of the illustrated embodiment can be used to perform corresponding operations. Figure 1 The steps in the method embodiments shown are implemented in a similar manner and have similar technical effects, and will not be repeated here.

[0168] See Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device 40 includes: a processor 41, a memory 42, and a computer program; wherein, The memory 42 is used to store computer programs, and the memory may also be flash memory. Computer programs may be, for example, application programs or functional modules that implement the methods described above.

[0169] The processor 41 is used to execute the computer program stored in the memory to implement the various steps performed by the device in the above method. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0170] Alternatively, the memory 42 can be either standalone or integrated with the processor 41.

[0171] When the memory 42 is a device independent of the processor 41, the device may also include: Bus 43 is used to connect memory 42 and processor 41.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for constructing a multi-modal feature library of a UAV, characterized in that, include: Acquire an image sequence from a dual-mode sensor above a power transmission channel. The image sequence includes a first UAV object and a second UAV object after they intersect, and the first UAV object and the second UAV object are not equal. In response to the first UAV object entering the shadow area of ​​the pole crossarm and causing an occlusion event, the first continuous trajectory vector of the second UAV object moving straight is obtained; When the first drone object flies out of the obstruction area and becomes visible again, obtain the first modal detection box set and the first infrared hot spot set of the first drone object; The geometric overlap probability is calculated based on the first modal detection box set and the first infrared hot spot set, and an association cost matrix is ​​generated based on the geometric overlap probability and the first continuous trajectory vector. Based on the association cost matrix, an instance binding operation is performed on the first UAV object to generate a binding feature vector, and the binding feature vector is aggregated into the UAV multimodal feature library.

2. The method for constructing a multimodal feature library for unmanned aerial vehicles according to claim 1, characterized in that, The acquisition of image sequences from dual-modal sensors above the power transmission channel includes: The high-frequency jitter signal sequence of the gimbal of the dual-modal sensor is obtained, and a jitter displacement compensation matrix is ​​generated by performing a frequency domain integral transform on the high-frequency jitter signal sequence of the gimbal. Perform a dot product operation between the jitter displacement compensation matrix and the original pixel coordinate matrix of the image sequence to generate a stable image sequence with pixel-level alignment; The relative motion vectors of the first UAV object and the second UAV object at the moment of intersection are extracted from the stable image sequence; Based on the relative motion vector, the first UAV object and the second UAV object are assigned unequal identity indexes.

3. The method for constructing a multimodal feature library for unmanned aerial vehicles according to claim 1, characterized in that, The step of generating the association cost matrix based on the geometric overlap probability and the first continuous trajectory vector includes: Extract the historical appearance feature vector of the first UAV object before the occlusion event occurs, and calculate the cosine similarity between the historical appearance feature vector and the current appearance feature vector corresponding to the first modality detection box set; When the cosine similarity is lower than a preset similarity threshold, an uncertainty suppression flag is added to the time smoothing constraint term corresponding to the first continuous trajectory vector, and the numerical feature of the time smoothing constraint term is modified to a zero vector sequence. Based on the uncertainty suppression flag, the time continuity cost component in the association cost matrix is ​​set to a zero vector, while the geometric cost component and appearance cost component are retained.

4. The method for constructing a multimodal feature library for unmanned aerial vehicles according to claim 3, characterized in that, The step of adding an uncertainty suppression flag to the time smoothing constraint term corresponding to the first continuous trajectory vector and modifying the numerical characteristics of the time smoothing constraint term to a zero vector sequence includes: Calculate the joint confidence score of the first UAV object based on the geometric cost component and the appearance cost component; When the joint confidence score is in the first interval, the feature data stream corresponding to the first UAV object is marked as reserved, and the feature data stream is allowed to enter the next processing node through the transmission channel; If the joint confidence score is in the second interval, the feature data stream corresponding to the first UAV object is marked as pending verification, and a pause forwarding operation is performed on the feature data stream. When the joint confidence score is in the third interval, the feature data stream corresponding to the first UAV object is marked as discarded and removed from the current processing queue.

5. The method for constructing a multimodal feature library for unmanned aerial vehicles according to claim 4, characterized in that, The step of marking the feature data stream corresponding to the first UAV object as pending verification and performing a pause forwarding operation on the feature data stream includes: In response to the pending verification status, the first historical feature sequence of the first UAV object within a preset time window before the rendezvous is obtained; Obtain the second historical feature sequence of the second UAV object within a preset time window before the rendezvous; The current feature vector of the first modality detection box set is used to perform distance metric calculations with the first historical feature sequence and the second historical feature sequence respectively to generate a first distance matrix and a second distance matrix; When the trace of the first distance matrix is ​​less than the trace of the second distance matrix, it is determined that the first drone object is not associated with the trajectory of the second drone object.

6. The method for constructing a multimodal feature library for unmanned aerial vehicles according to claim 5, characterized in that, When the trace of the first distance matrix is ​​less than the trace of the second distance matrix, after determining that the first UAV object is not associated with the trajectory of the second UAV object, the following steps are included: Perform a forward polynomial extrapolation on the location coordinates of the second UAV object that were lost during the occlusion event to generate a set of extrapolated location points; Search for the second modal detection box of the second UAV object within a preset neighborhood of the set of inferred location points; If the second modal detection box is found, the second modal detection box is re-associated and bound to the old identity of the second drone object.

7. The method for constructing a multimodal feature library for unmanned aerial vehicles according to claim 1, characterized in that, The calculation of the geometric overlap probability based on the first modal detection box set and the first infrared hotspot set includes: The coordinates of the four vertices of the first modal detection box set are extracted to form a first coordinate matrix, and the coordinates of the vertices of the smallest bounding rectangle of the first infrared hot spot set are extracted to form a second coordinate matrix. Calculate the intersection area and the union area of ​​the first coordinate matrix and the second coordinate matrix, where both the intersection area and the union area are positive real numbers; Divide the intersection area value by the union area value to generate the scalar value of the geometric overlap probability; When the scalar value is greater than the preset overlap threshold, it is determined that the first modal detection box set and the first infrared hot spot set belong to the same physical entity.

8. The method for constructing a multimodal feature library for unmanned aerial vehicles according to claim 5, characterized in that, The step of performing distance metric calculations on the current feature vector of the first modality detection box set with the first historical feature sequence and the second historical feature sequence respectively to generate a first distance matrix and a second distance matrix includes: The current feature vector is mapped to a preset dimension space to generate a current high-dimensional vector, and the first historical feature sequence is mapped to a preset dimension space to generate a first historical high-dimensional matrix; Calculate the dot product between the current high-dimensional vector and the transpose of the first historical high-dimensional matrix to generate the first distance matrix; Obtain the minimum element value in the first distance matrix, and perform a comparison operation between the minimum element value and a preset distance threshold to generate a reverse verification result.

9. The method for constructing a multimodal feature library for unmanned aerial vehicles according to claim 4, characterized in that, When the joint confidence score is in the third interval, marking the feature data stream corresponding to the first UAV object as discarded and removing the feature data stream from the current processing queue includes: Extract the error fusion feature vector of the first UAV object marked as abandoned; An interception and blocking operation is performed on the erroneous fusion feature vector to prevent it from being written into the storage array of the UAV multimodal feature library; The erroneous fusion feature vector is transmitted to the active learning data pool for isolated storage.

10. The method for constructing a multimodal feature library for unmanned aerial vehicles according to claim 6, characterized in that, If the second modal detection box is found, the step of re-associating and binding the second modal detection box with the old identity of the second drone object includes: Extract the pure multimodal feature vector of the second UAV object, wherein the pure multimodal feature vector includes visible light channel features and infrared channel features; Perform a channel concatenation operation on the visible light channel features and the infrared channel features to generate a cross-modal joint feature vector; The cross-modal joint feature vector is written into the UAV multimodal feature library and stored as independent prototype data.

11. The method for constructing a multimodal feature library for unmanned aerial vehicles according to claim 1, characterized in that, The step of performing an instance binding operation on the first UAV object based on the association cost matrix to generate a binding feature vector, and aggregating the binding feature vector into the UAV multimodal feature library, includes: Extract the modality candidate pair corresponding to the minimum cost element from the correlation cost matrix. The modality candidate pair includes the target box in the first modality detection box set and the target hot spot in the first infrared hot spot set. The visible light pixel features of the target bounding box and the infrared radiation features of the target hot spot are concatenated to generate the binding feature vector; In response to the generation event of the bound feature vector, the comparison distance between the bound feature vector and the existing prototype feature vector in the UAV multimodal feature library is calculated; If the comparison distance is greater than the preset entry threshold, the bound feature vector is written into the UAV multimodal feature library as a new prototype.

12. A UAV multimodal feature library construction system, employing the UAV multimodal feature library construction method as described in any one of claims 1 to 11, characterized in that, The acquisition module is used to acquire image sequences from dual-mode sensors above the power transmission channel. The image sequences include a first UAV object and a second UAV object after intersection, and the first UAV object and the second UAV object are not equal. The response module, in response to the occlusion event caused by the first UAV object entering the shadow area of ​​the pole crossarm, obtains the first continuous trajectory vector of the second UAV object moving straight. The detection module is used to acquire the first modal detection box set and the first infrared hot spot set of the first drone object when the first drone object flies out of the occlusion area and becomes visible again; The calculation module calculates the geometric overlap probability based on the first modal detection box set and the first infrared hot spot set, and generates an association cost matrix based on the geometric overlap probability and the first continuous trajectory vector. The aggregation module is used to perform an instance binding operation on the first UAV object according to the association cost matrix to generate a binding feature vector, and aggregate the binding feature vector into the UAV multimodal feature library.

Images