Unmanned aerial vehicle assembly detection method and system based on video stream

Abstract

The application discloses a method and system for detecting unmanned aerial vehicle assembly based on video stream. Real-time video stream containing unmanned aerial vehicle structure is acquired and asymmetric landmark features are identified to establish absolute orientation coordinate system and divide assembly area into non-overlapping quadrant areas; assembly operation points in the video stream are detected, when falling into target quadrant area, symmetric assembly components in the area are endowed with unique identity ID bound with the quadrant and continuously locked; when detecting assembly action for the symmetric assembly components, assembly event is generated combining with the identity ID and corresponding process nodes in preset dynamic directed acyclic graph are matched to update the state thereof; when the visual axis rotates, the coordinate system is reconstructed according to the asymmetric landmark features and the mapping relationship is corrected to intercept repeated triggering events; when the whole machine assembly is completed, if there is an unfinished node in the graph, a blocking prompt is output. The application can effectively decouple the symmetric structure of the unmanned aerial vehicle and realize accurate and leak-proof detection under disordered processes.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) assembly and inspection technology, specifically to a UAV assembly and inspection method and system based on video streams. Background Technology

[0002] With the intelligent transformation of the drone industry, online inspection technology based on computer vision and video streams has been introduced into assembly lines to identify assembly actions in real time and verify the integrity of processes.

[0003] However, the various arms and accessories of multi-rotor drones are visually highly similar and exhibit extremely high symmetry. In actual assembly, because the camera lens's line of sight frequently switches, tilts, or rotates, traditional algorithms, lacking global absolute constraints, are prone to visual misalignment after the line of sight changes. This can lead to misjudging a completed arm as a different arm in a symmetrical orientation, resulting in repeated counting and misjudgments of the process.

[0004] Furthermore, most existing assembly inspection systems are based on linear state machine models, which strictly limit the assembly sequence of workers. However, in actual production, workers often perform overlapping, parallel, or skip-assembly processes when dealing with multiple highly similar symmetrical machine arms. Existing linear algorithms are prone to failure when faced with such disordered and nonlinear process jumps, and cannot achieve dynamic and adaptive assembly logic tracking, thus leading to serious potential for missed inspections.

[0005] Therefore, how to achieve accurate prevention of missed inspections under axis transformation and disordered processes is a technical problem that urgently needs to be solved. Summary of the Invention

[0006] To address at least one of the technical deficiencies mentioned in the background art, the present invention aims to provide a method and system for unmanned aerial vehicle (UAV) assembly inspection based on video streams.

[0007] A first aspect of the present invention provides a method for unmanned aerial vehicle (UAV) assembly inspection based on video stream, the method comprising: Acquire real-time video streams containing drone structures and identify asymmetric landmark features within them to establish an absolute orientation coordinate system in the image, thereby dividing the entire assembly area into several non-overlapping quadrant regions. The assembly operation point in the video stream is detected. When it falls into the target quadrant region, the symmetrical assembly component in the target quadrant region is assigned a unique identity ID bound to that quadrant, and the identity ID is continuously locked using a time-series pixel tracker. When an assembly action is detected for the symmetrical assembly component, the assembly action is combined with the currently locked identity ID to generate an assembly event, and the assembly event is used to match the corresponding process node in the preset dynamic directed acyclic graph, and the process node is updated to the completed state. When a rotational transformation of the viewing axis is detected, the absolute orientation coordinate system is reconstructed based on the asymmetric landmark features to correct the mapping relationship between the identity ID and the quadrant in order to intercept repeated triggering events; when the assembly completion signal is received, if there are nodes in an incomplete state in the dynamic directed acyclic graph, a blocking prompt is output.

[0008] A second aspect of the present invention provides a video stream-based unmanned aerial vehicle (UAV) assembly inspection system, the system comprising: The region segmentation module is used to acquire real-time video streams containing the drone structure and identify asymmetric landmark features in them, thereby establishing an absolute orientation coordinate system in the image and dividing the entire assembly area into several non-overlapping quadrant regions. The identity locking module is used to detect assembly operation points in the video stream. When the point falls into the target quadrant region, the module assigns a unique identity ID to the symmetrical assembly parts in the target quadrant region that are bound to that quadrant, and uses a time-series pixel tracker to continuously lock the identity ID. The event mapping module is used to generate an assembly event by combining the assembly action with the currently locked identity ID when an assembly action for the symmetrical assembly component is detected, and to use the assembly event to match the corresponding process node in the preset dynamic directed acyclic graph and update the process node to the completed state. The dynamic verification module is used to reconstruct the absolute orientation coordinate system based on the asymmetric landmark features when a rotational transformation of the view axis is detected, thereby correcting the mapping relationship between the identity ID and the quadrant to intercept repeated triggering events; when the assembly completion signal is received, if there are nodes in the dynamic directed acyclic graph that are not in a completed state, a blocking prompt is output.

[0009] Compared with existing technologies, this invention establishes an absolute orientation coordinate system and divides quadrant regions by using asymmetric landmark features. Combined with identity ID binding triggered by assembly operation points and a time-series pixel tracker, it achieves continuous identity locking of symmetrical components under complex occlusion and viewing angle changes. At the same time, it uses a dynamic directed acyclic graph (DAG) to perform topological deconstruction of the assembly process, allowing parallel processes to be executed in any order while retaining only rigid front and rear constraints, thus completely eliminating the limitation of linear sequence. When the viewing axis rotates, it uses self-occluding edge residuals and 3D CAD digital twin models to perform multi-view projection comparison, reconstructs the complete geometric reference axis and corrects the coordinate system, effectively intercepts repeated triggering events caused by viewing angle misalignment, and outputs blocking prompts for unfinished nodes when the whole machine is finished.

[0010] Therefore, under actual production conditions of frequent changes in the line of sight and disordered processes, the present invention simultaneously solves the technical problems of repeated misjudgments and missed inspections in processes, thereby significantly improving the robustness and accuracy of UAV assembly inspection. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating a video stream-based drone assembly inspection method disclosed in an embodiment of the present invention. Figure 2 This is a schematic diagram of the network structure of the temporal pixel tracker disclosed in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a video stream-based unmanned aerial vehicle (UAV) assembly and inspection system disclosed in an embodiment of the present invention; Figure 4 This is another structural schematic diagram of a video stream-based drone assembly and inspection system disclosed in an embodiment of the present invention. Detailed Implementation

[0012] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0013] This invention provides a video stream-based method for drone assembly inspection. This embodiment uses the inspection of a quadcopter industrial drone assembly line as an example for detailed explanation. The four arms of this drone, along with its associated ESC module and motors, exhibit a high degree of central symmetry in physical appearance.

[0014] Reference Figure 1 As shown, the method includes the following steps: S1. Acquire a real-time video stream containing the drone structure and identify asymmetric landmark features in it, thereby establishing an absolute orientation coordinate system in the image and dividing the entire assembly area into several non-overlapping quadrant regions. After the drone enters the assembly station, an industrial camera located above the station captures a real-time video stream containing the overall structure of the drone's fuselage. Because the quadcopter drone has a highly symmetrical appearance, to prevent line-of-sight confusion caused by the rotation of the robotic arm or camera, a global spatial dimension division is first performed. Specifically, unique asymmetrical landmark features are automatically identified in the first few frames of the real-time video stream, and these features are used to establish an absolute orientation coordinate system in image space that is bound to the drone's physical structure, thereby dividing the entire assembly area into multiple non-overlapping quadrant regions.

[0015] For detailed implementation procedures, please refer to steps S11-S12 below: S11, Identify asymmetric landmark features in the real-time video stream formed by the head gimbal slot or the tail battery clip, and extract the relative geometric orientation between the asymmetric landmark features; The asymmetric landmark features are specifically the unique physical edge of the gimbal slot at the nose and the asymmetric battery clip at the rear of the fuselage. First, target detection is performed on each frame of the real-time video stream to identify the two physically unique features on the entire device: the gimbal slot and the battery clip.

[0016] Subsequently, the geometric center points of these two features on the image plane are calculated, and their relative geometric orientation is extracted, such as the vector from the battery clip to the gimbal slot. It should be understood that this relative geometric orientation does not change with the translation of the drone on the workstation, but is only affected by the rotation of the camera's line of sight, and therefore can serve as a stable reference for the subsequent establishment of the coordinate system.

[0017] S12, using the relative geometric orientation as the reference feature vector, establish a rectangular coordinate system with the geometric center of the UAV as the origin in the image space of the real-time video stream, and divide the assembly area into four mutually symmetrical and non-overlapping quadrant regions.

[0018] After obtaining the baseline feature vector, the direction of this vector is taken as the positive X-axis of the Cartesian coordinate system (e.g., pointing towards the nose), and its perpendicular direction is taken as the positive Y-axis. The origin of the coordinate system is set as the geometric center of the UAV calculated by the target detection algorithm. Based on this Cartesian coordinate system, the entire UAV assembly area is divided into four symmetrical and non-overlapping quadrant regions (quadrants one to four) in the image space, corresponding to the four symmetrical arm mounting areas of the UAV: ​​right front, left front, left rear, and right rear.

[0019] It should be understood that through the above division of spatial dimensions, the originally symmetrical physical structure has completed the decoupling of absolute orientation in the digital image space.

[0020] S2, detect the assembly operation point in the video stream, and when it falls into the target quadrant area, assign a unique identity ID to the symmetrical assembly component in the target quadrant area that is bound to the quadrant, and use a time-series pixel tracker to continuously lock the identity ID. After completing the global spatial dimension division and establishing the absolute orientation coordinate system, the location of assembly actions is further identified in real time, and highly symmetrical assembly parts are distinguished and continuously tracked. To this end, this step detects assembly operation points in the video stream, such as the operator's fingertips, the head of a screwdriver, or the end effector of an assembly robot arm. When an assembly operation point falls within a defined quadrant region, it is determined that the corresponding symmetrical assembly part (such as a robot arm) is being operated. At this point, a unique identity ID bound to that quadrant is assigned to the symmetrical assembly part within that quadrant region, thereby achieving individualized identification of the symmetrical part.

[0021] However, during continuous assembly, the camera lens may switch perspectives due to shooting needs, or the operator's hands, tools, etc., may partially obstruct the workpiece, causing the bound identity ID to be lost or misaligned. To solve this problem, the present invention further incorporates a time-series pixel tracker to continuously lock the symmetrical assembly parts that have been assigned identity IDs, ensuring the continuity and stability of their identity identification throughout the entire assembly process.

[0022] For detailed implementation procedures, please refer to steps S21-S22 below: S21, when the camera lens viewpoint is switched or the symmetrical assembly component is partially visually obscured, the temporal pixel tracker is used to track the last known pixel centroid of the symmetrical assembly component before it is obscured in real time. In actual assembly processes, delicate operations such as tightening screws on the machine arm will inevitably cause the operator's hand or pneumatic screwdriver bit to repeatedly obscure local areas of the machine arm, and may even completely obscure key features used for identification, such as the gimbal slot. When a symmetrical assembly component within the target quadrant is detected to be about to be obscured, the temporal pixel tracker immediately records the last known centroid coordinates of that component before obscuration. This centroid can be calculated based on the component's contour, texture, or edge features, effectively representing the component's centroid position in image space. Even after obscuration occurs, the temporal pixel tracker continues predictive tracking in the background, maintaining the pixel centroid trajectory based on the motion model, thus ensuring that the tracking status of the component is not lost during obscuration.

[0023] It should be noted that, referring to Figure 2As shown, the network structure of the temporal pixel tracker consists of a feature extraction backbone network, a similarity calculation head, and a Kalman filter prediction module. The feature extraction backbone network adopts the MobileNetV3 architecture, outputting a high-dimensional feature map of the input image block while ensuring real-time performance. The similarity calculation head shares weights in a twin manner, processing the target template (i.e., the last visible part image region before occlusion) and the current search region (the possible part positions in the new frame) respectively. A response heatmap is generated through cross-correlation operations, and the peak position of the heatmap corresponds to the predicted position of the part in the current frame.

[0024] In the last frame before the component is completely occluded, the temporal pixel tracker uses the center point of the component detection box in that frame as the initial pixel centroid and extracts target template features based on the image content within the detection box. Upon entering the occlusion state, the temporal pixel tracker activates its built-in Kalman filter module. This module assumes a uniform motion model for state transition, using the pixel centroid position and velocity from the previous frame as input to predict the state of the pixel centroid and its covariance matrix in the current frame. The observation update phase of the Kalman filter is suspended during occlusion, thus continuously outputting the predicted pixel centroid coordinates, forming a smooth predicted motion trajectory.

[0025] While occlusion persists, the temporal pixel tracker continuously performs time update steps, recursively estimating the position of the pixel centroid based on the motion model, while the predicted covariance gradually increases to characterize the rise in uncertainty. This design allows the temporal pixel tracker to still provide sub-pixel accuracy continuous centroid trajectories even during short-term occlusion (e.g., no more than 10 frames), preserving reliable historical motion information for realignment after the occlusion ends.

[0026] S22, when the occlusion ends and the symmetrical assembly component reappears, region alignment is performed according to the dynamically tracked pixel centroid trajectory to maintain the binding state between the unique identity ID and the symmetrical assembly component, so as to maintain the temporal tracking continuity of the unique identity ID.

[0027] When the occlusion ends and the component reappears in the video frame, the identification and ID assignment are not re-executed. Instead, the currently displayed candidate region is aligned and matched with the previously maintained pixel centroid trajectory. Specifically, the spatial distance and shape similarity between the current centroid of the re-displayed region and the end of the predicted trajectory are calculated. If the matching error is lower than a preset threshold, the displayed component is determined to be the same component as before the occlusion, and the original unique ID is re-bound to the component.

[0028] It should be understood that, through the aforementioned region alignment method based on temporal pixel trackers, even under conditions of frequent occlusion or brief absence from the field of view, the binding state between the identity ID and the symmetrically assembled parts can be seamlessly maintained, ensuring the continuity of temporal tracking.

[0029] It should be noted that the dynamic tracking and alignment methods described above are not only applicable to local occlusions caused by hands or tools, but also to camera lens perspective changes required for shooting. When the viewing axis rotates, causing changes in the position and orientation of a part in the image, the temporal pixel tracker can also use the last known state and motion trajectory to perform cross-frame correlation, thereby maintaining the stability of the identity ID before coordinate system reconstruction.

[0030] S3, when an assembly action is detected for the symmetrical assembly component, the assembly action is combined with the currently locked identity ID to generate an assembly event, and the assembly event is used to match the corresponding process node in the preset dynamic directed acyclic graph, and the process node is updated to the completed state. After identifying and continuously tracking the symmetrically assembled components, the system further determines in real time whether an assembly action has occurred and associates this action with the currently locked identity ID, thereby driving changes in the process state. Traditional assembly inspection systems typically use linear state machine models, strictly limiting the execution order of each process. However, in actual UAV assembly, the installation of the four arms is a parallel process, and operators often perform disordered, overlapping, or even skipped assembly based on the placement of parts or personal habits. If linear sequence constraints are used, failures or missed inspections are easily detected when faced with non-linear process jumps.

[0031] To this end, this invention pre-deconstructs the entire assembly process into a dynamic directed acyclic graph (DAG). Simultaneously, a visual recognition module detects assembly actions in real time and combines the action type with a continuously locked identity ID to generate standardized assembly events. Subsequently, this assembly event is used to retrieve matching process nodes in the DAG, and the corresponding node status is updated to "completed." Thus, through this graph-based nonlinear process management mechanism, the assembly process in any order can be adaptively tracked, retaining only the necessary rigid constraints between preceding and subsequent processes.

[0032] For detailed implementation procedures, please refer to steps S31-S32 below: S31, the assembly process of the whole machine is deconstructed into a directed acyclic graph containing multiple process nodes and unidirectional dependent edges in the background of the system; wherein, the unidirectional dependent edges are used to constrain rigid preceding and following processes with sequential relationships, and process nodes without dependent edge constraints are allowed to be assembled in an unordered manner. In the assembly process of unmanned aerial vehicles (UAVs), some processes have strict sequential relationships. For example, the motherboard installation must be completed before the arm installation; otherwise, subsequent operations cannot be performed. Other processes, such as the installation of the four arms, are technologically independent and do not depend on each other. To simultaneously represent rigid sequence constraints and parallel free sequences, this embodiment uses a directed acyclic graph for modeling.

[0033] Specifically, each independently identifiable assembly operation is defined as a process node. Node attributes include: process name, the assembly component it belongs to (e.g., "right front arm"), the required action type (e.g., "tightening"), and a completion status flag (initially "incomplete"). For two processes with a sequential relationship, a unidirectional dependency edge is added between their corresponding nodes, pointing from the preceding process to the following process. It should be noted that a dependency edge means that the state update of the following process node depends on the preceding process node having been completed. If there is no dependency edge between two process nodes, they can be executed in any order, including parallel, overlapping, or skip-assembly.

[0034] In the quadcopter drone assembly scenario of this embodiment, no dependency edges are set between the process nodes corresponding to the four arms. The operator can install the right front arm first and then the left rear arm, or pre-assemble the two arms first and then tighten them all at once, without generating false alarms or blockages. The "Install Motherboard" node will point to the four arm installation nodes through a one-way dependency edge, ensuring that the motherboard must be assembled before any of the arms.

[0035] S32, the type of the assembly action and the unique identity ID are encapsulated into an assembly event in the form of a standard tuple. The assembly event is used to retrieve the target process node with the same action type and ID attribute in the directed acyclic graph, and the retrieved target process node is updated to the completed state.

[0036] Based on temporal image sequences in a video stream, a lightweight 3D convolutional neural network combined with keypoint heatmaps is used for end-to-end recognition of assembly actions. Specifically, the trajectory sequence of the assembly operation points (e.g., the center of the hand or the tip of a tool) output above is used as the region of interest for action detection. For each frame, a fixed-size local image patch is cropped centered on the operation point, and N consecutive frames (e.g., N=16) are stacked to form a spatiotemporal cube. This spatiotemporal cube is fed into a pre-trained lightweight 3D CNN network (such as a 3D version based on MobileNetV3), and the network outputs an action class probability distribution.

[0037] The preset action types in this embodiment include, but are not limited to, "tightening," "clamping in," "plugging in / out of connector," "placing," and "removing." Considering that assembly actions are usually accompanied by the use of specific tools or changes in hand posture, heatmaps of key hand points can also be used as auxiliary features. Specifically, an Hourglass network is used to detect the two-dimensional positions of 21 key points on the hand, and the heatmaps are fused with the RGB image as an additional channel to improve the recognition accuracy of subtle hand gestures.

[0038] When the operation point is located within a certain quadrant and the action classification confidence exceeds a set threshold (e.g., 0.7) for multiple consecutive frames, an assembly action is determined to have occurred, and the corresponding action type code is output, such as "TYPE_SCREW_TIGHTEN" (tightening type).

[0039] Since the location of the assembly action (operation point) and the target symmetrical assembly component (such as the right front arm) are spatially highly overlapped, the previously established spatial mapping relationship is used to directly associate the currently detected assembly action with a unique identity ID located within the same quadrant. The specific process is as follows: First, the identity binding relationship established in step S2 is continuously maintained as a real-time mapping table from quadrant regions to unique identity IDs. For example, the first quadrant region is mapped to ID_001, the second quadrant region is mapped to ID_002, and so on. Meanwhile, the processing of the temporal pixel tracker ensures that even in the event of brief occlusion or line-of-sight jitter, the binding relationship between the identity ID and the component remains unchanged.

[0040] When the assembly action identification step outputs a valid action type, it immediately obtains the quadrant region number where the current assembly operation point is located and queries the unique identity ID corresponding to that region according to the mapping table. If the query is successful, the action type and the ID are combined into an assembly event in the form of a standard tuple, denoted as Event=(Action_Type, Component_ID). If the operation point is not currently located in any of the defined quadrant regions (e.g., outside the assembly area), the action is discarded and no event is generated.

[0041] To prevent duplicate events caused by hand tremors, tool rebound, or brief false alarms, timing checks are performed before event generation. Specifically, a most recently triggered timestamp is maintained for each (Action_Type, Component_ID) combination. When a new event is about to be generated, the difference between the current time and the last time the same combination of events was triggered is checked. If the time difference is less than the preset minimum trigger interval (e.g., 500 milliseconds), it is determined to be a duplicate trigger, and the event is blocked; otherwise, the event is accepted and the trigger timestamp is updated.

[0042] In addition, the continuity of actions is determined: for certain assembly actions that require a certain duration (such as tightening multiple screws), an event is generated only once at the beginning of the action ("start event") and once at the end of the action ("completion event"). In this embodiment, the completion event is used by default as the basis for updating the process node state; that is, the final assembly event is only generated when the action is completed (e.g., the screwdriver torque reaches the set value and stops rotating). This avoids repeatedly updating the node state during long-duration actions.

[0043] The process iterates through all process nodes in the pre-defined dynamic directed acyclic graph. Each process node has a pre-defined matching label, which is also composed of (Action_Type, Component_ID). The assembly event finds the target process node with the same label through equality matching. After a successful match, the process checks the node's prerequisites: if all prerequisite nodes pointing to this node through unidirectional dependency edges are in the "completed" state, the node's state is updated to "completed"; if there are incomplete prerequisite nodes, the state is not updated temporarily, and a prompt message can be output to inform the operator that the prerequisite process needs to be completed first.

[0044] It should be understood that by using the assembly event mapping method based on directed acyclic graphs, the traditional linear state machine can be extended to a state graph that supports arbitrary topology, thereby allowing unordered cross-assembly of parallel processes while retaining necessary process sequence constraints. This solves the technical problem of linear algorithms easily failing when faced with unordered process jumps, enabling dynamic adaptive tracking of the actual assembly logic.

[0045] S4, when a rotational transformation of the viewing axis is detected, the absolute orientation coordinate system is reconstructed based on the asymmetric landmark features to correct the mapping relationship between the identity ID and the quadrant in order to intercept repeated triggering events; when the assembly completion signal is received, if there are nodes in an incomplete state in the dynamic directed acyclic graph, a blocking prompt is output.

[0046] During the continuous assembly of drones, industrial cameras above the workstation may take multi-view panoramic shots along with the robotic arm, or operators may actively rotate the drone to facilitate the installation of other components. Both of these situations will cause the camera's line of sight to rotate relative to the drone. Since the aforementioned absolute orientation coordinate system relies on the projection position of asymmetric landmark features in the image, and these features (such as the gimbal slot) themselves have a three-dimensional structure, when the line of sight rotates to a large angle, these features will undergo non-rigid perspective shortening in the two-dimensional image, or even produce local self-occlusion. In this case, if the original coordinate system is used directly, it will lead to the misalignment of quadrant regions, causing symmetrical components that have already been assembled to be incorrectly mapped to another quadrant, triggering repeated assembly events and causing misjudgments of the process status.

[0047] To address the aforementioned issues, this invention, upon detecting a rotational transformation of the view axis, recalculates the complete 3D spatial pose based on asymmetric landmark features, thereby reconstructing the absolute orientation coordinate system and correcting the mapping relationship between the identity ID and quadrants, intercepting repeated triggers caused by viewpoint misalignment at the source. Furthermore, at the end of the overall assembly process, a global verification of the completion status of all process nodes in the dynamic directed acyclic graph is performed. If any incomplete nodes exist, a blocking prompt is output to prevent defective products from flowing into the next process.

[0048] For detailed implementation procedures, please refer to steps S41-S42 below: As an example, the absolute orientation coordinate system is reconstructed based on the asymmetric landmark features to correct the mapping relationship between the identity ID and the quadrant in order to intercept repeated triggering events, including: S41, when a rotational transformation of the view axis is detected, the three-dimensional local convex hull contour of the asymmetric landmark feature is extracted and its self-occlusion edge residual is calculated. The self-occlusion edge residual is used as an optimization constraint condition and compared with the preset UAV three-dimensional CAD digital twin model through multi-view projection. In this way, the real-time three-dimensional spatial pose of the asymmetric landmark feature is calculated, and then the complete geometric reference axis of the asymmetric landmark feature is completed and reconstructed. When the line of sight rotates at a large angle, asymmetric landmark features such as the gimbal slot on the nose cone will experience drastic perspective shortening in the 2D image, with some edges even becoming completely invisible due to self-occlusion. Directly using the 2D contour for coordinate system reconstruction would result in significant orientation errors. Therefore, this step extracts the 3D local convex hull contour of this feature from the current video frame—that is, the smallest convex polygon enclosing all visible pixels—and records its edge point set as the observed contour. Simultaneously, the system pre-stores a 3D CAD digital twin model of this UAV model, from which the baseline contour edge point set under a standard orthographic view can be extracted.

[0049] To quantify the contour distortion caused by self-occlusion and perspective shortening, the self-occlusion edge residual vector is calculated. The specific method is as follows: Let the set of edge points of the reference profile under the standard view be... Each These are two-dimensional image coordinates. The actual set of observed contour edge points extracted in the current frame is... Establish using shape context matching or nearest neighbor search. and The point pair correspondence between subsets. For each successfully matched point pair... Define the point location residual as For the benchmark set For points that fail to match (i.e. edges that are completely invisible due to self-occlusion), a fixed penalty vector is introduced. Its magnitude is significantly larger than that of the normal matching residual, for example, more than 5 times the maximum magnitude of the normal residual. Therefore, the self-occluding edge residual vector... Defined as: ; Among them, if the benchmark point There are matching points ,but ;otherwise .

[0050] It should be understood that this residual vector norm It comprehensively reflects the overall geometric distortion of the current observed profile relative to the standard profile, with the self-occlusion part highlighting the cost of information loss through a large penalty term.

[0051] The residual vector is used as an optimization constraint and compared with the 3D CAD digital twin model through multi-view projection. Specifically, let the candidate pose parameters be... (Including three rotation angles and three translations), the 3D CAD digital twin model is arranged according to... Projecting onto the image plane yields the set of projection contour edge points. Define the projection matching error. for Observation contours in the current frame The average bidirectional distance between them is used to sum the projection matching error with the weighted square norm of the self-occluding edge residual term, forming the total loss function: ; in, To balance the weighting coefficients of projection matching and self-occlusion penalty (typically between 0.5 and 1.0), the pose parameters that minimize the total loss are found using gradient descent or particle swarm optimization algorithms. This parameter represents the real-time three-dimensional spatial pose of the asymmetric landmark features.

[0052] After obtaining the real-time 3D spatial pose, the complete geometric reference axis of the feature is extracted from the CAD model, such as the symmetrical centerline of the nose gimbal slot along the forward and backward direction of the UAV. Projecting this axis onto the current image space according to the calculated pose yields a complete reference axis that is unaffected by self-occlusion and perspective shortening.

[0053] S42, reconstruct the absolute orientation coordinate system and correct the mapping relationship using the complete geometric reference axis as the new principal axis; if the symmetrical assembly component corresponding to the assembly operation point has been marked as completed in the historically generated assembly events, then intercept the repeated triggering events caused by viewpoint misalignment.

[0054] After obtaining the complete geometric reference axis, this axis is used as the new principal axis to re-establish the absolute orientation coordinate system. Specifically, the projection direction of this axis onto the image is determined as the positive X-axis direction (i.e., the nose direction) of the Cartesian coordinate system, with its perpendicular direction as the positive Y-axis direction. The origin of the coordinate system remains the geometric center of the UAV obtained through target detection. Based on the new coordinate system, the assembly area is re-divided into four symmetrical and non-overlapping quadrants.

[0055] After the coordinate system reconstruction is completed, the mapping relationship between the identity IDs and the quadrant regions is updated. Each original unique identity ID (such as ID_001) was originally bound to a symmetric assembly component in the old quadrant (such as the right front arm). At this point, the mapping table is re-established according to the quadrant where the centroid of the component falls in the new coordinate system. Since the temporal pixel tracker continuously maintains the association between the IDs and the physical components, even if the position of the component in the new image changes, its identity can still be confirmed to remain unchanged, and only the corresponding relationship from the ID to the new quadrant is updated.

[0056] When the visual axis continues to rotate and sweeps over the already assembled components again, it will be detected that the assembly operation point falls into the quadrant region where the component is located, and the assembly action is identified. At this time, before generating an assembly event from this action, the historical completion status of the identity ID corresponding to the component is queried. Specifically, it is retrieved whether the process node associated with this ID in the dynamic directed acyclic graph has been marked as "completed". If the query result is "yes", it is determined that the assembly action detected this time is a false trigger caused by view misalignment, the event is directly intercepted, no node status update is performed, and no prompt information is output.

[0057] Through the above coordinate system reconstruction, mapping correction and repeated interception mechanism, false positives caused by visual axis transformation can be effectively eliminated. When the assembly of the whole machine is finished, an assembly end signal is received (which can be triggered by the operator pressing the station completion button, or automatically generated when the system detects that all process nodes are completed). At this time, the entire dynamic directed acyclic graph is traversed to check whether there is a process node whose status is still "uncompleted". If it exists, a blocking prompt is output on the station display or the production management system, for example, "The assembly of the left rear arm is not completed, please check", and the current drone is prevented from flowing into the next process. If all nodes are completed, the assembly is determined to be qualified and release is allowed.

[0058] Reference Figure 3 , Figure 4 as shown, an embodiment of the present invention further provides a video stream-based drone assembly detection system 100, the system comprising: A region division module 10, configured to acquire a real-time video stream including the drone structure and identify asymmetric landmark features therein, so as to establish an absolute azimuth coordinate system in the image, and further divide the entire assembly region into a plurality of non-overlapping quadrant regions; An identity locking module 20, configured to detect an assembly operation point in the video stream, and when it falls into a target quadrant region, assign a unique identity ID bound to this quadrant to the symmetric assembly component in the target quadrant region, and use a temporal pixel tracker to continuously lock this identity ID; The event mapping module 30 is used to generate an assembly event by combining the assembly action with the currently locked identity ID when an assembly action for the symmetrical assembly component is detected, and to use the assembly event to match the corresponding process node in the preset dynamic directed acyclic graph and update the process node to the completed state. The dynamic verification module 40 is used to reconstruct the absolute orientation coordinate system based on the asymmetric landmark features when a rotation transformation of the viewing axis is detected, so as to correct the mapping relationship between the identity ID and the quadrant to intercept repeated triggering events; when the assembly completion signal is received, if there are nodes in the dynamic directed acyclic graph that are not in a completed state, a blocking prompt is output.

[0059] As an example, the region division module 10 specifically includes: Landmark recognition submodule 101 is used to identify asymmetric landmark features in the real-time video stream formed by the head gimbal slot or the tail battery clip, and to extract the relative geometric orientation between the asymmetric landmark features; The coordinate establishment submodule 102 is used to establish a rectangular coordinate system with the geometric center of the UAV as the origin in the image space of the real-time video stream, using the relative geometric orientation as the reference feature vector, and to divide the assembly area into four mutually symmetrical and non-overlapping quadrant regions.

[0060] As an example, the identity locking module 20 specifically includes: The timing tracking submodule 201 is used to track the last known pixel centroid of the symmetrical assembly component before it is occluded in real time using a timing pixel tracker when the camera lens viewpoint is switched or when the symmetrical assembly component is partially visually occluded. The state maintenance submodule 202 is used to perform region alignment based on the dynamically tracked pixel centroid trajectory when the occlusion ends and the symmetrical assembly component reappears, so as to maintain the binding state between the unique identity ID and the symmetrical assembly component and maintain the temporal tracking continuity of the unique identity ID.

[0061] As an example, the event mapping module 30 specifically includes: The topology deconstruction submodule 301 is used to preconstruct the whole machine assembly process into a directed acyclic graph containing multiple process nodes and unidirectional dependent edges in the system background; wherein, the unidirectional dependent edges are used to constrain rigid preceding and following processes with sequential relationships, and process nodes without dependent edge constraints are allowed to be assembled in an unordered manner. The retrieval and update submodule 302 is used to encapsulate the type of the assembly action and the unique identity ID into an assembly event in the form of a standard tuple, use the assembly event to retrieve the target process node with the same action type and ID attribute in the directed acyclic graph, and update the retrieved target process node to the completed state.

[0062] As an example, the dynamic verification module 40 specifically includes: The axis completion submodule 401 is used to extract the three-dimensional local convex hull contour of the asymmetric landmark feature and calculate its self-occlusion edge residual when a rotation transformation of the view axis is detected. The self-occlusion edge residual is used as an optimization constraint condition to perform multi-view projection comparison with a preset UAV three-dimensional CAD digital twin model to solve the real-time three-dimensional spatial pose of the asymmetric landmark feature, and then complete and reconstruct the complete geometric reference axis of the asymmetric landmark feature. The reconstruction interception submodule 402 is used to reconstruct the absolute orientation coordinate system and correct the mapping relationship using the complete geometric reference axis as the new principal axis; if the symmetrical assembly component corresponding to the assembly operation point has been marked as completed in the historically generated assembly events, then the repeated triggering events caused by viewpoint misalignment are intercepted.

[0063] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for inspecting the assembly of unmanned aerial vehicles (UAVs) based on video streams, characterized in that, The method includes: Acquire real-time video streams containing drone structures and identify asymmetric landmark features within them to establish an absolute orientation coordinate system in the image, thereby dividing the entire assembly area into several non-overlapping quadrant regions. The assembly operation point in the video stream is detected. When it falls into the target quadrant region, the symmetrical assembly component in the target quadrant region is assigned a unique identity ID bound to that quadrant, and the identity ID is continuously locked using a time-series pixel tracker. When an assembly action is detected for the symmetrical assembly component, the assembly action is combined with the currently locked identity ID to generate an assembly event, and the assembly event is used to match the corresponding process node in the preset dynamic directed acyclic graph, and the process node is updated to the completed state. When a rotational transformation of the viewing axis is detected, the absolute orientation coordinate system is reconstructed based on the asymmetric landmark features to correct the mapping relationship between the identity ID and the quadrant in order to intercept repeated triggering events; when the assembly completion signal is received, if there are nodes in an incomplete state in the dynamic directed acyclic graph, a blocking prompt is output.

2. The UAV assembly inspection method based on video stream according to claim 1, characterized in that, Identify asymmetric landmark features to establish an absolute orientation coordinate system in the image, and then divide the entire assembly area into several non-overlapping quadrant regions, including: Identify asymmetric landmark features in the real-time video stream formed by the head gimbal slot or the tail battery clip, and extract the relative geometric orientation between the asymmetric landmark features; Using the relative geometric orientation as the reference feature vector, a rectangular coordinate system with the geometric center of the UAV as the origin is established in the image space of the real-time video stream, and the assembly area is divided into four mutually symmetrical and non-overlapping quadrant regions.

3. The UAV assembly inspection method based on video stream according to claim 1, characterized in that, The identity ID is continuously locked using a time-series pixel tracker, including: When the camera lens perspective changes or the symmetrical assembly component is partially visually obstructed, a time-series pixel tracker is used to track the last known pixel centroid of the symmetrical assembly component before it is obstructed in real time. When the occlusion ends and the symmetrical assembly component reappears, region alignment is performed based on the dynamically tracked pixel centroid trajectory to maintain the binding state between the unique identity ID and the symmetrical assembly component, thereby ensuring the temporal tracking continuity of the unique identity ID.

4. The UAV assembly inspection method based on video stream according to claim 1, characterized in that, The assembly action is combined with the currently locked identity ID to generate an assembly event. This assembly event is then used to match the corresponding process node in a preset dynamic directed acyclic graph, and the process node is updated to the completed state, including: In the system background, the whole machine assembly process is deconstructed into a directed acyclic graph containing multiple process nodes and unidirectional dependent edges; wherein, the unidirectional dependent edges are used to constrain rigid preceding and following processes with sequential relationships, and process nodes without dependent edge constraints are allowed to be assembled in an unordered manner. The assembly event is encapsulated into a standard tuple form by the type of the assembly action and the unique identity ID. The assembly event is used to retrieve the target process node with the same action type and ID attribute in the directed acyclic graph, and the retrieved target process node is updated to the completed state.

5. The UAV assembly inspection method based on video stream according to claim 1, characterized in that, Reconstructing the absolute orientation coordinate system based on the asymmetric landmark features, thereby correcting the mapping relationship between the identity ID and the quadrant to intercept duplicate triggering events, including: When a rotational transformation of the view axis is detected, the three-dimensional local convex hull contour of the asymmetric landmark feature is extracted and its self-occlusion edge residual is calculated. The self-occlusion edge residual is used as an optimization constraint condition and compared with the preset UAV three-dimensional CAD digital twin model through multi-view projection. In this way, the real-time three-dimensional spatial pose of the asymmetric landmark feature is calculated, and then the complete geometric reference axis of the asymmetric landmark feature is completed and reconstructed. The absolute orientation coordinate system is reconstructed using the complete geometric reference axis as the new principal axis, and the mapping relationship is corrected; if the symmetrical assembly component corresponding to the assembly operation point has been marked as completed in the historically generated assembly events, the repeated triggering events caused by viewpoint misalignment are intercepted.

6. A video stream-based UAV assembly inspection system, characterized in that, The system includes: The region segmentation module is used to acquire real-time video streams containing the drone structure and identify asymmetric landmark features in them, thereby establishing an absolute orientation coordinate system in the image and dividing the entire assembly area into several non-overlapping quadrant regions. The identity locking module is used to detect assembly operation points in the video stream. When the point falls into the target quadrant region, the module assigns a unique identity ID to the symmetrical assembly parts in the target quadrant region that are bound to that quadrant, and uses a time-series pixel tracker to continuously lock the identity ID. The event mapping module is used to generate an assembly event by combining the assembly action with the currently locked identity ID when an assembly action for the symmetrical assembly component is detected, and to use the assembly event to match the corresponding process node in the preset dynamic directed acyclic graph and update the process node to the completed state. The dynamic verification module is used to reconstruct the absolute orientation coordinate system based on the asymmetric landmark features when a rotational transformation of the view axis is detected, thereby correcting the mapping relationship between the identity ID and the quadrant to intercept repeated triggering events; when the assembly completion signal is received, if there are nodes in the dynamic directed acyclic graph that are not in a completed state, a blocking prompt is output.

7. The UAV assembly inspection system based on video stream according to claim 6, characterized in that, The region division module specifically includes: The landmark recognition submodule is used to identify asymmetric landmark features in the real-time video stream formed by the head gimbal slot or the tail battery clip, and to extract the relative geometric orientation between the asymmetric landmark features; The coordinate establishment submodule is used to establish a rectangular coordinate system with the geometric center of the UAV as the origin in the image space of the real-time video stream, using the relative geometric orientation as the reference feature vector, and to divide the assembly area into four mutually symmetrical and non-overlapping quadrant regions.

8. The UAV assembly inspection system based on video stream according to claim 6, characterized in that, The identity locking module specifically includes: The temporal tracking submodule is used to track the last known pixel centroid of the symmetrical assembly component before it is occluded in real time using a temporal pixel tracker when the camera lens viewpoint is switched or when the symmetrical assembly component is partially visually occluded. The state maintenance submodule is used to perform region alignment based on the dynamically tracked pixel centroid trajectory when the occlusion ends and the symmetrical assembly component reappears, so as to maintain the binding state between the unique identity ID and the symmetrical assembly component and maintain the temporal tracking continuity of the unique identity ID.

9. The UAV assembly and inspection system based on video stream according to claim 6, characterized in that, The event mapping module specifically includes: The topology deconstruction submodule is used to pre-deconstruct the whole machine assembly process into a directed acyclic graph containing multiple process nodes and unidirectional dependent edges in the system background; wherein, the unidirectional dependent edges are used to constrain rigid preceding and following processes with sequential relationships, and process nodes without dependent edge constraints are allowed to be assembled in an unordered manner. The retrieval and update submodule is used to encapsulate the type of the assembly action and the unique identity ID into an assembly event in the form of a standard tuple, use the assembly event to retrieve the target process node with the same action type and ID attribute in the directed acyclic graph, and update the retrieved target process node to the completed state.

10. The UAV assembly inspection system based on video stream according to claim 6, characterized in that, The dynamic verification module specifically includes: The axis completion submodule is used to extract the three-dimensional local convex hull contour of the asymmetric landmark feature and calculate its self-occlusion edge residual when a rotation transformation of the view axis is detected. The self-occlusion edge residual is used as an optimization constraint condition to perform multi-view projection comparison with a preset UAV three-dimensional CAD digital twin model to solve the real-time three-dimensional spatial pose of the asymmetric landmark feature, and then complete and reconstruct the complete geometric reference axis of the asymmetric landmark feature. The reconstructed interception submodule is used to reconstruct the absolute orientation coordinate system and correct the mapping relationship using the complete geometric reference axis as the new principal axis; if the symmetrical assembly component corresponding to the assembly operation point has been marked as completed in the historically generated assembly events, then the repeated triggering events caused by viewpoint misalignment are intercepted.

Images