Machine vision positioning method and system applied to unmanned aerial vehicle parts

Abstract

The embodiment of the application provides a kind of machine vision positioning method and system applied to unmanned aerial vehicle parts, it is related to unmanned aerial vehicle intelligent maintenance vision positioning technical field, receive the multi-view optical sensing device synchronous acquisition unmanned aerial vehicle parts multi-view optical observation data stream set, it is executed between view optical texture correlation mapping processing, and cross-view optical texture correlation mapping tensor is established;To the tensor executes multi-scale texture structure decomposition, obtains global contour texture structure component and local detail texture structure component set;To both execute structure complementary fusion reconstruction processing, generate three-dimensional space positioning information component containing spatial position coordinate positioning information and spatial attitude orientation positioning information.The present application realizes the cross-view correlation modeling of multi-view observation data and multi-scale structure complementary fusion, improves unmanned aerial vehicle parts three-dimensional positioning precision.

Description

Technical Field

[0001] This application relates to the field of visual positioning technology for intelligent maintenance of unmanned aerial vehicles (UAVs), and more specifically, to a machine vision positioning method and system for UAV components. Background Technology

[0002] During the assembly, inspection, and maintenance of drones, precise positioning of numerous components is required for automated operation. Existing machine vision positioning methods typically employ single-view 2D image matching techniques, extracting edge features or key point descriptors of components and comparing them with pre-stored templates to obtain positional information. However, single-view methods experience a significant decrease in positioning accuracy and robustness when faced with scenarios involving partial component occlusion, changes in lighting, or the differentiation of similar components. Some existing technologies have introduced multi-view stereo vision solutions, acquiring depth information through parallax calculations using binocular or multi-view cameras. However, these solutions still rely on frame-by-frame independent feature matching, lacking unified semantic association modeling between observation data from different perspectives. This results in fragmented processing of texture information for the same structure from different perspectives, failing to form a consistent understanding of the overall shape of the component. Furthermore, existing multi-view methods typically employ simple feature stitching or weighted superposition when fusing structural information at different scales. This fails to effectively distinguish the structural complementarity between the global contour information and local detail information of components. Consequently, the positioning results may be accurate in overall position coordinates but have systematic biases in attitude and orientation estimation. This is especially true for UAV components with complex geometry and rich surface textures, making it difficult to achieve high-precision spatial position and attitude positioning simultaneously. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a machine vision positioning method and system for use in unmanned aerial vehicle (UAV) components.

[0004] In conjunction with the first aspect of this application, a machine vision positioning method for drone components is provided, which is applied to a machine vision positioning system for drone components. The method includes: The system receives a set of multi-view optical observation data streams of UAV components synchronously collected by a multi-view optical sensing device. The set of multi-view optical observation data streams includes optical observation frame sequences from at least two acquisition perspectives. Each acquisition perspective optical observation frame sequence consists of multiple optical observation frame units at consecutive time nodes. Perform inter-view optical texture association mapping processing on the multi-view optical observation data stream set, establish the optical texture association mapping relationship between optical observation frame units under different acquisition views, and generate a cross-view optical texture association mapping tensor; Multi-scale texture structure decomposition processing is performed on the cross-view optical texture association mapping tensor to obtain a set of global contour texture structure components that characterize the overall contour shape of UAV parts and a set of local detail texture structure components that characterize the local detail shape of UAV parts. The global contour texture structure component and the local detail texture structure component set are subjected to structural complementary fusion reconstruction processing to generate the three-dimensional spatial positioning information component of the UAV component. The three-dimensional spatial positioning information component includes the spatial position coordinate positioning information and spatial attitude orientation positioning information of the UAV component.

[0005] In conjunction with the second aspect of this application, a machine vision positioning system for drone components is provided. The machine vision positioning system for drone components includes a machine-readable storage medium and a processor. The machine-readable storage medium stores machine-executable instructions. When the processor executes the machine-executable instructions, the machine vision positioning system for drone components implements the aforementioned machine vision positioning method for drone components.

[0006] In conjunction with a third aspect of this application, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when the computer-executable instructions are executed, the aforementioned machine vision positioning method applied to unmanned aerial vehicle (UAV) components is implemented.

[0007] Combining any of the above aspects, by performing inter-view optical texture association mapping processing on the multi-view optical observation data stream set and generating a cross-view optical texture association mapping tensor, a structured semantic association between optical observation frame units under different acquisition views is established. This makes the observation data of each view no longer an independent set of features, but a related whole with a clear correspondence in a unified tensor space. On this basis, the association tensor is separated into a set of global contour texture structure components and local detail texture structure components through multi-scale texture structure decomposition. This achieves structural decoupling of the overall shape and local shape of UAV parts, avoiding the positioning contradictions caused by the mutual interference of global and local information in traditional methods. Furthermore, through structural complementary fusion reconstruction processing, the overall spatial constraints provided by the global contour components and the fine structural constraints provided by the local detail components are organically complementary in the three-dimensional spatial positioning information components. This allows the spatial position coordinate positioning information to benefit from the stability of the global contour, while the spatial attitude orientation positioning information benefits from the recognizability of the local details. The two are output collaboratively under a unified fusion framework, overcoming the systematic errors caused by the fragmentation of information sources when estimating position and attitude separately in the prior art. This significantly improves the three-dimensional spatial positioning accuracy and attitude estimation reliability of complex UAV components under multi-view conditions. Attached Figure Description

[0008] Figure 1 This application provides a schematic flowchart of a machine vision positioning method for drone components. Detailed Implementation

[0009] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0010] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0011] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0012] Figure 1 This document illustrates a flowchart of a machine vision positioning method for UAV components provided in an embodiment of this application. It should be noted that the acquisition, transmission, and storage of all multi-view optical observation data streams in this embodiment are performed within the authorized scope of the UAV component manufacturer. The scope of the acquired data is strictly limited to the appearance optical texture information and three-dimensional spatial positioning information of the UAV components, and does not involve any personal privacy data. The data transmission process uses Transport Layer Security (TLS) encryption, and the storage process uses Advanced Encryption Standard (AES) encryption algorithms. The multi-view optical sensing device consists of three industrial cameras, respectively positioned above, to the left front, and to the right front of the UAV component under test. The optical axes of the three industrial cameras converge at the center of the measurement area.

[0013] This embodiment uses a drone fuselage frame part as the test object. The part is an irregularly shaped structural component made of carbon fiber composite material, with a cross-woven texture and multiple assembly positioning holes on its surface. The part is placed on an optical measurement platform, and a multi-view optical sensing device simultaneously acquires optical images from three viewpoints at a preset frame rate. The resolution of each optical image is a preset pixel width multiplied by a preset pixel height.

[0014] Step S110: Receive a set of multi-view optical observation data streams of UAV components synchronously collected by the multi-view optical sensing device. The set of multi-view optical observation data streams contains optical observation frame sequences of at least two acquisition perspectives. Each acquisition perspective optical observation frame sequence consists of multiple optical observation frame units at consecutive time nodes.

[0015] The computing environment receives three optical observation data streams synchronously output from three industrial cameras of the multi-view optical sensing device via a gigabit Ethernet data acquisition card. The optical observation data streams are in RAW format, representing a two-dimensional pixel matrix. Each pixel contains grayscale values ​​for three color channels: red, green, and blue, with a preset bit depth for each channel. Each industrial camera has undergone intrinsic parameter calibration before leaving the factory, and its intrinsic parameter matrix records the focal length and principal point coordinates. The extrinsic parameter matrices between the three industrial cameras are obtained through bundle adjustment calibration.

[0016] The data structure of the multi-view optical observation data stream collection is a JSON object, containing a field for the number of views and a field for the view data array. Each element in the view data array corresponds to an acquisition view and includes a view identifier field and an optical observation frame sequence field. The view identifier field is a string representing the serial number of the industrial camera. The optical observation frame sequence field is a JSON array, where each element is an optical observation frame unit object. Each frame unit object contains a frame number field, a frame acquisition timestamp field, and a frame pixel matrix field. The frame acquisition timestamp field is an integer value in Unix timestamp format, measured in microseconds. The frame pixel matrix field is a three-dimensional floating-point array, where the indices of the three dimensions correspond to the pixel row coordinates, pixel column coordinates, and color channels, respectively. The pixel row and column coordinates are measured in pixels, and the color channels include red, green, and blue channels.

[0017] The computing environment aligns the optical observation frame sequences from the three perspectives according to their frame acquisition timestamps. The alignment method involves taking three frames whose timestamp differences are within a preset synchronization tolerance range as a group of synchronized frames at the same time node. Each group of synchronized frames after alignment contains three optical observation frame units from different perspectives.

[0018] Step S120: Perform optical texture association mapping processing between viewpoints on the multi-view optical observation data stream set, establish the optical texture association mapping relationship between optical observation frame units under different acquisition viewpoints, and generate cross-view optical texture association mapping tensor.

[0019] Step S121: Perform optical texture primitive extraction processing on the optical observation frame sequence of each acquisition view in the multi-view optical observation data stream set, extract multiple optical texture primitives from each optical observation frame unit, and each optical texture primitive consists of a local texture direction vector, a local texture density parameter, and a local texture contrast parameter.

[0020] The computational environment converts the frame pixel matrix of the optical observation frame unit from the red-green-blue color space to the grayscale space. The conversion is achieved by multiplying the grayscale value of the red channel by a preset red weight, the grayscale value of the green channel by a preset green weight, and the grayscale value of the blue channel by a preset blue weight; the sum of these three products is taken as the grayscale value. For each pixel position in the grayscale image matrix, a grayscale gradient direction histogram is calculated within a square neighborhood window centered on that pixel and with a side length of a preset window size. The grayscale gradient direction histogram evenly divides the circumferential angle range into a preset number of direction intervals, and the gradient magnitude of each pixel within the window is weighted and counted based on the frequency of its gradient direction falling into each interval. The direction angle corresponding to the direction interval with the highest frequency in the histogram is taken as the local texture orientation angle of that pixel.

[0021] The local texture direction angle is converted into a local texture direction vector. This vector is a unit-length vector whose horizontal component equals the cosine of the local texture direction angle, and whose vertical component equals the sine of the local texture direction angle. The local texture density parameter is calculated as the ratio of the number of pixels within the window whose gray-level gradient magnitude exceeds a preset texture response threshold to the total number of pixels in the window. The local texture contrast parameter is calculated as the difference between the maximum and minimum gray-level values ​​within the window, divided by the sum of the maximum and minimum gray-level values.

[0022] Texture primitives, measured in pixels, are sparsely sampled according to their spatial spacing, with the sampling step size preset to the pixel spacing value. Each texture primitive corresponding to a sampling position constitutes an optical texture primitive. The data structure of the optical texture primitive is a structure containing a pixel coordinate row value field, a pixel coordinate column value field, a local texture direction vector field, a local texture density parameter field, and a local texture contrast parameter field.

[0023] Step S122: Perform a viewpoint texture primitive correspondence search process on each optical texture primitive extracted from the optical observation frame unit of the first acquisition viewpoint. Search for the corresponding optical texture primitive with the maximum texture direction consistency with the optical texture primitive in the corresponding optical observation frame unit of the second acquisition viewpoint. Establish the viewpoint texture primitive correspondence between the optical texture primitive of the first acquisition viewpoint and the optical texture primitive of the second acquisition viewpoint. The viewpoint texture primitive correspondence includes the first pixel coordinate position of the optical texture primitive of the first acquisition viewpoint in its respective optical observation frame unit and the second pixel coordinate position of the corresponding optical texture primitive of the second acquisition viewpoint in its respective optical observation frame unit.

[0024] The computational environment calculates the epipolar equations of the optical texture primitives from the first acquisition viewpoint within the optical observation frame units of the second acquisition viewpoint, based on the extrinsic parameter matrix between the first and second acquisition viewpoints. The epipolar line is the straight line representing the projection trajectory of a pixel in the first acquisition viewpoint onto the image plane of the second acquisition viewpoint. The epipolar equation is obtained by multiplying the fundamental matrix by the homogeneous coordinates of the pixel coordinates from the first acquisition viewpoint.

[0025] In the second acquisition viewpoint optical observation frame unit, all optical texture primitives traversed by the epipolar line are extracted to form a candidate matching optical texture primitive set. For each candidate optical texture primitive in the candidate matching optical texture primitive set, the angle between its local texture direction vector and the local texture direction vector of the first acquisition viewpoint optical texture primitive after epipolar geometric transformation is calculated. The angle is the inverse cosine of the absolute value of the dot product of the two normalized direction vectors, with the dimension in radians. Simultaneously, the relative deviation of the local texture density parameter is calculated, which is the absolute value of the difference between the two parameters divided by the arithmetic mean of the two parameters. The relative deviation of the local texture contrast parameter is also calculated. The overall score for texture direction consistency is calculated as the angle value multiplied by a preset angle weight, plus the relative deviation of the local texture density parameter multiplied by a preset density weight, plus the relative deviation of the local texture contrast parameter multiplied by a preset contrast weight. The candidate optical texture primitive with the smallest overall score is taken as the matching result.

[0026] A correspondence between texture primitives from different viewpoints is established. This correspondence is a structure containing a first pixel coordinate position field and a second pixel coordinate position field. The value of the first pixel coordinate position field is a two-dimensional integer pair consisting of the row and column values ​​of the pixel coordinates of the optical texture primitive corresponding to the first acquisition viewpoint. The value of the second pixel coordinate position field is a two-dimensional integer pair consisting of the row and column values ​​of the pixel coordinates of the corresponding optical texture primitive for the second acquisition viewpoint.

[0027] Step S123: Perform a viewpoint-to-view texture primitive correspondence search process on all optical texture primitives in the optical observation frame unit to which the first acquisition viewpoint optical texture primitive belongs, and obtain a viewpoint-to-view texture primitive correspondence set covering all optical texture primitives in the entire optical observation frame unit.

[0028] The computing environment iterates through all optical texture primitives in the first acquisition viewpoint optical observation frame unit, repeating step S122 for each optical texture primitive to generate the corresponding viewpoint texture primitive correspondence. The viewpoint texture primitive correspondences of all optical texture primitives are collected into an array, forming a set of viewpoint texture primitive correspondences.

[0029] Step S124: Based on the first pixel coordinate position and the second pixel coordinate position contained in each interview texture primitive correspondence relationship in the interview texture primitive correspondence relationship set, calculate the corresponding pixel coordinate position of each pixel coordinate position of the first acquisition view optical observation frame unit in the second acquisition view optical observation frame unit, generate the interview optical texture association mapping relationship of pixel coordinate position, and establish the mapping correspondence from the pixel coordinate position set of the first acquisition view optical observation frame unit to the pixel coordinate position set of the second acquisition view optical observation frame unit.

[0030] The computational environment uses matching point pairs from the set of texture primitive correspondences between viewpoints as control points. A thin-plate spline interpolation algorithm is employed to construct a nonlinear mapping function from the pixel coordinate system of the first acquisition viewpoint to the pixel coordinate system of the second acquisition viewpoint. Thin-plate spline interpolation solves the interpolation function by minimizing the bending energy functional, and the interpolation function strictly passes through the given matching coordinates at the control points. For each pixel coordinate position in the optical observation frame unit of the first acquisition viewpoint, the thin-plate spline interpolation function is substituted to calculate the corresponding pixel coordinate position in the optical observation frame unit of the second acquisition viewpoint. The corresponding pixel coordinate position is a sub-pixel precision floating-point coordinate value. The mapping relationship of all pixel coordinate positions constitutes a viewpoint optical texture association mapping relationship for pixel-by-pixel coordinate positions. The data structure of this mapping relationship is a two-dimensional vector field with the same resolution as the optical observation frame unit of the first acquisition viewpoint. Each element of the vector field is a two-dimensional floating-point vector representing the corresponding pixel coordinate in the second acquisition viewpoint.

[0031] Step S125: For each optical observation frame unit in the optical observation frame sequence of each acquisition viewpoint, optical texture primitive extraction processing, inter-viewpoint texture primitive correspondence search processing, and generation processing of the inter-viewpoint optical texture association mapping relationship at pixel-by-pixel coordinate positions are performed on the corresponding optical observation frame units of the other acquisition viewpoints excluding its own acquisition viewpoint. This process obtains the inter-viewpoint optical texture association mapping relationship at pixel-by-pixel coordinate positions between the optical observation frame unit and the corresponding optical observation frame units of each of the other acquisition viewpoints excluding its own acquisition viewpoint.

[0032] The computing environment iterates through all pairwise view combinations among the three viewpoints, including combinations of the first and second viewpoints, the first and third viewpoints, and the second and third viewpoints. For each viewpoint combination, one viewpoint is used as the first acquisition viewpoint and the other viewpoint as the second acquisition viewpoint. The processing flow from steps S121 to S124 is repeated to generate the inter-viewpoint optical texture association mapping relationship for the pixel-by-pixel coordinate position corresponding to that viewpoint combination.

[0033] Step S126: Stack and combine the viewpoint optical texture association mapping relationship between each optical observation frame unit and the corresponding optical observation frame units of each other acquisition viewpoints other than its own acquisition viewpoint along the viewpoint dimension to generate a cross-viewpoint optical texture association mapping tensor with the optical observation frame unit as the basic unit and the viewpoint dimension as the tensor dimension.

[0034] For each set of synchronization frames, the inter-view optical texture association mapping relationships at pixel-by-pixel coordinate positions of all viewpoint combinations generated in step S125 are stacked along the newly added viewpoint combination dimension in the order of viewpoint combination, forming a cross-view optical texture association mapping tensor. The data structure of the cross-view optical texture association mapping tensor is a four-dimensional floating-point array, with the indices of the four dimensions corresponding to the intra-frame pixel row coordinate dimension, the intra-frame pixel column coordinate dimension, the viewpoint combination index dimension, and the mapping vector component dimension, respectively.

[0035] Step S130: Perform multi-scale texture structure decomposition on the cross-view optical texture association mapping tensor to obtain a set of global contour texture structure components that characterize the overall contour shape of the UAV parts and a set of local detail texture structure components that characterize the local detail shape of the UAV parts.

[0036] Step S131: Perform texture structure hierarchical extraction processing on the cross-view optical texture association mapping tensor. Decompose the optical texture association mapping relationship between optical observation frame units under different acquisition views contained in the cross-view optical texture association mapping tensor along the texture structure hierarchy direction to obtain multiple texture structure hierarchy components. Each texture structure hierarchy component corresponds to a different texture structure scale range.

[0037] The computational environment performs Gaussian pyramid decomposition on the two-dimensional vector field of each viewpoint combination of the cross-view optical texture association mapping tensor. The bottom layer of the Gaussian pyramid is the vector field at the original resolution. The first layer of the Gaussian pyramid is obtained by convolving the bottom layer with a Gaussian kernel and then downsampling every other row and column, with the downsampling coefficients halved along each dimension. The second layer of the Gaussian pyramid is obtained by convolving the first layer with the same Gaussian kernel and downsampling operation. This process is repeated to generate a multi-layer vector field with a preset number of pyramid layers. The standard deviation of the Gaussian kernel is preset to be the exponential function value of the base standard deviation multiplied by the pyramid layer index value. Each pyramid layer corresponds to a texture structure hierarchical component, and the texture structure scale range of the texture structure hierarchical component is related to the scale factor of the pyramid layer.

[0038] Step S132: Perform texture structure contour extraction processing on the texture structure hierarchical component with the largest texture structure scale range among multiple texture structure hierarchical components, extract the continuous texture region with gentle texture direction change and continuous texture density distribution in the texture structure hierarchical component, and take the closed contour of the texture boundary of the continuous texture region as the global contour texture structure component.

[0039] The texture structure hierarchy component with the largest texture structure scale range is the top layer of the Gaussian pyramid. The computational environment calculates the vector field divergence value for each pixel location within this top-level vector field. The divergence value is the sum of the partial derivatives of the horizontal component of the vector field with respect to the pixel's row coordinates and the partial derivatives of the vertical component with respect to the pixel's column coordinates. The partial derivatives are approximated using a central difference scheme. Pixel regions with an absolute divergence value less than a preset divergence threshold and a local texture density parameter greater than a preset continuous density threshold are marked as continuous texture regions. A boundary tracing algorithm based on topology analysis is used to extract the closed contours of the continuous texture regions. The boundary tracing algorithm traces point-by-point along the boundary between the continuous and non-continuous texture regions to obtain the pixel coordinate sequence of the closed contour. This closed contour is the global contour texture structure component. The data structure of the global contour texture structure component is a two-dimensional array of integer pairs, with array elements arranged in order of contour direction.

[0040] Step S133: Perform texture structure detail extraction processing on each of the texture structure hierarchical components except for the texture structure hierarchical component with the largest texture structure scale range. Extract discrete texture regions in each texture structure hierarchical component where the texture direction changes drastically and the texture density distribution changes abruptly. Use the texture boundary contour and texture internal direction distribution of each extracted discrete texture region as the local detail texture structure sub-component corresponding to that texture structure hierarchical component.

[0041] For the remaining layers of the Gaussian pyramid, the computational environment calculates the curl value of the vector field at each pixel location. The curl value is the difference between the partial derivative of the vertical component of the vector field with respect to the pixel's row coordinates and the partial derivative of the horizontal component of the vector field with respect to the pixel's column coordinates. Pixels with an absolute curl value exceeding a preset curl threshold are marked as points of drastic texture change. A region growing algorithm is used to aggregate spatially connected points of drastic texture change into discrete texture regions. For each discrete texture region, its boundary contour is extracted, and a histogram of the direction statistics of the texture direction vectors of each pixel within the region is calculated. The direction angle corresponding to the direction interval with the highest frequency in the histogram is taken as the texture internal direction distribution of that region. The boundary contours and texture internal direction distributions of the discrete texture regions together constitute the local detail texture structure sub-components corresponding to that pyramid layer.

[0042] Step S134: Combine the local detail texture structure sub-components corresponding to the remaining texture structure hierarchy components along the texture structure hierarchy direction to generate a local detail texture structure component set containing multiple local detail texture structure sub-components.

[0043] The computing environment arranges all the local detail texture structure sub-components generated in step S133 according to the pyramid layer index values ​​from high-resolution layers to low-resolution layers, forming a set of local detail texture structure components. The data structure of the set of local detail texture structure components is an array of local detail texture structure sub-components.

[0044] Step S140: Perform structural complementary fusion reconstruction processing on the global contour texture structure component and the local detail texture structure component set to generate the three-dimensional spatial positioning information component of the UAV parts. The three-dimensional spatial positioning information component includes the spatial position coordinate positioning information and spatial attitude orientation positioning information of the UAV parts.

[0045] Step S141: Perform contour space parameter parsing processing on the global contour texture structure components to extract the contour centroid space coordinates, contour orientation direction vector and contour scale parameters of the global contour texture structure components. The contour centroid space coordinates represent the center position of the UAV component in three-dimensional space, the contour orientation direction vector represents the principal axis orientation of the UAV component in three-dimensional space, and the contour scale parameter represents the overall scale of the UAV component in three-dimensional space.

[0046] The computational environment reconstructs a 3D point cloud from the 2D closed contour coordinate sequence of the global contour texture structure components using triangulation principles. Triangulation utilizes the correspondence between pairwise viewpoints from three perspectives, calculating the 3D spatial coordinates of the contour points using least squares triangulation. The contour centroid spatial coordinates are the arithmetic mean vector of all 3D contour point coordinates. The contour orientation direction vector is the eigenvector corresponding to the largest eigenvalue of the covariance matrix of the 3D contour point coordinates, representing the principal axis direction of the point cloud distribution. The contour scale parameter is the standard deviation of the Euclidean distance from the 3D contour point coordinates to the contour centroid spatial coordinates. The dimensions of the contour centroid spatial coordinates are millimeters, the dimensions of the contour orientation direction vector are dimensionless unit vectors, and the dimensions of the contour scale parameter are millimeters.

[0047] Step S142: Perform detail space parameter parsing processing on each local detail texture structure sub-component in the local detail texture structure component set, and extract the detail position space coordinates and detail orientation direction vector of each local detail texture structure sub-component. The detail position space coordinates represent the relative position of the local detail texture structure sub-component in the overall outline of the UAV component, and the detail orientation direction vector represents the local surface orientation of the local detail texture structure sub-component in the overall outline of the UAV component.

[0048] The computational environment reconstructs a 3D point cloud from the 2D coordinate sequence of the boundary contours of each local detail texture structure sub-component using triangulation. The spatial coordinates of the detail location are the centroid coordinates of this 3D point cloud. The detail orientation vector is the eigenvector corresponding to the smallest eigenvalue of the covariance matrix of this 3D point cloud, representing the local surface normal direction.

[0049] Step S143: Based on the spatial coordinates of the outline centroid and the spatial coordinates of the detail position of each local detail texture structure sub-component, calculate the relative displacement vector of each local detail texture structure sub-component relative to the spatial coordinates of the outline centroid, and based on the detail orientation direction vector and the outline orientation direction vector of each local detail texture structure sub-component, calculate the relative deflection angle of each local detail texture structure sub-component relative to the outline orientation direction vector.

[0050] The relative displacement vector is the difference between the spatial coordinates of the detail location and the spatial coordinates of the profile centroid. The relative deflection angle is calculated by taking the inverse cosine of the absolute value of the dot product of the detail orientation vector and the profile orientation vector, with the dimension in radians.

[0051] Step S144: Use the relative displacement vectors and relative deflection angles of all local detail texture structure sub-components to perform detail constraint correction processing on the spatial coordinates of the contour centroid and the contour orientation direction vector. Correct the spatial coordinates of the contour centroid to the detail constraint centroid spatial coordinates that satisfy the spatial constraints of the relative displacement vectors of all local detail texture structure sub-components, and correct the contour orientation direction vector to the detail constraint orientation direction vector that satisfies the direction constraints of the relative deflection angles of all local detail texture structure sub-components. The detail constraint correction processing is carried out by iterative approximation of spatial coordinate positions and gradual convergence of orientation vectors.

[0052] The computational environment constructs the detail constraint correction problem as a nonlinear least squares optimization problem. The optimization variables are the corrected centroid spatial coordinates and the corrected orientation vector. The objective function consists of two weighted parts: the first part requires that the corrected centroid coordinates and orientation vector be as close as possible to the original centroid coordinates and orientation vector; the second part requires that for all local detail texture structure sub-components, the Euclidean distance between the corrected centroid coordinates plus the relative displacement vector and the detail position spatial coordinates should be minimized, and the angle between the corrected orientation vector and the detail orientation vector should be consistent with the relative deflection angle. The Levenburg-Marquardt algorithm is used iteratively to solve the optimization problem, with the iteration termination condition being that the change in the optimization variables is lower than a preset convergence threshold. The solution yields the detail constraint centroid spatial coordinates and the detail constraint orientation vector.

[0053] Step S145: Use the centroid spatial coordinates of the detail constraint as the spatial position coordinates of the UAV component, and use the orientation vector of the detail constraint as the spatial attitude orientation of the UAV component, and combine them to generate the three-dimensional spatial positioning information components of the UAV component.

[0054] The spatial position coordinates are a 3D floating-point vector of centroid spatial coordinates with detailed constraints, measured in millimeters. The spatial attitude and orientation are 3D floating-point vectors of orientation direction vectors with detailed constraints, measured as dimensionless unit vectors. The combination of these two constitutes the 3D spatial positioning information component, which is structured as a JSON object containing both position coordinates and attitude / orientation fields.

[0055] Step S150: This method also includes a component motion state tracking step.

[0056] Step S210: After generating the three-dimensional spatial positioning information components of the UAV components, receive the subsequent multi-view optical observation data stream set of the UAV components synchronously collected by the multi-view optical sensing device at subsequent time nodes.

[0057] After processing the first set of synchronization frames, the computing environment continues to receive the second set of synchronization frames acquired by the multi-view optical sensing device at subsequent time points. The time interval between the second set of synchronization frames and the first set of synchronization frames is equal to the frame sampling period of the optical observation frame sequence.

[0058] Step S220: Perform optical texture association mapping processing between viewpoints on the subsequent multi-view optical observation data stream set, establish the optical texture association mapping relationship between optical observation frame units under different acquisition viewpoints at subsequent time nodes, and generate the subsequent cross-view optical texture association mapping tensor.

[0059] The computing environment takes the second set of synchronization frames as input and repeats all the sub-steps of step S120 to generate the subsequent cross-view optical texture association mapping tensor.

[0060] Step S230: Perform multi-scale texture structure decomposition on the subsequent cross-view optical texture association mapping tensor to obtain the subsequent global contour texture structure components and subsequent local detail texture structure components that characterize the overall contour shape of the UAV parts at subsequent time nodes.

[0061] The computing environment takes the subsequent cross-view optical texture association mapping tensor as input and repeats all sub-steps of step S130 to generate the subsequent global contour texture structure components and the subsequent local detail texture structure components set.

[0062] Step S240: Perform structural complementary fusion reconstruction processing on the subsequent global contour texture structure components and the subsequent local detail texture structure components to generate the subsequent three-dimensional spatial positioning information components of the UAV parts.

[0063] The computing environment takes the contour texture structure components and local detail texture structure component sets of the second set of synchronized frames as input, and repeatedly executes all sub-steps of step S140 to generate subsequent three-dimensional spatial positioning information components.

[0064] Step S250: Perform position offset vector extraction processing on the spatial position coordinate positioning information in the three-dimensional spatial positioning information component of the UAV component and the subsequent spatial position coordinate positioning information in the subsequent three-dimensional spatial positioning information component to obtain the spatial position offset vector of the UAV component between continuous time nodes and subsequent time nodes.

[0065] The spatial position offset vector is the difference vector between the subsequent spatial position coordinate positioning information and the spatial position coordinate positioning information, with the dimension of millimeters.

[0066] Step S260: Perform attitude change vector extraction processing on the spatial attitude orientation positioning information in the three-dimensional spatial positioning information component of the UAV component and the subsequent spatial attitude orientation positioning information in the subsequent three-dimensional spatial positioning information component to obtain the spatial attitude change vector of the UAV component between continuous time nodes and subsequent time nodes.

[0067] The attitude change vector is a three-dimensional rotation vector representation of the subsequent spatial attitude orientation positioning information and spatial attitude orientation positioning information, which is calculated from two unit orientation vectors using the Rodriguez rotation formula.

[0068] Step S270: Combine the spatial position offset vector and the spatial attitude change vector into a component motion state description component for the UAV component. The component motion state description component is used to characterize the direction of position change and attitude change trend of the UAV component in the time dimension.

[0069] The data structure for describing the motion state of a component is a JSON object, which includes a position offset vector field and an attitude change vector field.

[0070] Step S160: This method also includes a component picking pose generation step.

[0071] Step S310: After obtaining the three-dimensional spatial positioning information components of the UAV components, extract the three-dimensional spatial coordinate values ​​corresponding to the spatial position coordinate positioning information in the three-dimensional spatial positioning information components of the UAV components.

[0072] The computing environment extracts three components of the three-dimensional spatial coordinates from the position coordinate field of the three-dimensional spatial positioning information: the X-axis coordinate, the Y-axis coordinate, and the Z-axis coordinate, all in millimeters.

[0073] Step S320: Extract the attitude orientation quaternion value corresponding to the spatial attitude orientation positioning information from the three-dimensional spatial positioning information components of the UAV parts.

[0074] The computational environment converts the unit direction vector of spatial attitude orientation positioning information into a quaternion representation. The conversion method involves calculating the rotation axis and rotation angle from the reference direction vector to this direction vector. The rotation axis is the normalized vector of the cross product of the two direction vectors, and the rotation angle is the inverse cosine of the dot product of the two direction vectors. The real part of the quaternion is the cosine of the rotation angle, and the imaginary part is the product of each component of the rotation axis and the sine of the rotation angle.

[0075] Step S330: Based on the three-dimensional coordinates of the spatial position and the preset coordinates of the origin of the end effector reference coordinate system of the UAV end effector, generate a spatial translation transformation command to translate the origin of the end effector reference coordinate system to the three-dimensional coordinates of the spatial position.

[0076] The default origin of the end effector's reference coordinate system is a fixed three-dimensional point in the robotic arm's base coordinate system. The translation vector of the spatial translation transformation command is equal to the difference vector between the three-dimensional coordinates of the spatial position and the coordinates of the origin of the end effector's reference coordinate system.

[0077] Step S340: Based on the attitude orientation quaternion value and the preset reference orientation value of the end effector reference coordinate system of the UAV end effector, generate a spatial rotation transformation command to rotate the reference orientation of the end effector reference coordinate system to the attitude orientation quaternion value.

[0078] The preset reference coordinate system of the UAV end effector is a quaternion representation of the end effector's default attitude. The rotation quaternion of the spatial rotation transformation command is the conjugate of the attitude orientation quaternion value multiplied by the reference orientation quaternion.

[0079] Step S350: Combine the spatial translation transformation command and the spatial rotation transformation command into a component picking pose command. The component picking pose command is used to drive the UAV's component picking end effector to transform from the current attitude to the picking attitude corresponding to the three-dimensional coordinate values ​​of the spatial position and the quaternary values ​​of the attitude orientation.

[0080] The component picking pose instruction is a JSON object containing a translation vector field and a rotation quaternion field.

[0081] Step S360: Send the component pickup pose command to the UAV's end effector control unit, triggering the end effector control unit to drive the component pickup end effector to perform a component pickup operation on the UAV.

[0082] The computing environment publishes the part picking pose command to the motion control topic subscribed to by the end effector control unit through the topic publishing mechanism of the robot operating system. After receiving the command, the end effector control unit calculates the target angle values ​​of each joint through the inverse kinematics solver, and drives the servo motor to move the robotic arm to the picking posture.

[0083] Step S370: During the process of the end effector control unit driving the component picking end effector to perform the picking operation on the UAV component, the real-time multi-view optical observation data stream set of the UAV component synchronously collected by the multi-view optical sensing device is continuously received, and the three-dimensional spatial positioning information components of the UAV component are updated according to the real-time multi-view optical observation data stream set. The updated three-dimensional spatial positioning information components are used as the real-time pose correction reference of the component picking end effector.

[0084] During the pickup operation, the computing environment repeatedly executes steps S110 to S140 at a preset frequency, continuously updating the three-dimensional spatial positioning information components of the UAV parts, and using the updated positioning information to correct the motion trajectory of the pickup end effector, thus forming a visual servo closed-loop control.

[0085] Step S170: This method also includes a component assembly path planning step.

[0086] Step S410: After generating the three-dimensional spatial positioning information components of the UAV parts, read the preset UAV parts assembly target position coordinate information and the preset UAV parts assembly target attitude orientation information from the UAV parts assembly process database.

[0087] The UAV component assembly process database is a key-value pair storage system, with component type codes as keys and assembly target information as values. The assembly target information includes assembly target position coordinates and assembly target attitude orientation fields.

[0088] Step S420: Perform assembly path spatial planning processing on the spatial position coordinates of the three-dimensional spatial positioning information of the UAV parts and the preset assembly target position coordinates of the UAV parts. Generate a collision-free spatial displacement path sequence from the current spatial position coordinates to the assembly target position coordinates in three-dimensional space. The collision-free spatial displacement path sequence consists of multiple spatial path nodes.

[0089] The computational environment employs a fast expanding random tree algorithm for path planning. Starting with the current spatial coordinates as the starting node and the assembly target coordinates as the target node, nodes are randomly sampled in the 3D workspace. The new node is expanded towards its nearest neighbor node by a preset step size. If the expanded path does not collide with environmental obstacles, the new node is added to the search tree. This sampling and expansion process is repeated until the distance between the search tree and the target node is less than a preset threshold. The path is then backtracked from the target node to the starting node along the search tree to obtain a collision-free spatial displacement path sequence.

[0090] Step S430: Perform assembly attitude transformation planning processing on the spatial attitude orientation positioning information in the three-dimensional spatial positioning information of the UAV parts and the preset UAV parts assembly target attitude orientation information to generate an attitude transformation sequence that gradually transforms from the current attitude orientation to the assembly target attitude orientation. The attitude transformation sequence consists of multiple attitude transformation nodes.

[0091] The computational environment employs a spherical linear interpolation algorithm to interpolate the current attitude quaternion and the target attitude quaternion. The interpolation parameter increases from 0 at equal intervals to 1, with the number of intervals equal to the number of nodes in the spatial path. Each interpolation parameter corresponds to a quaternion that represents an attitude transformation node.

[0092] Step S440: Perform time synchronization alignment processing on each spatial path node in the collision-free spatial displacement path sequence and the corresponding attitude transformation node in the attitude transformation sequence, so that each spatial path node corresponds to a synchronized attitude transformation node, and generate a component assembly path planning instruction that synchronizes spatial displacement and attitude transformation.

[0093] The computing environment pairs spatial path nodes with attitude transformation nodes one by one according to node sequence number to form assembly path planning instructions. Each paired node contains position coordinates and attitude quaternions.

[0094] Step S450: Extract the path node position coordinates corresponding to each spatial path node in the component assembly path planning instruction and the node attitude orientation corresponding to the attitude transformation node synchronized with the spatial path node, and generate an assembly actuator control instruction sequence for controlling the UAV component assembly actuator.

[0095] The assembly actuator control instruction sequence is an instruction array, where each instruction element contains the path node position coordinates and node orientation.

[0096] Step S460: Send the component assembly path planning instruction and assembly actuator control instruction sequence to the component assembly actuator control unit of the UAV, triggering the component assembly actuator control unit to drive the component assembly actuator to complete the assembly operation of the UAV components.

[0097] The computing environment sends instruction sequences to the assembly actuator control unit through the topic publishing mechanism of the robot operating system. The assembly actuator control unit executes each path node sequentially according to the instruction sequence, controlling the assembly actuator to complete the assembly operation.

[0098] Step S470: During the process of the component assembly actuator control unit driving the component assembly actuator to complete the assembly operation of the UAV components, the multi-view optical observation data stream of the assembly area collected by the multi-view optical sensing device is received simultaneously to perform optical texture monitoring on the assembly gap and the bonding state of the assembly surface during the assembly process.

[0099] The computing environment extracts texture features of the assembly contact surfaces from the multi-view optical observation data stream of the assembly area, calculates the pixel dimensions of the assembly gap width, and converts them into physical dimensions using triangulation. If the gap width exceeds a preset assembly tolerance threshold, an assembly anomaly alarm is generated.

[0100] Step S180: This method also includes a component positioning information tracing step.

[0101] Step S510: After generating the three-dimensional spatial positioning information components of the UAV parts, assign a part identification code to the UAV parts. The part identification code is composed of the part type code field, the part batch code field, and the part sequence code field.

[0102] The computing environment retrieves the part type code string, part batch code string, and part serial number string for the current workstation from the Manufacturing Execution System. These three strings are concatenated using a hyphen (short hyphen) as a separator. The concatenated string is the part identification code. The part type code represents the part name code, the part batch code represents the production batch number, and the part serial number represents the sequence number within that batch.

[0103] Step S520: Extract the spatial coordinate axis component values ​​corresponding to the spatial position coordinate positioning information in the three spatial coordinate axis directions from the three-dimensional spatial positioning information components of the UAV parts.

[0104] The computing environment extracts three floating-point values—X-axis, Y-axis, and Z-axis—from the position coordinate field of the three-dimensional spatial positioning information components.

[0105] Step S530: Extract the spatial attitude axis component values ​​in the three spatial attitude orientation directions corresponding to the spatial attitude orientation positioning information from the three-dimensional spatial positioning information components of the UAV parts.

[0106] The computing environment extracts three component values ​​of the unit direction vector from the pose orientation field of the three-dimensional spatial positioning information components. The three components are the projection values ​​of the direction vector on the X-axis, Y-axis and Z-axis, respectively.

[0107] Step S540: Combine the component identification code, the spatial coordinate axis component values ​​in the three spatial coordinate axes, and the spatial attitude axis component values ​​in the three spatial attitude axes into a component positioning information record, and store the component positioning information record in the positioning information data table of the UAV component positioning information database.

[0108] The UAV component positioning information database is a relational database. The positioning information data table includes columns for component identification codes, X-axis coordinate values, Y-axis coordinate values, Z-axis coordinate values, X-axis component, Y-axis component, and Z-axis component, as well as a timestamp column. The computing environment executes SQL insert statements to write these values ​​into the corresponding columns.

[0109] Step S550: When receiving a location tracking request for UAV parts, parse the tracking part identification code carried in the location tracking request, and retrieve the corresponding part location information record in the location information data table of the UAV part location information database according to the tracking part identification code.

[0110] The location and traceability request is a JSON object containing a traceable component identification code field. The computing environment uses the traceable component identification code string as the query condition, executes an SQL query, retrieves all records in the location information data table whose component identification code column value equals the query condition value, sorts them in descending order by record timestamp, and takes the first record as the search result.

[0111] Step S560: Extract the spatial coordinate axis component values ​​in the three spatial coordinate axes and the spatial attitude axis component values ​​in the three spatial attitude axes from the retrieved component positioning information records, and reconstruct the traceability spatial position coordinates and traceability spatial attitude orientation of the UAV component in three-dimensional space.

[0112] The computing environment extracts three floating-point values—X-axis, Y-axis, and Z-axis—from the search results and combines them into a three-dimensional vector representing the spatial position coordinates. It also extracts three floating-point components—the X-axis, Y-axis, and Z-axis—and combines them into a three-dimensional unit vector representing the spatial attitude orientation.

[0113] Step S570: Map the traced spatial position coordinates and traced spatial attitude orientation to the 3D visualization display interface of the UAV component visualization display terminal. In the 3D visualization display interface, draw the spatial position point corresponding to the traced spatial position coordinates and the attitude pointing arrow corresponding to the traced spatial attitude orientation in the form of 3D marked primitives.

[0114] The computing environment pushes the trace spatial position coordinates and trace spatial orientation to the visualization display terminal via the WebSocket protocol. The visualization display terminal creates a sphere primitive in the 3D scene and places it at the trace spatial position coordinates as the spatial position point. It also creates an arrow primitive with the trace spatial position coordinates as the starting point and the vector of the trace spatial position coordinates plus the trace spatial orientation multiplied by the preset arrow length value as the ending point, serving as the orientation arrow.

[0115] Step S190: This method also includes a multi-view optical sensing parameter adaptive adjustment step.

[0116] Step S610: After generating the three-dimensional spatial positioning information components of the UAV parts, obtain the current optical sensing parameter set of the optical sensing device corresponding to each acquisition view in the multi-view optical sensing device. The current optical sensing parameter set includes the current photosensitivity parameter, the current exposure time parameter, and the current focal length parameter.

[0117] The computing environment obtains the current photosensitivity parameters of each industrial camera through the parameter reading interface of the industrial camera software development kit. The photosensitivity parameters are expressed in ISO standard sensitivity values, with the dimensionless ISO value as the unit. The current exposure time parameter is also obtained, with the unit being microseconds. Finally, the current focal length parameter is obtained, with the unit being millimeters.

[0118] Step S620: Based on the spatial coordinate positioning information in the three-dimensional spatial positioning information of the UAV components, calculate the Euclidean spatial distance between the spatial position of the UAV components and the spatial position of the optical sensing device corresponding to each acquisition viewpoint.

[0119] The spatial position of the optical sensor is the three-dimensional coordinate value of the optical center of the industrial camera in the world coordinate system, given by the translation vector of the camera's extrinsic parameter matrix. The Euclidean distance between the spatial positions of the components and the optical sensor is the L2 norm of the coordinate difference vector between the two points, with dimensions in millimeters.

[0120] Step S630: Based on the Euclidean space distance value corresponding to each acquisition viewpoint and the preset photosensitive sensitivity and distance mapping table, determine the updated photosensitive sensitivity parameter that matches the Euclidean space distance value, and replace the current photosensitive sensitivity parameter of the optical sensing device corresponding to the acquisition viewpoint with the updated photosensitive sensitivity parameter.

[0121] The preset photosensitivity-distance mapping table is a lookup table that maps distance ranges to corresponding photosensitivity values. The computing environment looks up the corresponding photosensitivity value based on the distance range that the Euclidean distance value falls into, and uses this value to update the photosensitivity parameter. The update method involves writing the updated value into the industrial camera's photosensitivity register.

[0122] Step S640: Based on the Euclidean space distance value corresponding to each acquisition viewpoint and the preset focal length and distance mapping table, determine the updated focal length parameter that matches the Euclidean space distance value, and replace the current focal length parameter of the optical sensing device corresponding to the acquisition viewpoint with the updated focal length parameter.

[0123] A pre-defined focal length-distance mapping table maps distance ranges to corresponding focal length values. The calculation environment finds the corresponding focal length value based on the distance range in Euclidean space, and uses this value to update the focal length parameter. The update method involves sending a focal length adjustment command to the lens control motor via a serial communication interface.

[0124] Step S650: Based on the spatial attitude orientation positioning information in the three-dimensional spatial positioning information components of the UAV components, extract the surface normal vector of the UAV components, and calculate the angle between the surface normal vector of the components and the optical axis of the optical sensing device corresponding to each acquisition viewpoint.

[0125] The surface normal vector of a component is the unit direction vector that provides spatial attitude orientation and positioning information. The optical axis direction vector of an optical sensing device is a unit vector extending from the optical center of the camera to the center of the image plane along the optical axis. The included angle is the inverse cosine of the absolute value of the dot product of the two unit vectors, with the dimension of degrees.

[0126] Step S660: Based on the included angle value corresponding to each acquisition viewpoint and the preset exposure time and included angle mapping table, determine the updated exposure time parameter that matches the included angle value, and replace the current exposure time parameter of the optical sensing device corresponding to the acquisition viewpoint with the updated exposure time parameter.

[0127] A preset exposure time-angle mapping table maps the angle range to the corresponding exposure time value. The larger the angle, the more the part surface is tilted towards the optical axis, requiring an increased exposure time to compensate for the attenuation of reflected light. The calculation environment finds the corresponding exposure time value based on the angle range into which the angle value falls, and uses this value to update the exposure time parameter.

[0128] Step S670: Encapsulate the updated photosensitivity parameters, updated exposure time parameters, and updated focal length parameters of the optical sensor corresponding to each acquisition angle into a multi-view optical sensor parameter adjustment command, and send the multi-view optical sensor parameter adjustment command to the multi-view optical sensor to trigger the multi-view optical sensor to acquire subsequent optical observation data streams according to the updated parameters.

[0129] The multi-view optical sensor parameter adjustment command is a JSON array, where each element corresponds to a sampling viewpoint and includes fields for viewpoint identification, photosensitivity, exposure time, and focal length. The computing environment sends this JSON array to the parameter configuration service port of the multi-view optical sensor via a TCP socket. The parameter configuration service parses the JSON array and sets the corresponding parameters one by one using the respective industrial camera software development kits.

[0130] Step S1100: This method also includes a step for detecting surface topography deviations of components.

[0131] For example, in step S710: after generating the three-dimensional spatial positioning information components of the UAV parts, the standard three-dimensional model data corresponding to the UAV parts is obtained. The standard three-dimensional model data includes a standard surface point cloud dataset and a standard surface triangular patch topology.

[0132] The computing environment reads standard 3D model data of UAV components from the product data management system. The standard surface point cloud dataset is a 3D floating-point array, with each row containing the X, Y, and Z coordinates of the standard surface point cloud data points. The standard surface triangular patch topology is an integer array, with each row containing the indices of the three vertices constituting the triangular patch in the standard surface point cloud dataset.

[0133] Step S720: Perform spatial registration processing on the spatial position coordinates and spatial attitude orientation positioning information in the three-dimensional spatial positioning information components of the UAV components and the standard three-dimensional model data, align the standard coordinate system of the standard three-dimensional model data to the actual spatial coordinate system of the UAV components, and generate a spatial registration transformation relationship.

[0134] The computational environment employs the Iterative Closest Point (TLP) algorithm for spatial registration. The target point set is a UAV component contour point cloud, and the source point set is a standard surface point cloud dataset. In each iteration, the TLP searches for the nearest point in the target point set for each point in the source set, calculating the rigid transformation matrix from the source to the target point set. This rigid transformation matrix includes a rotation matrix and a translation vector. The source point set position is updated using the transformation matrix, and the iteration is repeated until the root mean square distance between corresponding point pairs converges. The final rotation matrix and translation vector represent the spatial registration transformation relationship.

[0135] Step S730: Based on the spatial registration transformation relationship, transform each standard surface point cloud data point in the standard 3D model data to the actual spatial coordinate system of the UAV component to obtain an aligned standard surface point cloud dataset aligned with the actual spatial coordinate system.

[0136] The coordinate value of each point in the aligned standard surface point cloud dataset is equal to the rotation matrix multiplied by the coordinate vector of the standard surface point cloud data point, plus the translation vector.

[0137] Step S740: Receive the actual surface point cloud data of the UAV components collected by the multi-view optical sensing device. The actual surface point cloud data is reconstructed from the multi-view optical observation data stream set of the multi-view optical sensing device through the optical triangulation principle.

[0138] The actual surface point cloud data is reconstructed by taking all pairs of viewpoints from the three viewpoints and calculating the corresponding three-dimensional spatial coordinates of each pixel using least squares triangulation. After summarizing all the three-dimensional spatial coordinates and removing outliers, the actual surface point cloud data is obtained.

[0139] Step S750: For each aligned standard surface point cloud data point in the aligned standard surface point cloud dataset, search for the actual surface point cloud data point that is spatially closest to the aligned standard surface point cloud data point in the actual surface point cloud data, and calculate the spatial position deviation vector between the two to obtain the surface point spatial position deviation vector corresponding to each aligned standard surface point cloud data point.

[0140] The computing environment employs a KD-tree to accelerate nearest neighbor search. For each aligned standard surface point cloud data point, the K nearest actual surface point cloud data points are queried in the KD-tree, and the nearest point is selected. The spatial position deviation vector is the difference vector between the coordinates of the nearest actual surface point cloud data point and the coordinates of the aligned standard surface point cloud data point.

[0141] Step S760: Combine the spatial position deviation vectors of all aligned standard surface point cloud data points into surface topography deviation distribution information of UAV parts. The surface topography deviation distribution information describes the point-by-point deviation between the actual surface of the UAV parts and the standard surface in three-dimensional space.

[0142] The surface topography deviation distribution information is a JSON object containing a deviation vector array field. Each element in the array contains a standard point coordinate field and a deviation vector field. The deviation vector field contains three floating-point values: X component, Y component, and Z component.

[0143] Step S770: Send the surface topography deviation distribution information to the machining process parameter control unit of the UAV component manufacturing equipment, triggering the machining process parameter control unit to compensate and correct the machining tool trajectory according to the amplitude and direction of the surface point spatial position deviation vector in the surface topography deviation distribution information.

[0144] After receiving the surface topography deviation distribution information, the machining process parameter control unit decomposes the deviation vector of each surface point along the normal direction of the machining tool path, and takes the normal direction component as the compensation offset of the tool path at that position. Tool path compensation correction is to superimpose the compensation offset on the corresponding tool path point in the original CNC code to generate the compensated CNC code.

[0145] In some embodiments, the machine vision positioning system for drone components used to perform the above-described method can be any electronic device with data computing, processing, and storage functions. This machine vision positioning system for drone components can be used to implement the machine vision positioning method for drone components provided in the above embodiments.

[0146] Typically, machine vision positioning systems used in drone components include a processor and memory. The processor may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor can be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), or PLA (Programmable Logic Array). The processor may also include a main processor and coprocessors. The main processor, also known as a CPU (Central Processing Unit), processes data in the wake-up state; the coprocessor is a low-power processor that processes data in the standby state. In some embodiments, the processor may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor may also include an AI (Artificial Intelligence) processor, which handles computational operations related to machine learning.

[0147] The memory may include one or more computer-readable storage media, which may be non-transitory. The memory may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory is used to store a computer program configured to be executed by one or more processors to implement the machine vision positioning method applied to UAV components described above.

[0148] In an illustrative embodiment, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor of a computer device, implements the aforementioned machine vision positioning method applied to UAV components. Optionally, the aforementioned computer-readable storage medium may be a ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (CompactDisc Read-Only Memory), magnetic tape, floppy disk, or optical data storage device, etc.

[0149] This application provides a computer program product, which includes computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the machine vision positioning method for UAV components provided in this application.

[0150] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the machine vision positioning method for UAV components provided in this application.

[0151] In some embodiments, the computer-readable storage medium may be a read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disk, or CD-ROM, etc.; or it may be a device that includes one or any combination of the above-mentioned memories.

[0152] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0153] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0154] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0155] Finally, it should be noted that the above-disclosed embodiments are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A machine vision positioning method for unmanned aerial vehicle (UAV) components, characterized in that, The method includes: The system receives a set of multi-view optical observation data streams of UAV components synchronously collected by a multi-view optical sensing device. The set of multi-view optical observation data streams includes optical observation frame sequences from at least two acquisition perspectives. Each acquisition perspective optical observation frame sequence consists of multiple optical observation frame units at consecutive time nodes. Perform inter-view optical texture association mapping processing on the multi-view optical observation data stream set, establish the optical texture association mapping relationship between optical observation frame units under different acquisition views, and generate a cross-view optical texture association mapping tensor; Multi-scale texture structure decomposition processing is performed on the cross-view optical texture association mapping tensor to obtain a set of global contour texture structure components that characterize the overall contour shape of UAV parts and a set of local detail texture structure components that characterize the local detail shape of UAV parts. The global contour texture structure component and the local detail texture structure component set are subjected to structural complementary fusion reconstruction processing to generate the three-dimensional spatial positioning information component of the UAV component. The three-dimensional spatial positioning information component includes the spatial position coordinate positioning information and spatial attitude orientation positioning information of the UAV component.

2. The machine vision positioning method for UAV components according to claim 1, characterized in that, The step of performing inter-view optical texture association mapping processing on the multi-view optical observation data stream set, establishing optical texture association mapping relationships between optical observation frame units under different acquisition views, and generating a cross-view optical texture association mapping tensor includes: Optical texture primitive extraction processing is performed on the optical observation frame sequence of each acquisition view in the multi-view optical observation data stream set. Multiple optical texture primitives are extracted from each optical observation frame unit. Each optical texture primitive consists of a local texture direction vector, a local texture density parameter, and a local texture contrast parameter. For each optical texture primitive extracted from the optical observation frame unit of the first acquisition viewpoint, a search process for the inter-viewpoint texture primitive correspondence is performed. In the corresponding optical observation frame unit of the second acquisition viewpoint, a corresponding optical texture primitive with the maximum texture direction consistency with the optical texture primitive is searched. An inter-viewpoint texture primitive correspondence is established between the optical texture primitives of the first acquisition viewpoint and the optical texture primitives of the second acquisition viewpoint. The inter-viewpoint texture primitive correspondence includes the first pixel coordinate position of the optical texture primitive of the first acquisition viewpoint in its respective optical observation frame unit and the second pixel coordinate position of the corresponding optical texture primitive of the second acquisition viewpoint in its respective optical observation frame unit. For all optical texture primitives in the optical observation frame unit to which the first acquisition viewpoint optical texture primitive belongs, the viewpoint texture primitive correspondence search process is performed sequentially to obtain a set of viewpoint texture primitive correspondences covering all optical texture primitives in the entire optical observation frame unit. Based on the first pixel coordinate position and the second pixel coordinate position contained in each interview texture primitive correspondence relationship in the set of interview texture primitive correspondence relationships, calculate the corresponding pixel coordinate position of each pixel coordinate position of the first acquisition view optical observation frame unit in the second acquisition view optical observation frame unit, generate the interview optical texture association mapping relationship of pixel coordinate position, and establish the mapping correspondence of the pixel coordinate position set of the first acquisition view optical observation frame unit to the pixel coordinate position set of the second acquisition view optical observation frame unit. For each optical observation frame unit in the optical observation frame sequence of each acquisition viewpoint, the optical texture primitive extraction process, the inter-viewpoint texture primitive correspondence search process, and the generation process of the inter-viewpoint optical texture association mapping relationship at pixel-by-pixel coordinate positions are performed on the corresponding optical observation frame units of the other acquisition viewpoints excluding its own acquisition viewpoint. This process obtains the inter-viewpoint optical texture association mapping relationship at pixel-by-pixel coordinate positions between the optical observation frame unit and the corresponding optical observation frame units of each of the other acquisition viewpoints excluding its own acquisition viewpoint. The viewpoint optical texture association mapping relationship between each optical observation frame unit and the corresponding optical observation frame units of each other acquisition viewpoint (excluding its own acquisition viewpoint) is stacked and combined along the viewpoint dimension to generate the cross-viewpoint optical texture association mapping tensor with the optical observation frame unit as the basic unit and the viewpoint dimension as the tensor dimension.

3. The machine vision positioning method for UAV components according to claim 1, characterized in that, The process of performing multi-scale texture structure decomposition on the cross-view optical texture association mapping tensor yields a set of global contour texture structure components representing the overall outline shape of the UAV components and a set of local detail texture structure components representing the local detail shape of the UAV components, including: The texture structure hierarchical extraction process is performed on the cross-view optical texture association mapping tensor. The optical texture association mapping relationship between optical observation frame units under different acquisition views contained in the cross-view optical texture association mapping tensor is decomposed along the texture structure hierarchy direction to obtain multiple texture structure hierarchical components. Each texture structure hierarchical component corresponds to a different texture structure scale range. For the texture structure hierarchical component with the largest texture structure scale range among the multiple texture structure hierarchical components, perform texture structure contour extraction processing to extract continuous texture regions with gentle texture direction changes and continuous texture density distribution in the texture structure hierarchical component, and take the closed contour of the texture boundary of the continuous texture region as the global contour texture structure component. For each of the multiple texture structure hierarchical components except for the texture structure hierarchical component with the largest texture structure scale range, texture structure detail extraction processing is performed to extract discrete texture regions with drastic changes in texture direction and abrupt changes in texture density distribution in each texture structure hierarchical component. The texture boundary contour and texture internal direction distribution of each extracted discrete texture region are used as the local detail texture structure sub-components corresponding to the texture structure hierarchical component. The local detail texture structure sub-components corresponding to the remaining texture structure hierarchy components are combined along the texture structure hierarchy direction to generate the local detail texture structure component set containing multiple local detail texture structure sub-components.

4. The machine vision positioning method for UAV components according to claim 1, characterized in that, The step of performing structural complementary fusion reconstruction processing on the global contour texture structure components and the local detail texture structure component sets to generate the three-dimensional spatial positioning information components of the UAV parts includes: The global contour texture structure components are subjected to contour space parameter parsing processing to extract the contour centroid space coordinates, contour orientation direction vector and contour scale parameter of the global contour texture structure components. The contour centroid space coordinates represent the center position of the UAV component in three-dimensional space, the contour orientation direction vector represents the principal axis orientation of the UAV component in three-dimensional space, and the contour scale parameter represents the overall scale of the UAV component in three-dimensional space. For each local detail texture structure sub-component in the set of local detail texture structure components, perform detail space parameter parsing processing to extract the detail position space coordinates and detail orientation direction vector of each local detail texture structure sub-component. The detail position space coordinates represent the relative position of the local detail texture structure sub-component in the overall outline of the UAV component, and the detail orientation direction vector represents the local surface orientation of the local detail texture structure sub-component in the overall outline of the UAV component. Based on the spatial coordinates of the centroid of the contour and the spatial coordinates of the detail position of each local detail texture structure sub-component, calculate the relative displacement vector of each local detail texture structure sub-component relative to the spatial coordinates of the centroid of the contour, and based on the detail orientation direction vector of each local detail texture structure sub-component and the contour orientation direction vector, calculate the relative deflection angle of each local detail texture structure sub-component relative to the contour orientation direction vector. The centroid spatial coordinates and the orientation direction vector of the contour are modified by using the relative displacement vectors and relative deflection angles of all local detail texture structure sub-components. The centroid spatial coordinates of the contour are modified to satisfy the spatial constraints of the relative displacement vectors of all local detail texture structure sub-components, and the orientation direction vector of the contour is modified to satisfy the orientation constraints of the relative deflection angles of all local detail texture structure sub-components. The detail constraint modification process is carried out by iterative approximation of spatial coordinate positions and gradual convergence of orientation vectors. The detailed constraint centroid spatial coordinates are used as the spatial position coordinates of the UAV component, and the detailed constraint orientation vector is used as the spatial attitude orientation of the UAV component. These components are combined to generate the three-dimensional spatial positioning information of the UAV component.

5. The machine vision positioning method for UAV components according to claim 1, characterized in that, The method further includes: After generating the three-dimensional spatial positioning information components of the UAV components, the system receives the subsequent multi-view optical observation data stream set of the UAV components synchronously collected by the multi-view optical sensing device at subsequent time nodes. The subsequent multi-view optical observation data stream set is processed by optical texture association mapping between viewpoints to establish the optical texture association mapping relationship between optical observation frame units under different acquisition viewpoints at subsequent time nodes, and to generate the subsequent cross-view optical texture association mapping tensor. Multi-scale texture structure decomposition processing is performed on the subsequent cross-view optical texture association mapping tensor to obtain the subsequent global contour texture structure components and subsequent local detail texture structure components that characterize the overall contour shape of the UAV parts at subsequent time nodes. Structural complementary fusion reconstruction processing is performed on the subsequent global contour texture structure components and the subsequent local detail texture structure component sets to generate the subsequent three-dimensional spatial positioning information components of the UAV parts; The spatial position coordinate positioning information in the three-dimensional spatial positioning information component of the UAV component and the subsequent spatial position coordinate positioning information in the subsequent three-dimensional spatial positioning information component are subjected to position offset vector extraction processing to obtain the spatial position offset vector of the UAV component between the continuous time node and the subsequent time node; The attitude change vector extraction process is performed on the spatial attitude orientation positioning information in the three-dimensional spatial positioning information component of the UAV component and the subsequent spatial attitude orientation positioning information in the subsequent three-dimensional spatial positioning information component to obtain the spatial attitude change vector of the UAV component between the continuous time node and the subsequent time node. The spatial position offset vector and the spatial attitude change vector are combined to form the component motion state description component of the UAV component. The component motion state description component is used to characterize the position change direction and attitude change trend of the UAV component in the time dimension.

6. The machine vision positioning method for UAV components according to claim 1, characterized in that, The method further includes: After obtaining the three-dimensional spatial positioning information components of the UAV components, the spatial position coordinates corresponding to the spatial position coordinates positioning information in the three-dimensional spatial positioning information components of the UAV components are extracted. Extract the attitude orientation quaternion value corresponding to the spatial attitude orientation positioning information from the three-dimensional spatial positioning information components of the UAV components; Based on the three-dimensional coordinates of the spatial position and the preset coordinates of the origin of the reference coordinate system of the end effector of the UAV, a spatial translation transformation command is generated to translate the origin of the reference coordinate system of the end effector to the three-dimensional coordinates of the spatial position. Based on the attitude orientation quaternion value and the preset reference orientation value of the end effector reference coordinate system of the UAV end effector, a spatial rotation transformation command is generated to rotate the reference orientation of the end effector reference coordinate system to the attitude orientation quaternion value. The spatial translation transformation command and the spatial rotation transformation command are combined into a component picking pose command. The component picking pose command is used to drive the UAV's component picking end effector to change from the current pose to a picking pose corresponding to the three-dimensional coordinate value of the spatial position and the quaternion value of the pose orientation. The component pickup pose command is sent to the UAV's end effector control unit, triggering the end effector control unit to drive the component pickup end effector to perform a pickup operation on the UAV component; During the process of the end effector control unit driving the component picking end effector to perform the picking operation on the UAV component, it continuously receives the real-time multi-view optical observation data stream set of the UAV component synchronously collected by the multi-view optical sensing device, and updates the three-dimensional spatial positioning information components of the UAV component according to the real-time multi-view optical observation data stream set, and uses the updated three-dimensional spatial positioning information components as the real-time pose correction reference of the component picking end effector.

7. The machine vision positioning method for UAV components according to claim 1, characterized in that, The method further includes: After generating the three-dimensional spatial positioning information components of the UAV components, the preset UAV component assembly target position coordinate information and preset UAV component assembly target attitude orientation information are read from the UAV component assembly process database. The spatial position coordinates of the three-dimensional spatial positioning information of the UAV component are combined with the preset assembly target position coordinates of the UAV component to perform assembly path spatial planning processing, thereby generating a collision-free spatial displacement path sequence from the current spatial position coordinates to the assembly target position coordinates in three-dimensional space. The collision-free spatial displacement path sequence consists of multiple spatial path nodes. The spatial attitude orientation positioning information in the three-dimensional spatial positioning information of the UAV component is compared with the preset UAV component assembly target attitude orientation information to perform assembly attitude transformation planning processing, generating an attitude transformation sequence that gradually transforms from the current attitude orientation to the assembly target attitude orientation, the attitude transformation sequence consisting of multiple attitude transformation nodes; Each spatial path node in the collision-free spatial displacement path sequence is time-synchronized and aligned with the corresponding attitude transformation node in the attitude transformation sequence, so that each spatial path node corresponds to a synchronized attitude transformation node, thereby generating a component assembly path planning instruction that synchronizes spatial displacement and attitude transformation. Extract the path node position coordinates corresponding to each spatial path node in the component assembly path planning instruction and the node attitude orientation corresponding to the attitude transformation node synchronized with the spatial path node to generate an assembly actuator control instruction sequence for controlling the UAV component assembly actuator. The component assembly path planning instruction and the assembly actuator control instruction sequence are sent to the component assembly actuator control unit of the UAV, triggering the component assembly actuator control unit to drive the component assembly actuator to complete the assembly operation of the UAV components; During the process of the component assembly actuator control unit driving the component assembly actuator to complete the assembly operation of the UAV components, the multi-view optical observation data stream of the assembly area collected by the multi-view optical sensing device is received simultaneously to perform optical texture monitoring on the assembly gap and the fit of the assembly surface during the assembly process.

8. The machine vision positioning method for UAV components according to claim 1, characterized in that, The method further includes: After generating the three-dimensional spatial positioning information components of the UAV parts, a part identification code is assigned to the UAV parts. The part identification code is composed of a part type code field, a part batch code field, and a part sequence code field. Extract the spatial coordinate axis component values ​​in the three spatial coordinate axes corresponding to the spatial position coordinate positioning information from the three-dimensional spatial positioning information components of the UAV components; Extract the spatial attitude axis component values ​​in the three spatial attitude orientation directions corresponding to the spatial attitude orientation positioning information from the three-dimensional spatial positioning information components of the UAV components; The component identification code, the spatial coordinate axis component values ​​in the three spatial coordinate axes, and the spatial attitude axis component values ​​in the three spatial attitude axes are combined into a component positioning information record, and the component positioning information record is stored in the positioning information data table of the UAV component positioning information database; When receiving a location tracking request for the UAV component, the system parses the tracking component identification code carried in the location tracking request and retrieves the corresponding component location information record in the location information data table of the UAV component location information database according to the tracking component identification code. Extract the spatial coordinate axis component values ​​in the three spatial coordinate axes and the spatial attitude axis component values ​​in the three spatial attitude axes from the retrieved component positioning information records, and reconstruct the traceability spatial position coordinates and traceability spatial attitude orientation of the UAV component in three-dimensional space; The traceability spatial position coordinates and traceability spatial attitude orientation are mapped onto the three-dimensional visualization display interface of the UAV component visualization display terminal. In the three-dimensional visualization display interface, the spatial position point corresponding to the traceability spatial position coordinates and the attitude pointing arrow corresponding to the traceability spatial attitude orientation are drawn in the form of three-dimensional marked primitives.

9. The machine vision positioning method for UAV components according to claim 1, characterized in that, The method further includes: After generating the three-dimensional spatial positioning information components of the UAV components, the current optical sensing parameter set of the optical sensing device corresponding to each acquisition view in the multi-view optical sensing device is obtained. The current optical sensing parameter set includes the current photosensitivity parameter, the current exposure time parameter, and the current focal length parameter. Based on the spatial coordinate positioning information in the three-dimensional spatial positioning information of the UAV components, calculate the Euclidean spatial distance between the spatial position of the UAV components and the spatial position of the optical sensing device corresponding to each acquisition viewpoint. Based on the Euclidean space distance value corresponding to each acquisition viewpoint and the preset photosensitive sensitivity and distance mapping table, an updated photosensitive sensitivity parameter matching the Euclidean space distance value is determined, and the current photosensitive sensitivity parameter of the optical sensing device corresponding to the acquisition viewpoint is replaced with the updated photosensitive sensitivity parameter. Based on the Euclidean space distance value corresponding to each acquisition viewpoint and the preset focal length and distance mapping table, an updated focal length parameter matching the Euclidean space distance value is determined, and the current focal length parameter of the optical sensing device corresponding to the acquisition viewpoint is replaced with the updated focal length parameter. Based on the spatial attitude orientation positioning information in the three-dimensional spatial positioning information of the UAV components, the surface normal vector of the UAV components is extracted, and the angle between the surface normal vector of the components and the optical axis of the optical sensing device corresponding to each acquisition viewpoint is calculated. Based on the included angle value corresponding to each acquisition viewpoint and the preset exposure time and included angle mapping table, determine the updated exposure time parameter that matches the included angle value, and replace the current exposure time parameter of the optical sensing device corresponding to the acquisition viewpoint with the updated exposure time parameter; The updated photosensitivity parameters, exposure time parameters, and focal length parameters of the optical sensor corresponding to each acquisition viewpoint are encapsulated into a multi-view optical sensor parameter adjustment command. The multi-view optical sensor parameter adjustment command is then sent to the multi-view optical sensor, triggering the multi-view optical sensor to acquire subsequent optical observation data streams according to the updated parameters.

10. A machine vision positioning system for use in unmanned aerial vehicle (UAV) components, characterized in that, The device includes a processor and a computer-readable storage medium storing machine-executable instructions, which, when executed by a computer, implement the machine vision positioning method for unmanned aerial vehicle (UAV) components as described in any one of claims 1-9.

Images