A six-degree-of-freedom motion platform posture anomaly monitoring method, system and device

Abstract

The application provides a six-degree-of-freedom motion platform posture anomaly monitoring method, system and device, relates to the technical field of posture anomaly monitoring, and mainly comprises the following steps: a camera is arranged on the lower surface of the upper platform; a two-dimensional code mark is arranged on the upper surface of the supporting surface; when the six-degree-of-freedom motion platform is controlled to operate, the camera is used to collect images containing the two-dimensional code mark in real time; the AR Track Alvar package in the robot operating system is used to estimate the posture information of the six-degree-of-freedom motion platform in real time; and a trained Transform model is used to detect abnormal postures and generate alarm information. The computer image recognition technology and the Transform model are used for posture anomaly monitoring in the scheme, the scheme is free from environmental interference, a large number of sensors need not be arranged, and therefore the difficulty and cost of equipment procurement, installation, calibration and maintenance are greatly reduced. The scheme also has the advantages of stable operation, reliable result, fast response speed and small demand for computing resources.

Description

Technical Field

[0001] This invention relates to the field of posture anomaly monitoring technology, and in particular to a method, system and device for monitoring posture anomalies of a six-degree-of-freedom motion platform. Background Technology

[0002] The six-degree-of-freedom motion platform mainly consists of an upper platform, upper and lower hinge support assemblies, and electric cylinder assemblies, and is installed on a support surface (ground). It is then equipped with an electrical control system including a driver, Ethernet, six-dimensional force sensor, and master controller for closed-loop control. Specifically, as shown... Figure 1 As shown; six degrees of freedom generally refers to three rotational degrees of freedom (pitch α, roll β, and yaw γ) and three translational degrees of freedom (vertical rise z, longitudinal displacement y, and lateral displacement x).

[0003] As a crucial component in simulated flight missions, the six-degree-of-freedom motion platform plays a vital role in providing dynamic simulation. It has gradually gained widespread application in fields such as aviation medical emergency rescue training. Ensuring the positional reliability of the six-degree-of-freedom motion platform and the safety of rescue trainees has become a major issue that urgently needs to be addressed in the industry.

[0004] Currently, although some methods for detecting pose anomalies in six-degree-of-freedom motion platforms have emerged, they still have many shortcomings:

[0005] Sound analysis and vibration detection methods are easily affected by environmental noise and require high levels of professional skills from personnel and advanced equipment detection capabilities.

[0006] Thermal imaging detection technology is also greatly affected by ambient temperature and has high equipment costs.

[0007] Sensor monitoring methods often require the deployment of a large number of sensors, which significantly increases the difficulty and cost of equipment procurement, installation, calibration, and maintenance. Summary of the Invention

[0008] The purpose of this invention is to provide a method, system, and device for monitoring the pose anomaly of a six-degree-of-freedom motion platform, so as to solve at least one of the above-mentioned technical problems existing in the prior art.

[0009] In a first aspect, to solve the above-mentioned technical problems, the present invention provides a method for monitoring the pose anomaly of a six-degree-of-freedom motion platform, comprising the following steps:

[0010] Step 1: Install a camera on the lower surface of the upper platform; install a QR code mark on the upper surface of the support surface; the QR code mark is within the shooting range of the camera; connect the electrical control system of the camera and the six-degree-of-freedom motion platform to the robot operating system (ROS).

[0011] Step 2: During the operation of the six-degree-of-freedom motion platform, images containing the QR code markers are acquired in real time using a camera; based on the images, the pose information of the six-degree-of-freedom motion platform is estimated in real time using the AR Track Alvar package in the robot operating system.

[0012] Step 3: Based on the pose information, detect abnormal poses and generate alarm information using the trained Transformer model.

[0013] Using the above method, the pose anomaly monitoring function of a six-degree-of-freedom motion platform can be realized by simply using a camera for computer vision detection. It is not affected by environmental factors, does not require the deployment of a large number of sensors, and can significantly reduce the difficulty and cost of related equipment procurement, installation, calibration and maintenance.

[0014] In one feasible implementation, the QR code is labeled as an AR Tag; the image within the tag is divided into a 6x6 black and white grid array, representing a 36-bit binary value, with the first 10 bits representing the tag ID and the remaining 26 bits used for error detection and correction, ensuring the uniqueness of the tag in four orientations.

[0015] In one feasible implementation, the method for estimating the pose information in step 2 specifically includes:

[0016] Step 21: Calibrate the camera using Zhang Zhengyou's camera calibration method to determine its internal parameters, including focal length, optical center, and distortion; and determine its external parameters, including its position and orientation in the world coordinate system, based on its installation location.

[0017] Step 22: Detect the location of AR Tag markers in the image through frame-by-frame analysis; identify and verify AR Tag markers to ensure the accuracy and validity of the marker IDs;

[0018] Step 23: Based on the AR Tag corner data and the camera's internal and external parameters, calculate the pose information of the six-degree-of-freedom motion platform relative to the AR Tag marker using the PnP (Perspective-n-Point) algorithm.

[0019] In one feasible implementation, when the closest distance between the upper platform of the six-degree-of-freedom motion platform and the support surface is 3180 mm, the farthest distance is 4040 mm, and the maximum offset angle is 25 degrees, the size of the AR Tag can be 20*20 cm. Actual verification shows that this allows the camera to clearly capture and effectively identify the AR Tag within the operating range of the six-degree-of-freedom motion platform.

[0020] In one feasible implementation, the network architecture of the Transformer model in step 3 specifically includes an input layer, a linear transformation layer, a convolutional layer, a positional encoding module, a feature extraction module, a fully connected layer, and an output layer arranged sequentially.

[0021] The input layer includes a label embedding layer and a position embedding layer. The label embedding layer maps pixels in the input sequence to vector representations in a high-dimensional space through linear transformation, thereby providing rich feature representations for each sequence element. The position embedding layer generates a unique embedding for each position in the input sequence and adds the embedding to the feature vector of the corresponding element. Through this design, the model can not only understand the features of each element, but also effectively utilize the sequential information of these elements.

[0022] The convolutional layer is used to expand the feature dimension;

[0023] The location encoding module is used to add location information;

[0024] The feature extraction module includes a Transformer structure composed of a preset number of Block units; each Block unit contains a multi-head attention layer and a multilayer perceptron; the multi-head attention layer is used to capture the dependencies between different positions in the input sequence; the multilayer perceptron is used to perform nonlinear transformations for feature extraction.

[0025] The above structure can be regarded as an encoder, which can process and reassemble the features of the input sequence layer by layer, thereby gradually enhancing the ability to understand and process the input information;

[0026] The fully connected layer is used to map the input vector to a new space for final abnormal pose prediction. The specific calculation formula is as follows:

[0027] ;

[0028] in, This represents the weight matrix of the fully connected layer; Represents the input vector; Represents the bias vector; The output vector is a two-dimensional vector used to represent abnormal or normal poses.

[0029] In one feasible implementation, the multi-head attention layer performs attention operations on the input in parallel across different representation subspaces and ultimately aggregates the results, thereby achieving detailed attention to each part of the input sequence. This helps the model capture diverse information simultaneously, significantly improving computational efficiency and shortening the model training cycle. The specific operation process includes:

[0030] Step a1: Input vector Divide into multiple subsets; for the first Each head, independently calculated query vector Key vector and value vector The specific formulas are as follows:

[0031] ;

[0032] ;

[0033] ;

[0034] in, This represents the query vector weight matrix; Represents the key vector weight matrix; Represents the value vector weight matrix;

[0035] Step a2, for the first Size, calculating attention score The specific formula is as follows:

[0036] ;

[0037] in, Indicates the dimension of the key vector;

[0038] Step a3: Normalize the attention score of each head using the Softmax function to obtain the corresponding attention weights. And the weights sum to 1, the specific formula is:

[0039] ;

[0040] Step a4: Perform a weighted average on the value vector to obtain the first... Weighted output of size The specific formula is as follows:

[0041] ;

[0042] Step a5: Concatenate the weighted outputs of all heads using a linear transformation. To obtain the final multi-head attention output The specific formula is as follows:

[0043]

[0044] in, Indicates the total number of heads;

[0045] By following the steps above, information captured by different heads can be effectively integrated, enhancing the model's ability to understand and represent input data.

[0046] In one feasible implementation, the Transformer model uses the cross-entropy loss function, which calculates the loss by comparing the probability distribution of the model output with that of the actual labels. This can effectively avoid the gradient vanishing problem and ensure stable and efficient training.

[0047] In one feasible implementation, the Transformer model is trained on a training dataset, tested on a test dataset, and validated on a validation dataset.

[0048] In cases where data is missing from the dataset, multiple interpolation methods are used to supplement it.

[0049] In one feasible implementation, the multiple interpolation method specifically includes:

[0050] Step b1: Extract 6D pose data, including three translational components and three rotational components; the rotational components are represented by quaternions, which can avoid gimbal lock problems, improve computational efficiency, and achieve smooth interpolation.

[0051] Step b2: Locate the missing values ​​in the 6D pose data and identify the location and number of missing data.

[0052] Step b3: For the pose data on both sides of the missing data, interpolate their translation and rotation components respectively:

[0053] For the translation component, linear interpolation is used. Linear interpolation has advantages such as high computational efficiency, zero error at interpolation nodes, and a simple and easy-to-understand calculation process. The specific calculation formula for linear interpolation is as follows:

[0054] ;

[0055] in, and This represents two consecutive translation components; Indicates the interpolation parameter, ranging from 0 to 1: when When, the interpolation result is ;when When, the interpolation result is ;when When varying between 0 and 1, the interpolation result is... and Linear variation between them;

[0056] For rotational components, using spherical linear interpolation (SLERP) ensures that the interpolation result is still a valid quaternion, and maintains the continuity and smoothness of the rotational path, avoiding rotational discontinuities.

[0057] The specific calculation formula for the spherical linear interpolation is as follows:

[0058] ;

[0059] in, and Represents two consecutive rotational components (represented as quaternions); Indicates the interpolation parameter, ranging from 0 to 1: when When, the interpolation result is ;when When, the interpolation result is ;when When varying between 0 and 1, the interpolation result is... and Spherical linear interpolation is performed between them based on the shortest path;

[0060] Step b4: Combine the interpolation results of the translation and rotation components to obtain new pose data.

[0061] In one feasible implementation, the Transformer model automatically adjusts the learning rate and optimizes the model parameters during training using the Adam optimization algorithm.

[0062] In one feasible implementation, the Transformer model further optimizes the learning rate during training using linear decay and cosine decay strategies, with the specific formulas as follows:

[0063] ;

[0064] ;

[0065] in, This represents the learning rate in the current training round; Indicates the current training round number; This represents the learning rate factor for the last training round, which is a multiple of the initial learning rate. This indicates the total number of training rounds.

[0066] In one feasible implementation, the training dataset for the Transformer model includes the Linemod public dataset and a custom dataset;

[0067] The Linemod public dataset is an existing dataset that includes toys, home appliances, tools, etc., and is often used as a performance evaluation standard for 3D object detection and pose estimation algorithms.

[0068] The specific method for creating the custom dataset includes:

[0069] Step c1: Fix the depth camera on a tripod and set up a turntable in front of the depth camera; place a marker board and a shooting model on the turntable; place the shooting model in the center of the marker board; the markers on the marker board are evenly distributed around the shooting model; adjust the depth camera to a preset height and preset angle, and as the turntable rotates, take a downward view and continuously shoot a sequence of images around the entire circle to fully capture the object; save it as an RGB image; the depth camera can not only record color images, but also provide depth data for each pixel, simplifying the pose annotation process and improving annotation efficiency;

[0070] Step c2: Based on the marker matching and point cloud registration algorithm, calculate the rigid body transformation between two point clouds by detecting matching corner points; after merging and filtering, the registered point clouds are stored in a data structure to generate a transformation matrix;

[0071] Step c3: Based on the internal parameters of the depth camera, convert the corresponding depth map of the RGB image into a point cloud; iteratively load multiple point cloud fragments into the world coordinate system, merge them, remove duplicate and isolated points, and save them as a mesh file of the depth map; convert the existing CAD stereo map of the captured model into point cloud format to avoid manual segmentation and 3D reconstruction.

[0072] Step c4: Based on the internal parameters of the depth camera, project the point cloud coordinates of the CAD stereo image onto the image plane, redirect the mesh and save it as a projected image; calculate the 2D coordinates and bounding box of the projected image, create the 6D pose label and mask image (pixel level) of the captured model, and save the transformation matrix of the captured model relative to the scene to obtain a custom dataset, including RGB image, depth map, mask image and 6D pose label.

[0073] In one feasible implementation, the custom dataset can also extract 6D pose data through a multimodal self-attention pose estimation network (MSAPEN) to save time and computing resources for creating 6D pose labels for the dataset, and greatly improve the construction efficiency and effect of the custom dataset. The specific network architecture includes a semantic segmentation module, a pixel feature extraction module, and a feature fusion and dimensionality reduction module.

[0074] The semantic segmentation module is used to perform semantic segmentation on the RGB image and depth image of the captured model, extract the color information and depth information of the region of interest, perform masking processing, and then combine the camera's internal parameters to convert them into a mask image and a mask point cloud, respectively.

[0075] The pixel feature extraction module includes a parallel PointNet network and a CNN network, as well as a pixel-level attention mechanism, used to perform multimodal feature extraction and fusion on the mask image and the mask point cloud to obtain a feature map;

[0076] The feature fusion and dimensionality reduction module includes a global max pooling layer and a global average pooling layer set sequentially, used to extract global features from the feature map and reduce its dimensionality to obtain the 6D pose data of the captured model as a label.

[0077] In one feasible implementation, the evaluation criteria for the 6D pose data include the ADD index and the ADD-S index.

[0078] The ADD metric is used to evaluate the accuracy of pose estimation for asymmetric objects by calculating the average Euclidean distance between the positions of points on the object surface in the true pose and the predicted pose. It mainly focuses on whether the positions of points on the object surface in space are accurate.

[0079] The ADD-S index is used to evaluate the accuracy of pose estimation for symmetrical objects by calculating the minimum distance between each point on the object surface and all points in the predicted pose. The evaluation threshold of the ADD-S index is set to 10% of the diameter of the object being measured. That is, if the ADD-S index value of the predicted pose is less than 10% of the diameter of the object being measured, the predicted pose is considered accurate.

[0080] Secondly, based on the same inventive concept, this application also provides a six-degree-of-freedom motion platform pose anomaly monitoring system, including the AR Track Alvar package based on the robot operating system, and constructs a data acquisition module, a data processing module and a result generation module;

[0081] The data acquisition module is used to acquire images containing QR code markers via a camera; the camera is mounted on the lower surface of the upper platform; the QR code markers are mounted on the upper surface of the support surface.

[0082] The data processing module includes a pose information unit and a detection unit;

[0083] The pose information unit estimates the pose information of the six-degree-of-freedom motion platform in real time based on the image.

[0084] The detection unit, based on the pose information, detects abnormal poses and generates alarm information using a trained Transformer model.

[0085] The result generation module is used to send out the pose information and the alarm information.

[0086] Thirdly, based on the same inventive concept, this application also provides a six-degree-of-freedom motion platform pose anomaly monitoring device, including a camera, a processor, a memory, and a bus. The memory stores instructions and data read by the processor, and the processor is used to call the instructions and data in the memory to execute the six-degree-of-freedom motion platform pose anomaly monitoring method as described above. The bus connects the various functional components for transmitting information.

[0087] By adopting the above technical solution, the present invention has the following beneficial effects:

[0088] This invention provides a method, system, and device for monitoring pose anomalies in a six-degree-of-freedom motion platform. It utilizes computer image recognition technology for pose anomaly monitoring, is unaffected by environmental interference, and eliminates the need for deploying numerous sensors, thereby significantly reducing the difficulty and cost of equipment procurement, installation, calibration, and maintenance. This solution also offers advantages such as stable operation, reliable results, fast response speed, and low computational resource requirements. Attached Figure Description

[0089] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0090] Figure 1 This is a schematic diagram of a six-degree-of-freedom motion platform;

[0091] Figure 2 A flowchart of a method for monitoring pose anomalies of a six-degree-of-freedom motion platform provided in an embodiment of the present invention;

[0092] Figure 3 Example image of AR tag marking;

[0093] Figure 4 This is a network architecture diagram of the Transformer model provided in an embodiment of the present invention;

[0094] Figure 5 This is a schematic diagram of a multi-head attention mechanism provided in an embodiment of the present invention;

[0095] Figure 6 Example images of the aircraft simulator model provided in the embodiments of the present invention: a) Image is an RGB image; b) Image is a depth image; c) Image is a mask image; d) Image is a CAD 3D image;

[0096] Figure 7 This is an architecture diagram of the multimodal self-attention pose estimation network provided in an embodiment of the present invention;

[0097] Figure 8 A diagram of a six-degree-of-freedom motion platform pose anomaly monitoring system provided in an embodiment of the present invention;

[0098] Figure 9 A comparison chart of model training set loss curves provided in the embodiments of the present invention;

[0099] Figure 10 A comparison chart of loss curves for the model validation set provided in this embodiment of the invention;

[0100] Figure 11 A comparison chart of model accuracy curves provided for embodiments of the present invention;

[0101] Figure 12 A comparison chart of F1 value curves for models provided in embodiments of the present invention;

[0102] Figure 13 A comparison chart of model performance indicators provided for embodiments of the present invention. Detailed Implementation

[0103] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0104] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0105] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0106] The present invention will be further explained below with reference to specific embodiments.

[0107] It should also be noted that the specific embodiments or implementation methods described below are a series of optimized settings listed by the present invention to further explain the specific content of the invention, and these settings can be combined or used in conjunction with each other.

[0108] Example 1:

[0109] like Figure 2 As shown in the figure, this embodiment provides a method for monitoring the pose anomaly of a six-degree-of-freedom motion platform, which includes the following steps:

[0110] Step 1: Install a camera (monocular) on the lower surface of the upper platform; install a QR code mark on the upper surface of the support surface; the QR code mark is within the shooting range of the camera; connect the electrical control system of the camera and the six-degree-of-freedom motion platform to the robot operating system (ROS).

[0111] Step 2: During the operation of the six-degree-of-freedom motion platform, images containing the QR code markers are acquired in real time using a camera; based on the images, the pose information of the six-degree-of-freedom motion platform is estimated in real time using the AR Track Alvar package in the robot operating system.

[0112] Step 3: Based on the pose information, detect abnormal poses and generate alarm information using the trained Transformer model.

[0113] Using the above method, the pose anomaly monitoring function of a six-degree-of-freedom motion platform can be realized by simply using a camera for computer vision detection. It is not affected by environmental factors, does not require the deployment of a large number of sensors, and can significantly reduce the difficulty and cost of related equipment procurement, installation, calibration and maintenance.

[0114] Furthermore, the QR code is marked as an AR Tag, such as... Figure 3 As shown, the image within the marker is divided into a 6x6 black and white grid array, representing a 36-bit binary value. The first 10 bits represent the marker ID, and the remaining 26 bits are used for error detection and correction, ensuring the uniqueness of the marker in the four orientations.

[0115] Furthermore, the method for estimating pose information in step 2 specifically includes:

[0116] Step 21: Calibrate the camera using Zhang Zhengyou's camera calibration method to determine its internal parameters, including focal length, optical center, and distortion; and determine its external parameters, including its position and orientation in the world coordinate system, based on its installation location.

[0117] Step 22: Detect the location of AR Tag markers in the image through frame-by-frame analysis; identify and verify AR Tag markers to ensure the accuracy and validity of the marker IDs;

[0118] Step 23: Based on the AR Tag corner data and the camera's internal and external parameters, calculate the pose information of the six-degree-of-freedom motion platform relative to the AR Tag marker using the PnP (Perspective-n-Point) algorithm.

[0119] Furthermore, when the closest distance between the upper platform of the six-degree-of-freedom motion platform and the support surface is 3180 mm, the farthest distance is 4040 mm, and the maximum offset angle is 25 degrees, the size of the AR Tag can be 20*20 cm. Actual verification shows that this ensures that the camera can clearly capture and effectively identify the AR Tag within the operating range of the six-degree-of-freedom motion platform.

[0120] Furthermore, such as Figure 4 As shown, the network architecture of the Transformer model in step 3 specifically includes an input layer, a linear transformation layer, a convolutional layer, a positional encoding module, a feature extraction module, a fully connected layer, and an output layer arranged sequentially.

[0121] The input layer includes a label embedding layer and a position embedding layer. The label embedding layer maps pixels in the input sequence (e.g., 7-dimensional) to vector representations in a high-dimensional space (e.g., 128-dimensional) through a linear transformation, thereby providing rich feature representations for each sequence element. The position embedding layer generates a unique embedding for each position in the input sequence and adds the embedding to the feature vector of the corresponding element. Through this design, the model can not only understand the features of each element but also effectively utilize the sequential information of these elements.

[0122] The convolutional layer is used to expand the feature dimension (e.g., 196 dimensions).

[0123] The location encoding module is used to add location information;

[0124] The feature extraction module includes a Transformer structure composed of a preset number (e.g., 8) of Block units; each Block unit contains a multi-head attention layer and a multilayer perceptron; the multi-head attention layer is used to capture the dependencies between different positions in the input sequence; the multilayer perceptron is used to perform nonlinear transformations on the feature extraction.

[0125] The above structure can be regarded as an encoder, which can process and reassemble the features of the input sequence layer by layer, thereby gradually enhancing the ability to understand and process the input information;

[0126] The fully connected layer is used to map the input vector to a new space for final abnormal pose prediction. The specific calculation formula is as follows:

[0127] ;

[0128] in, This represents the weight matrix of the fully connected layer; Represents the input vector; Represents the bias vector; The output vector is a two-dimensional vector used to represent abnormal or normal poses.

[0129] Furthermore, the multi-head attention layer performs attention operations on the input in parallel across different representation subspaces and ultimately aggregates these results, thereby achieving detailed attention to each part of the input sequence. This helps the model capture diverse information simultaneously, significantly improving computational efficiency and shortening the model training cycle; for example... Figure 5 As shown, the specific calculation process includes:

[0130] Step a1: Input vector Divide into multiple subsets; for the first Each head, independently calculated query vector Key vector and value vector The specific formulas are as follows:

[0131] ;

[0132] ;

[0133] ;

[0134] in, This represents the query vector weight matrix; Represents the key vector weight matrix; Represents the value vector weight matrix;

[0135] Step a2, for the first Size, calculating attention score The specific formula is as follows:

[0136] ;

[0137] in, Indicates the dimension of the key vector;

[0138] Step a3: Normalize the attention score of each head using the Softmax function to obtain the corresponding attention weights. And the weights sum to 1, the specific formula is:

[0139] ;

[0140] Step a4: Perform a weighted average on the value vector to obtain the first... Weighted output of size The specific formula is as follows:

[0141] ;

[0142] Step a5: Concatenate the weighted outputs of all heads using a linear transformation. To obtain the final multi-head attention output The specific formula is as follows:

[0143]

[0144] in, Indicates the total number of heads;

[0145] By following the steps above, information captured by different heads can be effectively integrated, enhancing the model's ability to understand and represent input data.

[0146] Furthermore, the Transformer model employs the cross-entropy loss function, calculating the loss by comparing the probability distribution of the model output with that of the actual labels. This effectively avoids the vanishing gradient problem, ensuring stable and efficient training. The specific formula is as follows:

[0147] ;

[0148] in, The total number of samples representing pose information; The pose information represents the first The true label of each pose information sample; The model represents the first The predicted probability of each pose information sample.

[0149] Furthermore, the Transformer model is trained using the training dataset, tested using the test dataset, and validated using the validation dataset; the proportions of each dataset are as follows:

[0150] Training dataset: Test dataset: Validation dataset = 8:2:2;

[0151] The ratio of normal poses to abnormal poses is 5:1.

[0152] In cases where data is missing from the dataset, multiple interpolation methods are used to supplement it.

[0153] Furthermore, the multiple interpolation method specifically includes:

[0154] Step b1: Extract 6D pose data, including three translational components and three rotational components; the rotational components are represented by quaternions, which can avoid gimbal lock problems, improve computational efficiency, and achieve smooth interpolation.

[0155] Step b2: Locate the missing values ​​in the 6D pose data and identify the location and number of missing data.

[0156] Step b3: For the pose data on both sides of the missing data, interpolate their translation and rotation components respectively:

[0157] For the translation component, linear interpolation is used. Linear interpolation has advantages such as high computational efficiency, zero error at interpolation nodes, and a simple and easy-to-understand calculation process. The specific calculation formula for linear interpolation is as follows:

[0158] ;

[0159] in, and This represents two consecutive translation components; Indicates the interpolation parameter, ranging from 0 to 1: when When, the interpolation result is ;when When, the interpolation result is ;when When varying between 0 and 1, the interpolation result is... and Linear variation between them;

[0160] For rotational components, using spherical linear interpolation (SLERP) ensures that the interpolation result is still a valid quaternion, and maintains the continuity and smoothness of the rotational path, avoiding rotational discontinuities.

[0161] The specific calculation formula for the spherical linear interpolation is as follows:

[0162] ;

[0163] in, and Represents two consecutive rotational components (represented as quaternions); Indicates the interpolation parameter, ranging from 0 to 1: when When, the interpolation result is ;when When, the interpolation result is ;when When varying between 0 and 1, the interpolation result is... and Spherical linear interpolation is performed between them based on the shortest path;

[0164] Step b4: Combine the interpolation results of the translation and rotation components to obtain new pose data.

[0165] Furthermore, during the training process, the Transformer model automatically adjusts the learning rate and optimizes the model parameters through the Adam optimization algorithm.

[0166] Furthermore, during training, the Transformer model further optimizes the learning rate using linear and cosine decay strategies, with the specific formulas as follows:

[0167] ;

[0168] ;

[0169] in, This represents the learning rate in the current training round; Indicates the current training round number; This represents the learning rate factor for the last training round, which is a multiple of the initial learning rate. This indicates the total number of training rounds.

[0170] Furthermore, the training dataset for the Transformer model includes the Linemod public dataset and a custom dataset;

[0171] The Linemod public dataset is an existing dataset that includes toys, home appliances, tools, etc., and is often used as a performance evaluation standard for 3D object detection and pose estimation algorithms.

[0172] The specific method for creating the custom dataset includes:

[0173] Step c1: Fix the depth camera on a tripod and set up a turntable directly in front of the depth camera; place a marker board (e.g., 13 ArUco markers evenly distributed along a rectangular path) and a model to be photographed (e.g., a doll model and an airplane simulator model) on the turntable; place the model to be photographed in the center of the marker board; the markers on the marker board are evenly distributed around the model to be photographed; adjust the depth camera to a preset height and preset angle, and as the turntable rotates, take a downward view and continuously shoot a sequence of images around the entire circle to fully capture the object; save as RGB images (1228 images for the doll model; 1152 images for the airplane simulator model, etc.). Figure 6(As shown in Figure a); the depth camera (e.g., Intel RealSense D435i) can not only record color images, but also provide depth data for each pixel, simplifying the pose annotation process and improving annotation efficiency;

[0174] Step c2: Based on the marker matching and point cloud registration algorithm, calculate the rigid body transformation between two point clouds by detecting matching corner points; after merging and filtering, the registered point clouds are stored in a data structure to generate a transformation matrix;

[0175] Step c3: Based on the internal parameters of the depth camera, convert the RGB image into the corresponding depth map (e.g., ...). Figure 6 (As shown in Figure b) is converted into a point cloud; multiple point cloud fragments are iteratively loaded into the world coordinate system, merged, and then duplicate and isolated points are removed and saved as a mesh file of the depth map; existing CAD stereoscopic images of the photographed model (such as...) are then used to... Figure 6 (As shown in Figure d) is converted to point cloud format, thus avoiding manual segmentation and 3D reconstruction;

[0176] Step c4: Based on the intrinsic parameters of the depth camera, project the point cloud coordinates of the CAD stereo image onto the image plane, redirect the mesh, and save it as a projected image; calculate the 2D coordinates and bounding box of the projected image, and create a 6D pose label and mask image (pixel level, such as...) for the captured model. Figure 6 As shown in Figure c), the transformation matrix of the captured model relative to the scene (i.e., the transformation matrix generated in step c2) is saved to obtain a custom dataset, including RGB image, depth map, mask image and 6D pose label.

[0177] Furthermore, the custom dataset can also extract 6D pose data using a multimodal self-attention pose estimation network (MSAPEN) to save time and computational resources in creating 6D pose labels for the dataset, significantly improving the efficiency and effectiveness of building the custom dataset. The specific network architecture includes a semantic segmentation module, a pixel feature extraction module, and a feature fusion and dimensionality reduction module, as follows: Figure 7 As shown;

[0178] The semantic segmentation module is used to perform semantic segmentation on the RGB image and depth image of the captured model, extract the color information and depth information of the region of interest, perform masking processing, and then combine the camera's internal parameters to convert them into a mask image and a mask point cloud, respectively; the masking processing is used to mask the areas in the image that do not need to be processed.

[0179] The pixel feature extraction module includes a parallel PointNet network and a CNN network, as well as a pixel-level attention mechanism, used to perform multimodal feature extraction and fusion on the mask image and the mask point cloud to obtain a feature map;

[0180] Specifically, the PointNet network receives a mask point cloud generated from a depth map at its input, and its output is connected to the input of a first convolutional layer (conv), whose output is connected to the first input of a first concatenation layer (concat). The CNN network receives a mask image generated from an RGB image at its input, and its output is connected to the input of a second convolutional layer, whose output is connected to the second input of the first concatenation layer. The output of the first concatenation layer is connected to the input of a first per-pixel feature extractor (PFAM). The output of the first per-pixel feature extractor is connected to the input of a third convolutional layer. The output of the third convolutional layer is connected to the input of a second per-pixel feature extractor. The output of the second per-pixel feature extractor is connected to the input of a second concatenation layer. The output of the second concatenation layer is connected to the input of a third per-pixel feature extractor. The output of the third per-pixel feature extractor is connected to the input of a feature fusion and dimensionality reduction module.

[0181] The pixel-by-pixel feature extractor is a prior art technique used to enhance useful information in feature maps and suppress irrelevant pixel features;

[0182] The pixel-level attention mechanism is an existing technology used for feature extraction pixel by pixel. It helps the network focus more on the target region, improves the expressive power of the target region's features, and enables downstream modules to process features more efficiently.

[0183] The feature fusion and dimensionality reduction module includes a global max pooling layer and a global average pooling layer set sequentially, used to extract global features from the feature map and reduce its dimensionality to obtain the 6D pose data of the captured model as a label.

[0184] Furthermore, the evaluation criteria for the 6D pose data include the ADD index and the ADD-S index;

[0185] The ADD metric is used to evaluate the accuracy of pose estimation for asymmetric objects by calculating the average Euclidean distance between the positions of points on the object surface in the true pose and the predicted pose. It mainly focuses on whether the positions of points on the object surface in space are accurate.

[0186] The ADD-S index is used to evaluate the accuracy of pose estimation for symmetrical objects by calculating the minimum distance between each point on the object surface and all points in the predicted pose. The evaluation threshold of the ADD-S index is set to 10% of the diameter of the object being measured. That is, if the ADD-S index value of the predicted pose is less than 10% of the diameter of the object being measured, the predicted pose is considered accurate.

[0187] Example 2:

[0188] like Figure 8As shown, this embodiment provides a six-degree-of-freedom motion platform pose anomaly monitoring system, including the AR Track Alvar package based on the robot operating system, and constructs a data acquisition module, a data processing module, and a result generation module;

[0189] The data acquisition module is used to acquire images containing QR code markers via a camera; the camera is mounted on the lower surface of the upper platform; the QR code markers are mounted on the upper surface of the support surface.

[0190] The data processing module includes a pose information unit and a detection unit;

[0191] The pose information unit estimates the pose information of the six-degree-of-freedom motion platform in real time based on the image.

[0192] The detection unit, based on the pose information, detects abnormal poses and generates alarm information using a trained Transformer model.

[0193] The result generation module is used to send out the pose information and the alarm information.

[0194] Example 3:

[0195] This embodiment provides a six-degree-of-freedom motion platform pose anomaly monitoring device, including a camera, a processor, a memory, and a bus. The memory stores instructions and data read by the processor, and the processor is used to call the instructions and data in the memory to execute the six-degree-of-freedom motion platform pose anomaly monitoring method as described above. The bus connects the various functional components for information transmission.

[0196] In another implementation, this solution can be achieved through an integrated device, which may include corresponding modules that perform one or more steps in the various embodiments described above. A module may be one or more hardware modules specifically configured to perform the corresponding step, or implemented by a processor configured to perform the corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented through some combination thereof.

[0197] The processor executes the various methods and processes described above. For example, the method implementations in this scheme can be implemented as software programs tangibly contained in a machine-readable medium, such as memory. In some implementations, part or all of the software program can be loaded and / or installed via memory and / or a communication interface. When the software program is loaded into memory and executed by the processor, one or more steps of the methods described above can be performed. Alternatively, in other implementations, the processor can be configured to execute one of the methods described above by any other suitable means (e.g., by means of firmware).

[0198] This device can be implemented using a bus architecture. A bus architecture can include any number of interconnect buses and bridges, depending on the specific application of the hardware and overall design constraints. The bus connects various circuits, including one or more processors, memory, and / or hardware modules. The bus can also connect various other circuits such as peripherals, voltage regulators, power management circuitry, external antennas, etc.

[0199] Buses can be Industry Standard Architecture (ISA) buses, Peripheral Component Interconnect (PCI) buses, or Extended Industry Standard Component (EISA) buses, etc. Buses can be divided into address buses, data buses, control buses, etc.

[0200] Example 4:

[0201] The prediction performance of the Transformer model in Example 1 is compared with other deep learning models, as detailed below:

[0202] Comparison models: CNN and MobileNet;

[0203] Software development environment: Jupyter Notebook and PyCharm;

[0204] Software development language: Python;

[0205] Software libraries: NumPy, Pandas, and Sklearn;

[0206] Processor: 12th Gen Intel(R) Core(TM) i5-12400;

[0207] RAM: 16GB;

[0208] The model training parameters are set as shown in Table 1;

[0209]

[0210] Every 5 iterations of the training dataset, the validation dataset is evaluated, and the loss values ​​for both the training and validation datasets are recorded. Detailed comparison charts are shown below. Figure 9-10As shown in the figure, the loss of the CNN model decreases rapidly in the initial stage and then tends to stabilize; the loss of the MobileNet model fluctuates more and generally shows a decreasing trend; while the loss of this Transformer model is relatively stable and outperforms the other two.

[0211] For example, the specific values ​​of the loss function during the 390th to 450th iterations are shown in Table 2;

[0212]

[0213] As shown in Table 2, the Transformer model has a lower loss value and better convergence and generalization compared to the other two models.

[0214] A comparison of the accuracy and F1 score of this Transformer model and the MobileNet model when validated on the validation dataset, as follows: Figure 11-12 As shown, after training converges, the accuracy of this Transformer model reaches approximately 0.95, and the F1 score reaches approximately 0.92, both of which are better than the MobileNet model.

[0215] The bar charts showing the detailed comparison of the Transformer model with the MobileNet and CNN models in terms of accuracy, recall, precision, and F1 score are as follows: Figure 13 As shown in Table 3, the Transformer model performs better than other models in detecting pose anomalies and can better identify pose anomalies of a six-degree-of-freedom motion platform.

[0216]

[0217] Meanwhile, thanks to the method in Example 1, the main performance of the system in Example 2, as measured in actual tests, is as follows:

[0218] The average response time is no more than 500 milliseconds (at least 200 milliseconds), enabling real-time monitoring;

[0219] The time from abnormal pose detection to alarm response is no more than 1 second (at least 450 milliseconds), enabling rapid response;

[0220] CPU utilization should be below 60%, and memory usage should not exceed 70% of allocated resources to ensure efficient use of system resources.

[0221] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for monitoring pose anomalies in a six-degree-of-freedom motion platform, characterized in that, include: Step 1: Set up a camera on the lower surface of the upper platform; set up a QR code mark on the upper surface of the support surface; The QR code is marked within the camera's field of view; Integrate the electrical control system of the camera and the six-degree-of-freedom motion platform into the robot operating system; Step 2: While controlling the six-degree-of-freedom motion platform to run, capture images containing the QR code markers in real time using a camera; Based on the image, the pose information of the six-degree-of-freedom motion platform is estimated in real time using the AR Track Alvar package in the robot operating system; Step 3: Based on the pose information, detect abnormal poses and generate alarm information using the trained Transformer model; The method for estimating pose information in step 2 specifically includes: Step 21: Calibrate the camera using Zhang Zhengyou's camera calibration method to determine its internal parameters, including focal length, optical center, and distortion; and determine its external parameters, including its position and orientation in the world coordinate system, based on its installation location. Step 22: Detect the location of AR tags in the image through frame-by-frame analysis; identify and verify AR tags; Step 23: Based on the AR Tag corner data and the camera's internal and external parameters, calculate the pose information of the six-DOF motion platform relative to the AR Tag marker using the PnP algorithm; The network architecture of the Transformer model in step 3 specifically includes, in sequence, an input layer, a linear transformation layer, a convolutional layer, a positional encoding module, a feature extraction module, a fully connected layer, and an output layer; The input layer includes a label embedding layer and a position embedding layer; the label embedding layer maps pixels in the input sequence to vector representations in a high-dimensional space through a linear transformation; the position embedding layer generates a unique embedding for each position in the input sequence and adds the embedding to the feature vector of the corresponding element. The convolutional layer is used to expand the feature dimension; The location encoding module is used to add location information; The feature extraction module includes a Transformer structure composed of a preset number of Block units; each Block unit contains a multi-head attention layer and a multilayer perceptron. The fully connected layer is used to map the input vector to a new space, and the specific calculation formula is as follows: ； in, This represents the weight matrix of the fully connected layer; Represents the input vector; Represents the bias vector; The output vector is a two-dimensional vector used to represent abnormal or normal poses.

2. The monitoring method according to claim 1, characterized in that, The QR code is marked with an AR Tag.

3. The monitoring method according to claim 2, characterized in that, The AR Tag has a size of 20*20 cm.

4. The monitoring method according to claim 1, characterized in that, The specific computation process of the multi-head attention layer includes: Step a1: Input vector Divide into multiple subsets; for the first Each head, independently calculated query vector Key vector and value vector The specific formulas are as follows: ； ； ； in, This represents the query vector weight matrix; Represents the key vector weight matrix; Represents the value vector weight matrix; Step a2, for the first Size, calculating attention score The specific formula is as follows: ； in, Indicates the dimension of the key vector; Step a3: Normalize the attention score of each head using the Softmax function to obtain the corresponding attention weights. And the weights sum to 1, the specific formula is: ； Step a4: Perform a weighted average on the value vector to obtain the first... Weighted output of size The specific formula is as follows: ； Step a5: Concatenate the weighted outputs of all heads and perform a linear transformation. To obtain the final multi-head attention output The specific formula is as follows: in, This indicates the total number of heads.

5. The monitoring method according to claim 1, characterized in that, The Transformer model uses the cross-entropy loss function, which calculates the loss by comparing the probability distribution of the model output with that of the actual labels.

6. The monitoring method according to claim 1, characterized in that, When the Transformer model encounters missing data in the dataset, it supplements the data using multiple interpolation methods. The multiple interpolation method specifically includes: Step b1: Extract 6D pose data, including three translational components and three rotational components; the rotational components are represented by quaternions; Step b2: Locate the missing values ​​in the 6D pose data and identify the location and number of missing data. Step b3: For the pose data on both sides of the missing data, interpolate their translation and rotation components respectively: For the translation component, linear interpolation is used, and the specific calculation formula is as follows: ； in, and This represents two consecutive translation components; Indicates the interpolation parameter, ranging from 0 to 1: when When, the interpolation result is ;when When, the interpolation result is ;when When varying between 0 and 1, the interpolation result is... and Linear variation between them; For the rotational component, spherical linear interpolation is used, and the specific calculation formula is as follows: ； in, and This represents two consecutive rotational components; Indicates the interpolation parameter, ranging from 0 to 1: when When, the interpolation result is ;when When, the interpolation result is ;when When varying between 0 and 1, the interpolation result is... and Spherical linear interpolation is performed between them based on the shortest path; Step b4: Combine the interpolation results of the translation and rotation components to obtain new pose data.

7. A six-degree-of-freedom motion platform pose anomaly monitoring system, characterized in that, This includes the AR Track Alvar package based on the robot operating system, which builds data acquisition, data processing, and result generation modules. The data acquisition module is used to acquire images containing QR code markers via a camera; the camera is mounted on the lower surface of the upper platform; the QR code markers are mounted on the upper surface of the support surface. The data processing module includes a pose information unit and a detection unit; The pose information unit, based on the image, estimates the pose information of the six-degree-of-freedom motion platform in real time, specifically including: Step 21: Calibrate the camera using Zhang Zhengyou's camera calibration method to determine its internal parameters, including focal length, optical center, and distortion; and determine its external parameters, including its position and orientation in the world coordinate system, based on its installation location. Step 22: Detect the location of AR tags in the image through frame-by-frame analysis; identify and verify AR tags; Step 23: Based on the AR Tag corner data and the camera's internal and external parameters, calculate the pose information of the six-DOF motion platform relative to the AR Tag marker using the PnP algorithm; The detection unit, based on the pose information, detects abnormal poses and generates alarm information through a trained Transformer model. The network architecture of the Transformer model specifically includes an input layer, a linear transformation layer, a convolutional layer, a position encoding module, a feature extraction module, a fully connected layer, and an output layer arranged in sequence. The input layer includes a label embedding layer and a position embedding layer; the label embedding layer maps pixels in the input sequence to vector representations in a high-dimensional space through a linear transformation; the position embedding layer generates a unique embedding for each position in the input sequence and adds the embedding to the feature vector of the corresponding element. The convolutional layer is used to expand the feature dimension; The location encoding module is used to add location information; The feature extraction module includes a Transformer structure composed of a preset number of Block units; each Block unit contains a multi-head attention layer and a multilayer perceptron. The fully connected layer is used to map the input vector to a new space, and the specific calculation formula is as follows: ； in, This represents the weight matrix of the fully connected layer; Represents the input vector; Represents the bias vector; The output vector is a two-dimensional vector used to represent abnormal or normal poses. The result generation module is used to send out the pose information and the alarm information.

8. A device for monitoring the pose anomaly of a six-degree-of-freedom motion platform, characterized in that, It includes a camera, a processor, a memory, and a bus. The memory stores instructions and data read by the processor. The processor is used to call the instructions and data in the memory to execute the monitoring method as described in any one of claims 1-6. The bus connects the functional components for transmitting information.

Images