AR-based intelligent integrated production line dedicated circuit control method and system

Abstract

The present invention provides an AR-based intelligent integrated production line dedicated circuit control method and system, which relates to the field of production line control technology, including collecting a real-time point cloud model of production line equipment through AR glasses, and aligning it with a preset three-dimensional model to obtain equipment state deviation data. A time series convolutional network is used to analyze the equipment operation trend, and an operation status evaluation index and a control parameter correction amount are generated. The AR glasses project and display the circuit control nodes to be adjusted, and generate control instructions based on the operator's eye tracking and gesture interaction operations, and record the corresponding relationship between the instructions and the operation status evaluation indicators to form a control model. Based on the model, the edge computing unit constructs an AR-assisted control system, generates control instructions through graph neural network reasoning according to the new operation status evaluation indicators, and displays the operation guidance information on the AR interface. After the operator confirms, the instruction is sent to the production line controller, and the execution result is fed back to update the control model to achieve closed-loop control.

Description

Technical Field

[0001] The present invention relates to the field of production line control technology, and in particular to an AR-based intelligent integrated production line dedicated circuit control method and system. Background Art

[0002] With the development of industrial automation and intelligence, the control system of production line equipment is moving towards integration, efficiency and intelligence. The traditional production line control method mainly relies on fixed control panels and preset programs, and operators need to manually adjust the control parameters according to the operating status of the equipment. This method has problems such as low efficiency, easy errors, and difficulty in adapting to complex production line environments.

[0003] There are many defects and deficiencies in the existing technology. The traditional production line control method lacks flexibility and is difficult to adapt to the dynamic changes in the status of production line equipment. The preset control program cannot cope with the complex production environment, and the operator needs to spend a lot of time on manual adjustments, which affects production efficiency. The traditional control method is highly dependent on the operator's experience and is prone to human errors. The operator needs to remember a large number of control parameters and operating steps, which is prone to fatigue and misoperation, leading to production accidents and product quality problems. The traditional control method is difficult to achieve closed-loop control and lacks real-time feedback and optimization of the control effect. The adjustment of control parameters mainly depends on the operator's experience and judgment, lacks data support, and is difficult to achieve optimal control.

[0004] In summary, the rapid development of augmented reality technology and artificial intelligence technology provides new solutions for production line control. Virtual information can be superimposed on real scenes to provide operators with a more intuitive operation interface and richer equipment information. Control instructions are automatically generated according to the operating status of the equipment to improve control efficiency and accuracy. The present invention can solve the problems in the prior art. Summary of the invention

[0005] The embodiments of the present invention provide an AR-based intelligent integrated production line dedicated circuit control method and system, which can solve the problems in the prior art.

[0006] According to a first aspect of the embodiments of the present invention,

[0007] Provided is an AR-based intelligent integrated production line dedicated circuit control method, including:

[0008] Collect binocular camera images in the AR glasses worn by the operator and spatial posture data of the inertial measurement unit, build a real-time point cloud model of the production line equipment through the spatial reconstruction algorithm, align the real-time point cloud model with the preset three-dimensional model of the production line equipment, obtain equipment state deviation data, and use a temporal convolutional network to analyze the operation trend of the production line equipment based on the equipment state deviation data to generate operation status evaluation indicators and control parameter corrections;

[0009] According to the control parameter correction amount, the position of the circuit control node that needs to be adjusted is projected and displayed in real time in the AR glasses, the control node area that the operator is looking at is obtained through eye tracking, the circuit control panel and the operating status evaluation index are superimposed and displayed in the control node area, the operator's skeletal key point action data is collected, the gesture interaction operation on the circuit control panel is identified, the gesture interaction operation is converted into a circuit control instruction, the corresponding relationship between the circuit control instruction and the operating status evaluation index is recorded, and a control model is formed;

[0010] Based on the control model, an AR-assisted control system is constructed in the edge computing unit. When a new operating status evaluation index is detected, the AR-assisted control system generates a circuit control instruction through graph neural network reasoning, and superimposes and displays the operation guidance information corresponding to the circuit control instruction in the AR display interface. After confirmation by the operator, the circuit control instruction is sent to the production line controller, and the change of the operating status evaluation index corresponding to the execution result is fed back to update the control model, so as to realize closed-loop control of the production line equipment.

[0011] In an optional embodiment,

[0012] The spatial reconstruction algorithm includes:

[0013] Based on the binocular camera image, the binocular image sequence is decomposed according to the image frame, and the binocular image sequence is normalized to obtain the left image sequence and the right image sequence; for the left image sequence and the right image sequence, a four-layer Gaussian pyramid is constructed to generate a multi-scale image representation;

[0014] Performing Harris corner point response value calculation on each layer of the multi-scale image representation, selecting image points whose response values ​​are greater than a first preset threshold as a feature point candidate set; selecting, from the feature point candidate set, points with the largest response value within a domain range of a corresponding size centered on the feature point according to a preset domain size as final feature points; calculating the local image gradient of the final feature point, constructing a gradient direction histogram, and generating a feature description vector;

[0015] The feature description vectors in the left and right images are matched using the nearest neighbor matching method to obtain feature point matching pairs; based on the epipolar geometric constraints, the basic matrix of the feature point matching pairs is calculated, and the matching point pairs that do not meet the epipolar constraints are eliminated to obtain valid feature points;

[0016] Based on the spatial posture data, angular velocity data and acceleration data are obtained, the angular velocity data and acceleration data are aligned with the valid feature points according to the timestamps, visual reprojection error terms and inertial measurement pre-integration error terms are constructed, and feature point motion constraint equations are established;

[0017] According to the preset number of image frame extractions, the latest image frame is selected to construct a sliding window, within the sliding window, based on the feature point motion constraint equation, an objective function is constructed, the objective function is linearized, the Gauss-Newton method is used for iterative optimization, and the reprojection error of each iteration is calculated, and when the reprojection error is less than a second preset threshold, the three-dimensional spatial coordinates of the feature point are output;

[0018] Divide the space where the three-dimensional space coordinate point is located into cubic grids with a side length of a third preset value, calculate the spatial distribution characteristics of the points in each cubic grid to obtain a grid feature point set, calculate the mean of the grid feature point set, mark the grid feature points that deviate from the mean by more than a fourth preset threshold as outlier feature points, remove the outlier feature points, and obtain valid grid feature points;

[0019] A local quadratic surface fitting is performed on the effective grid feature points, a normal vector and a principal curvature of each effective grid feature point are calculated, and a real-time point cloud model is generated based on the normal vector and the principal curvature.

[0020] In an optional embodiment,

[0021] Performing local quadratic surface fitting on the effective grid feature points, calculating the normal vector and the principal curvature of each effective grid feature point, and generating a real-time point cloud model based on the normal vector and the principal curvature includes:

[0022] Calculate the point cloud density around the effective grid feature point, determine the search radius threshold according to the point cloud density, extract a local point set within the search radius threshold range with the effective grid feature point as the center; calculate the spatial Euclidean distance from each point in the local point set to the effective grid feature point, and construct a distance weight matrix; perform eigenvalue decomposition on the distance weight matrix, and determine the eigenvector corresponding to the minimum eigenvalue as the initial normal vector;

[0023] Constructing a vertical projection plane based on the initial normal vector, projecting the local point set onto the vertical projection plane to form a two-dimensional point set, triangulating the two-dimensional point set, obtaining a triangular mesh connection relationship, calculating a connection distance between adjacent points in the two-dimensional point set, performing a weighted combination of the connection distance and the spatial Euclidean distance to obtain a comprehensive distance weight, performing a weighted fitting on the two-dimensional point set through an iterative optimization algorithm, and obtaining a local quadratic surface equation;

[0024] Calculate the first-order derivative and the second-order derivative of the local quadratic surface equation at the effective grid feature points, construct a curvature matrix, solve the eigenvalue and eigenvector of the curvature matrix, determine the eigenvalue as the principal curvature, and determine the eigenvector as the principal direction; calculate the Gaussian curvature and the mean curvature according to the principal curvature, and combine the Gaussian curvature and the mean curvature to form a surface eigenvector;

[0025] The surface feature vector is input into a surface classification network to identify the surface type, a principal curvature correction coefficient is set according to the surface type, the principal curvature is multiplied by the principal curvature correction coefficient to obtain a corrected principal curvature, and a normal vector is updated based on the corrected principal curvature and the main direction; the normal vector and the corrected principal curvature are spatially filtered to obtain a final normal vector and principal curvature estimate.

[0026] In an optional embodiment,

[0027] The real-time point cloud model is aligned with the preset 3D model of the production line equipment to obtain the equipment status deviation data including:

[0028] Calculate the curvature value and normal vector of each point in the real-time point cloud model and the three-dimensional model of the production line equipment respectively, extract points whose curvature values ​​are greater than the preset curvature threshold, and determine the real-time point cloud feature points and model feature points; establish normal vector histograms and curvature histograms for the real-time point cloud feature points and the model feature points respectively, and obtain the geometric invariant features of the real-time point cloud feature points and the geometric invariant features of the three-dimensional model of the production line equipment;

[0029] Construct a KD tree search structure for the model feature points, determine the search reference tree, input the real-time point cloud feature points into the search reference tree in sequence, and search for K nearest neighbor points based on the Euclidean distance metric; calculate the cosine distance between the geometric invariant feature of each real-time point cloud feature point and the geometric invariant feature of the corresponding K nearest neighbor points as the feature similarity, select the point with the highest feature similarity as the matching corresponding point, and establish the corresponding relationship between the real-time point cloud feature point and the model feature point,

[0030] Based on the corresponding relationship, a feature point pairing set is obtained, and the spatial distance and the normal vector angle between each pair of feature points are calculated; a first error metric item is constructed based on the spatial distance, a second error metric item is constructed based on the normal vector angle, and corresponding weights are assigned according to preset weight coefficients to establish a registration cost function;

[0031] Calculate a first Jacobian matrix of the first error metric item with respect to the rotation matrix and a second Jacobian matrix with respect to the translation vector; calculate a third Jacobian matrix of the second error metric item with respect to the rotation matrix and a fourth Jacobian matrix with respect to the translation vector; combine the first Jacobian matrix, the second Jacobian matrix, the third Jacobian matrix and the fourth Jacobian matrix to obtain an overall Jacobian matrix;

[0032] The gradient value of the registration cost function is calculated based on the overall Jacobian matrix, and the annealing algorithm is used to adaptively adjust the gradient descent step size; when the gradient value of the registration cost function is less than the preset gradient threshold, the iteration is stopped, and the rotation matrix and translation vector of the real-time point cloud model to the three-dimensional model of the production line equipment are output; the rotation matrix and the translation vector are combined to generate a rigid body transformation matrix; and the real-time point cloud model is transformed into the coordinate system of the three-dimensional model of the production line equipment using the rigid body transformation matrix;

[0033] Calculate the Euclidean distance between the points in the transformed real-time point cloud model and the corresponding points in the three-dimensional model of the production line equipment; mark the points whose Euclidean distance exceeds the preset distance threshold as deviation points, record the spatial position and deviation value of the deviation points, and generate equipment status deviation data.

[0034] In an optional embodiment,

[0035] The control node areas where the operator's gaze is captured through eye tracking include:

[0036] Using a dual-wavelength infrared light source, a bright pupil image sequence and a dark pupil image sequence of the operator's eyes are collected, a difference operation is performed on the bright pupil image sequence and the dark pupil image sequence to obtain a pupil enhancement image, a grayscale gradient matrix of the pupil enhancement image is calculated, and a set of pupil edge candidate points is extracted based on the grayscale gradient matrix;

[0037] Calculating the gradient direction consistency and edge point continuity of adjacent edge points in the pupil edge candidate point set, constructing an edge point optimization function based on the gradient direction consistency and edge point continuity, executing the edge point optimization function on the pupil edge candidate point set, and obtaining an optimized pupil edge point set;

[0038] Calculate and optimize the shape regularity and time series smoothness of the pupil edge point set, construct an ellipse fitting optimization model, and solve the ellipse fitting optimization model to obtain pupil ellipse parameters;

[0039] Calculating the pupil space posture angle corresponding to the pupil ellipse parameter, extracting the time series change information of the eye corner reference feature, and building a sight line prediction model based on the pupil space posture angle and the time series change information of the eye corner reference feature;

[0040] Based on the gaze prediction model, a gaze direction output sequence is obtained, and based on the motion characteristics of the gaze direction output sequence, a state transfer matrix and an observation matrix of a recursive state estimator are constructed, and the gaze direction output sequence is filtered using the state transfer matrix and the observation matrix to obtain a filtered gaze direction sequence, and the filtered gaze direction sequence is mapped to a target plane to generate a gaze point trajectory sequence; spatial distribution characteristics and time duration characteristics are extracted from the gaze point trajectory sequence, and regional attention distribution entropy is calculated;

[0041] Based on the corresponding operation task, a preset decision threshold parameter is determined, and the attention distribution entropy is judged using the decision threshold parameter to determine the control node area that the operator is looking at.

[0042] In an optional embodiment,

[0043] Collecting the operator's skeleton key point motion data, identifying the gesture interaction operation on the circuit control panel, and converting the gesture interaction operation into circuit control instructions include:

[0044] According to a preset hand motion sampling rule, the key points of the hand are determined, and a three-dimensional coordinate sequence of the key points of the operator's hand is collected, wherein the key points of the hand include the key points of the palm and the key points of the finger joints, and a finger joint kinematic chain model is constructed, and a fixed distance between the key points of the palm and the finger joints is used as a first constraint parameter, and a relative angle between the key points of the finger joints is used as a second constraint parameter; the first constraint parameter and the second constraint parameter are input into a Kalman filter, and noise data in the three-dimensional coordinate sequence is corrected to obtain a standard three-dimensional coordinate sequence;

[0045] Constructing a time-varying graph structure based on the standard three-dimensional coordinate sequence, setting the palm key points and the finger joint key points as graph structure nodes, setting the skeletal connections between adjacent key points as graph structure edges, calculating the joint angles of the nodes at both ends of the graph structure edge, dividing the difference in joint angles corresponding to the graph structure edges of two adjacent frames by the time interval to obtain a joint angle change rate, setting the joint angle change rate as a weight coefficient of the graph structure edge, and obtaining a weighted time-varying graph structure; arranging the joint angle change rates in chronological order to form a joint angle change rate sequence;

[0046] Input the weighted time-varying graph structure into a spatial graph convolutional network to extract the topological features of the key points of the hand; input the joint angle change rate sequence into a temporal graph convolutional network to extract the dynamic features of the key points of the hand; input the topological features and the dynamic features into a variational encoder, encode the gesture sequence into a latent variable vector, construct a probability generation model for characterizing the probability distribution of gesture actions, train the probability generation model based on the latent variable vector, and optimize the probability generation model by maximizing the difference between the posterior probability and the prior probability of the latent variable vector;

[0047] The gesture sequence is segmented by conditional random fields to obtain segmented fragments. Based on the probability generation model, the interaction intention probability of each segmented fragment is calculated, and the segmented fragment with an interaction intention probability greater than a preset intention probability threshold is determined as a key interaction fragment; an interaction mapping matrix is ​​constructed to establish a correspondence between the key interaction fragment and the control area of ​​the circuit control panel; a Monte Carlo tree search method is used to perform trajectory sampling on the key interaction fragment to generate multiple groups of candidate control instruction sequences; the mapping probability of the candidate control instruction sequence in the interaction mapping matrix is ​​calculated, and the control instruction sequence with the highest mapping probability is determined as the gesture interaction operation recognition result.

[0048] In an optional embodiment,

[0049] Based on the control model, an AR-assisted control system is constructed in the edge computing unit. When a new operating status evaluation index is detected, the AR-assisted control system generates a circuit control instruction through graph neural network reasoning, and superimposes the operation guidance information corresponding to the circuit control instruction in the AR display interface. After confirmation by the operator, the circuit control instruction is sent to the production line controller, and the change of the operating status evaluation index corresponding to the execution result is fed back to update the control model, so as to realize the closed-loop control of the production line equipment, including:

[0050] Collecting operation status evaluation indicators of production line equipment, constructing a graph network structure according to the physical connection relationship between the production line equipment, using the operation status evaluation indicators as node features of the graph network structure, and calculating attention weight coefficients between adjacent nodes in the graph network structure according to the node features; performing weighted aggregation of the attention weight coefficients and the node features to obtain node fusion features; performing multi-layer feature transformation on the node fusion features to generate circuit control instructions;

[0051] Collect the spatial position information of the production line equipment, establish a real-time spatial positioning model of the production line equipment through the SLAM algorithm, parse the circuit control instructions into control target equipment information, operation part information, and operation parameter information, and based on the real-time spatial positioning model, generate a device contour projection corresponding to the control target equipment information, a position indication projection corresponding to the operation part information, and a numerical display projection corresponding to the operation parameter information in the AR display interface;

[0052] Receive the operator's confirmation signal for the projection information in the AR display interface, send the circuit control instruction to the production line controller, record the execution start time point of the circuit control instruction, and collect the first operation status evaluation index at the execution start time point; record the execution completion time point of the circuit control instruction, and collect the second operation status evaluation index at the execution completion time point; calculate the difference between the first operation status evaluation index and the second operation status evaluation index to obtain the state change index;

[0053] The state change index is multiplied by a preset weight coefficient to obtain a weighted state change index; a time sliding window is constructed, the weighted state change index is stored in the time sliding window, a variance value of the data in the time sliding window is calculated, and an adaptive weight coefficient is determined according to the variance value; the weighted state change index is multiplied by the adaptive weight coefficient to obtain a model update score;

[0054] A dynamic learning rate is calculated according to the model update score, and the dynamic learning rate decreases as the absolute value of the model update score increases; the network parameters of the graph network structure are updated based on the dynamic learning rate, and the first operating state evaluation index is respectively input into the graph network structure before and after the update, and the difference value of the circuit control instructions output twice is calculated; the weighted sum of the difference value and the model update score is used as a smoothing loss value, and the updated network parameters are adjusted based on the smoothing loss value to complete the optimization update of the graph network structure.

[0055] According to a second aspect of the embodiments of the present invention,

[0056] Provide an AR-based intelligent integrated production line dedicated circuit control system, including:

[0057] The first unit is used to collect the binocular camera images in the AR glasses worn by the operator and the spatial posture data of the inertial measurement unit, build a real-time point cloud model of the production line equipment through the spatial reconstruction algorithm, align the real-time point cloud model with the preset three-dimensional model of the production line equipment, obtain the equipment state deviation data, and use the time series convolutional network to analyze the operation trend of the production line equipment based on the equipment state deviation data to generate the operation status evaluation index and control parameter correction value;

[0058] The second unit is used to project and display the position of the circuit control node that needs to be adjusted in the AR glasses in real time according to the control parameter correction amount, obtain the control node area that the operator is looking at through eye tracking, superimpose and display the circuit control panel and the operating status evaluation index in the control node area, collect the operator's skeleton key point action data, identify the gesture interaction operation on the circuit control panel, convert the gesture interaction operation into a circuit control instruction, record the corresponding relationship between the circuit control instruction and the operating status evaluation index, and form a control model;

[0059] The third unit is used to build an AR-assisted control system in the edge computing unit based on the control model. When a new operating status evaluation index is detected, the AR-assisted control system generates a circuit control instruction through graph neural network reasoning, and superimposes and displays the operation guidance information corresponding to the circuit control instruction in the AR display interface. After confirmation by the operator, the circuit control instruction is sent to the production line controller, and the change of the operating status evaluation index corresponding to the execution result is fed back to update the control model, so as to realize closed-loop control of the production line equipment.

[0060] According to a third aspect of the embodiments of the present invention,

[0061] An electronic device is provided, comprising:

[0062] processor;

[0063] a memory for storing processor-executable instructions;

[0064] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0065] A fourth aspect of the embodiments of the present invention is:

[0066] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0067] In an embodiment of the present invention, the positions of circuit control nodes that need to be adjusted and the operating status evaluation indicators are displayed in real time through AR glasses, and combined with gesture interaction operations, circuit control can be performed directly on the production line, reducing the cumbersome operating steps and human errors in traditional control methods, and improving control efficiency and accuracy; by constructing a control model and an AR-assisted control system, combined with graph neural network reasoning, control instructions can be automatically generated according to the operating status of the equipment, and operation guidance information can be provided in the AR interface, reducing the professional skill requirements of the operator, realizing intelligent auxiliary control, and improving the automation level of production line control; by recording the correspondence between circuit control instructions and operating status evaluation indicators, and updating the control model based on execution result feedback, continuous optimization and closed-loop control of the control model are achieved, so that the control system can adaptively adjust the control strategy, thereby improving the stability and reliability of production line control. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a flow chart of a dedicated circuit control method for an intelligent integrated production line based on AR according to an embodiment of the present invention;

[0069] Figure 2 It is a structural schematic diagram of an intelligent integrated production line dedicated circuit control system based on AR according to an embodiment of the present invention. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0072] Figure 1 FIG. 1 is a flow chart of a dedicated circuit control method for an intelligent integrated production line based on AR according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0073] S101. Collect binocular camera images in the AR glasses worn by the operator and spatial posture data of the inertial measurement unit, build a real-time point cloud model of the production line equipment through a spatial reconstruction algorithm, align the real-time point cloud model with the preset three-dimensional model of the production line equipment, obtain equipment state deviation data, use a temporal convolutional network to analyze the operation trend of the production line equipment based on the equipment state deviation data, and generate operation status evaluation indicators and control parameter corrections;

[0074] First, the left eye image data and the right eye image data are collected from the binocular camera of the AR glasses. At the same time, the three-axis acceleration data and the three-axis angular velocity data are collected from the inertial measurement unit built into the AR glasses. The collected binocular images are distorted to eliminate the image deformation caused by lens distortion and obtain the corrected standard image pair.

[0075] Feature point extraction is performed on the corrected left and right eye images respectively. First, the prominent feature positions such as corners and edges in the image are detected as feature point candidates. Then, the gradient information of the local image block is extracted at each feature point position to construct a feature description vector. The feature points of the left and right eye images are matched based on the similarity of the feature description vectors to establish the corresponding relationship between the feature points.

[0076] The acceleration data and angular velocity data of the inertial measurement unit are used to calculate the position and posture changes of the AR glasses at adjacent moments. The approximate position of the feature point in the next frame image is predicted based on the position and posture changes. The matching feature points are searched in the area near the predicted position to achieve cross-frame tracking of the feature points. The corresponding relationship between the feature points of multiple consecutive frames of images is established through the tracking results.

[0077] Based on the known matching relationship between binocular camera parameters and feature points, the depth information of feature points is calculated through triangulation. The motion trajectory of feature points is established by combining the feature point tracking results in multi-frame image sequences. The camera motion is restored using the feature point trajectory, while the 3D coordinates of feature points are optimized to generate a real-time point cloud model of production line equipment.

[0078] Read the pre-established 3D model of production line equipment and extract the geometric feature points on the model surface. Pair the feature points in the real-time point cloud model with the geometric feature points of the 3D model. By iteratively optimizing the spatial transformation relationship between the two sets of feature points, accurate registration of the point cloud model and the 3D model is achieved.

[0079] According to the spatial transformation relationship after registration, the position offset and attitude deflection angle of the real-time point cloud model relative to the standard 3D model are calculated. The position offset and attitude deflection angle are combined into a state deviation vector. A time series database is constructed to store the continuously collected state deviation vectors in chronological order.

[0080] Set a time window of fixed length and extract a section of state deviation data from the time series database. Perform time series convolution operations on the state deviation data in the window to extract the change characteristics of different time scales. Through multi-layer convolution operations, the receptive field is gradually expanded to capture long-term change trends.

[0081] The extracted time series features are fed into a fully connected neural network to learn the evolution law of state deviation. The network output layer predicts the state deviation value at the future moment. The equipment operation trend is judged based on the predicted deviation value, and the operation status evaluation index including current deviation, temperature deviation, and vibration deviation is generated.

[0082] Compare the operating status evaluation index with the preset safety threshold. When the evaluation index exceeds the threshold, calculate the correction amount to be applied according to the degree of excess. Combined with the physical characteristics of the equipment and the control rules, the correction amount is converted into a specific control parameter adjustment value and output as the control parameter correction amount.

[0083] In this embodiment, based on the new operating status evaluation index, the graph neural network is used to accurately generate circuit control instructions to improve the intelligence and real-time performance of control decisions; the operation guidance information is superimposed on the AR display interface to facilitate operators to quickly understand and confirm control instructions, thereby improving operational efficiency and accuracy; by feeding back the changes in evaluation indicators corresponding to the execution results to the control model, the system performance is continuously optimized to achieve adaptive closed-loop control of production line equipment; the efficient processing capabilities of the edge computing unit are utilized to ensure low latency and high reliability to adapt to complex industrial control environments.

[0084] S102. Based on the control parameter correction amount, the position of the circuit control node that needs to be adjusted is projected and displayed in real time in the AR glasses, the control node area that the operator is looking at is obtained through eye tracking, the circuit control panel and the operating status evaluation index are superimposed and displayed in the control node area, the operator's skeleton key point action data is collected, the gesture interaction operation on the circuit control panel is identified, the gesture interaction operation is converted into a circuit control instruction, the corresponding relationship between the circuit control instruction and the operating status evaluation index is recorded, and a control model is formed;

[0085] After the operator wears AR glasses, the left and right cameras simultaneously collect images of the production line equipment. At the same time, the inertial measurement unit in the AR glasses continuously collects spatial posture data. The collected left and right eye images are preprocessed to eliminate deformation in the images so that they meet the requirements of subsequent processing. The processed image data and posture data are stored in chronological order.

[0086] Extract feature points from the processed left-eye image and right-eye image respectively. Look for obvious feature locations such as edges and corners in the image. Analyze the image area around each feature point and record the image feature information of the area. Pair and match the feature points in the left-eye image with the corresponding feature points in the right-eye image.

[0087] Based on the posture data collected by the inertial measurement unit, the position and rotation changes of the AR glasses at adjacent moments are determined. These change information is used to predict the approximate position of the feature point in the next frame of the image. Matching feature points are searched near the predicted position to establish the corresponding relationship between the feature points of the images at different moments.

[0088] The matching feature point information in the left and right eye images is used to calculate the position coordinates of these feature points in three-dimensional space. The three-dimensional feature points reconstructed at multiple times are combined to form a point cloud model representing the shape of the production line equipment. The point cloud model is optimized to remove noise points and make the model smoother and more complete.

[0089] Read the prepared 3D model file of the production line equipment. Select surface feature points from the 3D model. Align the point cloud model reconstructed in real time with the 3D model. Adjust the position and angle of the point cloud model to make it coincide with the 3D model as much as possible.

[0090] Calculate the position offset of the point cloud model relative to the standard 3D model. Measure the angle difference between the two models. Combine the position offset and angle difference data into state deviation data. Establish a time series record to store the state deviation data of continuous measurement.

[0091] Set a fixed-length observation time window. Extract a section of state deviation data from the time series record for analysis. Extract the change characteristics in this section of data. Predict the change trend of the equipment state. Generate evaluation indicators that reflect the equipment operating status based on the prediction results.

[0092] Compare the operating status evaluation index with the pre-set safety range. When the evaluation index exceeds the safety range, calculate the correction amount that needs to be adjusted. According to the control requirements of the equipment, convert the correction amount into a specific control parameter adjustment value. Output the control parameter correction amount for subsequent equipment control.

[0093] The AR glasses display interface displays the equipment's operating status information in real time. It marks the equipment parts that need special attention. It displays the current evaluation index values ​​and parameter corrections. It provides intuitive monitoring and operation guidance information for operators.

[0094] Continuously collect new image data and posture data. Update point cloud models and state deviation data. Adjust evaluation indicators and control parameters. Form a closed-loop real-time monitoring process. Save historical data for subsequent analysis and optimization.

[0095] In this embodiment, the three-dimensional point cloud model of the production line equipment is reconstructed in real time through a binocular camera and an inertial measurement unit, and the model is optimized to ensure high accuracy and integrity, laying the foundation for equipment status evaluation; the real-time point cloud model is aligned with the standard three-dimensional model, the position offset and angle difference are accurately measured, and the evaluation index of the equipment operation status is generated to achieve accurate diagnosis and prediction of the equipment status; the control parameter correction amount is calculated based on the deviation of the evaluation index, and an adjustment plan is automatically generated when it exceeds the safety range, supporting real-time closed-loop control and optimization of equipment operation; the equipment status, marking information of key parts, evaluation index values ​​and correction parameters are displayed in real time through the AR glasses interface, providing operators with clear monitoring and operation guidance to improve efficiency and safety; dynamic tracking of equipment status is achieved, and the control model is optimized in combination with historical data analysis to enhance system adaptability and intelligence.

[0096] S103. Based on the control model, an AR-assisted control system is constructed in the edge computing unit. When a new operating status evaluation index is detected, the AR-assisted control system generates a circuit control instruction through graph neural network reasoning, and superimposes and displays the operation guidance information corresponding to the circuit control instruction in the AR display interface. After confirmation by the operator, the circuit control instruction is sent to the production line controller, and the change of the operating status evaluation index corresponding to the execution result is fed back to update the control model, so as to realize closed-loop control of the production line equipment.

[0097] In this embodiment, based on the new operating status evaluation index, the graph neural network is used to accurately generate circuit control instructions to improve the intelligence and real-time performance of control decisions; the operation guidance information is superimposed on the AR display interface to facilitate operators to quickly understand and confirm control instructions, thereby improving operational efficiency and accuracy; by feeding back the changes in evaluation indicators corresponding to the execution results to the control model, the system performance is continuously optimized to achieve adaptive closed-loop control of production line equipment; the efficient processing capabilities of the edge computing unit are utilized to ensure low latency and high reliability to adapt to complex industrial control environments.

[0098] In an optional implementation, the space reconstruction algorithm includes:

[0099] Based on the binocular camera image, the binocular image sequence is decomposed according to the image frame, and the binocular image sequence is normalized to obtain the left image sequence and the right image sequence; for the left image sequence and the right image sequence, a four-layer Gaussian pyramid is constructed to generate a multi-scale image representation;

[0100] Performing Harris corner point response value calculation on each layer of the multi-scale image representation, selecting image points whose response values ​​are greater than a first preset threshold as a feature point candidate set; selecting, from the feature point candidate set, points with the largest response value within a domain range of a corresponding size centered on the feature point according to a preset domain size as final feature points; calculating the local image gradient of the final feature point, constructing a gradient direction histogram, and generating a feature description vector;

[0101] The feature description vectors in the left and right images are matched using the nearest neighbor matching method to obtain feature point matching pairs; based on the epipolar geometric constraints, the basic matrix of the feature point matching pairs is calculated, and the matching point pairs that do not meet the epipolar constraints are eliminated to obtain valid feature points;

[0102] Based on the spatial posture data, angular velocity data and acceleration data are obtained, the angular velocity data and acceleration data are aligned with the valid feature points according to the timestamps, visual reprojection error terms and inertial measurement pre-integration error terms are constructed, and feature point motion constraint equations are established;

[0103] According to the preset number of image frame extractions, the latest image frame is selected to construct a sliding window, within the sliding window, based on the feature point motion constraint equation, an objective function is constructed, the objective function is linearized, the Gauss-Newton method is used for iterative optimization, and the reprojection error of each iteration is calculated, and when the reprojection error is less than a second preset threshold, the three-dimensional spatial coordinates of the feature point are output;

[0104] Divide the space where the three-dimensional space coordinate point is located into cubic grids with a side length of a third preset value, calculate the spatial distribution characteristics of the points in each cubic grid to obtain a grid feature point set, calculate the mean of the grid feature point set, mark the grid feature points that deviate from the mean by more than a fourth preset threshold as outlier feature points, remove the outlier feature points, and obtain valid grid feature points;

[0105] A local quadratic surface fitting is performed on the effective grid feature points, a normal vector and a principal curvature of each effective grid feature point are calculated, and a real-time point cloud model is generated based on the normal vector and the principal curvature.

[0106] In a specific implementation, an image captured by a binocular camera is obtained. The image is decomposed in frames to form a left image sequence and a right image sequence. In order to eliminate the influence of different lighting conditions, the two image sequences need to be normalized, for example, the pixel values ​​are scaled to between 0 and 1.

[0107] Four layers of Gaussian pyramids are constructed for the normalized left and right image sequences. The construction method of the Gaussian pyramid is: downsample the image multiple times, and reduce the image size by half after each downsampling, so that image levels with different resolutions are formed to form a pyramid structure. Multi-scale image representation can improve the robustness of the algorithm.

[0108] On each layer of the Gaussian pyramid, the Harris corner point response value of each pixel is calculated. The Harris corner point response value is used to measure whether a pixel is a corner point. The image points with a response value greater than a first preset threshold (for example, 0.01) are taken as the feature point candidate set.

[0109] In order to avoid feature points being too dense in the candidate set of feature points, non-maximum suppression is required. The specific method is: set a preset area size (for example, 5x5 pixels), and select the point with the largest response value as the final feature point within the area of ​​the corresponding size centered on the feature point.

[0110] For each final feature point, calculate its local image gradient. The image gradient represents the speed and direction of the change of the image pixel value. Then, construct a gradient direction histogram. Divide the gradient direction into multiple intervals (for example, 8 intervals), count the cumulative value of the gradient in each interval, and form a feature description vector. This vector can be used to describe the local features of the feature point.

[0111] The feature description vectors in the left and right images are matched using the nearest neighbor matching method. The nearest neighbor matching method compares the description vector of each feature point in the left image with the description vectors of all feature points in the right image, and selects the point with the closest distance as the matching point. In this way, a feature point matching pair is obtained.

[0112] Epipolar geometry constraints are used to remove matching point pairs that do not satisfy the epipolar constraints. Epipolar geometry constraints mean that the matching feature points in the left and right images must be located on the corresponding epipolar lines. By calculating the basic matrix, it is possible to determine which matching point pairs do not satisfy the epipolar constraints and remove them. Finally, valid feature points are obtained.

[0113] Obtain angular velocity data and acceleration data through the spatial attitude sensor. Align the angular velocity data and acceleration data with the valid feature points according to the timestamp. Construct the visual reprojection error term and the inertial measurement pre-integration error term, and establish the feature point motion constraint equation. The visual reprojection error term represents the difference between the projected position of the feature point on the image and the actual position. The inertial measurement pre-integration error term represents the difference between the inertial measurement data and the visual measurement data.

[0114] Set an image frame extraction number (e.g. 10 frames), select the latest image frame to build a sliding window. In the sliding window, build the objective function based on the feature point motion constraint equation. Linearize the objective function, use the Gauss-Newton method for iterative optimization, and calculate the reprojection error of each iteration. When the reprojection error is less than the second preset threshold (e.g. 0.001), output the three-dimensional spatial coordinates of the feature point.

[0115] The space where the three-dimensional space coordinate point is located is divided into a cubic grid with a side length of a third preset value (e.g., 0.1 meters). The spatial distribution characteristics of the points in each cubic grid, such as the number of points, the average coordinates of the points, etc., are calculated to obtain a grid feature point set. The mean of the grid feature point set is calculated, and the grid feature points that deviate from the mean by more than a fourth preset threshold (e.g., 0.05 meters) are marked as outlier feature points, and they are removed to obtain valid grid feature points.

[0116] Perform local quadratic surface fitting on valid mesh feature points. Calculate the normal vector and principal curvature of each valid mesh feature point. The normal vector indicates the direction of the surface at that point, and the principal curvature indicates the degree of curvature of the surface at that point. Generate a real-time point cloud model based on the normal vector and principal curvature.

[0117] For example, in a cubic grid containing 10 points, the average coordinates of the points are (1, 2, 3), and the fourth preset threshold is 0.05. If the coordinates of a point are (1.1, 2.2, 3.3), and its distance from the average coordinates is 0.173, which is greater than the threshold 0.05, the point is marked as an outlier and removed.

[0118] In this embodiment, by fusing visual information and inertial measurement data, the movement of feature points can be effectively constrained and the accuracy of three-dimensional reconstruction can be improved; multi-scale image representation and non-maximum suppression can effectively reduce the impact of noise and mismatching and improve the robustness of the algorithm; the use of sliding windows and Gauss-Newton method can quickly optimize the objective function and realize the generation of real-time point cloud models.

[0119] In an optional implementation, performing local quadratic surface fitting on the effective grid feature points, calculating the normal vector and principal curvature of each effective grid feature point, and generating a real-time point cloud model based on the normal vector and principal curvature includes:

[0120] Calculate the point cloud density around the effective grid feature point, determine the search radius threshold according to the point cloud density, extract a local point set within the search radius threshold range with the effective grid feature point as the center; calculate the spatial Euclidean distance from each point in the local point set to the effective grid feature point, and construct a distance weight matrix; perform eigenvalue decomposition on the distance weight matrix, and determine the eigenvector corresponding to the minimum eigenvalue as the initial normal vector;

[0121] Constructing a vertical projection plane based on the initial normal vector, projecting the local point set onto the vertical projection plane to form a two-dimensional point set, triangulating the two-dimensional point set, obtaining a triangular mesh connection relationship, calculating a connection distance between adjacent points in the two-dimensional point set, performing a weighted combination of the connection distance and the spatial Euclidean distance to obtain a comprehensive distance weight, performing a weighted fitting on the two-dimensional point set through an iterative optimization algorithm, and obtaining a local quadratic surface equation;

[0122] Calculate the first-order derivative and the second-order derivative of the local quadratic surface equation at the effective grid feature points, construct a curvature matrix, solve the eigenvalue and eigenvector of the curvature matrix, determine the eigenvalue as the principal curvature, and determine the eigenvector as the principal direction; calculate the Gaussian curvature and the mean curvature according to the principal curvature, and combine the Gaussian curvature and the mean curvature to form a surface eigenvector;

[0123] The surface feature vector is input into a surface classification network to identify the surface type, a principal curvature correction coefficient is set according to the surface type, the principal curvature is multiplied by the principal curvature correction coefficient to obtain a corrected principal curvature, and a normal vector is updated based on the corrected principal curvature and the main direction; the normal vector and the corrected principal curvature are spatially filtered to obtain a final normal vector and principal curvature estimate.

[0124] In an optional specific implementation, the density of each point in the point cloud is calculated. For example, with a certain point as the center, the number of points in a small cube (for example, with a side length of 0.01 meters) is counted, and this number of points represents the density of the point. According to the calculated point cloud density, the search radius threshold is adaptively determined. In areas with higher density, the search radius can be set smaller, such as 0.05 meters; in areas with lower density, the search radius can be set larger, such as 0.1 meters. This ensures that enough neighboring points can be obtained in different density areas for local surface fitting.

[0125] With each valid grid feature point as the center, extract the local point set within its search radius threshold. Assume that the coordinates of a valid grid feature point are (1, 2, 3) and the search radius is 0.1 meters. Then, all points that are less than 0.1 meters away from this point will be extracted into the local point set.

[0126] Calculate the spatial Euclidean distance from each point in the local point set to the center point and construct a distance weight matrix. Suppose there are two points (1.01, 2.02, 3.03) and (0.98, 1.97, 2.95), and their distances to the center point (1, 2, 3) are approximately 0.037 meters and 0.054 meters respectively. The reciprocals of these two distances can be used as weights to construct a distance weight matrix.

[0127] Perform eigenvalue decomposition on the distance weight matrix and use the eigenvector corresponding to the minimum eigenvalue as the initial normal vector. Assuming that the eigenvector corresponding to the minimum eigenvalue is (0.2, 0.8, 0.56), this vector is the initial normal vector.

[0128] Construct a perpendicular projection plane based on the initial normal vector. Project the local point set onto this plane to form a two-dimensional point set. Assuming the initial normal vector is (0.2, 0.8, 0.56), the plane perpendicular to the vector can be expressed as 0.2x+0.8y+0.56z=d. Project each point in the local point set onto this plane to obtain a two-dimensional point set.

[0129] Triangulate the two-dimensional point set to obtain the connection relationship of the triangular mesh. For example, the Delaunay triangulation algorithm can be used to connect the two-dimensional point set into a triangular mesh.

[0130] Calculate the connection distance between adjacent points in the two-dimensional point set, and perform a weighted combination of the connection distance and the spatial Euclidean distance to obtain a comprehensive distance weight. Assume that the two-dimensional coordinates of two adjacent points are (0.1, 0.2) and (0.15, 0.22), and their connection distance is about 0.054. Combine this connection distance with the previously calculated spatial Euclidean distance, for example, taking a weight of 50% each, to obtain a comprehensive distance weight.

[0131] The local quadratic surface equation is obtained by weighted fitting the two-dimensional point set through an iterative optimization algorithm. For example, the least squares method can be used for fitting, and the goal is to minimize the sum of the weighted distances of all points to the fitting surface.

[0132] Calculate the first-order derivative and second-order derivative of the local quadratic surface equation at the effective grid feature points and construct the curvature matrix. The first-order derivative represents the slope of the surface, and the second-order derivative represents the curvature of the surface.

[0133] Solve the eigenvalues ​​and eigenvectors of the curvature matrix, determine the eigenvalues ​​as the principal curvatures, and the eigenvectors as the principal directions. Assuming that the eigenvalues ​​of the curvature matrix are 0.5 and 0.8, the corresponding eigenvectors are (0.6, 0.8) and (-0.8, 0.6), respectively. Then the principal curvatures are 0.5 and 0.8, and the principal directions are (0.6, 0.8) and (-0.8, 0.6), respectively.

[0134] Gaussian curvature and mean curvature are calculated based on the principal curvatures, and Gaussian curvature and mean curvature are combined to form the surface eigenvector. For example, Gaussian curvature is the product of two principal curvatures, and mean curvature is the average of the two principal curvatures.

[0135] The surface feature vector is input into the surface classification network to identify the surface type. For example, a neural network can be pre-trained to distinguish different types of surfaces such as planes, spheres, cylinders, etc.

[0136] Set the principal curvature correction factor according to the surface type. For example, for a plane, the correction factor can be set to 0; for a spherical surface, the correction factor can be set to 1.

[0137] The corrected principal curvature is obtained by multiplying the principal curvature by the principal curvature correction coefficient.

[0138] Update the normal vector based on the corrected principal curvature and principal direction.

[0139] The normal vector and the corrected principal curvature are spatially filtered to obtain final normal vector and principal curvature estimation values. For example, Gaussian filtering or bilateral filtering may be used for smoothing.

[0140] In this embodiment, through local quadratic surface fitting, the normal vector and principal curvature of the point cloud can be estimated more accurately, thereby improving the accuracy of the point cloud model and better reflecting the true shape of the object; accurate normal vector and principal curvature information can be used to calculate other geometric features of the point cloud, such as Gaussian curvature, mean curvature, etc., thereby enhancing the expressive power of the point cloud model and better describing the surface details of the object; the calculation efficiency is high, and point cloud data can be processed in real time, and high-quality point cloud models can be generated to meet the needs of real-time applications.

[0141] In an optional implementation, the real-time point cloud model is registered with a preset three-dimensional model of the production line equipment to obtain the equipment state deviation data including:

[0142] Calculate the curvature value and normal vector of each point in the real-time point cloud model and the three-dimensional model of the production line equipment respectively, extract points whose curvature values ​​are greater than the preset curvature threshold, and determine the real-time point cloud feature points and model feature points; establish normal vector histograms and curvature histograms for the real-time point cloud feature points and the model feature points respectively, and obtain the geometric invariant features of the real-time point cloud feature points and the geometric invariant features of the three-dimensional model of the production line equipment;

[0143] Construct a KD tree search structure for the model feature points, determine the search reference tree, input the real-time point cloud feature points into the search reference tree in sequence, and search for K nearest neighbor points based on the Euclidean distance metric; calculate the cosine distance between the geometric invariant feature of each real-time point cloud feature point and the geometric invariant feature of the corresponding K nearest neighbor points as the feature similarity, select the point with the highest feature similarity as the matching corresponding point, and establish the corresponding relationship between the real-time point cloud feature point and the model feature point,

[0144] Based on the corresponding relationship, a feature point pairing set is obtained, and the spatial distance and the normal vector angle between each pair of feature points are calculated; a first error metric item is constructed based on the spatial distance, a second error metric item is constructed based on the normal vector angle, and corresponding weights are assigned according to preset weight coefficients to establish a registration cost function;

[0145] Calculate a first Jacobian matrix of the first error metric item with respect to the rotation matrix and a second Jacobian matrix with respect to the translation vector; calculate a third Jacobian matrix of the second error metric item with respect to the rotation matrix and a fourth Jacobian matrix with respect to the translation vector; combine the first Jacobian matrix, the second Jacobian matrix, the third Jacobian matrix and the fourth Jacobian matrix to obtain an overall Jacobian matrix;

[0146] The gradient value of the registration cost function is calculated based on the overall Jacobian matrix, and the annealing algorithm is used to adaptively adjust the gradient descent step size; when the gradient value of the registration cost function is less than the preset gradient threshold, the iteration is stopped, and the rotation matrix and translation vector of the real-time point cloud model to the three-dimensional model of the production line equipment are output; the rotation matrix and the translation vector are combined to generate a rigid body transformation matrix; and the real-time point cloud model is transformed into the coordinate system of the three-dimensional model of the production line equipment using the rigid body transformation matrix;

[0147] Calculate the Euclidean distance between the points in the transformed real-time point cloud model and the corresponding points in the three-dimensional model of the production line equipment; mark the points whose Euclidean distance exceeds the preset distance threshold as deviation points, record the spatial position and deviation value of the deviation points, and generate equipment status deviation data.

[0148] In an optional specific implementation, the real-time point cloud model and the three-dimensional model of the production line equipment are preprocessed. For each point in the two models, the curvature value and normal vector are calculated. The curvature value describes the degree of curvature of the local surface where the point is located, and the normal vector describes the direction of the surface where the point is located. A curvature threshold is set, and points with curvature values ​​greater than this threshold are filtered out, and these points are defined as feature points. For example, if the curvature threshold is set to 0.05, points with curvature values ​​greater than 0.05 are selected as feature points. In this way, points with significant geometric features in the model, such as edges, corners, etc., can be extracted.

[0149] Next, the extracted feature points are characterized. Normal vector histograms and curvature histograms are established for real-time point cloud feature points and model feature points respectively. For example, the direction of the normal vector is divided into 32 intervals, the number of normal vectors in each interval is counted, and the normal vector histogram is constructed. Similarly, the curvature value is divided into several intervals and the curvature histogram is constructed. These histograms can be used as geometric invariant features of feature points for subsequent matching.

[0150] In order to quickly find matching points, a KD tree search structure is constructed for the model feature points. KD tree is an efficient spatial data structure that can quickly search for nearest neighbor points. The model feature points are used as nodes of the KD tree to build a search reference tree. Then, the real-time point cloud feature points are input into the search reference tree in sequence to search for K nearest neighbor points. For example, if K=5 is set, the 5 model feature points closest to each real-time point cloud feature point are searched.

[0151] Calculate the feature similarity of each real-time point cloud feature point and its K nearest neighbor points. Feature similarity is obtained by calculating the cosine distance between the geometric invariant features (normal vector histogram and curvature histogram) of two feature points. The smaller the cosine distance, the higher the feature similarity. Select the point with the highest feature similarity as the matching corresponding point, and establish the correspondence between the real-time point cloud feature point and the model feature point. For example, if the model feature point B has the highest feature similarity among the five nearest neighbors of the real-time point cloud feature point A, then pair A and B.

[0152] Based on the established correspondence, a set of feature point pairs is obtained. For each pair of feature points, the spatial distance and normal vector angle between them are calculated. The spatial distance represents the distance between two points in three-dimensional space, and the normal vector angle represents the angle between the two points on the surface. For example, the spatial distance between feature points A and B is 0.01, and the normal vector angle is 5 degrees.

[0153] In order to evaluate the accuracy of registration, a registration cost function is constructed. The first error metric is constructed based on the spatial distance of the feature point pair, and the second error metric is constructed based on the normal vector angle. The two error metrics are assigned preset weight coefficients, for example, the weight of the spatial distance is 0.8, and the weight of the normal vector angle is 0.2. The two weighted error metrics are added together to obtain the registration cost function.

[0154] In order to optimize the registration cost function, an iterative optimization method is used. The gradient value of the registration cost function with respect to the rotation matrix and the translation vector is calculated. The gradient value indicates the direction in which the cost function decreases fastest. The annealing algorithm is used to adaptively adjust the gradient descent step size to avoid falling into the local optimal solution. When the gradient value of the registration cost function is less than the preset gradient threshold, the iteration is stopped. For example, the gradient threshold is set to 0.001, and the iteration is stopped when the gradient value is less than 0.001.

[0155] After the iteration is completed, the rotation matrix and translation vector of the real-time point cloud model to the three-dimensional model of the production line equipment are output. The rotation matrix and translation vector are combined to generate a rigid body transformation matrix. Using the rigid body transformation matrix, the real-time point cloud model is transformed into the coordinate system of the three-dimensional model of the production line equipment.

[0156] Finally, calculate the Euclidean distance between the points in the transformed real-time point cloud model and the corresponding points in the 3D model of the production line equipment. Mark the points whose Euclidean distance exceeds the preset distance threshold as deviation points, and record the spatial position and deviation value of the deviation points. For example, if the distance threshold is set to 0.02, then mark the points whose Euclidean distance is greater than 0.02 as deviation points. Integrate the information of all deviation points to generate equipment status deviation data.

[0157] In this embodiment, through precise point cloud registration, the deviation between the real-time point cloud model and the three-dimensional model of the production line equipment can be accurately calculated, thereby improving the accuracy of equipment state deviation detection; using the KD tree search structure and iterative optimization algorithm, point cloud registration and deviation calculation can be completed quickly, thereby improving the efficiency of equipment state deviation detection; using the feature description method based on curvature and normal vector, it is possible to effectively deal with interference factors such as noise and occlusion, thereby enhancing the robustness of equipment state deviation detection.

[0158] In an optional implementation, obtaining the control node area where the operator is looking through eye tracking includes:

[0159] Using a dual-wavelength infrared light source, a bright pupil image sequence and a dark pupil image sequence of the operator's eyes are collected, a difference operation is performed on the bright pupil image sequence and the dark pupil image sequence to obtain a pupil enhancement image, a grayscale gradient matrix of the pupil enhancement image is calculated, and a set of pupil edge candidate points is extracted based on the grayscale gradient matrix;

[0160] Calculating the gradient direction consistency and edge point continuity of adjacent edge points in the pupil edge candidate point set, constructing an edge point optimization function based on the gradient direction consistency and edge point continuity, executing the edge point optimization function on the pupil edge candidate point set, and obtaining an optimized pupil edge point set;

[0161] Calculate and optimize the shape regularity and time series smoothness of the pupil edge point set, construct an ellipse fitting optimization model, and solve the ellipse fitting optimization model to obtain pupil ellipse parameters;

[0162] Calculating the pupil space posture angle corresponding to the pupil ellipse parameter, extracting the time series change information of the eye corner reference feature, and building a sight line prediction model based on the pupil space posture angle and the time series change information of the eye corner reference feature;

[0163] Based on the gaze prediction model, a gaze direction output sequence is obtained, and based on the motion characteristics of the gaze direction output sequence, a state transfer matrix and an observation matrix of a recursive state estimator are constructed, and the gaze direction output sequence is filtered using the state transfer matrix and the observation matrix to obtain a filtered gaze direction sequence, and the filtered gaze direction sequence is mapped to a target plane to generate a gaze point trajectory sequence; spatial distribution characteristics and time duration characteristics are extracted from the gaze point trajectory sequence, and regional attention distribution entropy is calculated;

[0164] Based on the corresponding operation task, a preset decision threshold parameter is determined, and the attention distribution entropy is judged using the decision threshold parameter to determine the control node area that the operator is looking at.

[0165] The method for obtaining the control node area where the operator is looking through eye tracking comprises the following steps:

[0166] First, a dual-wavelength infrared light source is used to illuminate the operator's eyes, and a camera is used to collect bright pupil and dark pupil image sequences. For example, infrared light sources with wavelengths of 850nm and 940nm are used to collect images at a frequency of 30 frames per second. Bright pupil images are images collected when the pupil reflects strong light under infrared light, and dark pupil images are images collected when the pupil reflects weak light.

[0167] Then, the bright pupil image and the dark pupil image corresponding to each frame are differentially calculated to obtain a pupil enhanced image. The differential operation can effectively remove noise and background interference in the eye image and highlight the pupil area. For example, the pixel value of the bright pupil image is subtracted from the pixel value of the dark pupil image to obtain a pupil enhanced image.

[0168] Next, the grayscale gradient matrix of the pupil enhanced image is calculated. The grayscale gradient matrix describes the change in the grayscale value of each pixel in the image and can be used to detect the edge of the image. For example, the Sobel operator is used to calculate the horizontal and vertical gradients of the image to obtain the gradient amplitude and direction.

[0169] Based on the grayscale gradient matrix, a set of pupil edge candidate points is extracted. Pixels whose gradient amplitude exceeds a preset threshold are selected as candidate points. For example, the threshold is set to 20, and pixels whose gradient amplitude is greater than 20 are selected as pupil edge candidate points.

[0170] Then, the gradient direction consistency and edge point continuity of adjacent edge points in the pupil edge candidate point set are calculated. Gradient direction consistency indicates whether the gradient directions of adjacent edge points are similar, and edge point continuity indicates whether the distance between adjacent edge points is close enough. For example, the cosine value of the gradient direction angle of adjacent candidate points is calculated. The closer the value is to 1, the higher the direction consistency. The Euclidean distance between adjacent candidate points is calculated. The smaller the distance, the higher the continuity.

[0171] Based on the gradient directional consistency and edge point continuity, an edge point optimization function is constructed. This function is used to evaluate the quality of candidate points and screen out the best pupil edge points. For example, the gradient directional consistency and edge point continuity are weighted and summed to obtain the optimization function. The weights can be adjusted according to actual conditions. For example, the directional consistency weight is set to 0.6 and the continuity weight is set to 0.4.

[0172] The edge point optimization function is executed on the pupil edge candidate point set to obtain an optimized pupil edge point set. For example, the candidate point with the largest optimization function value is selected as the optimized pupil edge point.

[0173] Next, the shape regularity and temporal smoothness of the optimized pupil edge point set are calculated. Shape regularity indicates whether the shape of the pupil edge is close to an ellipse, and temporal smoothness indicates whether the change of the pupil edge in the time series is smooth. For example, the fitted ellipticity of the optimized pupil edge point set is calculated. The closer the ellipticity is to 1, the higher the shape regularity. The distance between the pupil center positions of adjacent frames is calculated. The smaller the distance, the higher the temporal smoothness.

[0174] An ellipse fitting optimization model is constructed and solved to obtain pupil ellipse parameters. For example, the ellipse is fitted using the least square method to obtain the center coordinates, major and minor axis lengths, and rotation angles of the ellipse.

[0175] Then, the pupil space attitude angle corresponding to the pupil ellipse parameters is calculated. The pupil space attitude angle represents the rotation angle of the pupil in three-dimensional space. For example, the pupil rotation angle in the horizontal and vertical directions is calculated based on the length of the major and minor axes of the ellipse and the rotation angle.

[0176] Extract the temporal change information of the eye corner reference feature. The eye corner reference feature can be information such as the position and shape of the eye corner. For example, use image processing technology to extract the coordinates of the eye corner and calculate the change amount of the eye corner coordinates between adjacent frames.

[0177] Based on the time series change information of pupil space posture angle and eye corner reference features, a sight line prediction model is constructed. The model is used to predict the operator's sight line direction. For example, a linear regression model is used, with the time series change information of pupil space posture angle and eye corner reference features as input and the sight line direction as output.

[0178] Based on the gaze prediction model, a gaze direction output sequence is obtained. For example, the time series change information of the pupil spatial posture angle and the eye corner reference feature of each frame is input into the gaze prediction model to obtain the gaze direction of each frame.

[0179] Based on the motion characteristics of the sight direction output sequence, the state transfer matrix and observation matrix of the recursive state estimator are constructed. For example, assuming that the change of the sight direction follows a Gaussian distribution, the state transfer matrix and observation matrix of the Kalman filter are constructed.

[0180] The state transfer matrix and the observation matrix are used to filter the sight direction output sequence to obtain a filtered sight direction sequence. For example, a Kalman filter is used to filter the sight direction output sequence to remove noise and outliers.

[0181] The filtered sight direction sequence is mapped to the target plane to generate a gaze point trajectory sequence. For example, according to the camera parameters and the position of the target plane, the sight direction is converted into coordinates on the target plane to obtain a gaze point trajectory sequence.

[0182] Extract spatial distribution characteristics and temporal duration characteristics from the gaze point trajectory sequence and calculate the regional attention distribution entropy. For example, count the dwell time and distribution density of gaze points in different regions and calculate the regional attention distribution entropy.

[0183] Based on the corresponding operation task, a preset decision threshold parameter is determined. For example, the decision threshold parameter is set according to the complexity and accuracy requirements of the operation task.

[0184] The attention distribution entropy is judged by using the judgment threshold parameter to determine the control node area that the operator is looking at. For example, the area where the attention distribution entropy exceeds the threshold is determined as the control node area that the operator is looking at.

[0185] In this embodiment, by accurately capturing the operator's gaze point, more refined control can be achieved, especially in scenarios that require high-precision operations; through eye tracking technology, the system can better understand the operator's intentions, thereby providing a smarter and more natural way of interaction, reducing the operator's cognitive burden and improving operational efficiency; it can be widely used in various fields that require human-computer interaction, bringing new development opportunities to these fields.

[0186] In an optional implementation, collecting the operator's skeleton key point motion data, identifying the gesture interaction operation on the circuit control panel, and converting the gesture interaction operation into a circuit control instruction includes:

[0187] According to a preset hand motion sampling rule, the key points of the hand are determined, and a three-dimensional coordinate sequence of the key points of the operator's hand is collected, wherein the key points of the hand include the key points of the palm and the key points of the finger joints, and a finger joint kinematic chain model is constructed, and a fixed distance between the key points of the palm and the finger joints is used as a first constraint parameter, and a relative angle between the key points of the finger joints is used as a second constraint parameter; the first constraint parameter and the second constraint parameter are input into a Kalman filter, and noise data in the three-dimensional coordinate sequence is corrected to obtain a standard three-dimensional coordinate sequence;

[0188] Constructing a time-varying graph structure based on the standard three-dimensional coordinate sequence, setting the palm key points and the finger joint key points as graph structure nodes, setting the skeletal connections between adjacent key points as graph structure edges, calculating the joint angles of the nodes at both ends of the graph structure edge, dividing the difference in joint angles corresponding to the graph structure edges of two adjacent frames by the time interval to obtain a joint angle change rate, setting the joint angle change rate as a weight coefficient of the graph structure edge, and obtaining a weighted time-varying graph structure; arranging the joint angle change rates in chronological order to form a joint angle change rate sequence;

[0189] Input the weighted time-varying graph structure into a spatial graph convolutional network to extract the topological features of the key points of the hand; input the joint angle change rate sequence into a temporal graph convolutional network to extract the dynamic features of the key points of the hand; input the topological features and the dynamic features into a variational encoder, encode the gesture sequence into a latent variable vector, construct a probability generation model for characterizing the probability distribution of gesture actions, train the probability generation model based on the latent variable vector, and optimize the probability generation model by maximizing the difference between the posterior probability and the prior probability of the latent variable vector;

[0190] The gesture sequence is segmented by conditional random fields to obtain segmented fragments. Based on the probability generation model, the interaction intention probability of each segmented fragment is calculated, and the segmented fragment with an interaction intention probability greater than a preset intention probability threshold is determined as a key interaction fragment; an interaction mapping matrix is ​​constructed to establish a correspondence between the key interaction fragment and the control area of ​​the circuit control panel; a Monte Carlo tree search method is used to perform trajectory sampling on the key interaction fragment to generate multiple groups of candidate control instruction sequences; the mapping probability of the candidate control instruction sequence in the interaction mapping matrix is ​​calculated, and the control instruction sequence with the highest mapping probability is determined as the gesture interaction operation recognition result.

[0191] In an optional specific implementation, first, a data acquisition environment is built. The environment includes a depth camera and a data processing unit. The depth camera is used to capture three-dimensional image information of the operator's hand, and the data processing unit is responsible for processing the image information and extracting key point data.

[0192] Next, the three-dimensional coordinate sequence of the key points of the hand is collected. The operator performs preset hand movements in front of the depth camera, such as grabbing, moving, rotating, etc. The image information captured by the depth camera is transmitted to the data processing unit. The data processing unit uses open source libraries such as OpenPose to identify and extract the three-dimensional coordinates of the key points of the hand. Taking the index finger as an example, the extracted key points of the index finger include the fingertip, interphalangeal joint, and metacarpophalangeal joint. Assume that the three-dimensional coordinates of the index fingertip collected at a certain moment are (10, 20, 30), the interphalangeal joint coordinates are (12, 18, 28), and the metacarpophalangeal joint coordinates are (15, 15, 25). And so on, collect the three-dimensional coordinate sequence of the key points of all preset hand movements.

[0193] Then, the collected three-dimensional coordinate sequence is preprocessed. Since noise interference is inevitable during the acquisition process, the original data needs to be corrected. Here, the Kalman filter is used to remove noise. Taking the tip of the index finger as an example, the fixed distance between the fingertip and the interphalangeal joint (for example, 3 cm) is used as the first constraint parameter, and the relative angle between the interphalangeal joint and the metacarpophalangeal joint (for example, 15 degrees) is used as the second constraint parameter. These two parameters are input into the Kalman filter to correct the original three-dimensional coordinate sequence and obtain a standard three-dimensional coordinate sequence. For example, the corrected three-dimensional coordinates of the index fingertip are (10.1, 19.9, 29.8).

[0194] Next, construct a weighted time-varying graph structure. Set the palm key points and finger joint key points as graph structure nodes, and set the skeletal connections between adjacent key points as graph structure edges. Calculate the joint angles of the nodes at both ends of the graph structure edge, and divide the difference in joint angles corresponding to the graph structure edge of two adjacent frames by the time interval to obtain the joint angle change rate, which is set as the weight coefficient of the graph structure edge. Assuming that the joint angle between the index finger tip and the interphalangeal joint is 15 degrees at a certain moment, 16 degrees at the next moment, and the time interval is 0.1 seconds, the joint angle change rate is (16-15) / 0.1=10 degrees / second. Use this value as the weight coefficient of the graph structure edge. Arrange the joint angle change rates in chronological order to form a joint angle change rate sequence.

[0195] Then, gesture features are extracted. The constructed weighted time-varying graph structure is input into a spatial graph convolutional network (such as GCN) to extract the topological features of the key points of the hand. The joint angle change rate sequence is input into a temporal graph convolutional network (such as TCN) to extract the dynamic features of the key points of the hand. The extracted topological features and dynamic features are fused.

[0196] Then, a probabilistic generative model is constructed. The fused features are input into a variational encoder (such as VAE) to encode the gesture sequence into a latent variable vector. The probabilistic generative model is trained based on the latent variable vector, and the probabilistic generative model is optimized by maximizing the difference between the posterior probability and the prior probability of the latent variable vector so that it can represent the probability distribution of gesture actions.

[0197] Next, identify key interaction segments. Perform conditional random field (CRF) segmentation on the gesture sequence to obtain segmented segments. Based on the trained probability generation model, calculate the interaction intention probability of each segmented segment. Segmented segments with interaction intention probability greater than a preset intention probability threshold (e.g. 0.8) are identified as key interaction segments.

[0198] Construct an interaction mapping matrix to establish a correspondence between key interaction fragments and control areas of the circuit control panel. For example, the "grab" gesture corresponds to the "switch" control area, and the "move" gesture corresponds to the "brightness adjustment" control area.

[0199] Finally, a control instruction sequence is generated. The Monte Carlo Tree Search (MCTS) method is used to sample the trajectories of key interaction segments and generate multiple sets of candidate control instruction sequences. The mapping probability of the candidate control instruction sequence in the interaction mapping matrix is ​​calculated, and the control instruction sequence with the highest mapping probability is determined as the gesture interaction operation recognition result. For example, the candidate control instruction sequence corresponding to a "grab" gesture is "turn on the switch", and its mapping probability in the interaction mapping matrix is ​​0.9, so it is determined as the final control instruction.

[0200] In this embodiment, by constructing a weighted time-varying graph structure and a probability generation model, the spatiotemporal characteristics of gestures can be captured more accurately, thereby improving the accuracy of gesture recognition; by using a Kalman filter to correct noise data and using a conditional random field to segment gestures, the impact of noise and environmental changes on recognition results can be effectively reduced, thereby enhancing the robustness of the system; by converting gesture interaction operations into circuit control instructions, a more natural and intuitive human-computer interaction method can be achieved, thereby improving the user experience.

[0201] In an optional implementation, based on the control model, an AR-assisted control system is constructed in the edge computing unit. When a new operating status evaluation index is detected, the AR-assisted control system generates a circuit control instruction through graph neural network reasoning, and superimposes and displays the operation guidance information corresponding to the circuit control instruction in the AR display interface. After confirmation by the operator, the circuit control instruction is sent to the production line controller, and the change of the operating status evaluation index corresponding to the execution result is fed back to update the control model, so as to realize the closed-loop control of the production line equipment, including:

[0202] Collecting operation status evaluation indicators of production line equipment, constructing a graph network structure according to the physical connection relationship between the production line equipment, using the operation status evaluation indicators as node features of the graph network structure, and calculating attention weight coefficients between adjacent nodes in the graph network structure according to the node features; performing weighted aggregation of the attention weight coefficients and the node features to obtain node fusion features; performing multi-layer feature transformation on the node fusion features to generate circuit control instructions;

[0203] Collect the spatial position information of the production line equipment, establish a real-time spatial positioning model of the production line equipment through the SLAM algorithm, parse the circuit control instructions into control target equipment information, operation part information, and operation parameter information, and based on the real-time spatial positioning model, generate a device contour projection corresponding to the control target equipment information, a position indication projection corresponding to the operation part information, and a numerical display projection corresponding to the operation parameter information in the AR display interface;

[0204] Receive the operator's confirmation signal for the projection information in the AR display interface, send the circuit control instruction to the production line controller, record the execution start time point of the circuit control instruction, and collect the first operation status evaluation index at the execution start time point; record the execution completion time point of the circuit control instruction, and collect the second operation status evaluation index at the execution completion time point; calculate the difference between the first operation status evaluation index and the second operation status evaluation index to obtain the state change index;

[0205] The state change index is multiplied by a preset weight coefficient to obtain a weighted state change index; a time sliding window is constructed, the weighted state change index is stored in the time sliding window, a variance value of the data in the time sliding window is calculated, and an adaptive weight coefficient is determined according to the variance value; the weighted state change index is multiplied by the adaptive weight coefficient to obtain a model update score;

[0206] A dynamic learning rate is calculated according to the model update score, and the dynamic learning rate decreases as the absolute value of the model update score increases; the network parameters of the graph network structure are updated based on the dynamic learning rate, and the first operating state evaluation index is respectively input into the graph network structure before and after the update, and the difference value of the circuit control instructions output twice is calculated; the weighted sum of the difference value and the model update score is used as a smoothing loss value, and the updated network parameters are adjusted based on the smoothing loss value to complete the optimization update of the graph network structure.

[0207] First, collect the operating status evaluation indicators of the production line equipment, such as current, voltage, temperature, pressure, etc. Suppose there are two devices A and B. The collected current of device A is 10A, the voltage is 220V, and the temperature is 30°C; the current of device B is 15A, the voltage is 220V, and the temperature is 35°C. Construct a graph network structure based on the physical connection relationship between the production line equipment. Assuming that there is a physical connection between devices A and B, construct a graph network with two nodes, representing devices A and B respectively. The collected operating status evaluation indicators serve as node features of the graph network structure.

[0208] Next, the attention weight coefficients between adjacent nodes in the graph network structure are calculated based on the node features. For example, the attention weight coefficients can be calculated based on the similarity of the node features. Assuming that the current features of devices A and B are highly similar, the attention weight coefficients between them are relatively large, such as 0.8; if the similarity of other features is relatively low, the corresponding attention weight coefficients are relatively small, such as 0.2. The attention weight coefficients are weightedly aggregated with the node features to obtain the node fusion features. For example, the current, voltage, and temperature features of devices A and B are multiplied by the corresponding attention weight coefficients and then added to obtain new fusion features. Multi-layer feature transformation is performed on the node fusion features to generate circuit control instructions. For example, a multi-layer perceptron can be used to transform the fusion features and finally output control instructions, such as "adjust the current of device A to 8A".

[0209] At the same time, the spatial position information of the production line equipment is collected. For example, sensors are used to obtain the three-dimensional coordinates of equipment A and B. A real-time spatial positioning model of the production line equipment is established through the SLAM algorithm. The circuit control instructions are parsed into the control target device information, the operation part information, and the operation parameter information. For example, the instruction "adjust the current of device A to 8A" is parsed as: the control target device is A, the operation part is the current adjustment knob, and the operation parameter is 8A. Based on the real-time spatial positioning model, the device contour projection corresponding to the control target device information, the position indication projection corresponding to the operation part information, and the numerical display projection corresponding to the operation parameter information are generated in the AR display interface. For example, in the AR display interface, the outline of device A is highlighted, and an arrow indicator is displayed at the current adjustment knob position, and the numerical value "8A" is displayed next to it.

[0210] After the operator confirms the projection information in the AR display interface, the system sends the circuit control instruction to the production line controller. Record the execution start time of the circuit control instruction, and collect the first operating status evaluation index at the execution start time. For example, record the time when the instruction is issued, and collect the current value of device A at this time, assuming it is 10A. Record the execution completion time of the circuit control instruction, and collect the second operating status evaluation index at the execution completion time. For example, after the instruction is executed, collect the current value of device A, assuming it is 8.2A. Calculate the difference between the first operating status evaluation index and the second operating status evaluation index to obtain the state change index. For example, 10A-8.2A=1.8A.

[0211] Multiply the state change index by the preset weight coefficient to obtain the weighted state change index. For example, assuming the preset weight coefficient is 0.5, the weighted state change index is 1.8A*0.5=0.9A. Construct a time sliding window and store the weighted state change index in the time sliding window. Calculate the variance value of the data in the time sliding window. Determine the adaptive weight coefficient based on the variance value. For example, if the variance value is large, it means that the control effect fluctuates greatly, then reduce the adaptive weight coefficient; otherwise, increase the adaptive weight coefficient. Multiply the weighted state change index by the adaptive weight coefficient to obtain the model update score.

[0212] The dynamic learning rate is calculated according to the model update score. The dynamic learning rate decreases as the absolute value of the model update score increases. The network parameters of the graph network structure are updated based on the dynamic learning rate. The first operating state evaluation index is input into the graph network structure before and after the update, and the difference value of the circuit control instructions output twice is calculated. The weighted sum of the difference value and the model update score is used as the smoothing loss value. The updated network parameters are adjusted based on the smoothing loss value to complete the optimization update of the graph network structure.

[0213] In this embodiment, the operation guidance information is directly superimposed on the equipment through the AR display interface. The operator does not need to refer to complex circuit diagrams and operation manuals, but can directly perform operations, thereby improving operation efficiency; the AR-assisted control system can accurately indicate the operation parts and parameters, avoids human misoperation, and reduces the operation error rate; the system can adjust the control instructions in real time according to the operating status of the equipment, and continuously optimizes the control model through the feedback mechanism, thereby realizing closed-loop control of the production line equipment and improving control accuracy and stability.

[0214] Figure 2 FIG. 1 is a schematic diagram of a structure of a dedicated circuit control system for an intelligent integrated production line based on AR according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0215] The first unit is used to collect the binocular camera images in the AR glasses worn by the operator and the spatial posture data of the inertial measurement unit, build a real-time point cloud model of the production line equipment through the spatial reconstruction algorithm, align the real-time point cloud model with the preset three-dimensional model of the production line equipment, obtain the equipment state deviation data, and use the time series convolutional network to analyze the operation trend of the production line equipment based on the equipment state deviation data to generate the operation status evaluation index and control parameter correction value;

[0216] The second unit is used to project and display the position of the circuit control node that needs to be adjusted in the AR glasses in real time according to the control parameter correction amount, obtain the control node area that the operator is looking at through eye tracking, superimpose and display the circuit control panel and the operating status evaluation index in the control node area, collect the operator's skeleton key point action data, identify the gesture interaction operation on the circuit control panel, convert the gesture interaction operation into a circuit control instruction, record the corresponding relationship between the circuit control instruction and the operating status evaluation index, and form a control model;

[0217] The third unit is used to build an AR-assisted control system in the edge computing unit based on the control model. When a new operating status evaluation index is detected, the AR-assisted control system generates a circuit control instruction through graph neural network reasoning, and superimposes and displays the operation guidance information corresponding to the circuit control instruction in the AR display interface. After confirmation by the operator, the circuit control instruction is sent to the production line controller, and the change of the operating status evaluation index corresponding to the execution result is fed back to update the control model, so as to realize closed-loop control of the production line equipment.

[0218] According to a third aspect of the embodiments of the present invention,

[0219] An electronic device is provided, comprising:

[0220] processor;

[0221] a memory for storing processor-executable instructions;

[0222] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0223] A fourth aspect of the embodiments of the present invention is:

[0224] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0225] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0226] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. The AR-based intelligent integrated production line dedicated circuit control method is characterized by: include: Collect binocular camera images in the AR glasses worn by the operator and spatial posture data of the inertial measurement unit, build a real-time point cloud model of the production line equipment through the spatial reconstruction algorithm, align the real-time point cloud model with the preset three-dimensional model of the production line equipment, obtain equipment state deviation data, and use a temporal convolutional network to analyze the operation trend of the production line equipment based on the equipment state deviation data to generate operation status evaluation indicators and control parameter corrections; According to the control parameter correction amount, the position of the circuit control node that needs to be adjusted is projected and displayed in real time in the AR glasses, the control node area that the operator is looking at is obtained through eye tracking, the circuit control panel and the operating status evaluation index are superimposed and displayed in the control node area, the operator's skeletal key point action data is collected, the gesture interaction operation on the circuit control panel is identified, the gesture interaction operation is converted into a circuit control instruction, the corresponding relationship between the circuit control instruction and the operating status evaluation index is recorded, and a control model is formed; Based on the control model, an AR-assisted control system is constructed in the edge computing unit. When a new operating status evaluation index is detected, the AR-assisted control system generates a circuit control instruction through graph neural network reasoning, and superimposes and displays the operation guidance information corresponding to the circuit control instruction in the AR display interface. After confirmation by the operator, the circuit control instruction is sent to the production line controller, and the change of the operating status evaluation index corresponding to the execution result is fed back to update the control model, so as to realize closed-loop control of the production line equipment. Based on the control model, an AR-assisted control system is constructed in the edge computing unit. When a new operating status evaluation index is detected, the AR-assisted control system generates a circuit control instruction through graph neural network reasoning, and superimposes the operation guidance information corresponding to the circuit control instruction in the AR display interface. After confirmation by the operator, the circuit control instruction is sent to the production line controller, and the change of the operating status evaluation index corresponding to the execution result is fed back to update the control model, so as to realize the closed-loop control of the production line equipment, including: Collecting operation status evaluation indicators of production line equipment, constructing a graph network structure according to the physical connection relationship between the production line equipment, using the operation status evaluation indicators as node features of the graph network structure, and calculating attention weight coefficients between adjacent nodes in the graph network structure according to the node features; performing weighted aggregation of the attention weight coefficients and the node features to obtain node fusion features; performing multi-layer feature transformation on the node fusion features to generate circuit control instructions; Collect the spatial position information of the production line equipment, establish a real-time spatial positioning model of the production line equipment through the SLAM algorithm, parse the circuit control instructions into control target equipment information, operation part information, and operation parameter information, and based on the real-time spatial positioning model, generate a device contour projection corresponding to the control target equipment information, a position indication projection corresponding to the operation part information, and a numerical display projection corresponding to the operation parameter information in the AR display interface; Receive the operator's confirmation signal for the projection information in the AR display interface, send the circuit control instruction to the production line controller, record the execution start time point of the circuit control instruction, and collect the first operation status evaluation index at the execution start time point; record the execution completion time point of the circuit control instruction, and collect the second operation status evaluation index at the execution completion time point; calculate the difference between the first operation status evaluation index and the second operation status evaluation index to obtain the state change index; The state change index is multiplied by a preset weight coefficient to obtain a weighted state change index; a time sliding window is constructed, the weighted state change index is stored in the time sliding window, a variance value of the data in the time sliding window is calculated, and an adaptive weight coefficient is determined according to the variance value; the weighted state change index is multiplied by the adaptive weight coefficient to obtain a model update score; A dynamic learning rate is calculated according to the model update score, and the dynamic learning rate decreases as the absolute value of the model update score increases; the network parameters of the graph network structure are updated based on the dynamic learning rate, and the first operating state evaluation index is respectively input into the graph network structure before and after the update, and the difference value of the circuit control instructions output twice is calculated; the weighted sum of the difference value and the model update score is used as a smoothing loss value, and the updated network parameters are adjusted based on the smoothing loss value to complete the optimization update of the graph network structure.

2. The method according to claim 1, characterized in that The spatial reconstruction algorithm includes: Based on the binocular camera image, the binocular image sequence is decomposed according to the image frame, and the binocular image sequence is normalized to obtain the left image sequence and the right image sequence; for the left image sequence and the right image sequence, a four-layer Gaussian pyramid is constructed to generate a multi-scale image representation; Performing Harris corner point response value calculation on each layer of the multi-scale image representation, selecting image points whose response values ​​are greater than a first preset threshold as a feature point candidate set; selecting, from the feature point candidate set, points with the largest response value within a domain range of a corresponding size centered on the feature point according to a preset domain size as final feature points; calculating the local image gradient of the final feature point, constructing a gradient direction histogram, and generating a feature description vector; The feature description vectors in the left and right images are matched using the nearest neighbor matching method to obtain feature point matching pairs; based on the epipolar geometric constraints, the basic matrix of the feature point matching pairs is calculated, and the matching point pairs that do not meet the epipolar constraints are eliminated to obtain valid feature points; Based on the spatial posture data, angular velocity data and acceleration data are obtained, the angular velocity data and acceleration data are aligned with the valid feature points according to the timestamps, visual reprojection error terms and inertial measurement pre-integration error terms are constructed, and feature point motion constraint equations are established; According to the preset number of image frame extractions, the latest image frame is selected to construct a sliding window, within the sliding window, based on the feature point motion constraint equation, an objective function is constructed, the objective function is linearized, the Gauss-Newton method is used for iterative optimization, and the reprojection error of each iteration is calculated, and when the reprojection error is less than a second preset threshold, the three-dimensional spatial coordinates of the feature point are output; Divide the space where the three-dimensional space coordinate point is located into cubic grids with a side length of a third preset value, calculate the spatial distribution characteristics of the points in each cubic grid to obtain a grid feature point set, calculate the mean of the grid feature point set, mark the grid feature points that deviate from the mean by more than a fourth preset threshold as outlier feature points, remove the outlier feature points, and obtain valid grid feature points; A local quadratic surface fitting is performed on the effective grid feature points, a normal vector and a principal curvature of each effective grid feature point are calculated, and a real-time point cloud model is generated based on the normal vector and the principal curvature.

3. The method according to claim 2, characterized in that Performing local quadratic surface fitting on the effective grid feature points, calculating the normal vector and the principal curvature of each effective grid feature point, and generating a real-time point cloud model based on the normal vector and the principal curvature includes: Calculate the point cloud density around the effective grid feature point, determine the search radius threshold according to the point cloud density, extract a local point set within the search radius threshold range with the effective grid feature point as the center; calculate the spatial Euclidean distance from each point in the local point set to the effective grid feature point, and construct a distance weight matrix; perform eigenvalue decomposition on the distance weight matrix, and determine the eigenvector corresponding to the minimum eigenvalue as the initial normal vector; Constructing a vertical projection plane based on the initial normal vector, projecting the local point set onto the vertical projection plane to form a two-dimensional point set, triangulating the two-dimensional point set, obtaining a triangular mesh connection relationship, calculating a connection distance between adjacent points in the two-dimensional point set, performing a weighted combination of the connection distance and the spatial Euclidean distance to obtain a comprehensive distance weight, performing a weighted fitting on the two-dimensional point set through an iterative optimization algorithm, and obtaining a local quadratic surface equation; Calculate the first-order derivative and the second-order derivative of the local quadratic surface equation at the effective grid feature points, construct a curvature matrix, solve the eigenvalue and eigenvector of the curvature matrix, determine the eigenvalue as the principal curvature, and determine the eigenvector as the principal direction; calculate the Gaussian curvature and the mean curvature according to the principal curvature, and combine the Gaussian curvature and the mean curvature to form a surface eigenvector; The surface feature vector is input into a surface classification network to identify the surface type, a principal curvature correction coefficient is set according to the surface type, the principal curvature is multiplied by the principal curvature correction coefficient to obtain a corrected principal curvature, and a normal vector is updated based on the corrected principal curvature and the main direction; the normal vector and the corrected principal curvature are spatially filtered to obtain a final normal vector and principal curvature estimate.

4. The method according to claim 1, characterized in that: The real-time point cloud model is aligned with the preset 3D model of the production line equipment to obtain the equipment status deviation data including: Calculate the curvature value and normal vector of each point in the real-time point cloud model and the three-dimensional model of the production line equipment respectively, extract points whose curvature values ​​are greater than the preset curvature threshold, and determine the real-time point cloud feature points and model feature points; establish normal vector histograms and curvature histograms for the real-time point cloud feature points and the model feature points respectively, and obtain the geometric invariant features of the real-time point cloud feature points and the geometric invariant features of the three-dimensional model of the production line equipment; Construct a KD tree search structure for the model feature points, determine the search reference tree, input the real-time point cloud feature points into the search reference tree in sequence, and search for K nearest neighbor points based on the Euclidean distance metric; calculate the cosine distance between the geometric invariant feature of each real-time point cloud feature point and the geometric invariant feature of the corresponding K nearest neighbor points as the feature similarity, select the point with the highest feature similarity as the matching corresponding point, and establish the corresponding relationship between the real-time point cloud feature point and the model feature point, Based on the corresponding relationship, a feature point pairing set is obtained, and the spatial distance and the normal vector angle between each pair of feature points are calculated; a first error metric item is constructed based on the spatial distance, a second error metric item is constructed based on the normal vector angle, and corresponding weights are assigned according to preset weight coefficients to establish a registration cost function; Calculate a first Jacobian matrix of the first error metric item with respect to the rotation matrix and a second Jacobian matrix with respect to the translation vector; calculate a third Jacobian matrix of the second error metric item with respect to the rotation matrix and a fourth Jacobian matrix with respect to the translation vector; combine the first Jacobian matrix, the second Jacobian matrix, the third Jacobian matrix and the fourth Jacobian matrix to obtain an overall Jacobian matrix; The gradient value of the registration cost function is calculated based on the overall Jacobian matrix, and the annealing algorithm is used to adaptively adjust the gradient descent step size; when the gradient value of the registration cost function is less than the preset gradient threshold, the iteration is stopped, and the rotation matrix and translation vector of the real-time point cloud model to the three-dimensional model of the production line equipment are output; the rotation matrix and the translation vector are combined to generate a rigid body transformation matrix; and the real-time point cloud model is transformed into the coordinate system of the three-dimensional model of the production line equipment using the rigid body transformation matrix; Calculate the Euclidean distance between the points in the transformed real-time point cloud model and the corresponding points in the three-dimensional model of the production line equipment; mark the points whose Euclidean distance exceeds the preset distance threshold as deviation points, record the spatial position and deviation value of the deviation points, and generate equipment status deviation data.

5. The method according to claim 1, characterized in that The control node areas where the operator's gaze is captured through eye tracking include: Using a dual-wavelength infrared light source, a bright pupil image sequence and a dark pupil image sequence of the operator's eyes are collected, a difference operation is performed on the bright pupil image sequence and the dark pupil image sequence to obtain a pupil enhancement image, a grayscale gradient matrix of the pupil enhancement image is calculated, and a set of pupil edge candidate points is extracted based on the grayscale gradient matrix; Calculating the gradient direction consistency and edge point continuity of adjacent edge points in the pupil edge candidate point set, constructing an edge point optimization function based on the gradient direction consistency and edge point continuity, executing the edge point optimization function on the pupil edge candidate point set, and obtaining an optimized pupil edge point set; Calculate and optimize the shape regularity and time series smoothness of the pupil edge point set, construct an ellipse fitting optimization model, and solve the ellipse fitting optimization model to obtain pupil ellipse parameters; Calculating the pupil space posture angle corresponding to the pupil ellipse parameter, extracting the time series change information of the eye corner reference feature, and building a sight line prediction model based on the pupil space posture angle and the time series change information of the eye corner reference feature; Based on the gaze prediction model, a gaze direction output sequence is obtained, and based on the motion characteristics of the gaze direction output sequence, a state transfer matrix and an observation matrix of a recursive state estimator are constructed, and the gaze direction output sequence is filtered using the state transfer matrix and the observation matrix to obtain a filtered gaze direction sequence, and the filtered gaze direction sequence is mapped to a target plane to generate a gaze point trajectory sequence; spatial distribution characteristics and time duration characteristics are extracted from the gaze point trajectory sequence, and regional attention distribution entropy is calculated; Based on the corresponding operation task, a preset decision threshold parameter is determined, and the attention distribution entropy is judged using the decision threshold parameter to determine the control node area that the operator is looking at.

6. The method according to claim 1, characterized in that Collecting the operator's skeleton key point motion data, identifying the gesture interaction operation on the circuit control panel, and converting the gesture interaction operation into circuit control instructions include: According to a preset hand motion sampling rule, the key points of the hand are determined, and a three-dimensional coordinate sequence of the key points of the operator's hand is collected, wherein the key points of the hand include the key points of the palm and the key points of the finger joints, and a finger joint kinematic chain model is constructed, and a fixed distance between the key points of the palm and the finger joints is used as a first constraint parameter, and a relative angle between the key points of the finger joints is used as a second constraint parameter; the first constraint parameter and the second constraint parameter are input into a Kalman filter, and noise data in the three-dimensional coordinate sequence is corrected to obtain a standard three-dimensional coordinate sequence; Constructing a time-varying graph structure based on the standard three-dimensional coordinate sequence, setting the palm key points and the finger joint key points as graph structure nodes, setting the skeletal connections between adjacent key points as graph structure edges, calculating the joint angles of the nodes at both ends of the graph structure edge, dividing the difference in joint angles corresponding to the graph structure edges of two adjacent frames by the time interval to obtain a joint angle change rate, setting the joint angle change rate as a weight coefficient of the graph structure edge, and obtaining a weighted time-varying graph structure; arranging the joint angle change rates in chronological order to form a joint angle change rate sequence; Input the weighted time-varying graph structure into a spatial graph convolutional network to extract the topological features of the key points of the hand; input the joint angle change rate sequence into a temporal graph convolutional network to extract the dynamic features of the key points of the hand; input the topological features and the dynamic features into a variational encoder, encode the gesture sequence into a latent variable vector, construct a probability generation model for characterizing the probability distribution of gesture actions, train the probability generation model based on the latent variable vector, and optimize the probability generation model by maximizing the difference between the posterior probability and the prior probability of the latent variable vector; The gesture sequence is segmented by conditional random fields to obtain segmented fragments. Based on the probability generation model, the interaction intention probability of each segmented fragment is calculated, and the segmented fragment with an interaction intention probability greater than a preset intention probability threshold is determined as a key interaction fragment; an interaction mapping matrix is ​​constructed to establish a correspondence between the key interaction fragment and the control area of ​​the circuit control panel; a Monte Carlo tree search method is used to perform trajectory sampling on the key interaction fragment to generate multiple groups of candidate control instruction sequences; the mapping probability of the candidate control instruction sequence in the interaction mapping matrix is ​​calculated, and the control instruction sequence with the highest mapping probability is determined as the gesture interaction operation recognition result.

7. An AR-based intelligent integrated production line dedicated circuit control system, used to implement the method described in any one of claims 1 to 6, characterized in that: include: The first unit is used to collect the binocular camera images in the AR glasses worn by the operator and the spatial posture data of the inertial measurement unit, build a real-time point cloud model of the production line equipment through the spatial reconstruction algorithm, align the real-time point cloud model with the preset three-dimensional model of the production line equipment, obtain the equipment state deviation data, and use the time series convolutional network to analyze the operation trend of the production line equipment based on the equipment state deviation data to generate the operation status evaluation index and control parameter correction value; The second unit is used to project and display the position of the circuit control node that needs to be adjusted in the AR glasses in real time according to the control parameter correction amount, obtain the control node area that the operator is looking at through eye tracking, superimpose and display the circuit control panel and the operating status evaluation index in the control node area, collect the operator's skeleton key point action data, identify the gesture interaction operation on the circuit control panel, convert the gesture interaction operation into a circuit control instruction, record the corresponding relationship between the circuit control instruction and the operating status evaluation index, and form a control model; The third unit is used to build an AR-assisted control system in the edge computing unit based on the control model. When a new operating status evaluation index is detected, the AR-assisted control system generates a circuit control instruction through graph neural network reasoning, and superimposes and displays the operation guidance information corresponding to the circuit control instruction in the AR display interface. After confirmation by the operator, the circuit control instruction is sent to the production line controller, and the change of the operating status evaluation index corresponding to the execution result is fed back to update the control model, so as to realize closed-loop control of the production line equipment.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images