A valve state monitoring and safety interlocking method and system based on machine vision

Abstract

A valve state monitoring and safety interlocking method based on machine vision belongs to the technical field of industrial process control and safety. The steps are as follows: first, camera calibration and model establishment; second, image acquisition and pretreatment; third, key image feature extraction and state decision; fourth, opening degree calculation and state judgment; fifth, safety interlocking triggering; sixth, data feedback. The valve state monitoring and safety interlocking system realizes the valve state monitoring and safety interlocking method. The present application can realize real-time accurate measurement of valve opening displacement and angle at millimeter level or even sub-millimeter level, realize valve state monitoring, and improve the stability and reliability in the production process; by comparing the real-time opening value with the command opening value from the control system to make state judgment, generating a safety interlocking signal and transmitting it to the safety interlocking system, and finally triggering a physical safety action, a complete safety protection system is realized to avoid accidents.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of industrial process control and safety, and relates to a valve state monitoring and safety interlocking method and system based on machine vision. BACKGROUND

[0002] As a key control element in industrial fluid conveying systems, the state of the valve (such as opening, closing, and specific opening degree) is not only directly related to the continuity of the production process and the operating efficiency of the production system, but also bears key functions such as pressure control, safety isolation, and material proportioning. In the petroleum and chemical industry, energy and power, pharmaceutical industry, and other industries, any abnormal state of the valve due to misoperation, mechanical failure, or signal distortion may cause a chain reaction of production interruption, and even evolve into a catastrophic safety accident, such as medium leakage, pressure loss of control, and even explosion. To ensure safety, a series of key valve devices are introduced into safety interlocking, which automatically guides the production process to a safe state when the process parameters deviate from the normal range, so as to avoid the occurrence of catastrophic accidents.

[0003] At present, the monitoring and safety interlocking of the valve state have been widely studied. Zhong Shenghui et al. proposed a typical mechanical valve limit switch (Chinese invention patent CN202110099012.4), however, its invention method only relies on physical contact and mechanical transmission, which will inevitably cause wear of the dial, shaft, and trigger cam, and further cause jamming or trigger position drift, making it difficult to ensure reliability; at the same time, its output is only a simple on-off signal, which cannot provide the opening degree information of the valve at any intermediate position. Tan Zhuoya et al. proposed a valve position transmitter structure for nuclear power (Chinese invention patent CN202111664245.0), however, its invention method has extremely high manufacturing cost, and requires extremely precise mechanical integration and calibration with the valve actuator during installation, which also requires extremely high installation process, and maintenance or replacement often requires the entire valve to be taken offline, resulting in huge maintenance cost and time cost. In summary of existing research, the monitoring and safety interlocking of the valve state mainly rely on the following traditional technologies, which all have defects:

[0004] I. Adopting valve limit switch. A mechanical or magnetic proximity switch is installed on the valve actuator. When the valve reaches the fully open or fully closed position, the corresponding switch is triggered to output a dry contact signal. However, the disadvantage is that it can only provide a binary state (open / closed) and cannot obtain continuous opening degree information; the switch itself is prone to mechanical failure, wear, or false triggering; installation requires mechanical modification, and it cannot monitor manual valves without actuators.

[0005] II. Valve position transmitters. This type of sensor can output a continuous analog or digital signal proportional to the valve opening, representing the real-time opening of the valve. However, the disadvantages are usually high cost, the need for precise integration with the actuator, complex installation, and the risk of sensor drift and damage, resulting in high maintenance costs.

[0006] Therefore, in order to overcome the defects of the prior art, it is necessary to propose a valve state monitoring and safety interlocking method and system based on machine vision. SUMMARY

[0007] In view of the problems and deficiencies in the prior art, the present application provides a valve state monitoring and safety interlocking method and system based on machine vision, which obtains high-quality images by preprocessing the original visual images of the valve collected by the industrial camera, extracts key image features representing key components of the valve in the high-quality images using traditional algorithms and deep learning parallel paths, and performs multi-modal feature fusion and state decision to obtain the optimal valve state estimation and realize valve state monitoring. The established valve motion-pixel mapping mathematical model is used to convert the feature point coordinates output after state decision into real-time opening values, and the state is judged by comparing the real-time opening values with the command opening from the control system, a safety interlocking signal is generated and transmitted to the safety interlocking system, and finally a physical safety action is triggered to realize a complete safety protection system and prevent accidents.

[0008] In order to achieve the above purpose, the technical scheme adopted by the present application is:

[0009] A valve state monitoring and safety interlocking method based on machine vision, the valve state monitoring and safety interlocking method comprising the following steps:

[0010] First, camera calibration and model establishment. The internal and external parameters of the industrial camera are determined through industrial camera calibration and hand-eye calibration to establish the relative position relationship between the industrial camera and the valve, and the valve motion-pixel mapping model between the visual features of the valve motion components and the real-time opening is constructed for different types of valves, wherein the visual features of the valve motion components refer to pixel coordinates. Specifically as follows:

[0011] Step 1.1, industrial camera calibration. Zhang Zhengyou calibration method is adopted to obtain the intrinsic matrix K and distortion coefficient D of the industrial camera, which is used for subsequent image distortion correction and lays a foundation for accurate measurement.

[0012] The intrinsic matrix K is a matrix:

[0013] (1)

[0014] wherein, , respectively are the focal lengths of the industrial camera in x and y directions, in pixel; respectively are the optical center coordinates of the industrial camera in x and y directions, the optical center refers to the origin of the industrial camera coordinate system.

[0015] The distortion coefficient D is a matrix:

[0016] (2)

[0017] wherein, is the first radial distortion coefficient, is the second radial distortion coefficient, is the third radial distortion coefficient, the three radial distortion coefficients are determined according to the Levenberg-Marquardt nonlinear optimization algorithm; is the first tangential distortion coefficient, is the second tangential distortion coefficient, the two tangential distortion coefficients are determined according to the Levenberg-Marquardt nonlinear optimization algorithm.

[0018] Step 1.2, hand-eye calibration. Based on the internal parameter matrix K obtained in step 1.1, the conversion relationship between the valve world coordinate system and the industrial camera coordinate system is determined. Knowing each three-dimensional point and its corresponding two-dimensional projection, the pose of the industrial camera is solved by PnP algorithm, and the external parameter matrix of the industrial camera is obtained , wherein R is a rotation matrix, and t is a translation vector.

[0019] Define the point in the valve world coordinate system , the point in the industrial camera coordinate system , wherein T is the transpose symbol, and the projection relationship of the two is as follows:

[0020] (3)

[0021] wherein, respectively are the coordinates in x, y, z three-dimensional directions in the valve world coordinate system; U, V are respectively the coordinates in x, y two-dimensional directions in the industrial camera coordinate system; s is a scale factor; K is the internal parameter matrix of the industrial camera.

[0022] In actual application, the world coordinate system of the valve can be simplified, and the center point of the valve flange is set as the origin (0, 0, 0) of the world coordinate system, the axis is parallel to the movement direction of the valve stem. By measuring the actual physical distance from the optical center of the industrial camera to the origin of the world coordinate system, the actual physical distance is taken as the initial estimate of the translation vector t, which helps to solve and improve the calibration accuracy.

[0023] Step 1.3, Establishing the valve motion-pixel mapping model. Based on the industrial camera calibration results obtained in steps 1.1 and 1.2, a functional relationship between the visual features of the valve motion component and its real-time opening degree is established.

[0024] According to the different types of valves, two cases are divided:

[0025] The first case, for gate valves, stop valves and other valves with linear motion of the valve stem, first set high-contrast height visual marker points on the valve stem, then manually control the valve to move from full closing (0%) to full opening (100%), pause and collect an image every certain opening degree, and measure the actual physical height of the marker point on the valve stem at that opening degree through high-precision ruler . The number of records is 10-25 times.

[0026] Since the valve stem usually moves vertically, its physical displacement and its pixel displacement in the image are approximately linear in the case where the optical axis of the industrial camera is parallel or has a small angle with the motion direction of the valve stem, so the corresponding linear mapping model is finally established. Let the pixel vertical coordinate of the marker point when the valve is fully closed be , and when fully open , the full stroke physical displacement of the valve is , then the real-time opening degree of the valve Openness is calculated as:

[0027] (4)

[0028] where is the pixel vertical coordinate of the marker point in the current image. K is the pixel equivalent, which is obtained by fitting the calibration data:

[0029] (5)

[0030] The second case, for ball valves, butterfly valves, and faucet valves, the handwheel or actuator of which rotates, first set high-contrast angle visual marker points on the handwheel or actuator, then manually control the valve to rotate from full closing to full opening , record the real rotation angle and the corresponding image at multiple positions. Where N is the total number of turns, and the total number of the multiple positions is .

[0031] Since the opening degree of the valve is determined by the rotation angle, the corresponding angle mapping model is finally established. Let the handwheel rotate N turns to be fully open, i.e. the total angle is , then the real-time opening degree of the valve Openness is calculated as:

[0032] (6)

[0033] If the industrial camera is directly opposite the handwheel plane, the pixel angle in the image is linearly related to the real rotation angle:

[0034] (7)

[0035] wherein, is a proportional coefficient, which can be determined by calibration.

[0036] Second step, image acquisition and preprocessing. Based on the internal and external parameters of the industrial camera obtained in the first step, the original visual image of the valve is collected in real time by the industrial camera deployed near the valve, and the high-quality image is obtained by preprocessing the collected original visual image. The specific steps are as follows:

[0037] Step 2.1, image acquisition. High-resolution, high-frame-rate, global-shutter industrial cameras are used to collect original visual images of valve stems, handwheels or actuators at the moment of motion, and active lighting strategies are used to deploy professional light sources to ensure the stability and clarity of the acquisition process.

[0038] Step 2.2, image distortion correction. Based on the intrinsic matrix K and distortion coefficient D of the industrial camera obtained in step 1.1, the original visual image collected in step 2.1 is corrected for radial distortion and tangential distortion by calling the getPerspectiveTransform and warpPerspective perspective transformation functions in the opencv library, and the geometric distortion in the original image is eliminated.

[0039] Step 2.3, image optimization. Based on the distortion-corrected image obtained in step 2.2, the image is optimized to obtain a high-quality image. The optimization process includes:

[0040] Using the histogram equalization method, the image contrast is enhanced, especially the edge of the valve stem, the outline of the handwheel and the marker points and other detailed features.

[0041] Using the homomorphic filtering method to compress the gray dynamic range and suppress uneven lighting.

[0042] Using a combination of Gaussian filtering and median filtering to eliminate image noise, protect image edge information, and avoid edge blurring.

[0043] ​​Third step, key image feature extraction and state decision. Based on the high-quality image obtained in the second step, traditional algorithms and deep learning are used to detect and extract fine features in ROI, and key image features that can represent key components of the valve are extracted, where the key components of the valve refer to the valve stem, hand wheel and actuator, and the key image features refer to accurate position, posture and motion information. The key image features of the two parallel paths are fused and the state decision is executed to obtain the optimal valve state estimation and realize valve state monitoring. Specifically as follows:

[0044] Step 3.1, parallel fast detection is performed by using traditional algorithms and deep learning in parallel. Based on the pre-processed high-quality image obtained in the second step, the approximate area where the valve is located is quickly located from the whole image within milliseconds, greatly reducing the range of subsequent fine processing. Specifically as follows:

[0045] Path 1: traditional algorithm. Based on the pre-processed high-quality image obtained in the second step, Canny operator and other algorithms are used in parallel for fast edge detection, and then contour finding algorithm is used to extract all contours. According to the prior geometric features of the valve, which refer to the circular flange contour of the valve body and the length-width ratio of the rectangular housing of the actuator, the contours are quickly screened and positioned, and several candidate areas are output, and the output results of path 1 are obtained.

[0046] Path 2: deep learning. Based on the pre-processed high-quality image obtained in the second step, the high-quality image is scaled to a lower resolution and input into a lightweight convolutional neural network for fast inference, and a rough valve area heat map or bounding box is output, i.e. the output results of path 2 are obtained.

[0047] Step 3.2, ROI determination. Based on the output results of the two paths obtained in step 3.1, an optimal switching strategy is adopted to calculate a minimum rectangular area that can completely wrap the target valve, i.e. ROI, and the ROI is passed to the subsequent stage.

[0048] The optimal switching strategy is to preferentially select the output results of path 2, and when the confidence is lower than the threshold, the results of path 1 are used. The threshold range is 0.90-0.95.

[0049] Step 3.3, fine feature extraction in ROI. Based on the ROI determined in step 3.2, traditional algorithms and deep learning are used in parallel to extract rich and accurate image features from different dimensions, and the time-consuming is optimized.

[0050] Path 1: Traditional algorithm. Based on the ROI determined in step 3.2, sub-pixel level accurate positioning is performed on the visual marker points set in step 1.3 using the ORB feature detector. Let the visual marker point pixel coordinates be (x orb ,y orb ), if the marker is a specific pattern, template matching or marker detection library can be used for fast and high-precision pose estimation.

[0051] To estimate the motion speed and direction of feature points on the valve stem, hand wheel or actuator, to determine the motion trend of the valve, and ultimately obtain the key image features of the valve, the Lucas-Kanade optical flow method is used for tracking, where the feature points on the valve stem refer to the sharp corners of the valve stem nut, the connection between the top of the valve stem and the nut, and the visual marker points set, the feature points on the hand wheel refer to the edges of the hand wheel spokes, the sharp end of the hand wheel handle, and the visual marker points set, and the feature points on the actuator refer to the sharp corners of the actuator shell and the visual marker points set. The basic equation of the Lucas-Kanade optical flow method is as follows:

[0052] (8)

[0053] where, , are the image gradients in x and y directions respectively; is the time gradient; 、 are the optical flow vectors in x and y directions respectively.

[0054] The Lucas-Kanade optical flow method uses a small neighborhood of a feature point to establish a system of equations, and solves it using the least squares method to obtain the optical flow vector (u, v) of the feature point. By tracking the optical flow vectors of multiple feature points, the key image features of the valve can be obtained, and the pixel coordinates of the tracked feature points are output ; the motion trend of the valve can be accurately determined, and the valve is in a static, open or closed state.

[0055] Path 2: Deep learning. Based on the ROI determined in step 3.2, high-precision segmentation is performed.

[0056] First, the ROI region is cropped and scaled to a standard size, and input into the U-Net neural network for high-precision, pixel-level semantic segmentation. The U-Net neural network is trained by collecting a large number of valve images under different working conditions, and manually labeling the fine pixel level to form a training data set for repeated training.

[0057] Subsequently, U-Net neural network output and post-processing are performed. The U-Net neural network outputs the class probability of each pixel, including background, valve body, valve stem, handwheel, visual marker point, and generates the final segmentation mask through ArgMax operation. Subsequently, morphological closing operation is performed on the segmented target region to fill the holes, and the centroid coordinates are calculated , that is, the key image features of the valve are obtained.

[0058] Step 3.4, multi-modal feature fusion. Based on the image features obtained from the two parallel paths in step 3.3, feature alignment and comparison are performed. Specifically, the feature point pixel coordinates tracked by the traditional algorithm path are compared with the centroid coordinates segmented by the deep learning path, and the Euclidean distance d between them is calculated:

[0059] (9)

[0060] Step 3.5, state decision. Based on the Euclidean distance d obtained in step 3.4, confidence evaluation and arbitration are performed:

[0061] When the Euclidean distance d obtained in step 3.4 is less than or equal to the preset threshold, wherein the preset threshold range is , and the direction of the optical flow vector is consistent with the expected valve movement direction, it is determined that the results of the two parallel paths are consistent, and the confidence result is recorded as "high confidence state". The final output feature point coordinates are obtained by weighted average of the feature point pixel coordinates tracked by the traditional algorithm path and the centroid coordinates segmented by the deep learning path:

[0062] (10)

[0063] (11)

[0064] wherein x and y are the x-axis and y-axis coordinates of the final output feature point, respectively; is the weighting coefficient of the traditional algorithm, is the weighting coefficient of the deep learning.

[0065] When the Euclidean distance d obtained in step 3.4 is greater than the preset threshold, wherein the preset threshold range is , the arbitration mechanism is started. The internal confidence of the two parallel paths is checked, wherein the internal confidence of the traditional algorithm path is derived from the response strength of the feature points on the valve stem, handwheel or actuator or the continuity of optical flow tracking, and the internal confidence of the deep learning path is derived from the area size of the segmentation region output by the deep learning. ​

[0066] The rules of the arbitration mechanism are as follows:

[0067] Since the deep learning path is usually given higher weight, if the traditional algorithm path has low confidence, the feature point on the valve stem, hand wheel or actuator is accidentally blocked or overwhelmed by strong light, the centroid coordinates obtained by adopting the deep learning path segmentation As the final output feature point coordinates :

[0068] (12)

[0069] At the same time, the confidence result is recorded as "traditional algorithm path feature tracking failure".

[0070] Conversely, if the deep learning segmentation area is too small or the shape is abnormal, the feature point pixel coordinates tracked by the traditional algorithm path As the final output feature point coordinates :

[0071] (13)

[0072] At the same time, the confidence result is recorded as "low confidence of deep learning path segmentation".

[0073] Step 3.6, based on the final output feature point coordinates and the confidence result obtained in step 3.5, the output is calculated to obtain the optimal valve state estimation, and the valve state monitoring is realized.

[0074] Fourth step, opening degree calculation and state judgment. Based on the final output feature point coordinates obtained in the third step, according to the valve motion-pixel mapping model established in the first step, the real-time opening degree of the valve is calculated, and the final safety decision signal is output through state judgment. The specific steps are as follows:

[0075] Step 4.1, real-time opening degree calculation. Based on the final output feature point coordinates obtained in the third step, according to the valve motion-pixel mapping model established in the first step, it is converted into the real-time opening degree value Openness of the valve.

[0076] The first case, for gate valves, stop valves and other valves with straight-line motion of the valve stem, the conversion formula is shown in formulas (4) and (5).

[0077] The second case, for ball valves, butterfly valves, and plug valves, the hand wheel or actuator is a rotary motion valve, and the conversion formula is shown in formula (6). Through the final output feature point coordinates relative to the coordinates of the rotation center, the angle between it and the reference direction is calculated The real rotation angle is calculated by using formula (7) .

[0078] Step 4.2, state judgment. Based on the real-time opening degree of the valve obtained in step 4.1, the calculated real-time opening degree is compared with the instruction opening degree from the control system, and the system preset safety operation interval and alarm threshold are obtained, and a safety decision signal is obtained, which is specifically as follows:

[0079] If , the safety decision signal is "non-instructional displacement". Wherein is the allowable error, and the value range is 1%~3%.

[0080] If , the safety decision signal is "opening degree over limit".

[0081] If the system detects that a person is close to the valve through the human body recognition algorithm, and the valve opening degree changes, but the control system does not issue an instruction, the safety decision signal is "misoperation".

[0082] Step 5, safety interlock triggering. Based on the safety decision signal obtained in step 4, a safety interlock signal is generated, which is transmitted to the safety interlock system in time, and finally triggers the physical safety action. Specifically as follows:

[0083] Step 5.1, safety interlock signal generation. Based on the safety decision signal obtained in step 4, once the state is judged to be abnormal, the lightweight safety logic solver of the safety interlock system immediately generates a safety interlock signal.

[0084] Step 5.2, signal transmission. Based on the safety interlock signal generated in step 5.1, it is transmitted to the controller of the safety interlock system through the field bus and the standard protocol.

[0085] Step 5.3, trigger action. The safety function logic decision of the controller of the safety interlock system in step 5.2 is executed, and the signal is output to trigger the physical safety action, so that the valve returns to the safe state. The safety function logic decision is specifically as follows:

[0086] If the safety decision signal is "non-instructional displacement", trigger interlock, send alarm, and prompt control failure.

[0087] If the safety decision signal is "opening degree over limit", trigger the highest level interlock, send high-level alarm, and stop the pump or immediately close the reactor feed.

[0088] Step 6, data feedback. After the safety interlock is triggered based on step 5, the safety interlock system sends an alarm event, including the abnormal screenshot, the type of abnormality, the timestamp (the time of abnormality occurrence and interlock triggering), and the handling measures. After the interlock action is completed, the safety interlock system feeds back a "interlock executed" status signal, which is used to perfect the entire valve state monitoring and safety interlock state mechanism management.

[0089] A valve state monitoring and safety interlock system based on machine vision, which realizes the above-mentioned valve state monitoring and safety interlock method based on machine vision, the valve state monitoring and safety interlock system comprising an initial measurement module, a collection and preprocessing module, a data analysis and decision module, a state judgment module, a safety interlock and execution module, and a data feedback and management module. Specifically:

[0090] The initial measurement module is composed of a high-precision calibration board, a three-dimensional optical measurement system, and a length and angle measurement system, which is used to accurately measure the spatial pose relationship between the valve, the calibration board and the industrial camera, to establish an accurate valve motion-pixel mapping model, and is also the calibration cornerstone of the entire valve state monitoring and safety interlock system.

[0091] The collection and preprocessing module accesses the initial measurement module and is composed of a high-resolution industrial CCD / CMOS industrial camera with a wide-angle lens and a special lighting device, which is used to collect the original visual image of the valve and perform distortion correction and optimization processing, and is also the perception hub of the entire valve state monitoring and safety interlock system.

[0092] The data analysis and decision module accesses the collection and preprocessing module and the initial measurement module and is composed of a computer vision library, a training inference engine and a high-performance industrial computing platform, which is used to run all algorithms in the above-mentioned method, realize traditional algorithm calculation, deep learning inference, multi-modal feature fusion and state decision, get the optimal valve state estimation, realize valve state monitoring, and is also the decision center of the entire valve state monitoring and safety interlock system.

[0093] The state judgment module accesses the data analysis and decision module, the collection and preprocessing module and the initial measurement module and is composed of a high-performance industrial computing platform, which is used to calculate the real-time opening degree of the valve, make the final state judgment and safety decision, and is also the judgment core of the entire valve state monitoring and safety interlock system.

[0094] The safety interlock and execution module accesses the state judgment module, the data analysis and decision module, the acquisition and preprocessing module and the initial measurement module, is composed of a lightweight safety logic solver, a controller, a field bus and a standard protocol, is used to receive the safety decision signal output by the state judgment module, and generates a safety interlock signal, finally triggers a physical safety action, and is also a safety barrier of the entire valve state monitoring and safety interlock system.

[0095] The data feedback and management module accesses the safety interlock and execution module, the state judgment module, the data analysis and decision module, the acquisition and preprocessing module and the initial measurement module, is composed of a high-performance industrial computing platform, is used to send an alarm event, forms a complete intelligent solution that is traceable, analyzable, optimizable and integrable, provides a comprehensive and reliable decision basis, improves the long-term stability and reliability of the entire valve state monitoring and safety interlock system, and is also the information hub of the entire valve state monitoring and safety interlock system.

[0096] The beneficial effects of the present application are:

[0097] (1) The present application realizes real-time and accurate measurement of millimeter-level or even sub-millimeter-level displacement and angle of the valve opening by preprocessing the valve visual image collected by the industrial camera, adopting two parallel paths of traditional algorithm and deep learning to extract key image features representing the valve state in the image and performing multi-modal feature fusion and state decision, so as to obtain the optimal state estimation of the valve and realize valve state monitoring. At the same time, since the traditional algorithm is fast and accurate but easy to be disturbed and invalid, the deep learning is strong in anti-interference but large in calculation amount and slow in calculation speed, and the two parallel paths make the traditional algorithm and deep learning complement each other, greatly improving the stability and reliability in the production process.

[0098] (2) The present application converts the feature point coordinates output after state decision into real-time opening degree value of the valve by using the established valve motion-pixel mapping mathematical model, compares the real-time opening degree value with the command opening degree from the control system to judge the state, generates a safety interlock signal and transmits it to the safety interlock system, finally triggers a physical safety action, realizes a complete safety protection system, and thus avoids accidents. BRIEF DESCRIPTION OF DRAWINGS

[0099] Figure 1 It is a valve state monitoring and safety interlock method flowchart based on machine vision.

[0100] Figure 2 It is a valve state monitoring and safety interlock system structure diagram based on machine vision. DETAILED DESCRIPTION

[0101] The present application will be further described below in combination with specific implementation cases.

[0102] A valve state monitoring and safety interlocking method based on machine vision, comprising the following steps:

[0103] First, camera calibration and model establishment. The internal and external parameters of the industrial camera are determined through industrial camera calibration and hand-eye calibration, the relative position relationship between the industrial camera and the valve is established, and the valve movement-pixel mapping model between the visual features of the valve movement components and the real-time opening degree of different types of valves is constructed, wherein the visual features of the valve movement components refer to pixel coordinates. Specifically as follows:

[0104] Step 1.1, industrial camera calibration. Zhang Zhengyou calibration method is adopted to obtain the intrinsic matrix K and distortion coefficient D of the industrial camera, which is used for subsequent image distortion correction and lays a foundation for accurate measurement.

[0105] The intrinsic matrix K is a matrix, as shown in formula (1);

[0106] (1)

[0107] Wherein, , are the focal lengths of the industrial camera in x and y directions, respectively, with the unit of pixel; are the optical center coordinates of the industrial camera in x and y directions, respectively, and the optical center refers to the origin of the industrial camera coordinate system.

[0108] The distortion coefficient D is a matrix, as shown in formula (2).

[0109] (2)

[0110] Wherein, is the first radial distortion coefficient, is the second radial distortion coefficient, is the third radial distortion coefficient, and the three radial distortion coefficients are determined according to Levenberg-Marquardt nonlinear optimization algorithm; is the first tangential distortion coefficient, is the second tangential distortion coefficient, and the two tangential distortion coefficients are determined according to Levenberg-Marquardt nonlinear optimization algorithm.

[0111] Taking a key manual gate valve at the inlet of a raw material pump in a chemical plant as an example, the industrial camera is calibrated by using a chessboard calibration plate near the valve, and the intrinsic matrix K and the distortion coefficient D are obtained.

[0112] Step 1.2, hand-eye calibration. Based on the intrinsic matrix K obtained in step 1.1, the conversion relationship between the valve world coordinate system and the industrial camera coordinate system is determined. Given each three-dimensional point and its corresponding two-dimensional projection, the pose of the industrial camera is solved by the PnP algorithm, and the extrinsic matrix of the industrial camera is obtained , where R is the rotation matrix and t is the translation vector.

[0113] Define the point in the valve world coordinate system , the point in the industrial camera coordinate system , where , and their projection relationship is shown in equation (3).

[0114] (3)

[0115] , where are the coordinates in the x, y, z three-dimensional directions of the valve world coordinate system respectively; U, V are the coordinates in the x, y two-dimensional directions of the industrial camera coordinate system respectively; s is the scale factor; K is the intrinsic matrix of the industrial camera.

[0116] In actual application, the world coordinate system of the valve can be simplified. The center point of the valve flange can be set as the origin (0, 0, 0) of the world coordinate system, the axis is parallel to the motion direction of the valve stem. By measuring the actual physical distance from the optical center of the industrial camera to the origin of the world coordinate system, the actual physical distance is taken as the initial estimate of the translation vector t, which helps to solve and improve the calibration accuracy.

[0117] Taking a key manual gate valve at the inlet of a raw material pump in a chemical plant as an example, a high-precision chessboard calibration board is fixed on the valve flange, and a laser tracker is used to measure the world coordinates of the corner points of the calibration board , the industrial camera takes pictures and extracts the pixel coordinates of the corner points , using the PnP algorithm, combined with the known intrinsic matrix K of the industrial camera, the rotation matrix R and the translation vector t of the industrial camera relative to the valve world coordinate system can be solved, and the real coordinate back calculation comparison is carried out to ensure the calibration accuracy.

[0118] Step 1.3, establish the valve motion-pixel mapping model. Based on the industrial camera calibration results obtained in steps 1.1 and 1.2, a function relationship between the visual features of the valve motion components and their real-time opening degree is established.

[0119] According to the different types of valves, it can be divided into two cases:

[0120] The first case, for the valve stem for linear motion of the valve, first set the valve stem high contrast height visual marker point, then manually control the valve from the full close (0%) to the full open (100%), every certain degree pause and collect an image, and through high precision ruler to measure the actual physical height of the marker point on the valve stem at that opening .

[0121] Because the valve stem usually moves vertically, its physical displacement and its pixel displacement in the image are approximately linear when the optical axis of the industrial camera is parallel to the direction of motion of the valve stem or the angle is very small, so the corresponding linear mapping model is finally established. When the valve is fully closed, the pixel vertical coordinate of the marker point is , when it is fully open, it is , and the full stroke physical displacement of the valve is , then the real-time opening of the valve Openness is calculated by formula (4) and formula (5).

[0122] (4)

[0123] wherein, is the pixel vertical coordinate of the marker point in the current image. K is the pixel equivalent, which is obtained by fitting the calibration data:

[0124] (5)

[0125] The second case, for the valve stem for linear motion of the valve, first set the valve stem high contrast height visual marker point, then manually control the valve from the full close (0%) to the full open (100%), every certain degree pause and collect an image, and through high precision ruler to measure the actual physical height of the marker point on the valve stem at that opening . , record the real rotation angle and the corresponding image at multiple positions. Wherein N is the total number of turns, and the total number of the multiple positions is .

[0126] Because the opening of the valve is determined by the rotation angle, the corresponding angle mapping model is finally established. Set the hand wheel to rotate N turns to be fully open, that is, the total angle is , then the real-time opening of the valve Openness is calculated by formula (6) and formula (7).

[0127] (6)

[0128] If the industrial camera is directly opposite the hand wheel plane, the pixel angle in the image and the real rotation angle are in linear relationship:

[0129] (7)

[0130] wherein, is a proportional coefficient, which can be determined by calibration.

[0131] Taking a key manual gate valve at the inlet of a raw material pump in a chemical plant as an example, a 2 million pixel explosion-proof industrial camera can be selected, which is installed on a stand about 2 meters away from the valve, and the visual angle can completely cover the valve stem and its scale. At the top of the valve stem, a bright yellow fluorescent marker tape with a width of L = 20 mm is wound. At the full-closed and full-open positions of the valve, the physical displacement of the valve stem is measured by a high-precision ruler to obtain the full physical stroke . The valve is manually opened from full-closed to full-open, and an image is collected every 10% opening degree. In each image, the pixel coordinates of the lower edge of the marker tape are accurately measured . Taking the full-closed position as the origin (0%), the pixel coordinates at the full-closed position are . Through linear regression fitting, the pixel equivalent is obtained, so the real-time opening degree of the valve can be solved.

[0132] Second step, image acquisition and preprocessing. Based on the internal and external parameters of the industrial camera obtained in the first step, the original visual image of the valve is collected in real time by the industrial camera deployed near the valve, and the high-quality image is obtained by preprocessing the collected original visual image. Specifically as follows:

[0133] Step 2.1, image acquisition. A high-resolution, high-frame-rate, global-shutter industrial camera is used to collect the original visual image of the valve stem, hand wheel or actuator at the moment of movement, and a professional light source is deployed to ensure the stability and clarity of the collection process.

[0134] Step 2.2, image distortion correction. Based on the intrinsic matrix K and distortion coefficient D of the industrial camera obtained in step 1.1, the original visual image collected in step 2.1 is corrected for radial distortion and tangential distortion by calling the two perspective transformation functions getPerspectiveTransform and warpPerspective in the opencv library, to eliminate the geometric distortion in the original image.

[0135] Step 2.3, image optimization. Based on the distortion-corrected image obtained in step 2.2, the high-quality image is obtained by optimizing the image. The optimization process includes:

[0136] Using the histogram equalization method to enhance the contrast of the image, especially to enhance the details such as the edge of the valve stem, the outline of the hand wheel and the marker points.

[0137] Using the homomorphic filtering method to compress the gray scale dynamic range and suppress uneven lighting.

[0138] The combination strategy of Gaussian filter and median filter is adopted to eliminate image noise, protect edge information of the image, and avoid edge blur.

[0139] Taking a key manual gate valve at the inlet of a raw material pump in a chemical plant as an example, an industrial camera collects images at a rate of 5 fps. The images are first corrected for distortion using K and D. Due to large changes in light in the device area, a limited contrast adaptive histogram equalization algorithm can be used for image enhancement, and then Gaussian filtering is used to eliminate noise.

[0140] Third step, key image feature extraction and state decision. Based on the high-quality image obtained after optimization processing in the second step, traditional algorithms and deep learning are used to extract key image features that can represent key components of the valve, such as valve stem, handwheel, and actuator, through two parallel paths of parallel rapid detection and ROI fine feature extraction. The key image features of the two parallel paths are fused and the state decision is executed to obtain the optimal valve state estimation and realize valve state monitoring. The specific steps are as follows:

[0141] Step 3.1, two parallel paths of traditional algorithms and deep learning are used for parallel rapid detection. Based on the pre-processed high-quality image obtained in the second step, the approximate area of the valve is quickly located from the entire image within milliseconds, greatly reducing the range of subsequent fine processing. The specific steps are as follows:

[0142] Path 1: traditional algorithm. Based on the pre-processed high-quality image obtained in the second step, Canny operator and other rapid edge detection algorithms are used in parallel to extract all contours. According to the prior geometric features of the valve, such as the circular flange contour of the valve body and the length-width ratio of the rectangular housing of the actuator, the contours are quickly screened and positioned, and several candidate regions are output, obtaining the output results of path 1.

[0143] Path 2: deep learning. Based on the pre-processed high-quality image obtained in the second step, the high-quality image is scaled to a lower resolution and input into a lightweight convolutional neural network for rapid inference, outputting a rough valve region heat map or bounding box, i.e. the output results of path 2.

[0144] Step 3.2, ROI determination. Based on the output results of the two paths obtained in step 3.1, a minimum rectangular region that can completely wrap the target valve, i.e. ROI, is calculated using a selection strategy, and this ROI is passed to the subsequent stage.

[0145] The preferred switching strategy selection mode is: preferentially selecting the output result of path 2, and when the confidence degree is lower than the threshold value, the result of path 1 is adopted; the threshold value ranges from 0.90 to 0.95, and 0.92 is taken in the embodiment.

[0146] Step 3.3, fine feature extraction in ROI. Based on the ROI determined in step 3.2, two parallel paths of traditional algorithm and deep learning are simultaneously executed.

[0147] Path 1: traditional algorithm. Based on the ROI determined in step 3.2, the ORB feature detector is used to perform sub-pixel level fine positioning on the visual marker points set in step 1.3. Let the visual marker point pixel coordinates be If the marker is a preset specific pattern, template matching or marker detection library can be used for fast and high-precision pose estimation.

[0148] In order to estimate the motion speed and direction of the feature points on the valve stem, hand wheel or actuator, judge the motion trend of the valve, and finally obtain the key image features of the valve, the Lucas-Kanade optical flow method is used for tracking, in which the feature points on the valve stem refer to the sharp corners of the valve stem nut, the connection between the top of the valve stem and the nut, and the set visual marker points, the feature points on the hand wheel refer to the edges of the hand wheel spokes, the sharp end of the hand wheel handle, and the set visual marker points, and the feature points on the actuator refer to the sharp corners of the actuator shell and the set visual marker points. The basic equation of the Lucas-Kanade optical flow method is shown in formula (8).

[0149] (8)

[0150] wherein, , are the image gradients in x and y directions respectively; is the time gradient; , are the optical flow vectors in x and y directions respectively.

[0151] The Lucas-Kanade optical flow method uses a small neighborhood of a feature point to establish a system of equations, and solves it by least squares method to obtain the optical flow vector of the feature point . By tracking the optical flow vectors of multiple feature points, the key image features of the valve can be obtained, and the pixel coordinates of the tracked feature points are output ; the motion trend of the valve and whether the valve is in a static, open or closed state can be accurately judged.

[0152] Path 2: deep learning. Based on the ROI determined in step 3.2, high-precision segmentation is performed.

[0153] Firstly, the ROI region is cropped and scaled to a standard size, and input into the U-Net neural network for high-precision, pixel-level semantic segmentation. The U-Net neural network is trained repeatedly by collecting a large number of valve images under different working conditions, and manually labeling the fine pixel level to form a training data set.

[0154] Subsequently, U-Net neural network output and post-processing are performed. The U-Net neural network outputs the class probability of each pixel, including background, valve body, valve stem, hand wheel, and visual marker point, and generates the final segmentation mask through ArgMax operation. Subsequently, morphological closing operation is performed on the segmented target region to fill the holes, and the centroid coordinates are calculated , that is, the key image features of the valve are obtained.

[0155] Step 3.4, multi-modal feature fusion. Based on the image features obtained from the two parallel paths in step 3.3, feature alignment and comparison are performed, specifically: the feature point pixel coordinates tracked by the traditional algorithm path are compared with the centroid coordinates obtained by the deep learning path segmentation, and the Euclidean distance d between them is calculated by formula (9).

[0156] (9)

[0157] Step 3.5, state decision. Based on the Euclidean distance d obtained in step 3.4, confidence evaluation and arbitration are performed:

[0158] When the Euclidean distance d obtained in step 3.4 is less than or equal to the preset threshold, and the direction of the optical flow vector is consistent with the expected valve movement direction, it is determined that the results of the two parallel paths are consistent, and the confidence result is recorded as "high confidence state". The final output feature point coordinates are obtained by weighted average of the feature point pixel coordinates tracked by the traditional algorithm path and the centroid coordinates obtained by the deep learning path segmentation:

[0159] (10)

[0160] (11)

[0161] wherein, are the x-axis and y-axis coordinates of the final output feature point, respectively; is the weighting coefficient of the traditional algorithm, is the weighting coefficient of deep learning. Taking a key manual gate valve at the inlet of a raw material pump in a chemical plant as an example, ,

[0162] ​When the Euclidean distance d obtained in step 3.4 is greater than a preset threshold, wherein the preset threshold ranges from 0 to 1 , an arbitration mechanism is started. The internal confidence of the two parallel paths is checked, wherein the internal confidence of the traditional algorithm path is derived from the response strength of the feature points on the valve stem, hand wheel or actuator or the continuity of optical flow tracking, and the internal confidence of the deep learning path is derived from the area size of the segmented region of the deep learning output.

[0163] The rules of the arbitration mechanism are as follows:

[0164] Since the deep learning path is usually given a higher weight, if the confidence of the traditional algorithm path is low, the feature points on the valve stem, hand wheel or actuator are accidentally blocked or submerged in strong light, the centroid coordinates segmented by the deep learning path are adopted as the final output feature point coordinates , as shown in formula (12). At the same time, the confidence result is recorded as “traditional algorithm path feature tracking failure”. Formula (12) is:

[0165] (12)

[0166] On the contrary, if the area of the deep learning segmented region is too small or the shape is abnormal, the feature point pixel coordinates tracked by the traditional algorithm path are adopted as the final output feature point , as shown in formula (13). At the same time, the confidence result is recorded as “low confidence of deep learning path segmentation”. Formula (13) is:

[0167] (13)

[0168] Step 3.6, based on the final output feature point coordinates and confidence result obtained in step 3.5, the output is calculated to obtain the optimal valve state estimation, and the valve state monitoring is realized.

[0169] Taking a key manual gate valve at the inlet of a raw material pump in a chemical plant as an example, a method combining traditional algorithm and deep learning is adopted:

[0170] 5000 pictures of the valve under different illuminations, different angles and partial occlusion (such as being stained with oil) are pre-collected, three types of pixels, “valve stem”, “marking tape” and “background”, are labeled, and a U-Net neural network is trained. The neural network can accurately segment the valve stem and marking tape regions from the image, and even if the marking tape is partially stained, the approximate region can be inferred from the overall position of the valve stem. Subsequently, in the “marking tape” region segmented by the neural network, the Canny operator is used to extract the contour of the lower edge, and the least squares method is used to fit a straight line. The midpoint of the straight line is taken as the final output feature point pixelThe feature point coordinates used for this measurement are as follows.

[0171] The valve stem features are extracted through two parallel paths. The traditional algorithm accurately locates the valve stem nut corner point, and the deep learning U-Net neural network segments the overall valve stem region. The Euclidean distance d between the coordinates obtained by the traditional algorithm and the coordinates obtained by deep learning is calculated, and confidence evaluation and arbitration are performed (the preset threshold is set to 5 pixels). If the distance d is less than or equal to the preset threshold, the results of the two parallel paths are consistent, the system confidence is high, the results are fused, and the final valve stem top end coordinates are output.

[0172] Step 4, opening degree calculation and state judgment. Based on the final output feature point coordinates obtained in step 3, according to the valve movement-pixel mapping model established in step 1, the real-time opening degree of the valve is calculated, and the final safety decision signal is output through state judgment. The specific steps are as follows:

[0173] Step 4.1, real-time opening degree calculation. Based on the final output feature point coordinates obtained in step 3 , according to the valve movement-pixel mapping model established in step 1, it is converted into the real-time opening degree value Openness of the valve.

[0174] The first case is for gate valves, stop valves and other valves with straight-line motion of the valve stem. The conversion formula is shown in formulas (4) and (5).

[0175] The second case is for ball valves, butterfly valves, and other valves with rotary motion of the handwheel or actuator. The conversion formula is shown in formula (6). Through the final output feature point coordinates relative to the coordinates of the rotation center, the angle with the reference direction is calculated, and the real rotation angle is calculated using formula (7).

[0176] Step 4.2, state judgment. Based on the valve real-time opening degree obtained in step 4.1, the calculated real-time opening degree is compared with the command opening degree from the control system, the system preset safety operation interval and the alarm threshold are obtained, and the safety decision signal is obtained. The safety decision signal is as follows:

[0177] If , the safety decision signal is "non-instructional displacement". Wherein is the allowable error, the value range is 1%~3%.

[0178] If , the safety decision signal is "opening degree over limit".

[0179] If the system detects that a person is approaching the valve through the human body recognition algorithm, and the valve opening changes, but the control system does not issue instructions, the safety decision signal is "misoperation".

[0180] Take the key manual gate valve at the inlet of the raw material pump in a chemical plant as an example. The real-time opening calculated at a certain moment

[0181] Obviously, , the opening is judged to be "out of limit".

[0182] Step 5, safety interlock triggering. Based on the safety decision signal obtained in step 4, a safety interlock signal is generated, which is transmitted to the safety control system in time and finally triggers the physical safety action, thus forming a complete safety protection system. The specific steps are as follows:

[0183] Step 5.1, safety interlock signal generation. Based on the safety decision signal obtained in step 4, once the state is judged to be abnormal, the lightweight safety logic solver of the safety interlock system immediately generates a safety interlock signal.

[0184] Step 5.2, signal transmission. Based on the safety interlock signal generated in step 5.1, the signal is transmitted to the controller of the safety interlock system through the field bus and standard protocol.

[0185] Step 5.3, triggering action. The safety function logic decision of the controller of the safety interlock system in step 5.2 is executed, and the signal is output to trigger the physical safety action, so that the valve returns to the safe state. The safety function logic decision is as follows:

[0186] If the safety decision signal is "non-instruction displacement", trigger interlock, send alarm, and prompt control failure.

[0187] If the safety decision signal is "opening out of limit", trigger the highest level interlock, send high-level alarm, and stop the pump or immediately close the reactor feed.

[0188] Take the key manual gate valve at the inlet of the raw material pump in a chemical plant as an example. The safety control system receives the "opening out of limit" safety decision signal, triggers the highest level interlock, sends high-level alarm, and stops the pump and immediately closes the reactor feed.

[0189] ​​​​​Step 6, data feedback. After the safety interlock is triggered based on step 5, the safety interlock system sends an alarm event, including the abnormal screenshot, the type of abnormality, the timestamp (the time of abnormality occurrence and interlock triggering), and the handling measures. After the interlock action is completed, the safety interlock system feeds back a "interlock executed" status signal, which is used to perfect the entire valve state monitoring and safety interlock state mechanism management.

[0190] A valve state monitoring and safety interlock system based on machine vision, which realizes the above-mentioned valve state monitoring and safety interlock method based on machine vision, the valve state monitoring and safety interlock system comprising an initial measurement module, a collection and preprocessing module, a data analysis and decision module, a state judgment module, a safety interlock and execution module, and a data feedback and management module. Specifically:

[0191] The initial measurement module is composed of a high-precision calibration board, a three-dimensional optical measurement system, and a length and angle measurement system, which is used to accurately measure the spatial pose relationship between the valve, the calibration board and the industrial camera, to establish an accurate valve motion-pixel mapping model, and is also the calibration cornerstone of the entire valve state monitoring and safety interlock system.

[0192] The collection and preprocessing module accesses the initial measurement module and is composed of a high-resolution industrial CCD / CMOS industrial camera with a wide-angle lens and a special lighting device, which is used to collect the original visual image of the valve and perform distortion correction and optimization processing, and is also the perception hub of the entire valve state monitoring and safety interlock system.

[0193] The data analysis and decision module accesses the collection and preprocessing module and the initial measurement module and is composed of a computer vision library, a training inference engine and a high-performance industrial computing platform, which is used to run all algorithms in the above-mentioned method, realize traditional algorithm calculation, deep learning inference, multi-modal feature fusion and state decision, get the optimal valve state estimation, realize valve state monitoring, and is also the decision center of the entire valve state monitoring and safety interlock system.

[0194] The state judgment module accesses the data analysis and decision module, the collection and preprocessing module and the initial measurement module and is composed of a high-performance industrial computing platform, which is used to calculate the real-time opening degree of the valve, make the final state judgment and safety decision, and is also the judgment core of the entire valve state monitoring and safety interlock system.

[0195] The safety interlock and execution module accesses the state judgment module, the data analysis and decision module, the acquisition and preprocessing module and the initial measurement module, is composed of a lightweight safety logic solver, a controller, a field bus and a standard protocol, is used to receive a safety decision signal output by the state judgment module, and generates a safety interlock signal, finally triggers a physical safety action, and is also a safety barrier of the entire valve state monitoring and safety interlock system.

[0196] The data feedback and management module accesses the safety interlock and execution module, the state judgment module, the data analysis and decision module, the acquisition and preprocessing module and the initial measurement module, is composed of a high-performance industrial computing platform, is used to send an alarm event, forms a complete intelligent solution that is traceable, analyzable, optimizable and integrable, provides a comprehensive and reliable decision basis, improves long-term stability and reliability of the entire valve state monitoring and safety interlock, and is also an information hub of the entire valve state monitoring and safety interlock system.

[0197] The above-mentioned embodiments only express the implementation manners of the present application, and cannot be understood as a limitation on the scope of the present application. It should be pointed out that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which all belong to the protection scope of the present application.

Claims

1. A method for valve status monitoring and safety interlocking based on machine vision, characterized in that, The valve status monitoring and safety interlocking method includes the following steps: The first step is camera calibration and model building. The intrinsic and extrinsic parameters of the industrial camera are determined through industrial camera calibration and hand-eye calibration. The relative positional relationship between the industrial camera and the valve is established. For different types of valves, a valve motion-pixel mapping model is constructed between the visual features of the valve's moving parts and their real-time opening degree. The visual features of the valve's moving parts refer to pixel coordinates. The second step is image acquisition and preprocessing. Based on the internal and external parameters of the industrial camera obtained in the first step, the original visual images of the valve are acquired in real time by an industrial camera deployed near the valve, and the acquired original visual images are preprocessed to obtain high-quality images. The third step is key image feature extraction and state decision-making. Based on the high-quality optimized image obtained in the second step, parallel fast detection and fine feature extraction within the ROI are performed through two parallel paths: traditional algorithms and deep learning. Key image features characterizing the valve's critical components are extracted, including the valve stem, handwheel, and actuator. Key image features refer to precise position, orientation, and motion information. Multimodal feature fusion is performed on the key image features from the two parallel paths, and state decision-making is executed to obtain the optimal valve state estimate, thus achieving valve state monitoring. Specifically: Step 3.1 employs two parallel paths—traditional algorithms and deep learning—for rapid parallel detection. Based on the high-quality preprocessed image obtained in step 2, the region containing the valve is located from the entire image within milliseconds. Specifically: Path 1: Traditional algorithm; Based on the high-quality preprocessed image obtained in the second step, the Canny operator is used in parallel for fast edge detection, and then all contours are extracted; The contours are filtered and initially located according to the prior geometric features of the valve, and several candidate regions are output to obtain the output result of Path 1; Path 2: Deep learning; Based on the high-quality preprocessed image obtained in the second step, the high-quality image is scaled down to a lower resolution and input into a lightweight convolutional neural network for fast inference, outputting a coarse heatmap or bounding box of the valve region, which is the output result of Path 2. Step 3.2, ROI determination; Based on the output results of the two paths obtained in Step 3.1, a minimum rectangular area that can completely enclose the target valve is calculated using an optimal switching strategy, i.e., ROI, and this ROI is passed to the subsequent stages; Step 3.3, Fine-grained feature extraction within the ROI; Based on the ROI determined in Step 3.2, two parallel paths, traditional algorithm and deep learning, are executed simultaneously. Path 1: Traditional algorithm; Based on the ROI determined in step 3.2, perform sub-pixel-level precise localization of the visual markers set in step 1.3; Let the pixel coordinates of the visual markers be... If the marker is a preset specific pattern, then template matching or a marker detection library is used for fast and high-precision pose estimation. To estimate the velocity and direction of motion of feature points on the valve stem, handwheel, or actuator, determine the valve's motion trend, and ultimately obtain the valve's key image features, the Lucas-Kanade optical flow method is used for tracking to obtain the optical flow vectors of the feature points. By tracking the optical flow vectors of multiple feature points, key image features of the valve are obtained, and the pixel coordinates of the tracked feature points are output. It can accurately determine the movement trend of the valve and whether the valve is stationary, open, or closed. Path 2: Deep learning; high-precision segmentation based on the ROI determined in step 3.2; First, the ROI region is cropped and scaled to a standard size, then input into the U-Net neural network for semantic segmentation; Subsequently, the U-Net neural network output and post-processing are performed; the U-Net neural network outputs the class probability of each pixel and generates the final segmentation mask; then, morphological closing operations are performed on the segmented target regions to fill holes, and their centroid coordinates are calculated. The key image features of the valve are obtained; Step 3.4, Multimodal Feature Fusion: Based on the results of the two parallel paths obtained in Step 3.3, feature alignment and comparison are performed. Specifically, the pixel coordinates of feature points tracked by the traditional algorithm path are... Centroid coordinates obtained from deep learning path segmentation Compare the two and calculate the Euclidean distance between them; Step 3.5, State Decision: Based on the Euclidean distance d obtained in Step 3.4, confidence assessment and arbitration are performed. If the Euclidean distance d obtained in step 3.4 is less than or equal to the preset threshold, and the direction of the optical flow vector matches the expected valve movement direction, then the results of the two parallel paths are determined to be consistent, and the confidence result is recorded as a high confidence state; the final output is the feature point coordinates. Feature point pixel coordinates obtained by traditional algorithm path tracking Centroid coordinates obtained from deep learning path segmentation The weighted average yields: （10）； （11）； in, These are the x-axis and y-axis coordinates of the final output feature points, respectively. Weighting coefficients for traditional algorithms, Weighting coefficients for deep learning; When the Euclidean distance d obtained in step 3.4 is greater than a preset threshold, where the preset threshold range is... If so, the arbitration mechanism is initiated to check the internal confidence of the two parallel paths; Step 3.6: Based on the final output feature point coordinates obtained in Step 3.5 The confidence score results are used to perform subsequent calculations to obtain the optimal valve status estimate, thereby achieving valve status monitoring. The fourth step is to calculate the valve opening and determine the state. Based on the final output feature point coordinates obtained in the third step, the valve motion-pixel mapping model established in the first step is used to calculate the real-time valve opening and output the final safety decision signal through state determination. Fifth step, safety interlock triggering; based on the safety decision signal obtained in the fourth step, a safety interlock signal is generated, which is promptly transmitted to the safety interlock system and ultimately triggers physical safety actions; Step 6, data feedback; based on the safety interlock signal triggered in step 5, the safety interlock system sends an alarm event, including an abnormal screen screenshot, the type of abnormality, a timestamp, and the handling measures; after the interlock action is completed, the safety interlock system sends back a status signal indicating that the interlock has been executed, which is used to improve the overall valve status monitoring and the status mechanism management of the safety interlock itself.

2. The valve status monitoring and safety interlocking method based on machine vision according to claim 1, characterized in that, The first step is specifically as follows: Step 1.1, Industrial camera calibration; Obtain the intrinsic parameter matrix K and distortion coefficients D of the industrial camera for subsequent image distortion correction; The intrinsic parameter matrix K is a matrix: （1）； in, , These are the focal lengths of the industrial camera in the x and y directions, respectively, in pixels. These are the optical center coordinates of the industrial camera in the x and y directions, respectively. The optical center refers to the origin of the industrial camera's coordinate system. The distortion coefficient D is a matrix: （2）； in, The first radial distortion coefficient, The second radial distortion coefficient. The third radial distortion coefficient; The first tangential distortion coefficient, This is the second tangential distortion coefficient; Step 1.2, Hand-eye calibration; Based on the intrinsic parameter matrix K obtained in Step 1.1, determine the transformation relationship between the valve world coordinate system and the industrial camera coordinate system; Solve the pose of the industrial camera using the PnP algorithm to obtain the extrinsic parameter matrix of the industrial camera. , where R is the rotation matrix and t is the translation vector; Define points in the valve's world coordinate system Points in the industrial camera coordinate system ,in The sign for transpose is given, and the projection relationship between the two is as follows: （3）； in, These are the coordinates in the three-dimensional x, y, and z directions of the valve world coordinate system; U and V are the coordinates in the two-dimensional x and y directions of the industrial camera coordinate system; s is the scale factor; and K is the intrinsic parameter matrix of the industrial camera. Set the center point of the valve flange as the origin of the world coordinate system (0,0,0). The axis is parallel to the direction of movement of the valve stem; by measuring the actual physical distance from the optical center of the industrial camera to the origin of the world coordinate system, the actual physical distance is used as the initial estimate of the translation vector t; Step 1.3: Establish a valve motion-pixel mapping model; based on the industrial camera calibration results obtained in Steps 1.1 and 1.2, establish a functional relationship between the visual features of the valve's moving parts and its real-time opening degree.

3. The valve status monitoring and safety interlocking method based on machine vision according to claim 2, characterized in that, In step 1.3, depending on the type of valve, there are two scenarios: In the first scenario, for valves with linear motion, firstly, set high-contrast visual height markers on the valve stem. Then, manually control the valve to move from fully closed to fully open, pausing at regular intervals and capturing an image. Measure and record the actual physical height of the markers on the valve stem at that opening degree. ; Establish a linear mapping model; assume that when the valve is fully closed, the ordinate of the marked pixel is... When fully open The physical displacement of the valve throughout its entire stroke is Then the real-time opening degree of the valve is calculated as follows: （4）； in, Here, represents the ordinate of the marked point in the current image; K is the pixel equivalent, obtained by fitting the calibration data. （5）； In the second scenario, for valves where the handwheel or actuator rotates, first, set high-contrast visual markers for the angle on the handwheel or actuator. Then, manually control the valve to rotate from fully closed to fully open, recording the actual rotation angles at multiple positions. and the corresponding image; where N is the total number of circles, and the total number of the plurality of positions is ; Establish an angle mapping model; assume that N rotations of the handwheel constitute full opening, i.e., the total angle is... Then the real-time opening degree of the valve The calculation is as follows: （6）； If the industrial camera is directly facing the handwheel plane, the pixel angle in the image... Compared to the actual rotation angle A linear relationship exists: （7）； in, It is a proportionality coefficient, determined through calibration.

4. The valve status monitoring and safety interlocking method based on machine vision according to claim 3, characterized in that, The second step is as follows: Step 2.1, Image Acquisition; Acquire raw visual images of the valve stem, handwheel, or actuator at the moment of motion; Step 2.2, Image distortion correction; Based on the intrinsic parameter matrix K and distortion coefficients D of the industrial camera obtained in Step 1.1, radial distortion correction and tangential distortion correction are performed on the original visual image acquired in Step 2.

1. Step 2.3, Image Optimization; Based on the distortion-corrected image obtained in step 2.2, it is optimized to obtain a high-quality image; the optimization process includes a combination of histogram equalization, homomorphic filtering, Gaussian filtering, and median filtering.

5. The valve status monitoring and safety interlocking method based on machine vision according to claim 4, characterized in that, In the third step mentioned above: In path 1 of step 3.1, the prior geometric features of the valve refer to the circular flange profile of the valve body and the aspect ratio of the rectangular housing of the actuator. In step 3.2, the optimal switching strategy is selected as follows: the output result of path 2 is selected first, and the result of path 1 is used when its confidence level is lower than the threshold; the threshold range is 0.90~0.

95. In path 1 of step 3.3, the feature points on the valve stem refer to the sharp corners of the valve stem nut, the connection between the valve stem tip and the nut, and the visual markers. The feature points on the handwheel refer to the edges of the handwheel spokes, the tip of the handwheel handle, and the visual markers. The feature points on the actuator refer to the sharp corners of the actuator housing and the visual markers. The basic equations of the Lucas-Kanade optical flow method are as follows: （8）； in, , These are the image gradients in the x and y directions, respectively; It is the time gradient; , These are the optical flow vectors in the x and y directions, respectively; In path 2 of step 3.3, the U-Net neural network collects a large number of valve images under different working conditions, performs fine pixel-level annotation, forms a training dataset, and trains repeatedly.

6. The valve status monitoring and safety interlocking method based on machine vision according to claim 5, characterized in that, In step 3.5, the preset threshold range is 3~10 pixels; the internal confidence of the traditional algorithm path comes from the response intensity of feature points on the valve stem, handwheel or actuator or the continuity of optical flow tracking, while the internal confidence of the deep learning path comes from the size of the segmented region output by the deep learning. The specific rules of the arbitration mechanism are as follows: Because deep learning paths are assigned high weights, if the confidence level of traditional algorithm paths is low, or if feature points on valve stems, handwheels, or actuators are occluded or obscured by strong light, then the centroid coordinates obtained from deep learning path segmentation are adopted. As the final output feature point coordinates : （12）； Meanwhile, the confidence score result indicates that the traditional algorithm for path feature tracking has failed. Conversely, the pixel coordinates of feature points tracked by traditional algorithms are adopted. As the final output feature point coordinates : （13）； Meanwhile, the confidence score was recorded as low for deep learning path segmentation.

7. The valve status monitoring and safety interlocking method based on machine vision according to claim 6, characterized in that, The fourth step is specifically as follows: Step 4.1, Real-time opening calculation; based on the final output feature point coordinates obtained in step 3. Based on the valve motion-pixel mapping model established in the first step, it is converted into the valve's real-time opening value, Openness. In the first case, for valves such as gate valves and globe valves where the valve stem moves in a linear motion, the conversion formulas are shown in formulas (4) and (5); In the second case, for valves such as ball valves, butterfly valves, and plug valves, where the handwheel or actuator is rotary, the conversion formula is shown in formula (6); The final output is the feature point coordinates. Calculate the angle between the coordinates relative to the center of rotation and the reference direction. Then, the actual rotation angle is calculated using formula (7). ; Step 4.2, Status determination; Based on the real-time valve opening obtained in step 4.1, the calculated real-time opening... With instruction opening Compare and preset safe operating ranges. and alarm threshold The security decision signal is obtained, and the specific security decision signal is as follows: if Then the safety decision signal is a non-command displacement; where To allow for error, the value range is 1% to 3%; if If so, the safety decision signal is that the opening degree exceeds the limit; If a person is detected approaching the valve and the valve opening changes, but the control system does not issue a command, the safety decision signal is a misoperation.

8. The valve status monitoring and safety interlocking method based on machine vision according to claim 7, characterized in that, The fifth step is specifically as follows: Step 5.1, safety interlock signal generation; based on the safety decision signal obtained in step 4, once the state is judged to be abnormal, the lightweight safety logic solver of the safety interlock system immediately generates a safety interlock signal; Step 5.2, Signal Transmission: Based on the safety interlock signal generated in Step 5.1, it is transmitted to the controller of the safety interlock system via fieldbus and standard protocol; Step 5.3, Trigger Action: The controller of the safety interlock system in Step 5.2 executes the preset safety function logic decision and outputs a signal to trigger a physical safety action, restoring the valve to a safe state. The specific safety function logic decision is as follows: If the safety decision signal is a non-command displacement, the interlock is triggered, an alarm is sent, and a control failure is indicated. If the safety decision signal indicates that the opening is out of limit, the highest level of interlocking is triggered, an advanced alarm is sent, and the pump is stopped immediately or the reactor feed is shut off immediately.

9. A valve status monitoring and safety interlocking system based on machine vision, characterized in that, The valve status monitoring and safety interlocking system described in claims 1-8 implements the machine vision-based valve status monitoring and safety interlocking method. The valve status monitoring and safety interlocking system includes an initial measurement module, a data acquisition and preprocessing module, a data analysis and decision-making module, a status judgment module, a safety interlocking and execution module, and a data feedback and management module. Specifically: The initial measurement module consists of a high-precision calibration plate, a three-dimensional optical measurement system, and a length and angle measurement system, and is used to accurately measure the spatial pose relationship between the valve, the calibration plate, and the industrial camera. The acquisition and preprocessing module is connected to the initial measurement module and consists of an industrial camera and lighting equipment. It is used to acquire the original visual image of the valve and perform distortion correction and optimization processing. The data analysis and decision-making module is connected to the acquisition and preprocessing module and the initial measurement module to obtain the optimal valve state estimate and realize valve state monitoring. The status judgment module is connected to the data analysis and decision module, the acquisition and preprocessing module and the initial measurement module, and is used to calculate the real-time opening degree of the valve to obtain the final status judgment and safety decision. The safety interlock and execution module is connected to the status judgment module, data analysis and decision module, acquisition and preprocessing module and initial measurement module. It is used to receive the safety decision signal output by the status judgment module, generate the safety interlock signal, and finally trigger the physical safety action. The data feedback and management module is connected to the safety interlock and execution module, the status judgment module, the data analysis and decision-making module, the acquisition and preprocessing module, and the initial measurement module to send alarm events.

Images