System and method for matching entity objects in image based on VR panorama

Abstract

The invention discloses an in-image entity object matching system and method based on VR panorama, belongs to the field of VR panorama image management systems, and provides a system and method integrating high-precision positioning, emotional feature quantification and dynamic risk early warning. The blank of personnel state comprehensive monitoring and risk dynamic assessment in a complex industrial environment in the prior art is filled up, the system can recognize high-risk states such as fatigue and tension of personnel in advance by converting emotion features into quantifiable safety risk indexes, key technical support is provided for transformation of the petrochemical industry from passive safety to active prevention, and the system has a wide application prospect. The risk of misoperation caused by abnormal emotional states of personnel is reduced, risk assessment is improved from'lag response 'to'real-time prediction', the accuracy of early warning grade division is remarkably improved, the strict requirements of the petrochemical industry for'early discovery and early intervention 'of safety risks are met, and an efficient technical solution is provided for safety management and control in a high-risk scene.

Description

Technical Field

[0001] The present invention relates to the field of VR panoramic image management systems, and more specifically, to a system and method for matching entity objects in a VR panorama image. Background Art

[0002] In the dense device area of the petrochemical industry, due to the complex distribution of equipment and the high-risk operation environment, the accurate positioning and safety monitoring of key equipment, isolation point equipment and personnel are the core requirements for ensuring production safety.

[0003] In the prior art, the safety monitoring of personnel by traditional related systems is mostly limited to position tracking and behavior compliance inspection, lacking the real-time collection and analysis of personnel's physiological states (such as heart rate, skin electrical signals) and behavioral and emotional characteristics (such as body movements, facial expressions). And a large number of accident cases show that emotional states of personnel such as fatigue and tension are also important factors inducing operation errors;

[0004] Moreover, the risk warning mechanism of related systems is usually established based on static models constructed from equipment operation parameters and historical accident data, failing to dynamically integrate personnel's emotional characteristics and real-time spatial position information, resulting in a lag in risk assessment and a relatively rough division of warning levels, making it difficult to meet the petrochemical industry's demand for "early detection and early intervention" of safety risks;

[0005] Therefore, how to achieve the accurate positioning of key entity objects in the dense device area of the petrochemical industry, and at the same time, by collecting and analyzing personnel's emotional characteristics in real time, construct a dynamic safety risk model to achieve intelligent matching and accurate warning of risk levels has become a technical problem urgently to be solved in this field. So we propose a system and method for matching entity objects in a VR panorama image to solve the above existing problems. Summary of the Invention

[0006] 1. Technical Problems to be Solved

[0007] Aiming at the problems existing in the prior art, the purpose of the present invention is to provide a system and method for matching entity objects in a VR panorama image. It integrates technologies such as VR panoramic vision, multi-sensor fusion positioning, deep learning emotional feature extraction and machine learning risk modeling, and proposes a system and method integrating "high-precision positioning - emotional feature quantification - dynamic risk warning", filling the gap in the comprehensive monitoring of personnel status and dynamic risk assessment in complex industrial environments in the prior art.

[0008] 2. Technical Solutions

[0009] To solve the above problems, the present invention adopts the following technical solutions.

[0010] A system for matching entity objects in a diagram based on VR panoramas, comprising:

[0011] A VR panorama data acquisition module for acquiring VR panorama image data of a dense device area in the petrochemical industry. The VR panorama data acquisition module is communicatively connected to a number of VR panorama cameras distributed at key positions in the device area. The key positions include each corner of the device area, the tops of equipment, and above the main activity paths of personnel. The VR panorama cameras are internally integrated with IMU sensors for acquiring lens attitude data;

[0012] A data preprocessing module, communicatively connected to the VR panorama data acquisition module, for preprocessing the acquired VR panorama image data. The preprocessing includes image stitching, geometric correction, color balance, and noise filtering. The data preprocessing module includes a feature extraction unit, a matching and screening unit, and a fusion processing unit for implementing image stitching through BRIEF descriptors, the RANSAC algorithm, and multi-resolution pyramid construction technology;

[0013] An entity object recognition and positioning module, communicatively connected to the data preprocessing module, for recognizing key equipment, isolation point equipment, and personnel from the preprocessed VR panoramas and determining their positions in the device area. The entity object recognition and positioning module is internally equipped with an equipment feature database, a personnel feature database, and a positioning algorithm unit. The entity object recognition and positioning module uses a deep learning model to recognize objects and performs coordinate transformation based on a BIM model. The equipment feature database stores three-dimensional models, appearance features, and installation position information of key equipment and isolation point equipment. The personnel feature database stores human features of personnel and feature information of the safety equipment they wear. The positioning algorithm unit determines the positions of entity objects based on a visual positioning algorithm combined with a three-dimensional coordinate model of the device area, and the positioning algorithm unit includes a visual positioning subunit and a UWB positioning fusion subunit for aggregating ranging data through a ZigBee network and fusing positioning based on an extended Kalman filter algorithm;

[0014] A personnel emotion feature acquisition module for acquiring emotion-related data of personnel. The personnel emotion feature acquisition module is communicatively connected to a physiological sensor provided on a personnel safety jacket and a behavior camera provided in the device area. The physiological sensor includes a PPG sensor and a skin conductance sensor for acquiring physiological data such as the heart rate, blood pressure, and skin electrical signals of personnel. The behavior camera is used for acquiring behavior data such as the limb movements and facial expressions of personnel and supports H.265 encoding and transmits data through an industrial Ethernet;

[0015] An emotional feature processing module, communicatively connected to the personnel emotional feature collection module, is configured to process the collected emotion-related data and extract the emotional features of personnel. The emotional feature processing module includes a physiological data processing unit and a behavioral data processing unit. The emotional feature processing module further includes a signal conditioning circuit and a digital signal processing module, and outputs emotional features through the TensorFlowServing interface. The physiological data processing unit converts physiological data into emotional parameters through a preset physiological emotion model. The behavioral data processing unit analyzes behavioral data through a deep learning model to extract behavioral emotional features, and the behavioral data processing unit extracts spatial features and temporal features based on the deep learning model to generate a fused feature vector;

[0016] A safety risk modeling module, communicatively connected to the emotional feature processing module, is configured to establish a safety risk model based on the extracted emotional features of personnel and convert the personnel emotions into quantifiable safety risk indicators. The safety risk modeling module combines the location of personnel, operation tasks, and historical accident data, and establishes a mapping relationship between emotional features and safety risks through machine learning algorithms. A three-dimensional space construction unit is provided inside the safety risk modeling module. The safety risk modeling module accesses DCS system data through the OPCUA protocol and generates a risk probability density based on the three-dimensional space construction unit;

[0017] An early warning level matching module, communicatively connected to the safety risk modeling module, the early warning level matching module includes a fuzzy reasoning unit and an early warning execution unit. Multiple early warning measures are preset inside the early warning execution unit, and are configured to match the early warning level according to the safety risk indicators according to a preset early warning level logic chain and trigger corresponding early warning mechanisms. The early warning level logic chain is divided into multiple early warning levels according to the magnitude of the safety risk indicators, and each early warning level corresponds to different early warning methods and countermeasures.

[0018] Furthermore, the VR panoramic cameras form a star network through a PoE switch, and the raw image data is transmitted to the data preprocessing module in real time using the RTSP protocol. Moreover, the VR panoramic cameras obtain IP addresses through the DHCP protocol and transmit the raw data based on the TCP protocol. The three-axis angular velocity and acceleration data collected by the IMU sensor are synchronously transmitted to the image sensor control unit through the I2C bus, and the lens attitude correction parameters generated by the Kalman filter algorithm are fed back to the camera drive module in real time using the UDP protocol.

[0019] Further, the BRIEF descriptors output by the feature extraction unit of the data preprocessing module are transmitted to the matching and screening unit through shared memory. The effective feature point pairs screened by the RANSAC algorithm are stored in XML format and transmitted to the fusion processing unit through a message queue. The image data at different levels generated by the multi-resolution pyramid construction module are transmitted to the weighted fusion operation module through the PCIe bus to output a seamless stitched panoramic image of 8192×4096 pixels.

[0020] Further, the device feature point image coordinates output by the visual positioning subunit of the entity object recognition and positioning module are interacted with the CAD engineering drawing coordinate mapping table through the REST API interface. The generated homography matrix is transmitted to the UWB positioning fusion subunit through Ethernet, and the world coordinates of the entity object are output through the extended Kalman filtering algorithm.

[0021] Further, the PPG sensor and the skin conductance sensor of the personnel emotion feature acquisition module transmit the original signals to the microcontroller built into the safety helmet through the Bluetooth 5.0 protocol. After analog-to-digital conversion, they are sent to the edge computing node through Wi-Fi. The H.265 encoded video stream of the behavior camera is connected to the industrial Ethernet through the RJ45 interface and transmitted to the behavior data processing unit of the emotion feature processing module after being mapped by the NAT gateway. The physiological sensor and the behavior camera are synchronized in time through the NTP server, and the data is stored in HDF5 format.

[0022] Further, the analog signal output by the signal conditioning circuit of the emotion feature processing module is converted into a digital signal through a 16-bit DAC and transmitted to the digital signal processing module through the SPI bus. The processed 3D emotion feature vector is pushed to the safety risk modeling module through the TensorFlow Serving interface and stored in the time series database for historical data analysis at the same time.

[0023] Further, the AU unit feature vector output by the spatial feature extraction branch of the behavior data processing unit and the action trajectory feature output by the time feature extraction branch are concatenated by tensors to form a 3072-dimensional fusion feature, which is transmitted to the safety risk modeling module in ProtocolBuffers format through the gRPC interface after dimensionality reduction by the fully connected layer.

[0024] Further, the data fusion interface of the safety risk modeling module obtains the device working condition data of the DCS system in real time through the OPC UA protocol, inputs them into the three-dimensional space construction unit together with the emotion feature vector and the operation task code. The generated risk probability density value is output to the fuzzy inference unit of the warning level matching module through the Modbus TCP protocol and is marked in real time on the three-dimensional visualization interface.

[0025] Furthermore, the early warning level signal output by the fuzzy inference unit of the early warning level matching module is transmitted to the early warning execution unit through the industrial fieldbus. Among them, the first-level early warning signal triggers the control instruction of the safety helmet vibration module and starts the red strobe light in the equipment area. At the same time, JSON format data containing the coordinates of the entity object and risk indicators is sent to the central control room through the WebSocket protocol to achieve multi-terminal synchronous early warning.

[0026] A method for matching entity objects in a figure based on VR panorama includes the following steps:

[0027] S1. VR panorama data acquisition and preprocessing: Collect VR panorama image data through VR panorama cameras distributed at key positions in the dense device area of the petrochemical industry. Perform preprocessing on the collected image data, including image stitching, geometric correction, color balance, and noise filtering, to obtain the preprocessed VR panorama image;

[0028] S2. Entity object recognition and positioning: Input the preprocessed VR panorama image into the entity object recognition and positioning module. Utilize the information in the device feature database and personnel feature database to identify key devices, isolation point devices, and personnel through image recognition algorithms. Then, combine the three-dimensional coordinate model and positioning algorithm of the device area to determine their specific positions;

[0029] S3. Personnel emotion feature collection and processing: Collect physiological data such as heart rate, blood pressure, and skin electrical signal of personnel through physiological sensors set on personnel safety equipment. Collect behavioral data such as limb movements and facial expressions of personnel through behavior cameras set in the device area. Use the physiological data processing unit and behavioral data processing unit to process the collected data and extract the emotion features of personnel;

[0030] S4. Safety risk modeling: Input the extracted personnel emotion features, the location of personnel, operation tasks, and historical accident data into the safety risk modeling module. Establish a mapping relationship between emotion features and safety risks through machine learning algorithms, and convert personnel emotions into quantifiable safety risk indicators;

[0031] S5. Early warning level matching and early warning: According to the safety risk indicators, match the early warning level according to the preset early warning level logic chain. When the safety risk indicators reach the corresponding early warning level threshold, trigger the corresponding early warning mechanism and send out an early warning signal.

[0032] 3. Beneficial effects

[0033] Compared with the prior art, the advantages of the present invention are as follows:

[0034] (1) In this solution, through the personnel emotional feature acquisition module and the emotional feature processing module, multi-dimensional real-time acquisition and in-depth analysis of personnel physiological states (heart rate, blood pressure, skin electrical signals) and behavioral emotional features (body movements, facial expressions) are achieved. That is, physiological sensors (PPG sensors, skin conductance sensors) and behavioral cameras respectively collect physiological data and behavioral data. After being processed by signal conditioning circuits, digital signal processing modules and deep learning models, a 3072-dimensional emotional feature vector and a 3072-dimensional fusion feature vector are generated, filling the technical gap in the monitoring of personnel emotional states in existing systems. By converting emotional features into quantifiable safety risk indicators, the system can identify high-risk states such as personnel fatigue and tension in advance, providing key technical support for the transformation of the petrochemical industry from "passive safety" to "active prevention", and effectively reducing the risk of operation errors caused by abnormal personnel emotional states.

[0035] (2) In this solution, through the safety risk modeling module and the warning level matching module, a dynamic and precise safety risk assessment system is constructed. That is, the safety risk modeling module, based on machine learning algorithms, dynamically fuses personnel emotional features, real-time spatial positions, operation tasks and DCS system equipment condition data, and uses three-dimensional space construction units to generate risk probability densities, breaking through the limitations of traditional static models. The warning level matching module, through fuzzy inference units and preset multi-level warning logic chains, realizes the intelligent matching and precise warning of risk levels (such as the first-level warning triggering helmet vibration, red strobe lights and multi-terminal synchronous warning in the central control room). This mechanism upgrades risk assessment from "lagged response" to "real-time prediction", significantly improving the accuracy of warning level division and meeting the strict requirements of the petrochemical industry for "early detection and early intervention" of safety risks, providing an efficient technical solution for safety control in high-risk scenarios. Brief Description of the Drawings

[0036] Figure 1 It is a schematic diagram of the system architecture of the present invention;

[0037] Figure 2 It is a schematic diagram of the technical details of the VR panoramic data acquisition module and the data preprocessing module of the present invention;

[0038] Figure 3 It is a schematic diagram of the technical details of the entity object recognition and positioning module and the personnel emotional feature acquisition module of the present invention;

[0039] Figure 4 It is a schematic diagram of the technical details of the emotional feature processing module, the safety risk modeling module and the warning level matching module of the present invention;

[0040] Figure 5 It is a schematic diagram of the method steps of the present invention;

[0041] Figure 6 It is the principle mind map of step S1 and step S2 of the present invention;

[0042] Figure 7 It is the principle mind map of step S3, step S4 and step S5 of the present invention.

[0043] Description of reference numerals in the figure:

[0044] 1. VR panoramic data acquisition module; 101. VR panoramic camera; 1011. IMU sensor; 2. Data preprocessing module; 201. Feature extraction unit; 202. Matching and screening unit; 203. Fusion processing unit; 3. Entity object recognition and positioning module; 301. Equipment feature database; 302. Personnel feature database; 303. Positioning algorithm unit; 4. Personnel emotion feature acquisition module; 401. Physiological sensor; 4011. PPG sensor; 4012. Skin conductance sensor; 402. Behavior camera; 5. Emotion feature processing module; 501. Physiological data processing unit; 502. Behavior data processing unit; 503. Signal conditioning circuit; 504. Digital signal processing module; 6. Safety risk modeling module; 601. Three-dimensional space construction unit; 7. Early warning level matching module; 701. Fuzzy inference unit; 702. Early warning execution unit. Specific implementation mode

[0045] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings of the present invention; obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0046] Embodiment 1: Please refer to Figures 1 - 7 , a matching system for entity objects in a graph based on VR panorama, including: The VR panoramic data acquisition module 1 is used to collect VR panoramic image data of the dense device area in the petrochemical industry. The VR panoramic data acquisition module 1 is communicatively connected to a plurality of VR panoramic cameras 101 distributed at key positions in the device area. The key positions include each corner of the device area, the top of the equipment, and above the main activity paths of personnel. The VR panoramic camera 101 is internally integrated with an IMU sensor 1011 for collecting lens attitude data; The data preprocessing module 2, which is communicatively connected to the VR panoramic data acquisition module 1, is used to preprocess the acquired VR panoramic image data. The preprocessing includes image stitching, geometric correction, color balance, and noise filtering. The data preprocessing module 2 includes a feature extraction unit 201, a matching and screening unit 202, and a fusion processing unit 203, and is used to implement image stitching through BRIEF descriptors, the RANSAC algorithm, and multi-resolution pyramid construction technology; The entity object recognition and positioning module 3, which is communicatively connected to the data preprocessing module 2, is used to identify key devices, isolation point devices, and personnel from the preprocessed VR panoramic images and determine their positions in the device area. The entity object recognition and positioning module 3 internally houses a device feature database 301, a personnel feature database 302, and a positioning algorithm unit 303. The entity object recognition and positioning module 3 uses a deep learning model to identify objects and performs coordinate transformation based on the BIM model. The device feature database 301 stores the 3D models, appearance features, and installation position information of key devices and isolation point devices. The personnel feature database 302 stores the human body features of personnel and the feature information of the safety equipment they wear. The positioning algorithm unit 303 determines the positions of entity objects based on a visual positioning algorithm in combination with the 3D coordinate model of the device area, and the positioning algorithm unit 303 includes a visual positioning subunit and a UWB positioning fusion subunit, which converge ranging data through a ZigBee network and perform fusion positioning based on the extended Kalman filter algorithm; The personnel emotional feature acquisition module 4 is used to acquire data related to the emotions of personnel. The personnel emotional feature acquisition module 4 is communicatively connected to a physiological sensor 401 disposed on the personnel safety jacket and a behavior camera 402 disposed in the device area. The physiological sensor 401 includes a PPG sensor 4011 and a skin conductance sensor 4012, and is used to acquire physiological data such as the heart rate, blood pressure, and skin electrical signals of personnel. The behavior camera 402 is used to acquire behavior data such as the limb movements and facial expressions of personnel, and supports H.265 encoding and transmits data through an industrial Ethernet; An emotional feature processing module 5, communicatively connected to the personnel emotional feature acquisition module 4, is configured to process the collected emotion-related data and extract the emotional features of the personnel. The emotional feature processing module 5 includes a physiological data processing unit 501 and a behavior data processing unit 502. The emotional feature processing module 5 further includes a signal conditioning circuit 503 and a digital signal processing module 504, and outputs emotional features through the TensorFlowServing interface. The physiological data processing unit 501 converts physiological data into emotional parameters through a preset physiological emotion model. The behavior data processing unit 502 analyzes the behavior data through a deep learning model to extract behavior emotional features, and the behavior data processing unit 502 extracts spatial features and time features based on the deep learning model to generate a fused feature vector; A safety risk modeling module 6, communicatively connected to the emotional feature processing module 5, is configured to establish a safety risk model according to the extracted emotional features of the personnel and convert the personnel emotion into a quantifiable safety risk index. The safety risk modeling module 6 combines the location of the personnel, the operation task, and historical accident data, and establishes a mapping relationship between emotional features and safety risks through a machine learning algorithm. A three-dimensional space construction unit 601 is provided inside the safety risk modeling module 6. The safety risk modeling module 6 accesses DCS system data through the OPCUA protocol and generates a risk probability density based on the three-dimensional space construction unit 601; An early warning level matching module 7, communicatively connected to the safety risk modeling module 6, includes a fuzzy inference unit 701 and an early warning execution unit 702. Multiple early warning measures are preset inside the early warning execution unit 702, which is configured to match the early warning level according to the safety risk index according to a preset early warning level logic chain and trigger the corresponding early warning mechanism. The early warning level logic chain is divided into multiple early warning levels according to the magnitude of the safety risk index, and each early warning level corresponds to different early warning methods and countermeasures; The VR panoramic camera 101 forms a star network through a PoE switch, and transmits the original image data to the data preprocessing module 2 in real time using the RTSP protocol. The VR panoramic camera 101 obtains an IP through the DHCP protocol and transmits the original data based on the TCP protocol. The three-axis angular velocity and acceleration data collected by the IMU sensor 1011 are synchronously transmitted to the image sensor control unit through the I2C bus, and the lens attitude correction parameters generated by the Kalman filter algorithm are fed back to the camera drive module in real time through the UDP protocol; The BRIEF descriptors output by the feature extraction unit 201 of the data preprocessing module 2 are transmitted to the matching and screening unit 202 through shared memory. The effective feature point pairs screened by the RANSAC algorithm are stored in XML format and transmitted to the fusion processing unit 203 through a message queue. The image data of different levels generated by the multi-resolution pyramid construction module are transmitted to the weighted fusion operation module through the PCIe bus, and a seamless stitched panoramic image of 8192×4096 pixels is output; The device feature point image coordinates output by the visual positioning subunit of the entity object recognition and positioning module 3 are interacted with the CAD engineering drawing coordinate mapping table through the RESTAPI interface. The generated homography matrix is transmitted to the UWB positioning fusion subunit through Ethernet, and the world coordinates of the entity object are output through the extended Kalman filter algorithm; The PPG sensor 4011 and the skin conductance sensor 4012 of the personnel emotion feature acquisition module 4 transmit the original signals to the microcontroller built into the safety helmet through the Bluetooth 5.0 protocol. After analog-to-digital conversion, they are sent to the edge computing node through Wi-Fi. The H.265 encoded video stream of the behavior camera 402 is accessed to the industrial Ethernet through the RJ45 interface and transmitted to the behavior data processing unit 502 of the emotion feature processing module 5 after being mapped by the NAT gateway. The physiological sensor 401 and the behavior camera 402 are synchronized in time through the NTP server, and the data is stored in HDF5 format; The analog signal output by the signal conditioning circuit 503 of the emotion feature processing module 5 is converted into a digital signal through 16-bit DAC and transmitted to the digital signal processing module 504 through the SPI bus. The processed emotion feature vector is pushed to the safety risk modeling module 6 through the TensorFlowServing interface and stored in the time series database for historical data analysis at the same time; The AU unit feature vector output by the spatial feature extraction branch of the behavior data processing unit 502 and the action trajectory feature output by the time feature extraction branch are spliced into a 3072-dimensional fusion feature through tensors, and are transmitted to the safety risk modeling module 6 in ProtocolBuffers format through the gRPC interface after dimensionality reduction by the fully connected layer; The data fusion interface of the safety risk modeling module 6 obtains the device working condition data of the DCS system in real time through the OPCUA protocol, and inputs the emotion feature vector and the operation task code into the three-dimensional space construction unit 601 through the data bus. The generated risk probability density value is output to the fuzzy inference unit 701 of the warning level matching module 7 through the ModbusTCP protocol and is marked in real time on the three-dimensional visualization interface; The warning level signal output by the fuzzy reasoning unit 701 of the warning level matching module 7 is transmitted to the warning execution unit 702 through the industrial field bus, wherein the first-level warning signal triggers the control instruction of the safety helmet vibration module and starts the red strobe light in the equipment area. At the same time, JSON format data containing the coordinates of the entity object and the risk index are sent to the central control room through the WebSocket protocol to realize multi-terminal synchronous warning.

[0047] A method for matching entity objects in a picture based on VR panorama, comprising the following steps: S1. VR panoramic data acquisition and preprocessing: VR panoramic image data is collected by VR panoramic cameras 101 distributed in key positions in the intensive device area of ​​the petrochemical industry. The collected image data is preprocessed by image stitching, geometric correction, color balance and noise filtering to obtain a preprocessed VR panoramic image. The VR panoramic camera 101 automatically obtains an IP address through the DHCP protocol, establishes a TCP connection with the data preprocessing server, and uses a custom binary protocol to transmit the original image data. The preprocessed panoramic image is published in Base64 encoding format through the message middleware to the input queue of the entity object recognition and positioning module. The queue backlog data volume is controlled within 50 frames to ensure real-time performance. S2. Entity object recognition and positioning: The pre-processed VR panoramic image is input into the entity object recognition and positioning module 3. The information in the equipment feature database 301 and the personnel feature database 302 is used to identify key equipment, isolation point equipment and personnel through the image recognition algorithm. Then, the specific location is determined by combining the three-dimensional coordinate model of the device area and the positioning algorithm. The device boundary box coordinates output by the improved FasterR-CNN network are mapped to the world coordinate system coordinates through the BIM model coordinate conversion matrix of the device area. The result is published to the positioning algorithm unit in the form of a ROS message. At the same time, the device type code and the sensor data address are mapped through a hash table and stored in the memory database for subsequent risk modeling calls. S3. Collection and processing of personnel's emotional characteristics: The physiological sensor 401 installed on the personnel's safety equipment collects the personnel's heart rate, blood pressure, and skin electrical signal physiological data, and the behavioral camera 402 installed in the device area collects the personnel's body movements and facial expression behavioral data. The physiological data processing unit 501 and the behavioral data processing unit 502 are used to process the collected data and extract the personnel's emotional characteristics. The physiological sensor and the behavioral camera are synchronized through the NTP server, and the generated UTC timestamp is used as the data frame header. The physiological data and video frames are synchronized at the edge computing node through the timestamp alignment algorithm, and abnormal data with a time difference of >50ms is eliminated. The synchronized data stream is stored in the HDF5 format and input into the emotional feature processing module; S4. Safety Risk Modeling: Input the extracted personnel emotional characteristics, personnel location, operation tasks, and historical accident data into the safety risk modeling module 6. Establish a mapping relationship between emotional characteristics and safety risks through machine learning algorithms, and convert personnel emotions into quantifiable safety risk indicators. S5. Early Warning Level Matching and Warning: According to the safety risk indicators, match the early warning level according to the preset early warning level logic chain. When the safety risk indicator reaches the corresponding early warning level threshold, trigger the corresponding early warning mechanism and send out a warning signal.

[0048] Through the collaborative operation of multiple modules, this system realizes the precise positioning of key entity objects in the intensive device area of the petrochemical industry, the real-time acquisition and analysis of personnel emotional characteristics, and the dynamic safety risk modeling and warning. Its core working principle is as follows:

[0049] I. VR Panoramic Data Acquisition and Preprocessing 1. Multi-source Data Synchronous Acquisition The VR panoramic cameras 101 distributed at key positions in the device area (corners, tops of equipment, above personnel activity paths) form a star network through PoE switches, automatically obtain IP addresses based on the DHCP protocol, and transmit the original image data in real time in the form of a TCP long connection through the RTSP protocol (supporting a resolution of 8192×4096 pixels). The IMU sensors 1011 built into the cameras collect three-axis angular velocity (range of ±2000° / s) and acceleration (range of ±16g) data at a frequency of 100Hz, synchronize them to the image sensor control unit through the I2C bus (transmission rate of 400kHz), and calculate the lens attitude correction parameters (translation vector t, rotation matrix R) through the extended Kalman filter algorithm (state transition matrix F = [[1,Δt,0.5Δt²],[0,1,Δt],[0,0,1]]), and feedback them to the camera driver module in real time through the UDP protocol (port number 50001) to achieve motion blur correction and perspective deviation compensation during image acquisition.

[0050] 2. Image Preprocessing Pipeline The original image data is transmitted to the data preprocessing module 2 through the TCP protocol (port number 50002) and processed according to the following process: Feature Extraction: The feature extraction unit 201 divides each frame of image into 8×8 pixel blocks, generates a 64-bit binary feature vector using the BRIEF descriptor, and transmits it to the matching and screening unit 202 at a rate of 400MB / s through shared memory (ShmOpen interface).

[0051] Robust Matching: The matching and screening unit 202 eliminates mismatched point pairs based on the RANSAC algorithm (with 1000 iterations and an inlier threshold of 3 pixels), and the remaining valid feature point pairs are stored in the memory buffer in XML format (including key point coordinates and descriptor hash values), and are transmitted to the fusion processing unit 203 through a ZeroMQ message queue (subscribe-publish mode).

[0052] Multi-scale Mosaic: The fusion processing unit 203 constructs a 3-layer multi-resolution pyramid (with a bottom layer resolution of 8192×4096 and a top layer of 1024×512), transmits each layer of the image to the weighted fusion operation module through a PCIe 3.0 bus (bandwidth 8GB / s), and uses the Laplacian pyramid fusion algorithm (fusion weight matrix W(x,y)=0.5+0.5×sin(πx / W)) to generate a seamless stitched panoramic image, and simultaneously completes geometric correction (based on the OpenCV distortion correction model), color balance (gray world algorithm), and median filtering (kernel size 3×3). The preprocessed image is published to the input queue of the entity object recognition and positioning module in Base64 encoding through a Kafka message middleware (with 8 partitions and 2 replicas). The maximum backlog of the queue is set to 50 frames, and the data flow rate is controlled by the token bucket algorithm (token generation rate 20 frames / s) to ensure real-time performance.

[0053] II. Entity Object Recognition and Positioning 1. Cross-modal Target Recognition The preprocessed VR panoramic image is input into the entity object recognition and positioning module 3, and target classification and positioning are achieved through a dual-branch deep learning model: Device Recognition Branch: An improved Faster R-CNN network (with a backbone network of ResNet-50 and feature pyramid levels P2-P5) is trained based on the device feature database 301. The database stores 3D CAD models (STL format), appearance feature vectors (fused features of HOG+LBP), and installation location BIM coordinates (accuracy ±5cm) of more than 200 types of key devices / isolation point devices. The network outputs the device bounding box coordinates (pixel level), class confidence (threshold ≥0.8), and device type code (e.g., "P-001" represents a pump device).

[0054] Personnel Recognition Branch: The YOLOv5s network is optimized for personnel safety equipment features (helmet color, reflective strip pattern, gas detector model), with an input size of 640×640, and outputs human key point coordinates (17 COCO key points) and safety equipment status labels (such as "helmet worn normally", "breathing apparatus connection abnormal").

[0055] 2. 3D Space Coordinate Calculation Visual positioning fundamental matrix: Obtain the mapping relationship between the CAD engineering drawing of the device area and the world coordinate system through the BIM model, and calculate the homography matrix H using the eight-point method (degree of freedom 8, minimum number of matching point pairs 8). Convert the image coordinates (u, v) of the device feature points to world coordinates (X, Y, Z):

[0056]

[0057] The results are interacted in JSON format through the REST API interface (URI path / api / visual positioning), and the transmission delay ≤ 200 ms.

[0058] UWB integrated positioning: The UWB base stations (anchors) deployed in the device area collect ranging data through the ZigBee network (IEEE 802.15.4 protocol, transmission rate 250 kbps) (TOF ranging accuracy ±10 cm), and fuse with the visual positioning results through the extended Kalman filter algorithm (state vector [x, y, z, ẋ, ẏ, ż], process noise covariance Q = diag([0.1, 0.1, 0.1, 0.01, 0.01, 0.01])), output the world coordinates of the entity object (root mean square error ≤ 5 cm), the positioning results are published as ROS messages (topic name / target_pos), and establish the mapping relationship between the device type code and the sensor data address through the Redis in-memory database (hash table structure) (such as device code "V-002" corresponding to the temperature sensor address 0x0102) for real-time invocation by the security risk modeling module.

[0059] III. Collection and Processing of Personnel Emotional Characteristics 1. Synchronous collection of physiological-behavioral data Physiological data link: The PPG sensor 4011 collects the photoplethysmogram signal at a sampling rate of 500 Hz, and the skin conductance sensor 4012 collects the galvanic skin response signal at a sampling rate of 200 Hz. The original analog signal is transmitted to the STM32 microcontroller built into the safety helmet through the Bluetooth 5.0 protocol (transmission rate 2 Mbps, connection interval 7.5 ms), converted into a digital signal by a 16-bit ADC (sampling accuracy ±0.1% FS), and sent to the edge computing node (deployed on the edge server in the device area) through Wi-Fi 6 (802.11ax, channel bandwidth 80 MHz).

[0060] Behavior data link: The behavior camera 402 uses a global shutter CMOS sensor to collect a 1920×1080 pixel video stream at a frame rate of 30fps. After being encoded by H.265 (compression ratio 1:100, bit rate 2Mbps), it is connected to the industrial Ethernet through an RJ45 interface (the switch supports IEEE 1588v2 precision clock synchronization), and is transmitted to the behavior data processing unit 502 of the emotion feature processing module 5 through a NAT gateway (port mapping 8080→50003).

[0061] Spatio-temporal alignment: The physiological sensor and the behavior camera obtain the UTC time through an NTP server (time synchronization accuracy ≤1ms), and the generated timestamp (accurate to microseconds) is used as the data frame header. The edge computing node performs time alignment on the physiological data sequence and the video frame through the dynamic time warping (DTW) algorithm, eliminates abnormal data with a time difference >50ms, and the synchronized data stream is stored in the distributed file system in HDF5 format (group storage, physiological data group / behavior_data, video data group / video_frames).

[0062] 2. Multimodal feature engineering Physiological feature extraction: The original physiological signal is amplified (gain 1000 times) and filtered (50Hz notch filter) by the signal conditioning circuit 503, and then transmitted to the digital signal processing module 504 through the SPI bus (clock frequency 10MHz). Based on the cardiovascular health model, parameters such as heart rate variability (HRV, RMSSD index) and skin conductance level (SCL) are calculated, and mapped to emotion indicators through preset thresholds (e.g., HRV<50ms corresponds to "fatigue", SCL>10μS corresponds to "tension"), generating a 16-dimensional emotion feature vector (such as [HRV, SCL, respiratory rate,...]). It is pushed to the security risk modeling module 6 in ProtoBuffer format through the TensorFlow Serving interface (gRPC protocol, port number 8500), and is also stored in the InfluxDB time series database (time accuracy nanoseconds, retention policy 30 days).

[0063] Behavior feature extraction: The behavior data processing unit 502 adopts a two-stream CNN architecture: Spatial stream network: The input video frame extracts appearance features through ResNet-34, and outputs a 64-dimensional AU unit feature vector (such as the intensity of action units such as FACS-encoded Brow Raiser, Lip Corner Puller, etc.).

[0064] Time Flow Network: Extract the action trajectory features of 10 consecutive frames through 3D ConvLSTM to generate a 128-dimensional time-series feature vector (such as the spatio-temporal distribution of joint movement speed and acceleration). The two types of features are concatenated by tensors to form a 3072-dimensional fused feature vector, which is reduced to 128 dimensions through a fully connected layer (with 512 neurons and ReLU activation function), and then transmitted to the security risk modeling module 6 in Protocol Buffers format through the gRPC interface (service name BehaviorFeatureService), with a transmission delay ≤ 300ms.

[0065] IV. Security Risk Modeling and Dynamic Assessment 1. Multi-source Data Fusion Modeling The security risk modeling module 6 obtains the device operating condition data of the DCS system (such as temperature, pressure, flow rate, update frequency 1s) in real time through the OPC UA protocol (server address opc.tcp: / / dcs-system:4840), and inputs it into the three-dimensional space construction unit 601 through the data bus (bandwidth 1GB / s) together with the emotion feature vector, the world coordinates (X, Y, Z) of the entity object, and the operation task code (preset 10 types of tasks, such as "T-01" representing hot work). A risk prediction model is constructed based on the XGBoost algorithm, and the feature engineering includes: Personnel features: Emotion feature vector (16-dimensional), safety equipment status label (one-hot encoding, 5-dimensional) Spatial features: Distance from dangerous equipment (calculated by Euclidean distance), regional risk level (preset risk heat map of the device area) Device features: Normalized value of DCS parameters (Z-score standardization), historical failure frequency (Poisson distribution parameter) Task features: Task risk coefficient (such as hot work coefficient 1.8, regular inspection 0.5) 2. Risk Probability Density Generation The three-dimensional space construction unit 601 divides the device area into grids of 0.5m × 0.5m × 0.5m, and calculates the risk probability density (unit: times / cubic meter·hour) of each grid based on kernel density estimation (KDE, bandwidth h = 1.0). The formula is:

[0066]

[0067] where the kernel function K is a Gaussian kernel. The generated risk density field is rendered in real time on the three-dimensional visualization interface through the Three.js library, and the risk hot spots are marked with different colors (such as red ≥ 0.5 times / m³·h, yellow 0.1 - 0.5 times / m³·h). The risk probability density value is output to the warning level matching module 7 through the Modbus TCP protocol (function code 0x03, register address 40001 - 40100), with an update period of 500ms.

[0068] V. Early Warning Level Matching and Intelligent Response 1. Fuzzy Logic Inference Engine The fuzzy inference unit 701 of the early warning level matching module 7 adopts the Mamdani inference model. The input variable is the risk probability density value (universe of discourse [0, 1] times / m³·h, fuzzy set {low, medium, high}, and the membership function is triangular distribution), and the output variable is the early warning level (universe of discourse {Level III, Level II, Level I}, and the membership function is trapezoidal distribution). The rule base contains 9 fuzzy rules, for example: If the risk density is "low" and the personnel emotion is "normal", then the early warning level is "Level III" If the risk density is "high" and the personnel emotion is "nervous", then the early warning level is "Level I" 2. Hierarchical Early Warning Execution Mechanism The early warning execution unit 702 triggers multi-level responses according to the fuzzy inference results: Level I early warning (risk density ≥ 0.6): Hardware response: Send control instructions to the safety helmet through the CAN bus (baud rate 1Mbps) to drive the vibration module (frequency 200Hz, amplitude 0.5mm) to vibrate continuously for 5s; send Modbus RTU instructions (slave address 0x01, function code 0x0F) to the indicator light in the equipment area to start the red strobe light (frequency 2Hz).

[0069] Software response: Send JSON data (format {"entity_id": "P 003", "risk_score": 0.82, "location": 12.5, 3.2, 4.8]}) to the central control room through the WebSocket protocol (port number 8081), triggering the SCADA system to pop up an alarm, highlight on the electronic map, and voice broadcast ("High-risk area, please pay attention!").

[0070] Level II early warning (0.3 ≤ risk density < 0.6): Start the yellow strobe light through the RS-485 bus (transmission rate 9600bps), display the risk trend curve in the central control room, and synchronously push the early warning notice (including risk location and personnel emotion status) to the mobile APP of safety management personnel.

[0071] Level III early warning (risk density < 0.3): Only record the risk event in the system log, generate a 24-hour trend analysis report through the time series database, and automatically trigger the safety training system to push relevant case learning materials.

[0072] 3. Closed-loop feedback optimization The early warning execution results (such as the evacuation route of personnel and the shutdown status of equipment) are fed back to the safety risk modeling module 6 through the industrial fieldbus (PROFINET protocol) as real-time tag data for model update. The model retraining process is automatically triggered weekly to update the XGBoost model parameters based on 100,000 newly collected sample data (including positive and negative examples), ensuring that the risk assessment accuracy continues to improve over time.

[0073] The above is only a preferred specific embodiment of the present invention; however, the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution of the present invention and its improved concept, makes equivalent substitutions or changes, and should be covered by the protection scope of the present invention.

Claims

1. A system for matching entity objects in a graph based on VR panoramas, characterized in that, Including: A VR panoramic data acquisition module (1) for acquiring VR panoramic image data of intensive device areas in the petrochemical industry. The VR panoramic data acquisition module (1) is communicatively connected to several VR panoramic cameras (101) distributed at key positions in the device area. The key positions include each corner of the device area, the top of the equipment, and above the main activity paths of personnel. An IMU sensor (1011) is integrated inside the VR panoramic camera (101) for acquiring lens attitude data; A data preprocessing module (2), communicatively connected to the VR panoramic data acquisition module (1), for preprocessing the acquired VR panoramic image data. The preprocessing includes image stitching, geometric correction, color balance, and noise filtering. The data preprocessing module (2) includes a feature extraction unit (201), a matching and screening unit (202), and a fusion processing unit (203), and is used to implement image stitching through BRIEF descriptors, the RANSAC algorithm, and multi-resolution pyramid construction technology; An entity object recognition and positioning module (3), communicatively connected to the data preprocessing module (2), for recognizing key equipment, isolation point equipment, and personnel from the preprocessed VR panoramic images and determining their positions in the device area. An equipment feature database (301), a personnel feature database (302), and a positioning algorithm unit (303) are installed inside the entity object recognition and positioning module (3). The entity object recognition and positioning module (3) uses a deep learning model to recognize objects and performs coordinate conversion based on the BIM model. The equipment feature database (301) stores three-dimensional models, appearance features, and installation position information of key equipment and isolation point equipment. The personnel feature database (302) stores human body features of personnel and feature information of the safety equipment worn. The positioning algorithm unit (303) determines the positions of entity objects based on a visual positioning algorithm combined with a three-dimensional coordinate model of the device area, and the positioning algorithm unit (303) includes a visual positioning subunit and a UWB positioning fusion subunit, converges ranging data through a ZigBee network, and fuses positioning based on the extended Kalman filter algorithm; A personnel emotion feature acquisition module (4) for acquiring emotion-related data of personnel. The personnel emotion feature acquisition module (4) is communicatively connected to a physiological sensor (401) provided on the safety jacket of personnel and a behavior camera (402) provided in the device area. The physiological sensor (401) includes a PPG sensor (4011) and a skin conductance sensor (4012) for acquiring physiological data such as the heart rate, blood pressure, and skin electrical signals of personnel. The behavior camera (402) is used to acquire behavior data such as the limb movements and facial expressions of personnel, supports H.265 encoding, and transmits data through industrial Ethernet; The emotional feature processing module (5), communicatively connected to the personnel emotional feature acquisition module (4), is used to process the collected emotion-related data and extract the emotional features of the personnel. The emotional feature processing module (5) includes a physiological data processing unit (501) and a behavior data processing unit (502). The emotional feature processing module (5) further includes a signal conditioning circuit (503) and a digital signal processing module (504), and outputs the emotional features through the TensorFlowServing interface. The physiological data processing unit (501) converts the physiological data into emotional parameters through a preset physiological emotion model. The behavior data processing unit (502) analyzes the behavior data through a deep learning model to extract behavior emotional features, and the behavior data processing unit (502) extracts spatial features and time features based on the deep learning model to generate a fused feature vector; The safety risk modeling module (6), communicatively connected to the emotional feature processing module (5), is used to establish a safety risk model according to the extracted emotional features of the personnel and convert the personnel emotion into a quantifiable safety risk index. The safety risk modeling module (6) combines the location of the personnel, the operation task, and the historical accident data, and establishes a mapping relationship between the emotional features and the safety risks through a machine learning algorithm. The safety risk modeling module (6) is internally provided with a three-dimensional space construction unit (601). The safety risk modeling module (6) accesses the DCS system data through the OPCUA protocol and generates a risk probability density based on the three-dimensional space construction unit (601); The warning level matching module (7), communicatively connected to the safety risk modeling module (6), the warning level matching module (7) includes a fuzzy inference unit (701) and a warning execution unit (702). The warning execution unit (702) internally presets multiple warning measures, and is used to match the warning level according to the safety risk index according to a preset warning level logic chain and trigger the corresponding warning mechanism. The warning level logic chain is divided into multiple warning levels according to the size of the safety risk index, and each warning level corresponds to different warning methods and countermeasures.

2. The entity object matching system in a figure based on VR panorama according to claim 1, wherein: The VR panoramic camera (101) forms a star network through a PoE switch, and uses the RTSP protocol to transmit the original image data to the data preprocessing module (2) in real time. The VR panoramic camera (101) obtains an IP through the DHCP protocol and transmits the original data based on the TCP protocol. The three-axis angular velocity and acceleration data collected by the IMU sensor (1011) are synchronously transmitted to the image sensor control unit through the I2C bus, and the lens attitude correction parameters generated by the Kalman filtering algorithm are fed back to the camera drive module in real time through the UDP protocol.

3. The entity object matching system in a picture based on VR panorama according to claim 1, characterized in that: The BRIEF descriptors output by the feature extraction unit (201) of the data preprocessing module (2) are transmitted to the matching and screening unit (202) through shared memory. The valid feature point pairs screened by the RANSAC algorithm are stored in XML format and transmitted to the fusion processing unit (203) through a message queue. The different-level image data generated by the multi-resolution pyramid construction module are transmitted to the weighted fusion operation module through the PCIe bus, and a seamless stitched panoramic image of 8192×4096 pixels is output.

4. A system for matching entity objects in a graph based on VR panorama according to claim 1, characterized in that: The device feature point image coordinates output by the visual positioning subunit of the entity object recognition and positioning module (3) interact with the CAD engineering drawing coordinate mapping table through the RESTAPI interface. The generated homography matrix is transmitted to the UWB positioning fusion subunit through Ethernet, and the world coordinates of the entity object are output through the extended Kalman filter algorithm.

5. A system for matching entity objects in a graph based on VR panorama according to claim 1, characterized in that: The PPG sensor (4011) and the skin conductance sensor (4012) of the personnel emotion feature acquisition module (4) transmit the original signals to the microcontroller built into the safety helmet through the Bluetooth 5.0 protocol. After analog-to-digital conversion, they are sent to the edge computing node through Wi-Fi. The H.265 encoded video stream of the behavior camera (402) is accessed to the industrial Ethernet through the RJ45 interface, and after being mapped by the NAT gateway, it is transmitted to the behavior data processing unit (502) of the emotion feature processing module (5). The physiological sensor (401) and the behavior camera (402) synchronize time through the NTP server, and the data is stored in HDF5 format.

6. The entity object matching system in a picture based on VR panorama according to claim 1, wherein: The analog signal output by the signal conditioning circuit (503) of the emotion feature processing module (5) is converted into a digital signal through a 16-bit DAC and transmitted to the digital signal processing module (504) through the SPI bus. The processed dimensional emotion feature vector is pushed to the safety risk modeling module (6) through the TensorFlow Serving interface, and at the same time, it is stored in the time series database for historical data analysis.

7. The entity object matching system in a figure based on VR panorama according to claim 1, characterized in that: The AU unit feature vector output by the spatial feature extraction branch of the behavior data processing unit (502) and the action trajectory feature output by the time feature extraction branch are spliced by tensors to form a 3072-dimensional fusion feature, which is transmitted to the safety risk modeling module (6) in ProtocolBuffers format through the gRPC interface after dimensionality reduction by the fully connected layer.

8. The entity object matching system in a figure based on VR panorama according to claim 1, wherein: The data fusion interface of the safety risk modeling module (6) obtains the device working condition data of the DCS system in real time through the OPCUA protocol, and inputs the emotion feature vector and the operation task code into the three-dimensional space construction unit (601) through the data bus. The generated risk probability density value is output to the fuzzy inference unit (701) of the warning level matching module (7) through the ModbusTCP protocol, and at the same time, it is marked in real time on the three-dimensional visualization interface.

9. The entity object matching system in a figure based on VR panorama according to claim 1, wherein: The warning level signal output by the fuzzy inference unit (701) of the warning level matching module (7) is transmitted to the warning execution unit (702) through the industrial field bus. Among them, the first-level warning signal triggers the control instruction of the safety helmet vibration module and starts the red strobe light in the equipment area. At the same time, JSON format data containing the coordinates of the entity object and risk indicators is sent to the central control room through the WebSocket protocol to achieve multi-terminal synchronous warning.

10. A method for matching entity objects in a graph based on VR panorama, including applying to a system for matching entity objects in a graph based on VR panorama according to any one of claims 1-9, characterized in that, The method includes the following steps: S1. VR panoramic data collection and preprocessing: VR panoramic image data is collected by VR panoramic cameras (101) distributed at key positions in the dense device area of the petrochemical industry. The collected image data is preprocessed by image stitching, geometric correction, color balance and noise filtering to obtain the preprocessed VR panoramic image. S2. Entity object recognition and positioning: The preprocessed VR panoramic image is input into the entity object recognition and positioning module (3). Using the information in the device feature database (301) and the personnel feature database (302), key devices, isolation point devices and personnel are recognized through image recognition algorithms, and then their specific positions are determined in combination with the three-dimensional coordinate model and positioning algorithm of the device area. S3. Personnel emotional feature collection and processing: Physiological data such as heart rate, blood pressure and skin electrical signals of personnel are collected by physiological sensors (401) set on the personnel safety equipment, and behavioral data such as limb movements and facial expressions of personnel are collected by behavioral cameras (402) set in the device area. The collected data is processed by the physiological data processing unit (501) and the behavioral data processing unit (502) to extract the emotional features of personnel. S4. Safety risk modeling: The extracted personnel emotional features, the location of personnel, operation tasks and historical accident data are input into the safety risk modeling module (6). Through machine learning algorithms, a mapping relationship between emotional features and safety risks is established, and personnel emotions are converted into quantifiable safety risk indicators. S5. Warning level matching and warning: According to the safety risk indicators, the warning level is matched according to the preset warning level logic chain. When the safety risk indicators reach the corresponding warning level threshold, the corresponding warning mechanism is triggered to send a warning signal.

Images