A VR panorama-based entity object matching system and method

Abstract

The present invention discloses a system and method for matching entity objects in a VR panorama-based image, which belongs to the field of VR panoramic image management systems. It proposes a system and method that integrates "high-precision positioning-emotional feature quantification-dynamic risk warning", filling the gap in the existing technology for comprehensive monitoring of personnel status and dynamic risk assessment in complex industrial environments. By converting emotional features into quantifiable safety risk indicators, the system can identify high-risk conditions such as personnel fatigue and tension in advance, providing key technical support for the petrochemical industry to transform from "passive safety" to "active prevention", reducing the risk of operational errors caused by abnormal emotional states of personnel, and upgrading risk assessment from "delayed response" to "real-time prediction", and significantly improving the accuracy of warning level division, meeting the petrochemical industry's stringent requirements for "early detection and early intervention" of safety risks, and providing an efficient technical solution for safety management and control in high-risk scenarios.

Description

Technical Field

[0001] The present invention relates to the field of VR panoramic image management systems, and more particularly to a system and method for matching entity objects in a VR panoramic image. Background Art

[0002] In the densely populated areas of the petrochemical industry, due to the complex distribution of equipment and the high-risk operating environment, accurate positioning and safety monitoring of key equipment, isolation point equipment and personnel are core requirements for ensuring production safety.

[0003] In existing technologies, traditional systems for monitoring personnel safety are mostly limited to location tracking and behavioral compliance checks. They lack real-time collection and analysis of personnel's physiological status (such as heart rate and skin conduction signals) and behavioral and emotional characteristics (such as body movements and facial expressions). Numerous accident cases have shown that emotional states such as fatigue and tension are also important factors that induce operational errors.

[0004] Furthermore, the risk warning mechanisms of related systems are typically based on static models constructed from equipment operating parameters and historical accident data, failing to dynamically integrate personnel emotional characteristics with real-time spatial location information. This results in delayed risk assessments and a relatively crude classification of warning levels, making it difficult to meet the petrochemical industry's demand for "early detection and early intervention" of safety risks.

[0005] Therefore, how to accurately locate key physical objects in densely populated areas of the petrochemical industry, and at the same time, through real-time collection and analysis of personnel emotional characteristics, build a dynamic safety risk model to achieve intelligent matching and accurate early warning of risk levels has become a technical challenge that needs to be solved in this field. Therefore, we proposed a physical object matching system and method based on VR panorama to solve the above problems. Summary of the Invention

[0006] 1. Technical problems to be solved

[0007] In response to the problems existing in the prior art, the purpose of the present invention is to provide a system and method for matching entity objects in images based on VR panorama. It integrates VR panoramic vision, multi-sensor fusion positioning, deep learning emotion feature extraction and machine learning risk modeling technologies to propose a system and method that integrates "high-precision positioning-emotion feature quantification-dynamic risk warning", filling the gap in the prior art in comprehensive monitoring of personnel status and dynamic risk assessment in complex industrial environments.

[0008] 2. Technical solution

[0009] To solve the above problems, the present invention adopts the following technical solutions.

[0010] A system for matching entity objects in a VR panorama-based image, comprising:

[0011] A VR panoramic data acquisition module is used to collect VR panoramic image data from densely populated areas in the petrochemical industry. The module is connected to a number of VR panoramic cameras distributed at key locations within the area, including corners, tops of equipment, and above the main activity paths of personnel. The VR panoramic cameras are equipped with integrated IMU sensors to collect lens posture data.

[0012] a data preprocessing module, communicatively connected to the VR panoramic data acquisition module, for preprocessing the acquired VR panoramic image data, wherein the preprocessing includes image stitching, geometric correction, color balancing, and noise filtering. The data preprocessing module includes a feature extraction unit, a matching and screening unit, and a fusion processing unit, for implementing image stitching using a BRIEF descriptor, a RANSAC algorithm, and a multi-resolution pyramid construction technique;

[0013] An entity object recognition and positioning module is communicatively connected to the data preprocessing module and is used to identify key equipment, isolation point equipment, and personnel from the preprocessed VR panoramic image and determine their positions in the device area. The entity object recognition and positioning module is internally equipped with an equipment feature database, a personnel feature database, and a positioning algorithm unit. The entity object recognition and positioning module uses a deep learning model to identify objects and performs coordinate conversion based on the BIM model. The equipment feature database stores the three-dimensional models, appearance features, and installation location information of key equipment and isolation point equipment. The personnel feature database stores the human body features of personnel and feature information of the safety equipment they wear. The positioning algorithm unit determines the position of the entity object based on a visual positioning algorithm combined with a three-dimensional coordinate model of the device area. The positioning algorithm unit includes a visual positioning subunit and a UWB positioning fusion subunit, which aggregates ranging data through a ZigBee network and fuses positioning based on an extended Kalman filter algorithm.

[0014] A personnel emotion feature acquisition module is used to collect emotion-related data of personnel. The module is communicatively connected to physiological sensors provided on personnel safety jackets and behavioral cameras provided in the device area. The physiological sensors include PPG sensors and skin conductance sensors, which are used to collect physiological data such as heart rate, blood pressure, and skin electrical signals of personnel. The behavioral cameras are used to collect behavioral data such as body movements and facial expressions of personnel, and support H.265 encoding, transmitting data via industrial Ethernet.

[0015] An emotion feature processing module is communicatively connected to the personnel emotion feature acquisition module and is used to process the collected emotion-related data and extract the emotion features of the personnel. The emotion feature processing module includes a physiological data processing unit and a behavioral data processing unit. The emotion feature processing module also includes a signal conditioning circuit and a digital signal processing module, and outputs emotion features through a TensorFlowServing interface. The physiological data processing unit converts physiological data into emotion parameters through a preset physiological emotion model. The behavioral data processing unit analyzes the behavioral data through a deep learning model to extract behavioral emotion features. The behavioral data processing unit extracts spatial features and temporal features based on the deep learning model to generate a fused feature vector.

[0016] A safety risk modeling module is in communication with the emotion feature processing module and is used to establish a safety risk model based on the extracted emotional features of the personnel and convert the personnel emotions into quantifiable safety risk indicators. The safety risk modeling module combines the personnel's location, operation tasks and historical accident data to establish a mapping relationship between emotional features and safety risks through a machine learning algorithm. The safety risk modeling module is internally provided with a three-dimensional space construction unit. The safety risk modeling module accesses the DCS system data through the OPC UA protocol and generates risk probability density based on the three-dimensional space construction unit.

[0017] The warning level matching module is communicatively connected to the security risk modeling module. The warning level matching module includes a fuzzy reasoning unit and a warning execution unit. The warning execution unit has multiple levels of warning measures preset inside, which are used to match the warning level according to the security risk index and the preset warning level logic chain, and trigger the corresponding warning mechanism. The warning level logic chain is divided into multiple warning levels according to the size of the security risk index, and each warning level corresponds to different warning methods and response measures.

[0018] Furthermore, the VR panoramic camera forms a star network through a PoE switch, and uses the RTSP protocol to transmit raw image data to the data preprocessing module in real time. The VR panoramic camera obtains IP through the DHCP protocol and transmits raw data based on the TCP protocol. The three-axis angular velocity and acceleration data collected by the IMU sensor are synchronously transmitted to the image sensor control unit through the I2C bus, and the lens attitude correction parameters generated by the Kalman filter algorithm are fed back to the camera driver module in real time using the UDP protocol.

[0019] Furthermore, the BRIEF descriptor output by the feature extraction unit of the data preprocessing module is transmitted to the matching screening unit through shared memory, and the valid feature point pairs after screening by the RANSAC algorithm are stored in XML format and passed to the fusion processing unit through the message queue. The different levels of image data generated by the multi-resolution pyramid construction technology are transmitted to the weighted fusion operation module through the PCIe bus, and a seamless stitched panoramic image of 8192×4096 pixels is output.

[0020] Furthermore, the device feature point image coordinates output by the visual positioning subunit of the entity object recognition and positioning module interact with the CAD engineering drawing coordinate mapping table through the RESTAPI interface, and the generated homography matrix is ​​transmitted to the UWB positioning fusion subunit via Ethernet, and the world coordinates of the entity object are output through the extended Kalman filter algorithm.

[0021] Furthermore, the PPG sensor and skin conductance sensor of the personnel emotion feature acquisition module transmit the original signal to the microcontroller built into the helmet through the Bluetooth 5.0 protocol, and after analog-to-digital conversion, it is sent to the edge computing node via Wi-Fi. The H.265 encoded video stream of the behavior camera is connected to the industrial Ethernet through the RJ45 interface, and is transmitted to the behavior data processing unit of the emotion feature processing module after mapping through the NAT gateway. The physiological sensor and the behavior camera synchronize time through the NTP server, and the data is stored in HDF5 format.

[0022] Furthermore, the analog signal output by the signal conditioning circuit of the emotion feature processing module is converted into a digital signal through a 16-bit DAC and transmitted to the digital signal processing module through the SPI bus. The processed dimensional emotion feature vector is pushed to the security risk modeling module through the TensorFlowServing interface, and is stored in the time series database for historical data analysis.

[0023] Furthermore, the AU unit feature vector output by the spatial feature extraction branch of the behavior data processing unit and the motion trajectory feature output by the temporal feature extraction branch are concatenated through tensors to form a 3072-dimensional fusion feature, which is then reduced in dimension by the fully connected layer and transmitted to the security risk modeling module through the gRPC interface in ProtocolBuffers format.

[0024] Furthermore, the data fusion interface of the safety risk modeling module obtains the equipment operating condition data of the DCS system in real time through the OPCUA protocol, and inputs the data together with the emotional feature vector and the operation task code into the three-dimensional space construction unit through the data bus. The generated risk probability density value is output to the fuzzy reasoning unit of the warning level matching module through the ModbusTCP protocol, and is marked in real time on the three-dimensional visualization interface.

[0025] Furthermore, the warning level signal output by the fuzzy reasoning unit of the warning level matching module is transmitted to the warning execution unit through the industrial field bus, wherein the first-level warning signal triggers the control instruction of the safety helmet vibration module and starts the red strobe light in the equipment area, and at the same time sends JSON format data containing the coordinates of the entity object and risk indicators to the control room through the WebSocket protocol to realize multi-terminal synchronous warning.

[0026] A method for matching entity objects in a VR panorama image includes the following steps:

[0027] S1. VR panoramic data acquisition and preprocessing: VR panoramic cameras distributed at key locations in the petrochemical industry's densely populated areas collect VR panoramic image data. The collected image data is preprocessed by image stitching, geometric correction, color balancing, and noise filtering to obtain preprocessed VR panoramic images.

[0028] S2. Entity Object Recognition and Positioning: The pre-processed VR panoramic image is input into the entity object recognition and positioning module. Using the information in the equipment feature database and the personnel feature database, an image recognition algorithm is used to identify key equipment, isolation point equipment, and personnel. Their specific locations are then determined using a three-dimensional coordinate model of the device area and a positioning algorithm.

[0029] S3. Collection and processing of emotional characteristics of personnel: The physiological sensors installed on the personnel safety equipment collect the physiological data of the personnel, such as heart rate, blood pressure, and skin electrical signals. The behavioral cameras installed in the device area collect the behavioral data of the personnel's body movements and facial expressions. The collected data are processed by the physiological data processing unit and the behavioral data processing unit to extract the emotional characteristics of the personnel.

[0030] S4. Safety Risk Modeling: The extracted emotional characteristics of personnel, their locations, operational tasks, and historical accident data are input into the safety risk modeling module. A mapping relationship between emotional characteristics and safety risks is established through machine learning algorithms, converting personnel emotions into quantifiable safety risk indicators.

[0031] S5. Warning level matching and warning: According to the security risk indicators, the warning level is matched according to the preset warning level logic chain. When the security risk indicators reach the corresponding warning level threshold, the corresponding warning mechanism is triggered and a warning signal is issued.

[0032] 3. Beneficial effects

[0033] Compared with the prior art, the advantages of the present invention are:

[0034] (1) This solution, through the personnel emotion feature acquisition module and the emotion feature processing module, realizes the multi-dimensional real-time acquisition and in-depth analysis of personnel physiological status (heart rate, blood pressure, skin electrical signal) and behavioral emotion characteristics (body movements, facial expressions). That is, physiological sensors (PPG sensors, skin conductance sensors) and behavioral cameras collect physiological data and behavioral data respectively. After being processed by the signal conditioning circuit, digital signal processing module and deep learning model, 3072-dimensional emotion feature vectors and 3072-dimensional fusion feature vectors are generated, filling the technical gap of the existing system for monitoring personnel emotion status. By converting emotion features into quantifiable safety risk indicators, the system can identify high-risk states such as fatigue and tension in personnel in advance, providing key technical support for the petrochemical industry to transform from "passive safety" to "active prevention", and effectively reducing the risk of operational errors caused by abnormal personnel emotion status;

[0035] (2) This solution constructs a dynamic and precise safety risk assessment system through the safety risk modeling module and the warning level matching module. That is, the safety risk modeling module is based on a machine learning algorithm, dynamically integrating personnel emotional characteristics, real-time spatial location, operation tasks and DCS system equipment working condition data, and using three-dimensional space construction units to generate risk probability density, breaking through the limitations of traditional static models. The warning level matching module realizes intelligent matching and precise warning of risk levels (such as the first-level warning triggering the vibration of the safety helmet, the red strobe light and the simultaneous warning of multiple terminals in the central control room) through the fuzzy reasoning unit and the preset multi-level warning logic chain. This mechanism upgrades risk assessment from "delayed response" to "real-time prediction", and the accuracy of warning level division is significantly improved, meeting the petrochemical industry's stringent requirements for "early detection and early intervention" of safety risks, and providing an efficient technical solution for safety management in high-risk scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the system architecture of the present invention;

[0037] Figure 2 This is a schematic diagram of the technical details of the VR panoramic data acquisition module and data preprocessing module of the present invention;

[0038] Figure 3 This is a schematic diagram of the technical details of the entity object recognition and positioning module and the human emotion feature acquisition module of the present invention;

[0039] Figure 4 This is a schematic diagram of the technical details of the emotional feature processing module, security risk modeling module, and warning level matching module of the present invention;

[0040] Figure 5 Schematic diagram of the method steps of the present invention;

[0041] Figure 6This is a mind map of the principles of step S1 and step S2 of the present invention;

[0042] Figure 7 This is a mind map of the principles of step S3, step S4 and step S5 of the present invention.

[0043] Description of the numbers in the figure:

[0044] 1. VR panoramic data acquisition module; 101. VR panoramic data acquisition module; 1011. IMU sensor;

[0045] 2. Data preprocessing module; 201. Feature extraction unit; 202. Matching and screening unit; 203. Fusion processing unit;

[0046] 3. Entity object recognition and positioning module; 301. Equipment feature database; 302. Personnel feature database; 303. Positioning algorithm unit;

[0047] 4. Personnel emotional feature acquisition module; 401. Physiological sensor; 4011. PPG sensor; 4012. Skin conductance sensor; 402. Behavior camera;

[0048] 5. Emotional feature processing module; 501. Physiological data processing unit; 502. Behavioral data processing unit; 503. Signal conditioning circuit; 504. Digital signal processing module;

[0049] 6. Security risk modeling module; 601. Three-dimensional space construction unit;

[0050] 7. Warning level matching module; 701. Fuzzy reasoning unit; 702. Warning execution unit. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the specification of the present invention; it is obvious that the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0052] Example 1:

[0053] See also Figure 1-Figure 7 , a VR panorama-based entity object matching system, comprising:

[0054] A VR panoramic data acquisition module (1) is used to acquire VR panoramic image data of a densely populated device area in the petrochemical industry. The VR panoramic data acquisition module (1) is communicatively connected to a plurality of VR panoramic cameras (101) distributed at key positions in the device area. The key positions include corners of the device area, tops of equipment, and above main activity paths of personnel. The VR panoramic cameras (101) are internally integrated with an IMU sensor (1011) for acquiring lens posture data.

[0055] A data preprocessing module (2) is communicatively connected to the VR panoramic data acquisition module (1) and is used to preprocess the acquired VR panoramic image data, wherein the preprocessing includes image stitching, geometric correction, color balance and noise filtering. The data preprocessing module (2) includes a feature extraction unit (201), a matching screening unit (202) and a fusion processing unit (203), and is used to realize image stitching through a BRIEF descriptor, a RANSAC algorithm and a multi-resolution pyramid construction technology.

[0056] The entity object recognition and positioning module (3) is in communication with the data preprocessing module (2) and is used to recognize key equipment, isolation point equipment and personnel from the preprocessed VR panoramic image and determine their positions in the device area. The entity object recognition and positioning module (3) is internally equipped with an equipment feature database (301), a personnel feature database (302) and a positioning algorithm unit (303). The entity object recognition and positioning module (3) uses a deep learning model to recognize objects and performs coordinate conversion based on a BIM model. The equipment feature database (301) stores three-dimensional models, appearance features and installation location information of key equipment and isolation point equipment. The personnel feature database (302) stores human body features and feature information of safety equipment worn by personnel. The positioning algorithm unit (303) determines the position of the entity object based on a visual positioning algorithm combined with a three-dimensional coordinate model of the device area. The positioning algorithm unit (303) includes a visual positioning subunit and a UWB positioning fusion subunit, which aggregates ranging data through a ZigBee network and fuses positioning based on an extended Kalman filter algorithm.

[0057] A personnel emotion feature acquisition module (4) is used to acquire emotion-related data of a person, wherein the personnel emotion feature acquisition module (4) is communicatively connected to a physiological sensor (401) provided on a person's safety jacket and a behavior camera (402) provided in a device area, wherein the physiological sensor (401) includes a PPG sensor (4011) and a skin conductance sensor (4012) and is used to acquire physiological data of a person's heart rate, blood pressure, and skin electrical signals, and the behavior camera (402) is used to acquire behavioral data of a person's body movements and facial expressions, and supports H.265 encoding and transmits data via industrial Ethernet;

[0058] An emotion feature processing module (5) is communicatively connected to the personnel emotion feature acquisition module (4) and is used to process the collected emotion-related data and extract the emotion features of the personnel. The emotion feature processing module (5) includes a physiological data processing unit (501) and a behavioral data processing unit (502). The emotion feature processing module (5) also includes a signal conditioning circuit (503) and a digital signal processing module (504). The emotion features are output through a TensorFlowServing interface. The physiological data processing unit (501) converts the physiological data into emotion parameters through a preset physiological emotion model. The behavioral data processing unit (502) analyzes the behavioral data through a deep learning model to extract behavioral emotion features. The behavioral data processing unit (502) extracts spatial features and temporal features based on the deep learning model to generate a fusion feature vector.

[0059] A safety risk modeling module (6) is in communication with the emotion feature processing module (5) and is used to establish a safety risk model based on the extracted emotional features of the personnel, and convert the personnel emotions into quantifiable safety risk indicators. The safety risk modeling module (6) combines the personnel's location, operation tasks and historical accident data to establish a mapping relationship between the emotional features and the safety risks through a machine learning algorithm. A three-dimensional space construction unit (601) is provided inside the safety risk modeling module (6). The safety risk modeling module (6) accesses the DCS system data through the OPC UA protocol and generates a risk probability density based on the three-dimensional space construction unit (601).

[0060] An early warning level matching module (7) is in communication with the security risk modeling module (6). The early warning level matching module (7) includes a fuzzy reasoning unit (701) and an early warning execution unit (702). The early warning execution unit (702) has a preset multi-level early warning measure, which is used to match the early warning level according to the security risk index and a preset early warning level logic chain, and trigger a corresponding early warning mechanism. The early warning level logic chain is divided into multiple early warning levels according to the size of the security risk index, and each early warning level corresponds to a different early warning method and response measure.

[0061] The VR panoramic camera (101) forms a star network through a PoE switch, and uses the RTSP protocol to transmit raw image data to the data preprocessing module (2) in real time. The VR panoramic camera (101) obtains an IP address through the DHCP protocol and transmits raw data based on the TCP protocol. The three-axis angular velocity and acceleration data collected by the IMU sensor (1011) are synchronously transmitted to the image sensor control unit through the I2C bus, and the lens attitude correction parameters generated by the Kalman filter algorithm are fed back to the camera driver module in real time through the UDP protocol.

[0062] The BRIEF descriptor output by the feature extraction unit (201) of the data preprocessing module (2) is transmitted to the matching screening unit (202) through the shared memory, the valid feature point pairs screened by the RANSAC algorithm are stored in XML format and transmitted to the fusion processing unit (203) through the message queue, and the different levels of image data generated by the multi-resolution pyramid construction technology are transmitted to the weighted fusion operation module through the PCIe bus, and a seamless splicing panoramic image of 8192×4096 pixels is output;

[0063] The device feature point image coordinates output by the visual positioning subunit of the entity object recognition and positioning module (3) interact with the CAD engineering drawing coordinate mapping table through a REST API interface, and the generated homography matrix is ​​transmitted to the UWB positioning fusion subunit via Ethernet, and the world coordinates of the entity object are output through an extended Kalman filter algorithm;

[0064] The PPG sensor (4011) and the skin conductance sensor (4012) of the personnel emotion feature acquisition module (4) transmit the original signal to the microcontroller built into the helmet via the Bluetooth 5.0 protocol, and after analog-to-digital conversion, the signal is sent to the edge computing node via Wi-Fi. The H.265 encoded video stream of the behavior camera (402) is connected to the industrial Ethernet via the RJ45 interface, and after being mapped by the NAT gateway, it is transmitted to the behavior data processing unit (502) of the emotion feature processing module (5). The physiological sensor (401) and the behavior camera (402) synchronize time via the NTP server, and the data is stored in the HDF5 format.

[0065] The analog signal output by the signal conditioning circuit (503) of the emotion feature processing module (5) is converted into a digital signal via a 16-bit DAC and transmitted to the digital signal processing module (504) via the SPI bus. The processed dimensional emotion feature vector is pushed to the security risk modeling module (6) via the TensorFlowServing interface and stored in the time series database for historical data analysis.

[0066] The AU unit feature vector output by the spatial feature extraction branch of the behavior data processing unit (502) and the motion trajectory feature output by the temporal feature extraction branch are concatenated into a 3072-dimensional fusion feature through tensor splicing, and after dimensionality reduction by the fully connected layer, the fusion feature is transmitted to the security risk modeling module (6) in Protocol Buffers format through the gRPC interface;

[0067] The data fusion interface of the safety risk modeling module (6) obtains the equipment operating condition data of the DCS system in real time through the OPCUA protocol, and inputs the data together with the emotion feature vector and the operation task code into the three-dimensional space construction unit (601) through the data bus. The generated risk probability density value is output to the fuzzy reasoning unit (701) of the warning level matching module (7) through the ModbusTCP protocol, and is marked in real time on the three-dimensional visualization interface.

[0068] The warning level signal output by the fuzzy reasoning unit (701) of the warning level matching module (7) is transmitted to the warning execution unit (702) via the industrial field bus, wherein the first-level warning signal triggers the control instruction of the helmet vibration module and starts the red strobe light of the equipment area, and at the same time sends JSON format data containing the coordinates of the entity object and the risk index to the central control room via the WebSocket protocol, thereby realizing multi-terminal synchronous warning.

[0069] A method for matching entity objects in a VR panorama image includes the following steps:

[0070] S1. VR panoramic data acquisition and preprocessing: VR panoramic image data is collected by VR panoramic cameras (101) distributed at key locations in the densely populated area of ​​the petrochemical industry. The collected image data is preprocessed by image stitching, geometric correction, color balance and noise filtering to obtain a preprocessed VR panoramic image. The VR panoramic camera 101 automatically obtains an IP address through the DHCP protocol, establishes a TCP connection with the data preprocessing server, and uses a custom binary protocol to transmit the original image data. The preprocessed panoramic image is published in Base64 encoding format through the message middleware to the input queue of the entity object recognition and positioning module. The queue backlog data volume is controlled within 50 frames to ensure real-time performance.

[0071] S2. Entity object recognition and positioning: The pre-processed VR panoramic image is input into the entity object recognition and positioning module (3). The information in the equipment feature database (301) and the personnel feature database (302) is used to identify key equipment, isolation point equipment and personnel through the image recognition algorithm. Then, the specific location is determined by combining the three-dimensional coordinate model of the device area and the positioning algorithm. The device boundary box coordinates output by the improved FasterR-CNN network are mapped to the world coordinate system coordinates through the BIM model coordinate conversion matrix of the device area. The result is published to the positioning algorithm unit in the form of a ROS message. At the same time, the device type code and the sensor data address are mapped through a hash table and stored in the memory database for subsequent risk modeling calls.

[0072] S3. Collection and processing of personnel emotional characteristics: The physiological sensor (401) provided on the personnel safety equipment is used to collect the physiological data of the personnel's heart rate, blood pressure, and skin electrical signal; the behavioral camera (402) provided in the device area is used to collect the personnel's body movements and facial expression behavioral data; the physiological data processing unit (501) and the behavioral data processing unit (502) are used to process the collected data and extract the personnel's emotional characteristics; the physiological sensor and the behavioral camera are synchronized through the NTP server; the generated UTC timestamp is used as the data frame header; the physiological data and the video frame are synchronized through the timestamp alignment algorithm at the edge computing node; abnormal data with a time difference of more than 50ms is eliminated; the synchronized data stream is stored in the HDF5 format and input into the emotional characteristics processing module;

[0073] S4. Safety risk modeling: The extracted emotional characteristics of personnel, personnel location, operation tasks and historical accident data are input into the safety risk modeling module (6). A mapping relationship between emotional characteristics and safety risks is established through machine learning algorithms, and personnel emotions are converted into quantifiable safety risk indicators.

[0074] S5. Warning level matching and warning: According to the security risk indicators, the warning level is matched according to the preset warning level logic chain. When the security risk indicators reach the corresponding warning level threshold, the corresponding warning mechanism is triggered and a warning signal is issued.

[0075] This system uses multi-module collaboration to achieve precise positioning of key physical objects within densely populated areas of the petrochemical industry, real-time collection and analysis of personnel emotional characteristics, and dynamic safety risk modeling and early warning. Its core working principles are as follows:

[0076] 1. VR Panoramic Data Collection and Preprocessing

[0077] 1.Synchronous collection of multi-source data

[0078] The VR panoramic cameras (101) distributed at key locations in the device area (corners, top of equipment, and above the path of personnel activities) form a star network through a PoE switch, automatically obtain IP addresses based on the DHCP protocol, and transmit raw image data in real time through the RTSP protocol in a TCP long connection mode (supporting 8192×4096 pixel resolution). The IMU sensor (1011) built into the camera collects three-axis angular velocity (±2000° / s range) and acceleration (±16g range) data at a frequency of 100Hz, and synchronizes it to the image sensor control unit through the I2C bus (transmission rate 400kHz). After the extended Kalman filter algorithm (state transfer matrix F = [[1,Δt,0.5Δt 2],[0,1,Δt],[0,0,1]]) calculates the lens posture correction parameters (translation vector t, rotation matrix R), and feeds them back to the camera driver module in real time via the UDP protocol (port number 50001) to achieve motion blur correction and perspective deviation compensation during image acquisition.

[0079] 2. Image preprocessing pipeline

[0080] The original image data is transmitted to the data preprocessing module (2) via the TCP protocol (port number 50002) and processed according to the following process:

[0081] Feature extraction: The feature extraction unit (201) divides each frame of image into 8×8 pixel blocks, uses the BRIEF descriptor to generate a 64-bit binary feature vector, and transmits it to the matching and screening unit (202) at a rate of 400MB / s through the shared memory (ShmOpen interface).

[0082] Robust matching: The matching screening unit (202) removes incorrectly matched point pairs based on the RANSAC algorithm (1000 iterations, 3 pixels of internal point threshold), and stores the retained valid feature point pairs in the memory buffer in XML format (including key point coordinates and descriptor hash values), and transmits them to the fusion processing unit (203) through the ZeroMQ message queue (subscribe-publish mode).

[0083] Multi-scale stitching: The fusion processing unit (203) constructs a three-layer multi-resolution pyramid (bottom layer resolution 8192×4096, top layer 1024×512), and transmits each layer of image to the weighted fusion operation module via the PCIe 3.0 bus (bandwidth 8GB / s). The Laplace pyramid fusion algorithm (fusion weight matrix W(x,y)=0.5+0.5×sin(πx / W)) is used to generate a seamless stitching panoramic image. Geometric correction (based on the OpenCV distortion correction model), color balance (gray world algorithm) and median filtering (kernel size 3×3) are completed simultaneously. The pre-processed image is encoded in Base64 and published to the input queue of the entity object recognition and positioning module through the Kafka message middleware (number of partitions 8, number of copies 2). The maximum backlog of the queue is set to 50 frames. The data flow rate is controlled by the token bucket algorithm (token generation rate 20 frames / s) to ensure real-time performance.

[0084] 2. Entity Object Recognition and Positioning

[0085] 1. After cross-modal target recognition preprocessing, the VR panoramic image is input into the entity object recognition and positioning module (3), and the target classification and positioning are achieved through a dual-branch deep learning model:

[0086] Equipment identification branch: The improved Faster R-CNN network (backbone network is ResNet-50, feature pyramid level P2-P5) is trained based on the equipment feature database (301). The database stores 3D CAD models (STL format), appearance feature vectors (HOG+LBP fusion features) and installation location BIM coordinates (accuracy ±5cm) of more than 200 types of key equipment / isolation point equipment. The network outputs the device bounding box coordinates (pixel level), category confidence (threshold ≥0.8) and equipment type code (such as "P-001" for pump equipment).

[0087] Personnel identification branch: The YOLOv5s network is optimized for the characteristics of personnel safety equipment (helmet color, reflective strip pattern, gas detector model), with an input size of 640×640. It outputs the coordinates of key points on the human body (17 COCO key points) and the safety equipment status label (such as "helmet is worn normally" and "respirator connection is abnormal").

[0088] 2. 3D space coordinate calculation

[0089] Visual positioning basic matrix: The mapping relationship between the CAD engineering drawing of the device area and the world coordinate system is obtained through the BIM model. The homography matrix H is calculated using the eight-point method (8 degrees of freedom, 8 minimum matching point pairs), and the image coordinates (u, v) of the device feature points are converted to world coordinates (X, Y, Z):

[0090]

[0091] The results are exchanged in JSON format through the REST API interface (URI path / api / visual location), with a transmission delay of ≤200ms.

[0092] UWB fusion positioning: The UWB base station (anchor point) deployed in the device area aggregates the ranging data (TOF ranging accuracy ±10cm) through the ZigBee network (IEEE 802.15.4 protocol, transmission rate 250kbps), and fuses it with the visual positioning results through the extended Kalman filter algorithm (state vector The process noise covariance Q = diag([0.1, 0.1, 0.1, 0.01, 0.01, 0.01])) outputs the world coordinates of the entity object (root mean square error ≤ 5 cm). The positioning result is published as a ROS message (topic name / target_pos), and a mapping relationship between the device type code and the sensor data address is established through the Redis memory database (hash table structure) (for example, the device code "V-002" corresponds to the temperature sensor address 0x0102) for real-time call by the security risk modeling module.

[0093] 3. Collection and Processing of Personnel Emotional Characteristics

[0094] 1. Synchronous collection of physiological and behavioral data

[0095] Physiological data link: The PPG sensor (4011) collects photoplethysmographic signals at a sampling rate of 500Hz, and the skin conductance sensor (4012) collects galvanic skin response signals at a sampling rate of 200Hz. The original analog signals are transmitted to the STM32 microcontroller built into the helmet via the Bluetooth 5.0 protocol (transmission rate 2Mbps, connection interval 7.5ms), converted into digital signals by a 16-bit ADC (sampling accuracy ±0.1% FS), and sent to the edge computing node (deployed in the device area edge server) via Wi-Fi 6 (802.11ax, channel bandwidth 80MHz).

[0096] Behavioral data link: The behavioral camera (402) uses a global shutter CMOS sensor to capture 1920×1080 pixel video stream at a frame rate of 30fps, and is connected to the industrial Ethernet (the switch supports IEEE 1588v2 precision clock synchronization) via the RJ45 interface through H.265 encoding (compression ratio 1:100, bit rate 2Mbps), and is transmitted to the behavioral data processing unit (502) of the emotion feature processing module (5) through the NAT gateway (port mapping 8080→50003).

[0097] Spatiotemporal alignment: Physiological sensors and behavioral cameras obtain UTC time through an NTP server (time synchronization accuracy ≤ 1ms). The generated timestamp (accurate to microseconds) serves as the data frame header. Edge computing nodes use the Dynamic Time Warping (DTW) algorithm to time-align physiological data sequences with video frames, eliminating anomalous data with time differences greater than 50ms. The synchronized data streams are stored in a distributed file system in HDF5 format (clustered storage: physiological data group / behavior_data, video data group / video_frames).

[0098] 2. Multimodal Feature Engineering

[0099] Physiological feature extraction: The original physiological signal is amplified (gain 1000 times) and filtered (50Hz notch filter) by the signal conditioning circuit (503), and then transmitted to the digital signal processing module (504) via the SPI bus (clock frequency 10MHz). Based on the cardiovascular health model, parameters such as heart rate variability (HRV, RMSSD index) and skin conductance level (SCL) are calculated and mapped to emotional indicators through preset thresholds (such as HRV < 50ms corresponds to "fatigue", SCL > 10μS corresponds to "tension") to generate a 16-dimensional emotional feature vector (such as [HRV, SCL, respiratory rate, ...]). It is pushed to the security risk modeling module (6) in ProtoBuffer format through the TensorFlow Serving interface (gRPC protocol, port number 8500) and stored in the InfluxDB time series database (time accuracy nanoseconds, retention policy 30 days).

[0100] Behavior feature extraction: The behavior data processing unit (502) adopts a dual-stream CNN architecture:

[0101] Spatial Stream Network: Input video frames are processed through ResNet-34 to extract appearance features and output 64-dimensional AU unit feature vectors (such as FACS-encoded eyebrow raiser, lip corner puller, and other action unit intensities).

[0102] Temporal Stream Network: 10 consecutive frames of motion trajectory features are extracted through 3D ConvLSTM to generate a 128-dimensional temporal feature vector (such as the spatiotemporal distribution of joint motion speed and acceleration). The two types of features are concatenated into a 3072-dimensional fusion feature vector through tensor splicing. The dimensionality is reduced to 128 through a fully connected layer (512 neurons, ReLU activation function) and transmitted to the security risk modeling module (6) in Protocol Buffers format through the gRPC interface (service name BehaviorFeatureService). The transmission delay is ≤300ms.

[0103] 4. Security Risk Modeling and Dynamic Assessment

[0104] 1. Multi-source data fusion modeling

[0105] The safety risk modeling module (6) obtains the equipment operating data (such as temperature, pressure, flow, and update frequency of 1 second) of the DCS system in real time through the OPC UA protocol (server address opc.tcp: / / dcs-system:4840), and inputs the emotional feature vector, the world coordinates (X, Y, Z) of the entity object, and the operation task code (10 types of tasks are preset, such as "T-01" for hot work) into the three-dimensional space construction unit (601) through the data bus (bandwidth 1GB / s). The risk prediction model is constructed based on the XGBoost algorithm. The feature engineering includes:

[0106] Personnel characteristics: sentiment feature vector (16 dimensions), safety equipment status label (one-hot encoding, 5 dimensions)

[0107] Spatial characteristics: distance from dangerous equipment (Euclidean distance calculation), regional risk level (preset device area risk heat map)

[0108] Equipment characteristics: DCS parameter normalization values ​​(Z-score standardization), historical fault frequency (Poisson distribution parameters)

[0109] Task characteristics: Task risk coefficient (e.g., hot work coefficient 1.8, routine inspection coefficient 0.5)

[0110] 2. Risk Probability Density Generation The three-dimensional space construction unit (601) divides the device area into 0.5m×0.5m×0.5m grids and calculates the risk probability density (unit: times / cubic meter·hour) of each grid based on kernel density estimation (KDE, bandwidth h=1.0). The formula is:

[0111]

[0112] The kernel function K is a Gaussian kernel. The generated risk density field is rendered in real time on a 3D visualization interface using the Three.js library, and risk hotspots are marked with different colors (e.g. red ≥ 0.5 times / m 3 h, yellow 0.10.5 times / m 3 The risk probability density value is output to the warning level matching module (7) via the Modbus TCP protocol (function code 0x03, register address 40001-40100), with an update cycle of 500ms.

[0113] V. Warning Level Matching and Intelligent Response

[0114] 1. Fuzzy logic reasoning engine

[0115] The fuzzy reasoning unit (701) of the warning level matching module (7) adopts the Mamdani reasoning model, and the input variable is the risk probability density value (domain [0,1] times / m 3h, fuzzy set {low, medium, high}, membership function is triangular distribution), the output variable is the warning level (domain {level III, level II, level I}, membership function is trapezoidal distribution). The rule base contains 9 fuzzy rules, for example:

[0116] If the risk density is "low" and the emotions of the personnel are "normal", the warning level is "Level III"

[0117] If the risk density is "high" and the personnel are "tense", the warning level is "Level I"

[0118] 2. Graded early warning implementation mechanism

[0119] The early warning execution unit (702) triggers a multi-level response based on the fuzzy reasoning result:

[0120] Level Ⅰ warning (risk density ≥ 0.6):

[0121] Hardware response: A control command is sent to the helmet via the CAN bus (baud rate 1Mbps), driving the vibration module (frequency 200Hz, amplitude 0.5mm) to vibrate continuously for 5 seconds; a Modbus RTU command (slave address 0x01, function code 0x0F) is sent to the indicator light in the equipment area to start the red strobe light (frequency 2Hz).

[0122] Software response: Send JSON data (format {"entity_id":"P

[0123] 003","risk_score":

[0124] 0.82, "location": [

[0125] 12.5,

[0126] 3.2,

[0127] 4.8]}), triggering a pop-up alarm in the SCADA system, a red mark on the electronic map, and a voice announcement ("High-risk area, please pay attention!").

[0128] Level II warning (0.3≤risk density<0.6): The yellow strobe light is activated through the RS-485 bus (transmission rate 9600bps), the risk trend curve is displayed in the central control room, and a warning notification (including risk location and personnel emotional status) is simultaneously pushed to the security manager's mobile phone APP.

[0129] Level III warning (risk density <0.3): Risk events are only recorded in the system log, a 24-hour trend analysis report is generated through the time series database, and the security training system is automatically triggered to push relevant case study materials.

[0130] 3. Closed-loop feedback optimization

[0131] The results of early warning execution (such as personnel evacuation routes and equipment shutdown status) are fed back to the safety risk modeling module (6) via the industrial fieldbus (PROFINET protocol) as real-time label data for model updates. The model retraining process is automatically triggered every week, and the XGBoost model parameters are updated based on the newly collected 100,000 sample data (including positive and negative examples), ensuring that the risk assessment accuracy continues to improve over time.

[0132] The above description is merely a preferred embodiment of the present invention; however, the scope of protection of the present invention is not limited thereto. Any person skilled in the art who, within the technical scope disclosed by the present invention, makes equivalent substitutions or modifications based on the technical solutions and improved concepts of the present invention shall be covered by the scope of protection of the present invention.

Claims

1. A system for matching physical objects in images based on VR panoramas, characterized in that: include: A VR panoramic data acquisition module (1) is used to acquire VR panoramic image data of a densely populated device area in the petrochemical industry. The VR panoramic data acquisition module (1) is communicatively connected to a plurality of VR panoramic cameras (101) distributed at key positions in the device area. The key positions include corners of the device area, tops of equipment, and above main activity paths of personnel. The VR panoramic cameras (101) are internally integrated with an IMU sensor (1011) for acquiring lens posture data. A data preprocessing module (2) is communicatively connected to the VR panoramic data acquisition module (1) and is used to preprocess the acquired VR panoramic image data, wherein the preprocessing includes image stitching, geometric correction, color balance and noise filtering. The data preprocessing module (2) includes a feature extraction unit (201), a matching screening unit (202) and a fusion processing unit (203), and is used to realize image stitching through a BRIEF descriptor, a RANSAC algorithm and a multi-resolution pyramid construction technology; An entity object recognition and positioning module (3) is communicatively connected to the data preprocessing module (2) and is used to recognize key equipment, isolation point equipment and personnel from the preprocessed VR panoramic image and determine their positions in the device area. The entity object recognition and positioning module (3) is internally equipped with an equipment feature database (301), a personnel feature database (302) and a positioning algorithm unit (303). The entity object recognition and positioning module (3) uses a deep learning model to recognize objects and performs coordinate conversion based on a BIM model. The equipment feature database (301) stores three-dimensional models, appearance features and installation location information of key equipment and isolation point equipment. The personnel feature database (302) stores human body features and feature information of safety equipment worn by personnel. The positioning algorithm unit (303) determines the position of the entity object based on a visual positioning algorithm combined with a three-dimensional coordinate model of the device area. The positioning algorithm unit (303) includes a visual positioning subunit and a UWB positioning fusion subunit, which aggregates ranging data through a ZigBee network and fuses positioning based on an extended Kalman filter algorithm. A personnel emotion feature acquisition module (4) is used to acquire emotion-related data of a person, wherein the personnel emotion feature acquisition module (4) is communicatively connected to a physiological sensor (401) provided on a person's safety jacket and a behavior camera (402) provided in a device area, wherein the physiological sensor (401) includes a PPG sensor (4011) and a skin conductance sensor (4012) and is used to acquire physiological data of a person's heart rate, blood pressure, and skin electrical signals, and the behavior camera (402) is used to acquire behavioral data of a person's body movements and facial expressions, and supports H.265 encoding and transmits data via industrial Ethernet; An emotion feature processing module (5) is communicatively connected to the person emotion feature acquisition module (4) and is used to process the collected emotion-related data and extract the emotion features of the person. The emotion feature processing module (5) includes a physiological data processing unit (501) and a behavioral data processing unit (502). The emotion feature processing module (5) also includes a signal conditioning circuit (503) and a digital signal processing module (504). The emotion features are output through a TensorFlowServing interface. The physiological data processing unit (501) converts the physiological data into emotion parameters through a preset physiological emotion model. The behavioral data processing unit (502) analyzes the behavioral data through a deep learning model to extract behavioral emotion features. The behavioral data processing unit (502) extracts spatial features and temporal features based on the deep learning model to generate a fusion feature vector. A safety risk modeling module (6) is in communication with the emotion feature processing module (5) and is used to establish a safety risk model based on the extracted emotional features of the personnel, and convert the emotions of the personnel into quantifiable safety risk indicators. The safety risk modeling module (6) combines the personnel's location, operation tasks and historical accident data to establish a mapping relationship between the emotional features and the safety risks through a machine learning algorithm. A three-dimensional space construction unit (601) is provided inside the safety risk modeling module (6). The safety risk modeling module (6) accesses the DCS system data through the OPCUA protocol and generates a risk probability density based on the three-dimensional space construction unit (601); An early warning level matching module (7) is in communication with the security risk modeling module (6). The early warning level matching module (7) includes a fuzzy reasoning unit (701) and an early warning execution unit (702). The early warning execution unit (702) has a preset multi-level early warning measure for matching the early warning level according to the security risk index and the preset early warning level logic chain, and triggering the corresponding early warning mechanism. The early warning level logic chain is divided into multiple early warning levels according to the size of the security risk index, and each early warning level corresponds to a different early warning method and response measure.

2. The system for matching physical objects in a VR panorama image according to claim 1, characterized in that: The VR panoramic camera (101) forms a star network through a PoE switch, and uses the RTSP protocol to transmit raw image data to the data preprocessing module (2) in real time. The VR panoramic camera (101) obtains an IP through the DHCP protocol and transmits raw data based on the TCP protocol. The three-axis angular velocity and acceleration data collected by the IMU sensor (1011) are synchronously transmitted to the image sensor control unit through the I2C bus, and the lens attitude correction parameters generated by the Kalman filter algorithm are fed back to the camera driver module in real time through the UDP protocol.

3. The system for matching physical objects in a VR panorama image according to claim 1, characterized in that: The BRIEF descriptor output by the feature extraction unit (201) of the data preprocessing module (2) is transmitted to the matching screening unit (202) via a shared memory. The valid feature point pairs screened by the RANSAC algorithm are stored in XML format and transmitted to the fusion processing unit (203) via a message queue. The image data of different levels generated by the multi-resolution pyramid construction technology are transmitted to the weighted fusion operation module via a PCIe bus, and a seamless spliced ​​panoramic image of 8192×4096 pixels is output.

4. The system for matching physical objects in a VR panorama image according to claim 1, characterized in that: The device feature point image coordinates output by the visual positioning subunit of the entity object recognition and positioning module (3) interact with the CAD engineering drawing coordinate mapping table through a REST API interface, and the generated homography matrix is ​​transmitted to the UWB positioning fusion subunit via Ethernet, and the world coordinates of the entity object are output through an extended Kalman filter algorithm.

5. The system for matching physical objects in a VR panorama image according to claim 1, characterized in that: The PPG sensor (4011) and the skin conductance sensor (4012) of the personnel emotion feature acquisition module (4) transmit the original signal to the microcontroller built into the helmet via the Bluetooth 5.0 protocol, and after analog-to-digital conversion, the signal is sent to the edge computing node via Wi-Fi. The H.265 encoded video stream of the behavior camera (402) is connected to the industrial Ethernet via the RJ45 interface, and after being mapped by the NAT gateway, it is transmitted to the behavior data processing unit (502) of the emotion feature processing module (5). The physiological sensor (401) and the behavior camera (402) synchronize time via the NTP server, and the data is stored in the HDF5 format.

6. The system for matching physical objects in a VR panorama image according to claim 1, characterized in that: The analog signal output by the signal conditioning circuit (503) of the emotion feature processing module (5) is converted into a digital signal via a 16-bit DAC and transmitted to the digital signal processing module (504) via the SPI bus. The processed dimensional emotion feature vector is pushed to the security risk modeling module (6) via the TensorFlowServing interface and is stored in the time series database for historical data analysis.

7. The system for matching physical objects in a VR panorama image according to claim 1, characterized in that: The AU unit feature vector output by the spatial feature extraction branch of the behavior data processing unit (502) and the motion trajectory feature output by the temporal feature extraction branch are concatenated into a 3072-dimensional fusion feature through tensor splicing, and after dimensionality reduction by the fully connected layer, the fusion feature is transmitted to the security risk modeling module (6) in Protocol Buffers format through the gRPC interface.

8. The system for matching physical objects in a VR panorama image according to claim 1, characterized in that: The data fusion interface of the safety risk modeling module (6) obtains the equipment operating condition data of the DCS system in real time through the OPCUA protocol, and inputs the data together with the emotion feature vector and the operation task code into the three-dimensional space construction unit (601) through the data bus. The generated risk probability density value is output to the fuzzy reasoning unit (701) of the warning level matching module (7) through the ModbusTCP protocol, and is marked in real time on the three-dimensional visualization interface.

9. The system for matching physical objects in a VR panorama image according to claim 1, characterized in that: The warning level signal output by the fuzzy reasoning unit (701) of the warning level matching module (7) is transmitted to the warning execution unit (702) via the industrial field bus, wherein the first-level warning signal triggers the control instruction of the helmet vibration module and starts the red strobe light of the equipment area, and at the same time sends JSON format data containing the coordinates of the entity object and the risk index to the central control room via the WebSocket protocol, thereby realizing multi-terminal synchronous warning.

10. A method for matching entity objects in a VR panorama-based image, comprising the method applied to the VR panorama-based entity object matching system according to claims 1-9, characterized in that: The following steps are involved: S1. VR panoramic data acquisition and preprocessing: VR panoramic image data is collected by VR panoramic cameras (101) distributed at key locations in the petrochemical industry's densely populated device area, and the collected image data is preprocessed by image stitching, geometric correction, color balancing, and noise filtering to obtain a preprocessed VR panoramic image; S2. Entity object recognition and positioning: The pre-processed VR panoramic image is input into the entity object recognition and positioning module (3), and the information in the equipment feature database (301) and the personnel feature database (302) are used to identify key equipment, isolation point equipment and personnel through an image recognition algorithm, and then their specific locations are determined by combining the three-dimensional coordinate model of the device area and the positioning algorithm; S3. Collection and processing of personnel emotional characteristics: The physiological sensors (401) installed on the personnel safety equipment collect the personnel's heart rate, blood pressure, and skin electrical signal physiological data, and the behavioral cameras (402) installed in the device area collect the personnel's body movements and facial expression behavioral data, and the physiological data processing unit (501) and the behavioral data processing unit (502) process the collected data to extract the personnel's emotional characteristics; S4. Safety risk modeling: The extracted emotional characteristics of personnel, personnel location, operation tasks and historical accident data are input into the safety risk modeling module (6). The mapping relationship between emotional characteristics and safety risks is established through machine learning algorithms, and personnel emotions are converted into quantifiable safety risk indicators; S5. Warning level matching and warning: According to the security risk indicators, the warning level is matched according to the preset warning level logic chain. When the security risk indicators reach the corresponding warning level threshold, the corresponding warning mechanism is triggered and a warning signal is issued.

Images