Behavior recognition method, device, equipment, storage medium and computer program product

Abstract

The present application relates to the field of behavior recognition technology, and discloses a behavior recognition method, device, equipment, storage medium and computer program product, including: obtaining multi-source heterogeneous data collected by a multimodal sensor network, and performing spatiotemporal synchronization and fusion processing on the multi-source heterogeneous data based on a spatiotemporal synchronization algorithm to obtain target comprehensive data; extracting low-dimensional feature vectors from the target comprehensive data, and combining the attention mechanism to perform time series modeling on the low-dimensional feature vectors to construct a behavior dynamic model; performing behavior recognition on the real-time collected operation process information according to the behavior dynamic model, and outputting the personnel behavior category. By integrating multi-source heterogeneous data in the complex environment of the power plant, and accurately extracting key features, and capturing the dynamic change laws of behavior and state, the model can accurately distinguish various types of personnel behaviors under normal operating procedures or complex and changeable working conditions, and realizes real-time and accurate recognition of violations and abnormal behaviors in power plant operations.

Description

Technical Field

[0001] The present application relates to the field of behavior recognition technology, and in particular to a behavior recognition method, device, equipment, storage medium and computer program product. Background Art

[0002] At present, the operation process of power plants is complex and the behaviors of personnel are diverse. There are subtle differences between different behaviors, and there are various interference factors in the operating environment, which makes the detection system unable to quickly and accurately classify various behaviors and detect violations and abnormal behaviors in a timely manner. Summary of the invention

[0003] The main purpose of this application is to provide a behavior recognition method, device, equipment, storage medium and computer program product, which aims to solve the technical problems that the current power plant operation process is complex, the personnel behavior is diverse, there are slight differences between different behaviors, and there are various interference factors in the operating environment, which leads to the detection system being unable to quickly and accurately classify various behaviors and timely discover violations and abnormal behaviors.

[0004] To achieve the above objectives, the present application proposes a behavior recognition method, which includes:

[0005] Acquire multi-source heterogeneous data collected by a multimodal sensor network, and perform spatiotemporal synchronization and fusion processing on the multi-source heterogeneous data based on a spatiotemporal synchronization algorithm to obtain target comprehensive data;

[0006] Extracting low-dimensional feature vectors from the target comprehensive data, and combining the attention mechanism to perform time series modeling on the low-dimensional feature vectors to construct a behavior dynamic model;

[0007] The behavior dynamic model is used to perform behavior recognition on the operation process information collected in real time, and the behavior category of the personnel is output.

[0008] Optionally, the step of performing behavior recognition on the real-time collected work process information according to the behavior dynamic model and outputting the personnel behavior category includes:

[0009] Analyze the collected operation process information to obtain the personnel behavior process and equipment operation process;

[0010] Inputting the personnel behavior process and the equipment operation process into the behavior dynamic model, and calculating the behavior similarity between the operation process information and the predefined behavior pattern;

[0011] The personnel behavior category of the operation process information is determined according to the behavior similarity.

[0012] Optionally, the personnel behavior categories include normal behavior, illegal behavior and abnormal behavior;

[0013] The step of determining the personnel behavior category of the operation process information according to the behavior similarity includes:

[0014] If the behavior similarity is within the similarity threshold of the normal behavior pattern, and the operation process information is consistent with the overall operation process, it is determined to be a normal behavior;

[0015] If the behavior similarity is within the similarity threshold range of the violation behavior pattern, the operation detail features in the operation process information are matched with the preset violation behavior feature library, and when the match is successful, the personnel behavior category is determined as a violation behavior;

[0016] If the behavior similarity is within the similarity threshold range of the abnormal behavior pattern, it is determined to be an abnormal behavior.

[0017] Optionally, after the step of determining the behavior as abnormal behavior if the behavior similarity is within the similarity threshold range of the abnormal behavior pattern, the step further includes:

[0018] Parsing the operation process information through the multimodal sensor network to obtain target process information causing the abnormal behavior and a category of the target process information;

[0019] If the category is the personnel behavior process, then obtaining the historical operation data of the personnel before the abnormal behavior occurs, and determining the cause of the abnormal behavior according to the historical operation data;

[0020] If the category is the device operation process, then the operating parameter change information of the device before the abnormal behavior occurs is obtained, and the cause of the device failure is determined based on the operating parameter change information.

[0021] Optionally, the step of extracting low-dimensional feature vectors from the target comprehensive data, and performing time series modeling on the low-dimensional feature vectors in combination with an attention mechanism to construct a behavior dynamic model includes:

[0022] Inputting the target comprehensive data into an encoder of a deep autoencoder network to obtain a low-dimensional feature vector;

[0023] Assigning different weights to each element in the low-dimensional feature vector based on the attention mechanism to obtain a weighted target feature vector;

[0024] A long short-term memory network is used to perform time series modeling on the target feature vector, and the output of the fully connected layer of the long short-term memory network is used as the final output to construct the behavior dynamic model.

[0025] Optionally, the multi-source heterogeneous data includes historical personnel behavior, historical equipment operation status information and environmental condition information;

[0026] The step of acquiring multi-source heterogeneous data collected by the multimodal sensor network, and performing spatiotemporal synchronization and fusion processing on the multi-source heterogeneous data based on a spatiotemporal synchronization algorithm to obtain target comprehensive data includes:

[0027] Collect multi-source heterogeneous data through pre-deployed multimodal sensor networks;

[0028] Calibrate the time reference between the historical personnel behavior and the historical equipment operation status information based on the time-space synchronization algorithm to obtain the historical operation status information;

[0029] The historical operation status information and the environmental status information are fused by data layer fusion to obtain the target comprehensive data.

[0030] In addition, to achieve the above purpose, the present application also proposes a behavior recognition device, which includes:

[0031] A data acquisition module is used to acquire multi-source heterogeneous data collected by a multimodal sensor network, and perform spatiotemporal synchronization and fusion processing on the multi-source heterogeneous data based on a spatiotemporal synchronization algorithm to obtain target comprehensive data;

[0032] A model building module, used to extract low-dimensional feature vectors from the target comprehensive data, and perform time series modeling on the low-dimensional feature vectors in combination with an attention mechanism to build a behavior dynamic model;

[0033] The behavior recognition module is used to perform behavior recognition on the operation process information collected in real time according to the behavior dynamic model and output the personnel behavior category.

[0034] In addition, to achieve the above-mentioned purpose, the present application also proposes a behavior recognition device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the behavior recognition method described above.

[0035] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the behavior recognition method described above are implemented.

[0036] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the behavior recognition method described above are implemented.

[0037] The present application discloses the acquisition of multi-source heterogeneous data collected by a multimodal sensor network, and the spatiotemporal synchronization and fusion processing of the multi-source heterogeneous data based on the spatiotemporal synchronization algorithm to obtain target comprehensive data; extracting low-dimensional feature vectors from the target comprehensive data, and combining the attention mechanism to perform time series modeling on the low-dimensional feature vectors to construct a behavior dynamic model; performing behavior recognition on the real-time collected operation process information according to the behavior dynamic model, and outputting the personnel behavior category. By integrating multi-source heterogeneous data in the complex environment of the power plant, and accurately extracting key features, and capturing the dynamic change rules of behavior and state, various types of personnel behaviors can be accurately distinguished regardless of normal operating procedures or complex and changeable working conditions, realizing real-time and accurate recognition of violations and abnormal behaviors in power plant operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0040] Figure 1 This is a flow chart of the first embodiment of the behavior recognition method of the present application;

[0041] Figure 2 This is a flow chart of the data collection system for this application;

[0042] Figure 3 This is a flow chart of the second embodiment of the behavior recognition method of the present application;

[0043] Figure 4 This is a flow chart of the third embodiment of the behavior recognition method of the present application;

[0044] Figure 5 This is a schematic diagram of the module structure of the behavior recognition device according to an embodiment of the present application;

[0045] Figure 6 Schematic diagram of the device structure of the hardware operating environment involved in the behavior recognition method in the embodiment of the present application.

[0046] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0048] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0049] The main solution of the embodiment of the present application is: to obtain multi-source heterogeneous data collected by a multimodal sensor network, and to perform spatiotemporal synchronization and fusion processing on the multi-source heterogeneous data based on a spatiotemporal synchronization algorithm to obtain target comprehensive data; to extract low-dimensional feature vectors from the target comprehensive data, and to perform time series modeling on the low-dimensional feature vectors in combination with an attention mechanism to construct a behavior dynamic model; to perform behavior recognition on the job process information collected in real time according to the behavior dynamic model, and to output personnel behavior categories.

[0050] Due to the complex operation process of power plants and the diverse behaviors of personnel, there are subtle differences between different behaviors, and there are various interference factors in the operating environment, which makes it impossible for the detection system to quickly and accurately classify various behaviors and detect violations and abnormal behaviors in a timely manner.

[0051] Therefore, this application provides a behavior recognition method, which integrates the multi-source heterogeneous data collected from all directions by a multimodal sensor network into accurate and information-rich target comprehensive data after being processed by a spatiotemporal synchronization algorithm, effectively solving the spatiotemporal inconsistency of the data, greatly improving the data quality and availability, so that the data can fully and accurately reflect the actual state of the power plant operation. On this basis, a behavioral dynamic model constructed using a deep autoencoder network and an attention mechanism is used to accurately extract key features and capture the dynamic changes of behavior and state, greatly enhancing the accuracy and adaptability of behavior recognition. Whether it is a normal operating process or a complex and changeable working condition, it can accurately distinguish between various types of personnel behaviors. Ultimately, real-time and accurate identification of violations and abnormal behaviors in power plant operations is achieved.

[0052] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, etc. The following takes the behavior monitoring system as an example to illustrate this embodiment and the following embodiments.

[0053] Based on this, the present application embodiment provides a behavior recognition method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the behavior recognition method of the present application.

[0054] In this embodiment, the behavior recognition method includes:

[0055] Step S10, acquiring multi-source heterogeneous data collected by a multimodal sensor network, and performing spatiotemporal synchronization and fusion processing on the multi-source heterogeneous data based on a spatiotemporal synchronization algorithm to obtain target comprehensive data.

[0056] It should be noted that a multimodal sensor network is a network system composed of many different types of sensors, which can sense different aspects of information, including visual sensors (such as cameras), auditory sensors (such as microphones), environmental sensors (such as temperature sensors, humidity sensors, etc.), location sensors (such as GPS), and various sensors for monitoring device status. Multi-source heterogeneous data refers to data from multiple different data sources with different data formats, structures, semantics and other characteristics. In a multimodal sensor network, it can include image data collected by cameras, numerical data collected by temperature sensors, and three-dimensional vector data collected by accelerometers.

[0057] In addition, it should be noted that the spatiotemporal synchronization algorithm is an algorithm used to ensure that the data collected by different sensors can be accurately aligned in time and space, which can make the time base of each sensor consistent and make the collected data comparable on the time axis. The target comprehensive data is a data set obtained after spatiotemporal synchronization and fusion processing of multi-source heterogeneous data, which integrates the originally scattered, heterogeneous and inconsistent data in time and space to form a more comprehensive data set that can more comprehensively reflect the state of the monitored object or environment.

[0058] It is understandable that the clocks of various sensors are calibrated by referring to a unified time standard so that their time records remain consistent. According to the location information of each sensor, the data collected by different sensors are mapped to the same spatial reference system using a spatial transformation algorithm (such as based on coordinate transformation, feature matching, etc.), so that the data can be accurately fused in space.

[0059] In one example, reference Figure 2 , Figure 2This is a flow chart of the data acquisition system for this application. First, a variety of sensors of high-definition cameras are responsible for collecting data, including high-definition cameras, which are used to monitor the operating actions of personnel in the workshop, the appearance status of equipment, and the handling of materials in real time; temperature sensors are distributed in various areas of the workshop and around key equipment to measure ambient temperature and equipment surface temperature; vibration sensors are installed on large equipment to monitor the vibration of equipment; and personnel positioning tags (which can be regarded as a position sensor) are worn by workshop staff to track the location of personnel in real time. These data are transmitted through the communication data interface. Next, the data enters the data processing module for preliminary processing, and then is transmitted to the space-time synchronization module to ensure the consistency of the data in time and space. The data that has completed space-time synchronization is then sent to the fusion module for data fusion, and finally the fused data is stored in the database.

[0060] When monitoring a power plant workshop, a variety of sensors were installed in the workshop to form a multimodal sensor network. These include high-definition cameras, which are used to monitor the operating actions of personnel in the workshop, the appearance of equipment, and the handling of materials in real time; temperature sensors, which are distributed in various areas of the workshop and around key equipment, are used to measure the ambient temperature and the surface temperature of the equipment; vibration sensors, which are installed on large equipment, are used to monitor the vibration of the equipment; and personnel positioning tags (which can be regarded as a position sensor), which are worn by workshop staff to track the position of personnel in real time. These sensors collect data according to their respective settings. For example, the high-definition camera collects 25 frames of images per second, the temperature sensor measures the temperature value every 1 minute, the vibration sensor collects the acceleration data of the equipment vibration in real time, and the personnel positioning tag sends its own position information every 10 seconds. During data collection, the data collected by each sensor is accompanied by a timestamp accurate to the millisecond level. At the same time, the workshop is pre-modeled in three dimensions and the precise position coordinates of each sensor in the workshop are determined using technologies such as laser ranging. The collected data is directly mapped to the three-dimensional model space of the workshop according to its location coordinates, achieving spatial synchronization. The processed temperature data, panoramic images, equipment vibration data collected by vibration sensors, and personnel location data sent by personnel positioning tags are combined to form a data set that contains multiple information about personnel, equipment, environment, etc. in the workshop.

[0061] Step S20, extracting low-dimensional feature vectors from the target comprehensive data, and combining the attention mechanism to perform time series modeling on the low-dimensional feature vectors to construct a behavior dynamic model.

[0062] It should be noted that the behavioral dynamics model is designed to accurately describe and predict the dynamic changes in the behavior of monitored objects (such as personnel, equipment, etc.). It can output prediction results about the behavior of the object based on the input relevant data (such as the low-dimensional feature vector obtained after processing the real-time monitoring data from the sensor), such as predicting the next action of the personnel, whether the equipment will fail, etc.

[0063] Since the target comprehensive data may have a high dimension, directly processing it may face problems such as complex calculations and difficult model training. Through a specific feature extraction method, it is converted into a low-dimensional feature vector, that is, while retaining the key features of the data, the dimension of the data is reduced, making it more convenient for subsequent analysis and modeling operations.

[0064] It is understandable that the low-dimensional feature vectors in the target comprehensive data can be extracted by principal component analysis and autoencoders. When modeling time series, the long short-term memory network can be used. The long short-term memory network will learn the dynamic change rules of the data in the time series based on the input sequence data, such as the change pattern of personnel behavior over time, the fluctuation of equipment status at different times, etc. In the modeling process, the attention mechanism will assign weights to different elements of the low-dimensional feature vector at each time step. In this way, the model can pay more attention to those key feature elements according to the weights assigned by the attention mechanism when performing time series modeling, thereby more accurately capturing the key information of the dynamic changes of behavior and improving the accuracy of the model's prediction of the dynamic behavior.

[0065] Step S30, performing behavior recognition on the real-time collected work process information according to the behavior dynamic model, and outputting the personnel behavior category.

[0066] It should be noted that the personnel behavior category is a behavior type determined based on the personnel's action behavior. Common personnel behavior categories may include normal behavior (such as performing operations in accordance with standard operating procedures, activities within specified areas, etc.), abnormal behavior (such as operating equipment in violation of regulations, entering unauthorized areas, etc.), and illegal behavior (such as performing unsafe actions in dangerous areas, performing dangerous operations without taking necessary protective measures, etc.), etc.

[0067] It is understandable that in the actual operation process, various sensors or monitoring devices are used to obtain relevant information about personnel performing work tasks in real time. This information may include personnel position changes, movement postures, the status of operating equipment, and other aspects, and is continuously updated and collected as the operation progresses, which can reflect the actual operation status of personnel at the current moment. Using the constructed behavioral dynamic model, the real-time collected operation process information is analyzed and processed to determine the process of personnel behavior patterns or types corresponding to this information. By comparing and matching with the various behavioral characteristics and rules that have been learned in the model, it is possible to determine which category the specific behavior of the personnel belongs to.

[0068] In this embodiment, multi-source heterogeneous data collected by a multimodal sensor network is obtained, and the multi-source heterogeneous data is synchronized and fused based on a spatiotemporal synchronization algorithm to obtain target comprehensive data; low-dimensional feature vectors in the target comprehensive data are extracted, and the low-dimensional feature vectors are modeled in time series in combination with an attention mechanism to construct a behavioral dynamic model; behavior recognition is performed on the real-time collected operation process information according to the behavioral dynamic model, and the personnel behavior category is output. By integrating multi-source heterogeneous data in the complex environment of the power plant and accurately extracting key features, the dynamic change rules of behavior and state are captured. Whether it is a normal operating process or a complex and changeable working condition, various types of personnel behaviors can be accurately distinguished, and real-time and accurate identification of violations and abnormal behaviors in power plant operations is achieved.

[0069] Reference Figure 3 , Figure 3 This is a flow chart of the second embodiment of the behavior recognition method of the present application. Based on the above-mentioned first embodiment, the second embodiment of the behavior recognition method of the present application is proposed.

[0070] In the second embodiment, the step S30 includes:

[0071] Step S301, parse the collected work process information to obtain the personnel behavior process and equipment operation process.

[0072] It is understandable that the operation process information is collected through the multimodal sensor network (including high-definition cameras, position sensors, equipment status sensors, motion capture sensors, etc.) deployed in the operation area of ​​the power plant. This information includes the activity trajectory of personnel in various areas of the power plant, operation actions, and changes in the operation status of equipment. When parsing the collected operation process information, information parsing algorithms (such as feature extraction algorithms, target detection algorithms, and text information parsing algorithms, etc.) can be used to separate the personnel behavior process and equipment operation process from complex multi-source heterogeneous data. For example, the personnel behavior process information such as the action sequence and posture changes of personnel can be extracted from the video data captured by the camera through image recognition and analysis technology; the equipment operation process information such as the start, stop, and parameter adjustment of the equipment can be parsed from the data collected by the equipment status sensor.

[0073] Step S302: input the personnel behavior process and the equipment operation process into the behavior dynamic model, and calculate the behavior similarity between the operation process information and the predefined behavior pattern.

[0074] It should be understood that before inputting the personnel behavior process into the behavior dynamics model, it needs to be preprocessed. For example, if the personnel behavior process is represented in the form of a series of image sequences or the coordinates of key points of action postures, it may be necessary to convert it into a unified feature vector form. For personnel behavior data in time series, time alignment processing is also required to ensure that the behavior data collected in different time periods can be in the correct time sequence when the model is input, so that the model can accurately learn the temporal change law of the behavior. Equipment operation process data may contain information such as the start and stop time of the equipment, parameter adjustment values, and operation sequence. Preprocessing is also required to normalize these different types of data and convert them into a format suitable for model input. For example, the various parameter values ​​of the equipment are normalized so that their value range is within a specific interval, so that the model can better process and compare the operation data of different devices.

[0075] In one example, in the behavior dynamics model, the predefined behavior pattern and the input operation process information (personnel behavior process and equipment operation process after preprocessing) are expressed as feature vectors. The Euclidean distance is used to measure the distance between two feature vectors. Assume that the feature vector of the predefined behavior pattern is V p , the feature vector of the input operation process information after processing is V i , then the Euclidean distance between them is calculated as:

[0076]

[0077] Where n is the dimension of the feature vector, and are the values ​​of the th dimension of the predefined behavior pattern feature vector and the input operation process information feature vector, respectively. The smaller the Euclidean distance, the more similar the two feature vectors are, that is, the higher the behavioral similarity between the operation process information and the predefined behavior pattern.

[0078] Understandably, behavioral similarity can also be used as an important indicator for risk assessment. Lower behavioral similarity may mean that there is greater uncertainty or deviation from the normal state of the operation process, which may indicate potential risks. Based on this, different behavioral similarity thresholds can be set to trigger corresponding risk warning mechanisms. For example, when the behavioral similarity is lower than a certain threshold, the system can issue a warning signal to prompt managers to further check and intervene in the operation process to avoid possible accidents or safety hazards.

[0079] Step S303: determining the personnel behavior category of the operation process information according to the behavior similarity.

[0080] It can be understood that if the behavior similarity is within the similarity threshold range of the normal behavior pattern and the operation process information is consistent with the overall operation process, then it can be judged as normal behavior; if it is within the similarity threshold range of the illegal behavior pattern, it is further matched with the illegal behavior feature library to determine whether it is an illegal behavior; if it is within the similarity threshold range of the abnormal behavior pattern, it is judged as abnormal behavior.

[0081] Of course, in order to ensure the standardization and safety of power plant operations and reduce the probability of accidents, step S303 may include:

[0082] If the behavior similarity is within the similarity threshold range of the normal behavior pattern, and the operation process information is consistent with the overall operation process, it is judged as normal behavior; if the behavior similarity is within the similarity threshold range of the illegal behavior pattern, the operation detail features in the operation process information are matched with the preset illegal behavior feature library, and when the match is successful, the personnel behavior category is judged as illegal behavior; if the behavior similarity is within the similarity threshold range of the abnormal behavior pattern, it is judged as abnormal behavior.

[0083] It is understandable that the overall operation process refers to the complete and correct operation process steps and requirements determined by the standard operating procedures, safety regulations, etc. of the power plant (or other related scenarios). The violation behavior feature library is summarized and refined based on a large number of historical violation cases, safety regulations, etc., and contains various possible violation operation details, such as incorrect operation sequence, failure to take necessary protective measures for specific operations, and illegal use of tools.

[0084] It should be understood that when the behavior similarity is within the similarity threshold range of the abnormal behavior pattern, it is judged as abnormal behavior. Abnormal behavior refers to behavior that is neither obviously normal behavior nor clearly conforms to a known violation behavior pattern, but has a certain degree of deviation or abnormality compared to the normal behavior pattern. It may be due to some unexpected situations, negligence of personnel, or temporary failure of equipment, which causes the operation process to perform differently from the norm.

[0085] In one example, in the power plant inspection operation scenario, it is assumed that the similarity threshold range of the normal behavior pattern is set to 0.8-1.0. The behavioral dynamic model calculates that the behavioral similarity between the inspection operation process information of a certain inspector and the predefined normal behavior pattern is 0.9, which meets the condition within the normal behavior pattern similarity threshold range. Further inspection found that the inspector performed appearance inspections, instrument readings and other operations on each device in turn according to the prescribed inspection route. The operation sequence, steps and inspection content of each device are completely consistent with the overall inspection operation process formulated by the power plant. At this point, it can be determined that the inspector's operation behavior this time is normal.

[0086] In the power plant operation scenario, the similarity threshold range of the set violation behavior pattern is set to 0.3-0.5. A certain calculation shows that the behavior similarity between the operation process information of an operator and the predefined violation behavior pattern is 0.4, which meets the condition within the similarity threshold range of the violation behavior pattern. Then, the operation detail features in the operation process information of the operator are analyzed, and it is found that when operating high-voltage electrical equipment, he did not cut off the power supply as required and directly disassembled the internal components of the equipment. This operation detail feature is matched with the preset violation behavior feature library, and it is found that the violation feature of "disassembling the internal parts of high-voltage electrical equipment without cutting off the power supply" in the library is completely matched with it. Therefore, it is determined that the operator's behavior this time is a violation.

[0087] In the equipment maintenance work of the power plant, the similarity threshold range of abnormal behavior patterns is set to 0.5-0.8. The similarity between the operation process information of a maintenance personnel and the predefined abnormal behavior pattern is 0.6. When the maintenance personnel was maintaining an important equipment, the key steps that should have been completed in a short time took too long, which was obviously different from the normal behavior pattern of maintaining the equipment, but it did not fall into the prescribed illegal operation category. If the condition within the abnormal behavior pattern similarity threshold is met, the behavior is judged as abnormal behavior.

[0088] Of course, in order to further accurately trace the cause of abnormal behavior after determining it, provide a basis for taking targeted measures, facilitate rapid problem solving, restore normal production, and optimize management processes. If the behavior similarity is within the similarity threshold range of the abnormal behavior pattern, after determining it as abnormal behavior, it also includes:

[0089] The operation process information is parsed through the multimodal sensor network to obtain the target process information that causes the abnormal behavior and the category of the target process information; if the category is the personnel behavior process, the historical operation data of the personnel before the abnormal behavior occurs is obtained, and the cause of the abnormal behavior is determined based on the historical operation data; if the category is the equipment operation process, the operating parameter change information of the equipment before the abnormal behavior occurs is obtained, and the cause of the equipment failure is determined based on the operating parameter change information.

[0090] It should be understood that once an abnormal behavior is determined, it is necessary to conduct an in-depth analysis of the time period during which the abnormal behavior occurred and the surrounding related operation process information to determine the target process information and its category that caused the abnormal behavior.

[0091] It is understandable that once the category of abnormal behavior is determined to be the personnel behavior process, it is necessary to extract the operation data of the person before the abnormal behavior occurs from the historical data recorded by the multimodal sensor network. These data may include information such as the action sequence of the person within a period of time (such as the first few minutes or the first few operation steps), the operation time interval, the accuracy of the operation (such as the operation force of the device switch, whether the angle meets the standard), etc. For example, if the data is recorded by a camera and a motion capture sensor, a series of action video clips of the person before the abnormal behavior occurs can be reviewed retrospectively, and key action feature data can be extracted, such as the movement speed of the limbs, the change in the bending angle of the joints, etc.; the movement trajectory and residence time of the person in the operation area can be understood from the position sensor data. Determine whether the staff made mistakes in the previous operation link, such as accidentally touching the button, skipping the necessary inspection steps, etc.; or whether the operation action is deformed due to fatigue and lack of concentration, deviating from the normal operation specifications.

[0092] It is understandable that when the abnormal behavior category is the equipment operation process, it is necessary to obtain the information on the change of the equipment's operating parameters before the abnormal behavior occurs from the historical data recorded by the equipment status sensor. This includes the curve of the change of the equipment's temperature, pressure, speed, current, voltage and other parameters over time. For example, for a power generation equipment, check whether its temperature rises abnormally, whether the pressure fluctuates violently, whether the current has unstable peaks or valleys, and other data records within a few minutes before the abnormal behavior occurs. These data are stored in the form of time series, which can be easily analyzed for trends and abnormal point detection.

[0093] In this embodiment, the collected operation process information is parsed to obtain the personnel behavior process and the equipment operation process; the personnel behavior process and the equipment operation process are input into the behavior dynamic model, and the behavior similarity between the operation process information and the predefined behavior pattern is calculated; the personnel behavior category of the operation process information is determined based on the behavior similarity. The operation process information is parsed and the behavior category is determined based on the similarity calculated based on the behavior dynamic model, so as to accurately distinguish the characteristics of personnel behavior in different modes and improve the accuracy and pertinence of behavior recognition.

[0094] Reference Figure 4 , Figure 4 This is a flow chart of the third embodiment of the behavior recognition method of the present application. Based on the above second embodiment, the third embodiment of the behavior recognition method of the present application is proposed.

[0095] In the third embodiment, the step S10 includes:

[0096] Step S101, collecting multi-source heterogeneous data through a pre-deployed multimodal sensor network.

[0097] It is important to understand that before deploying a multimodal sensor network, a comprehensive and detailed analysis of the target monitoring scenarios (such as power plants, factory workshops, smart buildings, etc.) is required. Clarify the key areas in the scenario, key monitoring objects, and possible safety hazards or operational process links that require attention. For example, in a power plant, it is important to monitor areas such as generator sets, distribution rooms, and transmission pipelines; in a factory workshop, it is necessary to pay attention to production equipment, material handling channels, and personnel operating stations. Based on these analysis results, a sensor network deployment plan is developed to determine the installation locations, coverage areas, and collaborative relationships of different types of sensors to ensure that all types of information related to the scenario can be collected comprehensively and accurately.

[0098] Step S102, calibrating the time reference between the historical personnel behavior and the historical equipment operation status information based on the time-space synchronization algorithm to obtain the historical operation status information.

[0099] It should be understood that the historical operating status information is comprehensive information obtained after the time base calibration of historical personnel behavior and historical equipment operating status information through the spatiotemporal synchronization algorithm. It can accurately reflect the relationship between personnel behavior activities and the operating status of equipment on the same time scale.

[0100] It should be understood that before conducting a time benchmark, it is necessary to analyze the respective timestamp formats and recording methods in the historical personnel behavior information and the historical equipment operation status information in advance to ensure that each data point has an accurate timestamp.

[0101] It is understandable that for historical personnel behavior information, calibration is performed through the time synchronization module in the spatiotemporal synchronization algorithm based on its original timestamp and the determined unified time reference. If the timestamp of the personnel behavior information is a relative time, it needs to be converted into an absolute time corresponding to the unified time reference. For example, assuming that the time recorded by the personnel location sensor is a relative time calculated from the time when the device is started, and the unified time reference is GPS time, then it is necessary to convert the timestamps of the personnel location information one by one into absolute time under GPS time based on the correspondence between the time when the device is started and the GPS time.

[0102] In one example, the personnel location sensor records the position changes of worker A in the workshop, and the camera captures some operating action posture information of worker A. Assuming that the GPS time corresponding to the equipment startup moment is 9:00 a.m., then the GPS time corresponding to 10 minutes after the equipment is started (relative time) is 9:10 a.m. (absolute time). At the same time, at 9:15 a.m., it is recorded that worker A bent over at a certain position in the workshop. At the same time, the temperature of the equipment is 40°C and the current is 5A. This information forms a complete historical operating status information.

[0103] Step S103, using data layer fusion to fuse the historical operating status information and the environmental status information to obtain the target comprehensive data.

[0104] It should be noted that data layer fusion is to directly process and combine information from different data sources at the original level of the data. Through specific fusion algorithms and rules, multiple related but different types of data are integrated into a more comprehensive data set that can more comprehensively reflect the status of the monitored object or scene.

[0105] In one example, historical operating status information includes: personnel behavior (location changes of a certain on-duty personnel in the past hour and records of their equipment operation), equipment operation (temperature change data of a certain generator set in the past hour); environmental status information includes environmental temperature data, environmental humidity data and air quality status. The weighted average method is used for fusion processing. Considering that the equipment operation status is relatively more important in the overall status assessment of the power plant, the weight of the temperature data in the equipment operation is set to 0.6, and the weight of the temperature data in the environmental status information is set to 0.4. Similarly, the weight of the current data in the equipment operation is set to 0.7, and the weight of the humidity data in the environmental status information is set to 0.3. For temperature data, the temperature data in the equipment operation is T1, and the temperature data in the environmental status information is T2. It is known that T1 is 40°C at 30 minutes and T2 is 30°C at 30 minutes. According to the weighted average method formula, the fused temperature data is fused as follows:

[0106] T融合 =0.6×40+0.4×30=36℃

[0107] For the location change data and air quality conditions of personnel behavior, considering the possible impact of air quality conditions on personnel behavior, it is assumed that when the air quality is good, the range of personnel activities may be relatively large, while when the air quality is medium, the range of personnel activities may be relatively small. Based on this logical relationship, the air quality condition is converted into a weight relationship for the impact on the location change of personnel behavior. For example, when the air quality is good, the weight is 0.8, and when the air quality is medium, the weight is 0.6. Then, according to the air quality conditions in different time periods, the location change data of personnel behavior is adjusted accordingly to obtain the fused personnel behavior related data.

[0108] In the third embodiment, the step S20 includes:

[0109] Step S201, inputting the target comprehensive data into the encoder of the deep autoencoder network to obtain a low-dimensional feature vector.

[0110] It should be noted that the deep autoencoder network is a special neural network architecture, which is mainly used for data feature extraction, dimensionality reduction and data representation learning, and includes encoders and decoders. The encoder is used to gradually compress and transform the input data, and nonlinearly map the input data through a series of hidden layers, converting high-dimensional input data into low-dimensional feature vector representation.

[0111] Step S202: assigning different weights to each element in the low-dimensional feature vector based on the attention mechanism to obtain a weighted target feature vector.

[0112] It is understandable that the content-based attention mechanism mainly calculates weights based on the content of each element of the low-dimensional feature vector and the relationship between it and other elements. Specifically, the low-dimensional feature vector is processed by a learnable function (usually a neural network layer, such as a fully connected layer), which generates a weight vector based on the input low-dimensional feature vector. Each element of this weight vector corresponds to the weight of the corresponding element in the low-dimensional feature vector.

[0113] In one example, let the low-dimensional feature vector be Where m is the dimension of the low-dimensional feature vector. Input into the calculation module of the content-based attention mechanism. The calculation module has learnable parameters (such as weight matrix W and bias vector b), and then processes the low-dimensional feature vector according to the following formula to generate the weight vector:

[0114]

[0115] in, is a nonlinear function based on the fully connected layer, which can be It will be based on the input low-dimensional feature vector And internal parameters W, b calculate an intermediate vector. Then, the softmax function converts each element of this intermediate vector into a probability value between 0 and 1, and these probability values ​​constitute the weight vector.

[0116] Assume that a low-dimensional feature vector is extracted from the comprehensive data of personnel behavior and equipment operation status The dimension is 4. The content-based attention mechanism is used to assign weights to it. Assume that after being processed by the calculation module (fully connected layer) of the content-based attention mechanism, the weight vector According to the weighted calculation formula, the weighted target feature vector for:

[0117]

[0118] In the above example, the elements in the original low-dimensional feature vector are assigned weights and weighted by the attention mechanism to obtain the weighted target feature vector The second element 0.5 is relatively important in the original vector, and becomes 0.2 after weighting.

[0119] Step S203, using a long short-term memory network to perform time series modeling on the target feature vector, and using the output of the fully connected layer of the long short-term memory network as the final output to construct the behavior dynamic model.

[0120] It should be noted that the long short-term memory network is a special recurrent neural network that can effectively capture the long-term dependencies of data in time series.

[0121] It should be understood that before modeling, it is necessary to ensure that the target feature vectors have been arranged in chronological order. Because the long short-term memory network is based on time series data, it is necessary to clarify the time points corresponding to each target feature vector so that the changing pattern of the data in the time series can be accurately captured. At the same time, the number of neurons in the fully connected layer needs to be determined according to the specific task requirements. For example, if the final task is to perform binary classification (such as judging whether the behavior of a person is normal or abnormal), then the fully connected layer may only need to set one neuron, and its output value can be processed by a suitable activation function (such as the Sigmoid function) to represent the probability of belonging to a certain category; if it is a multi-classification task (such as judging whether the behavior of a person belongs to a variety of different normal or abnormal categories), it is necessary to set the corresponding number of neurons according to the number of categories to be classified, and use a suitable activation function to generate the probability distribution of each category.

[0122] In this embodiment, multi-source heterogeneous historical data are collected and subjected to spatiotemporal synchronization and fusion processing, and various types of information are fully integrated to provide a rich and comprehensive data basis for behavior recognition and analysis, so that the model can better learn and adapt to the power plant operating environment and improve the reliability of recognition. A behavior dynamic model is constructed using a deep autoencoder network and an attention mechanism to effectively extract key features and capture time series changes, thereby enhancing the model's ability to understand and predict complex behaviors.

[0123] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the behavior identification method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0124] This application also provides a behavior recognition device, please refer to Figure 5 , the behavior recognition device comprises:

[0125] The data acquisition module 10 is used to acquire multi-source heterogeneous data collected by the multi-modal sensor network, and perform spatiotemporal synchronization and fusion processing on the multi-source heterogeneous data based on a spatiotemporal synchronization algorithm to obtain target comprehensive data;

[0126] A model building module 20 is used to extract low-dimensional feature vectors from the target comprehensive data, and perform time series modeling on the low-dimensional feature vectors in combination with an attention mechanism to build a behavior dynamic model;

[0127] The behavior recognition module 30 is used to perform behavior recognition on the operation process information collected in real time according to the behavior dynamic model and output the personnel behavior category.

[0128] The behavior recognition device provided in the present application adopts the behavior recognition method in the above-mentioned embodiment, which can solve the current complex operating procedures and diverse personnel behaviors in power plants. There are subtle differences between different behaviors, and there are various interference factors in the operating environment, which leads to the technical problem that the detection system cannot quickly and accurately classify various behaviors and promptly detect violations and abnormal behaviors. Compared with the prior art, the beneficial effects of the behavior recognition device provided in the present application are the same as the beneficial effects of the behavior recognition method provided in the above-mentioned embodiment, and the other technical features in the behavior recognition device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0129] The present application provides a behavior recognition device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the behavior recognition method in the above-mentioned embodiment one.

[0130] Reference below Figure 6 , which shows a schematic diagram of the structure of a behavior recognition device suitable for implementing the embodiment of the present application. The behavior recognition device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The behavior recognition device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0131] like Figure 6As shown, the behavior recognition device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the behavior recognition device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the behavior recognition device to communicate with other devices wirelessly or wired to exchange data. Although the figure shows a behavior recognition device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0132] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0133] The behavior recognition device provided in the present application adopts the behavior recognition method in the above embodiment, which can solve the current complex operating procedures and diverse personnel behaviors in power plants. There are subtle differences between different behaviors, and there are various interference factors in the operating environment, which leads to the technical problem that the detection system cannot quickly and accurately classify various behaviors and promptly detect violations and abnormal behaviors. Compared with the prior art, the beneficial effects of the behavior recognition device provided in the present application are the same as the beneficial effects of the behavior recognition method provided in the above embodiment, and the other technical features in the behavior recognition device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0134] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0135] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0136] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the behavior recognition method in the above-mentioned embodiment.

[0137] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0138] The computer-readable storage medium may be included in the behavior recognition device; or may exist independently without being assembled into the behavior recognition device.

[0139] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the behavior recognition device, the behavior recognition device executes the behavior recognition method described above.

[0140] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0141] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0142] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0143] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned behavior recognition method, which can solve the current complex operation process of power plants and diverse personnel behaviors. There are subtle differences between different behaviors, and there are various interference factors in the operating environment, which leads to the technical problem that the detection system cannot quickly and accurately classify various behaviors and promptly detect violations and abnormal behaviors. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the behavior recognition method provided in the above-mentioned embodiments, which will not be repeated here.

[0144] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned behavior recognition method when executed by a processor.

[0145] The computer program product provided by this application can solve the current complex operating procedures and diverse personnel behaviors of power plants. There are subtle differences between different behaviors, and there are various interference factors in the operating environment, which leads to the technical problem that the detection system cannot quickly and accurately classify various behaviors and promptly detect violations and abnormal behaviors. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the behavior recognition method provided in the above embodiment, which will not be repeated here.

[0146] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A behavior recognition method, characterized in that: The behavior recognition method comprises: Acquire multi-source heterogeneous data collected by a multimodal sensor network, and perform spatiotemporal synchronization and fusion processing on the multi-source heterogeneous data based on a spatiotemporal synchronization algorithm to obtain target comprehensive data; Extracting low-dimensional feature vectors from the target comprehensive data, and combining the attention mechanism to perform time series modeling on the low-dimensional feature vectors to construct a behavior dynamic model; The behavior dynamic model is used to perform behavior recognition on the operation process information collected in real time, and the behavior category of the personnel is output.

2. The behavior recognition method according to claim 1, characterized in that: The step of performing behavior recognition on the real-time collected operation process information according to the behavior dynamic model and outputting the personnel behavior category includes: Analyze the collected operation process information to obtain the personnel behavior process and equipment operation process; Inputting the personnel behavior process and the equipment operation process into the behavior dynamic model, and calculating the behavior similarity between the operation process information and the predefined behavior pattern; The personnel behavior category of the operation process information is determined according to the behavior similarity.

3. The behavior recognition method according to claim 2, characterized in that: The categories of personnel behavior include normal behavior, irregular behavior, and abnormal behavior; The step of determining the personnel behavior category of the operation process information according to the behavior similarity includes: If the behavior similarity is within the similarity threshold of the normal behavior pattern, and the operation process information is consistent with the overall operation process, it is determined to be a normal behavior; If the behavior similarity is within the similarity threshold range of the violation behavior pattern, the operation detail features in the operation process information are matched with the preset violation behavior feature library, and when the match is successful, the personnel behavior category is determined as a violation behavior; If the behavior similarity is within the similarity threshold range of the abnormal behavior pattern, it is determined to be an abnormal behavior.

4. The behavior recognition method according to claim 3, characterized in that: After the step of determining that the behavior is abnormal if the behavior similarity is within the similarity threshold range of the abnormal behavior pattern, the method further includes: Parsing the operation process information through the multimodal sensor network to obtain target process information causing the abnormal behavior and a category of the target process information; If the category is the personnel behavior process, then obtaining the historical operation data of the personnel before the abnormal behavior occurs, and determining the cause of the abnormal behavior according to the historical operation data; If the category is the device operation process, then the operating parameter change information of the device before the abnormal behavior occurs is obtained, and the cause of the device failure is determined based on the operating parameter change information.

5. The behavior recognition method according to claim 1, characterized in that: The step of extracting the low-dimensional feature vector from the target comprehensive data, and performing time series modeling on the low-dimensional feature vector in combination with the attention mechanism to construct a behavior dynamic model includes: Inputting the target comprehensive data into an encoder of a deep autoencoder network to obtain a low-dimensional feature vector; Assigning different weights to each element in the low-dimensional feature vector based on the attention mechanism to obtain a weighted target feature vector; A long short-term memory network is used to perform time series modeling on the target feature vector, and the output of the fully connected layer of the long short-term memory network is used as the final output to construct the behavior dynamic model.

6. The behavior recognition method according to claim 1, characterized in that: The multi-source heterogeneous data includes historical personnel behavior, historical equipment operation status information and environmental condition information; The step of acquiring multi-source heterogeneous data collected by the multimodal sensor network, and performing spatiotemporal synchronization and fusion processing on the multi-source heterogeneous data based on a spatiotemporal synchronization algorithm to obtain target comprehensive data includes: Collect multi-source heterogeneous data through pre-deployed multimodal sensor networks; Calibrate the time reference between the historical personnel behavior and the historical equipment operation status information based on the time-space synchronization algorithm to obtain the historical operation status information; The historical operation status information and the environmental status information are fused by data layer fusion to obtain the target comprehensive data.

7. A behavior recognition device, characterized in that: The device comprises: A data acquisition module is used to acquire multi-source heterogeneous data collected by a multimodal sensor network, and perform spatiotemporal synchronization and fusion processing on the multi-source heterogeneous data based on a spatiotemporal synchronization algorithm to obtain target comprehensive data; A model building module, used to extract low-dimensional feature vectors from the target comprehensive data, and perform time series modeling on the low-dimensional feature vectors in combination with an attention mechanism to build a behavior dynamic model; The behavior recognition module is used to perform behavior recognition on the operation process information collected in real time according to the behavior dynamic model and output the personnel behavior category.

8. A behavior recognition device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the behavior recognition method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the behavior recognition method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the behavior recognition method according to any one of claims 1 to 6 are implemented.

Images