Personnel behavior detection method, controller, personnel behavior detection system and substation

Abstract

The present invention discloses a personnel behavior detection method, a controller, a personnel behavior detection system and a substation, and relates to the technical field of personnel behavior detection, wherein the personnel behavior detection method is used in a substation, and includes obtaining the video data captured by the photographic component; inputting the video data into a pre-trained multimodal feature generation model, and the multimodal feature generation model includes a visual coding module, a semantic coding module and a decoding module. The image features and semantic features in the video data are extracted by the visual coding module and the semantic coding module, and the image features and semantic features are combined by the decoding module to be used as the output of a multimodal feature signal, and the multimodal feature signal is used to characterize whether the personnel behavior in the substation is in an abnormal state or a normal state. In this way, by combining image features and corresponding semantic features, the robustness of the personnel behavior detection method can be improved, thereby improving the accuracy and reliability of the detection.

Description

Technical Field

[0001] The present invention relates to the technical field of substations, and in particular to a personnel behavior detection method, a controller, a personnel behavior detection system and a substation. Background Art

[0002] At present, personnel safety monitoring in substations mainly relies on a combination of cameras and manual monitoring. This system collects on-site video data through cameras, and operators monitor the screen in real time. However, for sudden abnormal behaviors of personnel, such as electric shock or falling to the ground, the existing system usually needs to combine multiple logical judgments to identify abnormalities. The stability and versatility are relatively poor and it is easy to make misjudgments or missed judgments, and it is impossible to accurately describe and record the scene. Summary of the invention

[0003] The main purpose of the present invention is to propose a personnel behavior detection method, a controller, a personnel behavior detection system and a substation, aiming to improve the robustness of the personnel behavior detection method, thereby improving the accuracy and reliability of the detection.

[0004] To achieve the above object, the present invention proposes a personnel behavior detection method, which is used in a substation, wherein at least one photographing component is provided in the substation, and the personnel behavior detection method comprises:

[0005] Acquiring video data captured by the photographic component;

[0006] Inputting the video data into a pre-trained multimodal feature generation model, wherein the multimodal feature generation model includes a visual encoding module, a semantic encoding module, and a decoding module;

[0007] Encoding the video data by the visual encoding module to obtain image features corresponding to the video data;

[0008] Encoding the video data by the semantic coding module to obtain semantic features corresponding to the video data, and / or encoding the image features by the semantic coding module to obtain semantic features corresponding to the video data;

[0009] The image features and semantic features are decoded by the decoding module to obtain a multimodal feature signal corresponding to the image features and semantic features, wherein the multimodal feature signal is used to characterize whether the behavior of personnel in the substation is in an abnormal state or a normal state;

[0010] The state of human behavior in the substation is determined according to the multimodal characteristic signal.

[0011] In one embodiment, the training step of the multimodal feature generation model includes:

[0012] Acquire at least one training sample from preset personnel behavior big data, wherein the training sample includes at least one visual image and a language label corresponding to the visual image;

[0013] The training samples are input into the multimodal feature generation model, and the parameters of the multimodal feature generation module are updated.

[0014] In one embodiment, the number of the training samples is multiple, and the multiple training samples are used to characterize that the behavior of the personnel in the substation is in the first abnormal state, and the language label of each of the training samples is also used to characterize the abnormal level of the behavior of the personnel in the substation in the first abnormal state;

[0015] Determining the behavior of personnel in the substation according to the multimodal characteristic signal includes:

[0016] When the video data is input into the multimodal feature generation model and the multimodal feature generation model determines that the scene corresponding to the video data includes a first abnormal state, an alarm signal is output to characterize that the behavior of personnel in the substation is in an abnormal state, and a level signal is output to characterize the abnormal level of the first abnormal state.

[0017] In one embodiment, the training samples include at least one training sample for characterizing that the personnel in the substation are in an abnormal state and at least one training sample for characterizing that the personnel in the substation are in a normal state;

[0018] Determining the behavior of personnel in the substation according to the multimodal characteristic signal includes:

[0019] In the case where the multimodal characteristic signal does not characterize the abnormal state and the normal state, it is determined that there is a potential abnormal state in the behavior of personnel in the substation, and the specific status of the personnel is determined through scene analysis.

[0020] In one embodiment, before inputting the training sample into the multimodal feature generation model, the method further includes:

[0021] Performing transformation processing on the visual images in the training samples to obtain at least one processed visual image, wherein the transformation processing includes image scaling and / or color adjustment and / or noise processing and / or clarity processing and / or mixed images;

[0022] According to the language tag corresponding to the visual image, a language tag of the processed visual image is determined.

[0023] In one embodiment, a plurality of photographic components are provided in the substation, and obtaining the video data captured by the photographic components specifically includes:

[0024] Acquire multiple video data captured by the multiple photographic components;

[0025] The multiple video data are subjected to image processing, wherein the image processing includes noise filtering processing, histogram equalization processing, contrast enhancement processing, and / or frame rate adjustment processing.

[0026] In one embodiment, the personnel behavior detection method further includes:

[0027] When the multimodal characteristic signal indicates that the behavior of personnel in the substation is abnormal, triggering an alarm mechanism and sending the video data to the cloud; and / or,

[0028] Generate an event report based on the video data, wherein the event report includes at least one of an image analysis result, a semantic analysis result, and a scene prediction result;

[0029] After the alarm information is issued, continue to detect the behavior of personnel in the substation until it is confirmed that the abnormal state has been eliminated;

[0030] When the multimodal characteristic signal indicates that the behavior of the personnel in the substation is in a normal state, the behavior of the personnel in the substation continues to be detected.

[0031] The present invention also provides a main controller, which is used to implement the personnel behavior detection method as described in any one of the above.

[0032] The present invention also provides a personnel behavior detection system, the behavior detection system comprising:

[0033] A data acquisition module having multiple high-resolution photographic components for capturing video data;

[0034] A multimodal feature generation module is connected to the data acquisition module, and comprises a visual encoding module, a semantic encoding module and a decoding module. The visual encoding module is used to perform visual encoding on the video data to obtain image features, the semantic encoding module is used to perform semantic encoding on the video data to obtain semantic features, and the decoding module is used to perform decoding processing on the image features and the semantic features, and output a multimodal feature signal;

[0035] A detection module connected to the multimodal feature generation module, the detection module is used to output a detection signal representing the behavior of the person according to the multimodal feature signal;

[0036] An alarm module connected to the detection module; when the detection signal indicates that the behavior of the person is in an abnormal state, an alarm signal is output;

[0037] The main controller as described above is connected to the multimodal feature generation module and the detection module, and the main controller is used to realize personnel behavior detection through the data acquisition module, the multimodal feature generation module, the detection module and the alarm module.

[0038] The present invention also provides a substation, the substation comprising:

[0039] A master controller as described above; or,

[0040] The personnel behavior detection system as described above.

[0041] The present invention provides a personnel behavior detection system, which improves the robustness of the personnel behavior detection method, thereby improving the accuracy and reliability of detection. Multi-photographic components ensure all-round, no-dead-angle real-time monitoring. The multimodal feature generation model combines visual and semantic coding technology, and improves the accuracy and reliability of behavior recognition through feature fusion and anomaly detection algorithms. Specifically, video data is captured by a photographic component, and then the video data is encoded and decoded by this pre-trained multimodal feature generation model. First, the image features in the video data are extracted by the visual coding module, and then the image features or the semantic features of the video data are extracted by the semantic coding module. Then, the image features and the semantic features are decoded to obtain a multimodal feature signal. The multimodal feature signal combines the image features and semantic features of the original video data. In this way, the system can more comprehensively and accurately identify and judge the personnel behavior in the substation, whether it is normal behavior or abnormal behavior, it can be detected in a timely and accurate manner, thereby significantly improving the accuracy and reliability of the system's detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0043] Figure 1 A flowchart of a first embodiment of a method for detecting human behavior provided by the present invention;

[0044] Figure 2 A flowchart of a second embodiment of the personnel behavior detection method provided by the present invention;

[0045] Figure 3 A flowchart of a third embodiment of the personnel behavior detection method provided by the present invention;

[0046] Figure 4 A flowchart of a fourth embodiment of the personnel behavior detection method provided by the present invention;

[0047] Figure 5 A flowchart of a fifth embodiment of the personnel behavior detection method provided by the present invention;

[0048] Figure 6 A flowchart of a sixth embodiment of the personnel behavior detection method provided by the present invention;

[0049] Figure 7 A flowchart of a seventh embodiment of the personnel behavior detection method provided by the present invention;

[0050] Figure 8 This is a system diagram of an embodiment of a personnel behavior detection system provided by the present invention.

[0051] Description of Figure Numbers:

[0052] 10. Data acquisition module; 20. Multimodal feature generation module; 30. Detection module; 40. Alarm module; 50. Main controller.

[0053] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components in a certain specific posture. If the specific posture changes, the directional indication will also change accordingly.

[0056] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features limited to "first" and "second" may explicitly or implicitly include at least one of the features. In addition, if "and / or" or "and / or" appears in the full text, its meaning includes three parallel solutions. Taking "A and / or B" as an example, it includes solution A, solution B, or solutions that satisfy both A and B. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0057] It should be noted that, in this article, step codes such as S100, S200, etc. are used for the purpose of expressing the corresponding content more clearly and concisely, and do not constitute a substantial limitation on the sequence. When implementing the step, those skilled in the art may execute S200 first and then S100, etc., but these should all be within the scope of protection of this application.

[0058] At present, personnel safety monitoring in substations mainly relies on a combination of cameras and manual monitoring. This system collects on-site video data through cameras, and operators monitor the screen in real time. However, for sudden abnormal behaviors of personnel, such as electric shock or falling to the ground, the existing system usually needs to combine multiple logical judgments to identify abnormalities. The stability and versatility are relatively poor and it is easy to make misjudgments or missed judgments, and it is impossible to accurately describe and record the scene.

[0059] It is understandable that the personnel detection methods in the prior art usually need to combine multiple logical judgments to identify sudden abnormal behaviors of personnel, such as electric shock, falling to the ground and other emergency situations. However, due to the suddenness and complexity of these behaviors, it is often difficult for the system to accurately capture and identify them, resulting in misjudgments or missed judgments from time to time. At the same time, although the camera can provide real-time video images, it ultimately needs to rely on manual real-time monitoring and judgment. This not only increases the workload of operators, but may also lead to safety problems caused by fatigue, negligence and other reasons. In summary, the existing substation personnel safety monitoring technology has certain defects in identifying sudden abnormal behaviors, stability and versatility, data description and recording capabilities, and personnel dependence.

[0060] Therefore, the main purpose of the present invention is to propose a personnel behavior detection method to achieve full-automatic detection, and also to improve the robustness of the personnel behavior detection method, thereby improving the accuracy and reliability of the detection. It can be understood that in this embodiment, the personnel behavior detection method is used in a substation, and in other application scenarios, it can also be applied to various places with high security requirements such as factory workshops, transportation hubs, and construction sites, providing strong technical support for various security monitoring systems.

[0061] In one embodiment, at least one photographic component is provided in the substation, that is, the number of photographic components can be one or more. When the number of photographic components is more than one, the video data output ends of the multiple photographic components can also be connected to a data center, so as to centrally process the video data of the multiple photographic components. At the same time, considering the complex lighting conditions that the substation may face, this photographic component adopts night vision and multi-spectral imaging functions. The night vision function allows the camera to still clearly image in a dimly lit or completely dark environment, ensuring the monitoring effect at night or in low-light environments. In this way, the robustness of the personnel behavior detection method can be improved, thereby improving the accuracy and reliability of the detection.

[0062] In addition, in order to reduce the risk of data loss, the system adopts a redundant transmission mechanism, that is, the video data will be backed up through multiple paths or methods during the transmission process. Once a transmission path fails, the system can quickly switch to other paths to continue transmitting data. In addition, the system is also equipped with a data cache function. When the network is congested or the processing unit is busy, the video data can be temporarily stored in the cache and transmitted and processed when the network is unblocked or the processing unit is idle. In this way, the integrity and reliability of the video data are ensured.

[0063] In one embodiment, if Figure 1 As shown, the personnel behavior detection method includes steps S100 to S600.

[0064] In this embodiment, step S100, obtaining video data captured by the photographic component.

[0065] It is understandable that at least one photographic component, which can be a camera, is arranged in the key area of ​​the substation. These cameras can support night vision and multi-spectral imaging to ensure that clear video data can be captured under different lighting conditions. The camera captures video data in real time and ensures the safe transmission of data through a redundant transmission mechanism, while using the data caching function to reduce the risk of data loss. The video data is then transmitted to the central processing unit for further processing. The video data can contain the original image information of the activities of people in the substation, which is the basic material for subsequent analysis and processing.

[0066] Furthermore, the video stream V(t) captured by the camera component is the core data source of the substation monitoring system, which is real-time and high-definition. After capturing the video stream, the system can immediately encode it in H.264 or H.265 to compress the data and reduce the transmission bandwidth. The compressed video stream is transmitted to the central processing unit via the TCP / IP protocol, and a redundant transmission mechanism is used to ensure the security and reliability of the data. Among them, the central processing unit, as the core of the system, is responsible for receiving, decoding, storing and analyzing video data.

[0067] In this embodiment, step S200, the video data is input into a pre-trained multimodal feature generation model, and the multimodal feature generation model includes a visual encoding module, a semantic encoding module and a decoding module.

[0068] It is understandable that the video data captured from step S100 is input into a pre-trained multimodal feature generation model. The model is used to process video data and extract various types of information (i.e., multimodal features) therefrom. The multimodal feature generation model in this embodiment may include a visual encoding module, a semantic encoding module, and a decoding module, which work together to achieve a comprehensive analysis of the video data. In this way, the model is pre-trained using a large-scale annotated data set, including common scenes and abnormal behavior data in a substation, to improve the generalization ability and robustness of the model.

[0069] It should be noted that this multimodal feature generation model uses a large visual-language model, which can be an LVLMs model. The large visual-language model is used to combine visual and semantic features for complex scene analysis and behavior recognition. It has slightly higher performance and relatively fewer misjudgments and missed judgments. In practical applications, the specific model can be selected according to the actual situation. In other applications, the ResNet model can also be used.

[0070] In this embodiment, step S300, encodes the video data through the visual encoding module to obtain image features corresponding to the video data.

[0071] It can be understood that the visual encoding module is used to extract key image features from video data. Image features may include information such as color, texture, shape, motion, etc., which serve as the basis for subsequent analysis and recognition.

[0072] In this embodiment, step S400, encodes the video data through the semantic coding module to obtain semantic features corresponding to the video data, and / or encodes the image features through the semantic coding module to obtain semantic features corresponding to the video data.

[0073] It is understandable that the semantic coding module in this embodiment can understand and analyze high-level semantic features in video data, such as human behavior, scene layout, object relationships, etc., to achieve stronger human-computer interaction. For example, when a fall occurs in a video, the semantic coding module generates text information of "fall" according to the corresponding image features, corresponding to the video.

[0074] In this embodiment, for step S400, the semantic coding module may encode the original video data, or may encode the image features encoded by the visual coding module, or may encode both at the same time.

[0075] It is understandable that in some examples, a multi-head self-attention mechanism can be used to achieve the combination of image features and semantic features.

[0076] Furthermore, this multimodal feature generation model uses a two-stream model architecture, including a visual encoding module and language encoding modules The visual encoding module is based on a convolutional neural network (CNN) to extract visual features. , the formula is: The language encoding module is based on a transformer and uses a multi-head self-attention mechanism. , aligning semantic features with image features, defined as: , where T is the text input, is the semantic feature of the output.

[0077] In this embodiment, step S500, the image features and semantic features are decoded by the decoding module to obtain a multimodal feature signal corresponding to the image features and semantic features, and the multimodal feature signal is used to characterize whether the behavior of personnel in the substation is in an abnormal state or a normal state.

[0078] It can be understood that the decoding module integrates the features output by the visual encoding module and the semantic encoding module to generate a multimodal feature signal. The multimodal feature signal contains comprehensive information of the video data, including both underlying image features and high-level semantic features. The system can accurately determine whether the behavior of personnel in the substation is abnormal by analyzing the multimodal feature signal. In this way, combining visual features with semantic features can achieve the fusion of multimodal features, thereby improving the accuracy and reliability of behavior recognition.

[0079] It should be explained that when the behavior of personnel in the substation is in a normal state, the behavior of personnel is not dangerous; when the behavior of personnel in the substation is in an abnormal state, it may indicate that the behavior of personnel may constitute a danger, or that the behavior of personnel is already in a dangerous situation. In other examples, the specific manifestations of the normal state and the abnormal state may be different. Therefore, the manifestations of the normal state and the abnormal state in this embodiment are not limited and can be determined according to the actual application scenario and situation. What is important is that this embodiment judges the state of personnel behavior through multimodal features, thereby improving the robustness of the personnel behavior detection method, and then improving the accuracy and reliability of the detection.

[0080] In this embodiment, step S600, determining the state of human behavior in the substation according to the multimodal characteristic signal.

[0081] It is understandable that based on multimodal feature signals, the system can more accurately judge the behavior of people in the substation. Compared with relying on a single feature alone, multimodal feature signals contain more comprehensive information and can better cope with complex and changing scenarios. Furthermore, machine learning models such as decision trees or random forests can be used to make final judgments based on multimodal feature signals to improve the accuracy and reliability of decision-making.

[0082] It should be explained that in practical applications, image features can include low-level information such as color, texture, shape, and motion, which can help the system identify specific objects and actions in the video. For example, image features can be used to identify that a person is walking or running. Semantic features can include higher-level understanding, such as the action category of a person, the layout of a scene, the relationship between objects, etc. For example, semantic features can be used to understand whether a person is operating a device or whether there is abnormal behavior (such as falling or electric shock). When image features and semantic features are combined to output multimodal feature signals, the system can understand the scene more comprehensively. For example, if a person falls near a device, image features can identify the action of falling, while semantic features can understand that the person fell while operating the device, thereby more accurately judging that this is an abnormal behavior.

[0083] In summary, the present invention realizes an accurate and reliable personnel behavior detection system. Multi-camera components ensure all-round, no-dead-angle real-time monitoring. The multimodal feature generation model combines visual and semantic coding technology, and improves the accuracy and reliability of behavior recognition through feature fusion and anomaly detection algorithms. Specifically, video data is captured by the photography component, and then the video data is encoded and decoded by the pre-trained multimodal feature generation model. First, the image features in the video data are extracted by the visual coding module, and then the image features or the semantic features of the video data are extracted by the semantic coding module. The image features and the semantic features are then decoded to obtain a multimodal feature signal. The multimodal feature signal combines the image features and semantic features of the original video data. In this way, the system can more comprehensively and accurately identify and judge the personnel behavior in the substation. Whether it is normal behavior or abnormal behavior, it can be detected in a timely and accurate manner, thereby significantly improving the accuracy and reliability of the system's detection.

[0084] In one embodiment, if Figure 2 As shown, the training steps of the multimodal feature generation model include steps S210 to S220.

[0085] Step S210: Acquire at least one training sample from preset personnel behavior big data, wherein the training sample includes at least one visual image and a language label corresponding to the visual image.

[0086] It is understandable that the preset personnel behavior big data refers to a database containing a large amount of annotated video data, covering various personnel behaviors, including normal behaviors and abnormal behaviors. These data usually come from real substations or other places with high safety requirements. Each training sample includes a visual image and a language label corresponding to the visual image. The visual image is an image frame captured from the video, which can be one or more images, used to provide visual information; the language label is used to provide semantic features. For example, if a person in the image is operating a device, the language label may be "operating the device"; if a person falls in the image, the language label may be "falling". A large amount of annotated data ensures that the model can learn the characteristics of various behaviors and improve the generalization ability of the model. At the same time, providing visual images and language labels enables the model to learn in combination with visual and semantic features, thereby improving the accuracy and robustness of recognition.

[0087] Step S220: input the training sample into the multimodal feature generation model, and update the parameters of the multimodal feature generation module.

[0088] It should be explained that the training samples containing visual images and language labels are input into the multimodal feature generation model. In a training example, the following training method can be used. The model calculates the predicted output under the current parameters through forward propagation. For example, for an image of "operating a device", the model may output a probability of "operating a device" of 0.7 and a probability of "falling" of 0.3. Subsequently, the predicted output is compared with the true label of the training sample to calculate the loss function value. For example, if the true label is "operating a device", the loss function value reflects the difference between the predicted output and the true label. Through the back-propagation algorithm, the gradient of the loss function to the model parameters is calculated, for example, which parameter changes can reduce the loss function value. Finally, according to the calculated gradient, the optimization algorithm (such as gradient descent, Adam, etc.) is used to update the parameters of the model, for example, the weights and biases are adjusted so that the model can be closer to the true label in the next prediction.

[0089] It can be understood that the image features output by the visual encoding module correspond to the visual images in the above training samples, and the semantic features output by the semantic encoding module correspond to the language labels in the above training samples.

[0090] Furthermore, the multimodal feature generation model in this embodiment uses the cross entropy loss function Perform training on multi-classification tasks. Specifically, ,in is the true label, is the predicted label probability value. Data augmentation techniques are used for training, including random cropping, rotation, and color jittering, to improve the robustness of the model. Finally, the multimodal feature generation module outputs a multimodal feature signal P, which is used to classify the behavior using the following formula: ,in, Indicates that given the visual features The probability of behavior category c under the condition of and semantic features T. By maximizing this probability, the system can determine the most likely behavior category P, thereby achieving accurate behavior detection.

[0091] In this way, through rich training data, the model can more effectively reduce misjudgments and missed judgments, ensuring the reliability and accuracy of the system's detection.

[0092] In actual application scenarios, for example, a person suddenly falls down near a device. The camera captures a person suddenly falling down near the device. The multimodal feature generation model extracts the image features of the fall through the visual encoding module, understands that this is the behavior of "falling" through the semantic encoding module, and finally outputs the label "falling". The system immediately triggers an alarm to notify the operator to take emergency measures.

[0093] In one embodiment, the number of training samples in step S210 may be multiple, and the multiple training samples are used to characterize that the behavior of the personnel in the substation is in the first abnormal state, and the language label of each training sample is also used to characterize the abnormal level of the behavior of the personnel in the substation in the first abnormal state. It can be understood that these training samples are used to characterize that the activities of the personnel in the substation are in the first abnormal state, and the first abnormal state may refer to a specific abnormal behavior, such as falling, electric shock, etc. At the same time, the language label of each training sample not only describes the behavior of the personnel, but also characterizes the abnormal level of the behavior. For example, the abnormal level can be divided into mild, moderate and severe. For example, the visual image of a training sample shows that a person falls slightly, and its language label may be "fall-mild"; the visual image of another training sample shows that a person is severely electric shocked, and its language label may be "electric shock-severe".

[0094] In this embodiment, if Figure 3 As shown, step S600 specifically includes step S610. Step S610, when the video data is input into the multimodal feature generation model, and the multimodal feature generation model determines that the scene corresponding to the video data includes a first abnormal state, outputs an alarm signal for characterizing that the behavior of personnel in the substation is in an abnormal state, and outputs a level signal for characterizing the abnormal level of the first abnormal state.

[0095] It can be understood that the model extracts image features through the visual encoding module, extracts semantic features through the semantic encoding module, and then generates a multimodal feature signal through the decoding module. The multimodal feature generation model determines whether the scene corresponding to the video data includes the first abnormal state based on the generated multimodal feature signal. If the model determines that the scene corresponding to the video data includes the first abnormal state, an alarm signal is output to characterize that the personnel in the substation are in an abnormal state, and a level signal is further output to characterize the abnormal level of the first abnormal state.

[0096] In practical applications, for example, if a person falls, the model will output a graded signal of "Fall - Mild" or "Fall - Severe". In this way, by introducing abnormality levels, the system can provide more refined abnormality detection and alarms. For example, mild abnormalities may only require recording and attention, while severe abnormalities require immediate emergency measures.

[0097] It should be noted that, in this embodiment, it is important to further train the multimodal feature generation model so that it can distinguish the severity of the person's state. There may be more than one abnormal state type in the first abnormal state, such as falling and electric shock. In other embodiments, it may include more than the first abnormal state, and may also include the second and third abnormal states. The present invention does not limit this. What is important is to train the level judgment for different states.

[0098] Furthermore, through multimodal features, the system classifies behaviors and can use the Softmax function to normalize the output into a probability value. The formula is: ,in is the activation value of the behavior category c. After detecting abnormal behavior, the system analyzes it in combination with the scene semantics. The severity of abnormal behavior is scored using the logistic regression model g(z), which is defined as: , where z is the input feature, g(z) is the severity, and e is a natural constant (value approximately equal to 2.71828).

[0099] In one embodiment, the number of training samples in step S210 may be multiple, and the multiple training samples include at least one training sample for representing an abnormal state input into the multimodal feature generation model and at least one training sample for representing a normal state input into the multimodal feature generation model. It is understandable that the abnormal state sample is used to indicate that the activities of personnel in the substation are abnormal, such as falling, electric shock, etc.; the normal state sample is used to indicate that the activities of personnel in the substation are normal, such as normal walking, operating equipment, etc.

[0100] In this embodiment, if Figure 4 As shown, step S600 specifically includes step S620. Step S620, when the multimodal feature signal does not represent the abnormal state and the normal state, determine that there is a potential abnormal state in the behavior of the personnel in the substation, and determine the specific personnel state through scene analysis. It should be explained that the multimodal feature generation model determines whether the scene corresponding to the video data includes a normal state or an abnormal state based on the generated multimodal feature signal. When it does not include a normal state or an abnormal state, the system will determine that there is a potential abnormal state for the personnel in the substation. After determining that there is a potential abnormal state, the system will further determine the specific personnel state through the scene analysis module. The scene analysis module can combine more contextual information, such as the location of the personnel, the duration of the action, changes in the surrounding environment, etc., to perform a more detailed analysis to determine the specific personnel state.

[0101] In this way, some potential abnormal conditions can be discovered, and it is not necessary to wait until the actual abnormal condition occurs before taking corresponding measures.

[0102] In one embodiment, if Figure 5 As shown, before the training samples are input into the multimodal feature generation model, steps S230 to S240 are also included.

[0103] In this embodiment, step S230, the visual images in the training samples are transformed to obtain at least one processed visual image, and the transformation processing includes image scaling and / or color adjustment and / or noise processing and / or clarity processing and / or mixed images.

[0104] It is understandable that transformation processing includes a variety of image processing techniques, which are designed to simulate different shooting conditions and environments and enhance the robustness of the model. Common transformation processing includes: Image scaling: changing the size of the image to simulate shooting effects at different distances. Color adjustment: adjusting the brightness, contrast, saturation, etc. of the image to simulate different lighting conditions. Noise processing: adding noise to the image to simulate noise interference in actual shooting. Clarity processing: adjusting the clarity of the image to simulate the shooting effects of cameras of different qualities. Mixed images: mixing multiple images together to generate new images and increase the diversity of training data. In this way, the robustness of the model can be enhanced and the generalization ability of the model can be enhanced.

[0105] In this embodiment, step S240, determining the language tag of the processed visual image according to the language tag corresponding to the visual image.

[0106] It is understood that each original visual image has a corresponding language label that describes the behavior in the image. The language label of the processed visual image should be consistent with the language label of the original visual image.

[0107] Therefore, through steps S230-S240, image transformation processing is performed before the training sample is input into the model, and the language label of the processed visual image is determined. This embodiment not only enhances the robustness and generalization ability of the multimodal feature generation model, but also improves the accuracy and reliability of the model. This enables the system to better adapt to complex scenarios in practical applications and improve the safety management level of substations and other places with high safety requirements.

[0108] In one embodiment, a plurality of photographic components are provided in the substation, such as Figure 6 As shown, step S100 may specifically include steps S110 to S150.

[0109] In this embodiment, step S110, multiple video data captured by the multiple camera components are obtained. It can be understood that multiple cameras are installed in the substation, each camera is responsible for monitoring a different area, and the video data captured by each camera will be collected and used for subsequent processing.

[0110] In this embodiment, step S120 performs noise filtering on the plurality of video data to filter out at least part of the noise in the video data. It is understandable that the noise in the video data may come from the sensor noise of the camera itself, changes in ambient light, electromagnetic interference, etc. One or more of median filtering, Gaussian filtering, bilateral filtering, etc. may be used for filtering. In this way, a more accurate detection result may be obtained.

[0111] Optionally, in some examples, an adaptive median filter is used to eliminate random noise and retain edge information. Specifically, the median filter M(x, y) is defined as: , where W(x y) is a filtering window, which is used to eliminate random noise and retain edge information. In this way, the system can better process images and improve the detection accuracy and reliability of the personnel behavior detection method.

[0112] In this embodiment, step S130, the multiple video data are subjected to histogram equalization processing to obtain higher definition video data. It can be understood that the video data captured by each camera is subjected to histogram equalization processing to make the brightness and contrast of the image more uniform and improve the clarity of the image. In this way, the accuracy and reliability of the detection can be improved.

[0113] Furthermore, for histogram equalization, the histogram equalization formula in this embodiment is specifically: , where H′(i) is the new grayscale value, L is the grayscale level, N is the number of pixels, and h(j) is the frequency of the pixel grayscale level. This algorithm improves image contrast and is particularly suitable for low-light environments, because in low-light environments, the dynamic range of images is usually small, and most pixels are concentrated in the lower grayscale value area. Through histogram equalization, the range of these low grayscale values ​​can be effectively expanded, making the details of the image more clearly visible. In this way, the system can better process images and can have higher detection accuracy and reliability in low-light environments.

[0114] In this embodiment, step S140 performs contrast enhancement processing on the plurality of video data to obtain video data with higher contrast. It is understandable that the contrast enhancement processing may be performed by one or more of linear stretching, nonlinear stretching, histogram normalization, etc.

[0115] In this embodiment, step S150 adjusts the frame rate of the plurality of video data so that the frame rate of the video data matches the encoding frame rate of the visual encoding module. In this way, it can be ensured that the video data can be processed smoothly when input into the multimodal feature generation model, avoiding processing delays or errors caused by frame rate mismatch, and improving the real-time performance and response speed of the system.

[0116] Furthermore, in order to meet the real-time requirements of the system reasoning module, the frame rate of the original video data needs to be increased from Adjust to First, the downsampling factor r is calculated, which is defined as . Select a suitable downsampling method, such as uniform frame extraction, content-based frame extraction, or motion-based frame extraction, to reduce the number of video frames. Taking uniform frame extraction as an example, the specific steps include initializing the current frame index to 0, reading the video data frame by frame, retaining the frame whenever the current frame index reaches a multiple of r, otherwise discarding the frame, and updating the current frame index after each frame is read. When all video frames are traversed, the downsampling operation is completed. In order to maintain the coherence of the video, some additional measures can be taken during the downsampling process. For example, interpolated frames can be inserted between the retained frames, and intermediate frames can be generated using linear interpolation or bicubic interpolation methods for smooth transitions. In this way, the coherence of the video is maintained, the real-time requirements of the system reasoning module are met, and the detection accuracy and reliability of the personnel behavior detection method are indirectly improved.

[0117] It is understandable that in actual applications, not all of the above steps may be used, and only one or any number of them may be used. In this way, by performing multiple preprocessing steps before inputting the video data into the multimodal feature generation model, not only the quality of the video data is improved, but also the performance and robustness of the model are enhanced. This enables the system to better adapt to complex scenarios in actual applications and improve the safety management level of substations and other places with high safety requirements.

[0118] In one embodiment, if Figure 7 As shown, the personnel behavior detection method also includes steps S710 to S740.

[0119] In this embodiment, in step S710, when the multimodal feature signal indicates that the behavior of the personnel in the substation is in an abnormal state, the alarm mechanism is triggered and the video data is sent to the cloud. It is understandable that when the multimodal feature generation model detects that the behavior of the personnel in the substation is in an abnormal state, the system will immediately trigger the alarm mechanism, generate an alarm signal, and notify the operator or security manager. The alarm method may include sound alarm, light alarm, SMS notification, email notification and other forms. At the same time, the system sends the video clips containing abnormal behavior to the cloud for storage for subsequent analysis.

[0120] In this embodiment, step S720, based on the video data, an event report is generated, and the event report includes at least one of the image analysis results, semantic analysis results, and scene prediction results. It can be understood that the system generates a detailed event report based on the video data, and the report content includes at least the image analysis results (describing the image features detected in the video, such as falling, electric shock, etc.), the semantic analysis results (describing the semantic features of the behavior detected in the video, such as "falling-severe", "electric shock-moderate", etc.) and the scene prediction results (based on the currently detected behavior, predicting possible subsequent scenes, such as "immediate medical assistance is required", "need to shut down related equipment", etc.).

[0121] In this embodiment, step S730, after the alarm information is issued, the behavior of the personnel in the substation continues to be detected until it is confirmed that the abnormal state has been eliminated. It is understandable that after the alarm mechanism is triggered, the system will continue to monitor the behavior of the personnel in the substation in real time to ensure that the abnormal state is effectively handled. When it is detected that the abnormal state has been resolved or corresponding measures have been taken, the system will stop the alarm and record the processing results to ensure that the abnormal situation is properly handled.

[0122] In this embodiment, step S740, when the multimodal feature signal indicates that the behavior of the personnel in the substation is in a normal state, the behavior of the personnel in the substation continues to be detected. It is understandable that when the multimodal feature generation model detects that the behavior of the personnel in the substation is in a normal state, the system will continue to monitor the behavior of the personnel in the substation to ensure that any abnormal situation is discovered in time. The system maintains a real-time monitoring state to ensure the safety of the substation.

[0123] It is understandable that in actual applications, not all of the above steps may be used, and only one or any number of them may be used. In this way, the response speed and accuracy of the system are improved, and the trust and reliance of operators on the system are enhanced, thereby significantly improving the human-computer interaction capabilities. At the same time, it also enables the system to better adapt to complex scenarios in actual applications and improve the safety and reliability of substations and other places with high safety requirements.

[0124] The present invention further provides a main controller 50, and the main controller 50 is used to implement the personnel behavior detection method as described in any one of the above.

[0125] The main controller 50 provided by the present application adopts the personnel behavior detection method in the above embodiment, which can solve the technical problem of personnel behavior detection. Compared with the prior art, the beneficial effects of the main controller 50 provided by the present application are the same as the beneficial effects of the personnel behavior detection method provided by the above embodiment, and other technical features in the main controller 50 are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0126] like Figure 8 As shown, the present invention also provides a personnel behavior detection system, which includes a data acquisition module 10, a multimodal feature generation module 20, a detection module 30 and an alarm module 40. Among them, the data acquisition module 10 has multiple high-resolution photography components for capturing video data; the multimodal feature generation module 20 is connected to the data acquisition module 10, and the multimodal feature generation module 20 has a visual encoding module, a semantic encoding module and a decoding module, the visual encoding module is used to perform visual encoding on the video data to obtain image features, the semantic encoding module is used to perform semantic encoding on the video data to obtain semantic features, and the decoding module is used to decode the image features and the semantic features and output a multimodal feature signal; the detection module 30 is used to generate a plurality of high-resolution photography components for capturing video data; the multimodal feature generation module 20 is connected to the data acquisition module 10, and the multimodal feature generation module 20 has a visual encoding module, a semantic encoding module and a decoding module, the visual encoding module is used to perform visual encoding on the video data to obtain image features, the semantic encoding module is used to perform semantic encoding on the video data to obtain semantic features, and the decoding module is used to perform decoding processing on the image features and the semantic features and output a multimodal feature signal; the detection module 30 is used to generate a plurality of high-resolution photography components for capturing video data; the multimodal feature generation module 2 ... 0 is connected to the multimodal feature generation module 20, the detection module 30 is used to output a detection signal representing the behavior of the personnel according to the multimodal feature signal; the alarm module 40 is connected to the detection module 30; when the detection signal represents that the behavior of the personnel is in an abnormal state, an alarm signal is output; the main controller 50 as described above is connected to the multimodal feature generation module 20 and the detection module 30, and the main controller 50 is used to realize personnel behavior detection through the data acquisition module 10, the multimodal feature generation module 20, the detection module 30 and the alarm module 40.

[0127] It should be noted that the main controller 50 provided in the present application, using the main controller 50 in the above embodiment, can solve the technical problem of personnel behavior detection. Compared with the prior art, the beneficial effects of the personnel behavior detection system provided in the present application are the same as the beneficial effects of the personnel behavior detection method provided in the above embodiment, and other technical features in the main controller 50 are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0128] The present invention further provides a substation, which includes the main controller 50 as described above; or the personnel behavior detection system as described above.

[0129] The substation provided by the present application can solve the technical problem of personnel behavior detection. Compared with the prior art, the beneficial effects of the substation provided by the present application are the same as the beneficial effects of the personnel behavior detection method provided by the above embodiment, which will not be repeated here.

[0130] In this embodiment, the substation includes a personnel behavior detection system, which realizes efficient detection of personnel behavior in the substation and timely response to abnormal conditions.

[0131] First, the data acquisition module 10 of the personnel behavior detection system is the basis of the entire system. It is equipped with multiple high-resolution photography components for capturing video data. These high-resolution photography components are distributed in different key areas of the substation, such as the high-voltage equipment area, the operating table area, the exit area, and the entrance area, ensuring all-round and no-dead-angle monitoring. The high-resolution photography components can capture clear images, providing a high-quality data foundation for subsequent feature extraction and behavior analysis.

[0132] Next, the multimodal feature generation module 20 is connected to the data acquisition module 10, and is responsible for extracting and generating multimodal features of the captured video data. The multimodal feature generation module 20 contains a visual coding module, a semantic coding module and a decoding module. The visual coding module performs visual coding on the video data and extracts image features, which include visual information such as the shape, color, and texture of the object. The semantic coding module performs semantic coding on the video data and extracts semantic features, which include high-level semantic content such as the character's actions, behaviors, and environmental descriptions. The decoding module fuses and decodes the extracted image features and semantic features to generate a multimodal feature signal. The multimodal feature signal combines visual and semantic features, and can more comprehensively and accurately describe the behavior of people in the video.

[0133] The detection module 30 is connected to the multimodal feature generation module 20 and is responsible for outputting a detection signal representing the behavior of the person according to the generated multimodal feature signal. The detection module 30 determines whether the behavior of the person in the video is normal or abnormal by analyzing the multimodal feature signal. The normal state includes normal walking, operating equipment and other behaviors, while the abnormal state includes falling, electric shock, fatigue and other behaviors. The detection module 30 can quickly and accurately identify the abnormal state and generate the corresponding detection signal.

[0134] The alarm module 40 is connected to the detection module 30. When the detection signal indicates that the behavior of the person is abnormal, the alarm module 40 will output an alarm signal. The alarm signal can be issued in a variety of ways, such as sound alarm, light alarm, SMS notification, email notification, etc., to ensure that the operator or security manager can receive the alarm in time and take corresponding measures. The alarm module 40 is also responsible for sending the video clip containing abnormal behavior to cloud storage for subsequent analysis and review, and also for preservation as evidence.

[0135] Finally, the main controller 50 is connected to the alarm module 40 and is responsible for coordinating and controlling the operation of the entire system. The main controller 50 realizes comprehensive detection of personnel behavior and timely response to abnormal conditions through the data acquisition module 10, the multimodal feature generation module 20, the detection module 30 and the alarm module 40. The main controller 50 not only manages the data transmission and processing between the various modules, but is also responsible for controlling the generation of detailed event reports, including image analysis results, semantic analysis results and scene prediction results, providing comprehensive information support. After the alarm information is issued, the main controller 50 will continue to monitor the behavior of personnel in the substation to ensure that the abnormal conditions are effectively handled, and continue to monitor under normal conditions to ensure that any potential problems are discovered in a timely manner.

[0136] In this way, the human behavior detection system of the present invention not only improves the response speed and accuracy of the system, but also enhances the operator's trust and reliance on the system, thereby significantly improving the human-computer interaction capability. The system can better adapt to complex scenarios in practical applications and improve the safety management level of substations and other places with high safety requirements.

[0137] The above description is only an exemplary embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the technical concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A method for detecting human behavior, characterized in that: Used in a substation, wherein at least one photographic component is provided in the substation, the personnel behavior detection method comprises: Acquiring video data captured by the photographic component; Inputting the video data into a pre-trained multimodal feature generation model, wherein the multimodal feature generation model includes a visual encoding module, a semantic encoding module, and a decoding module; Encoding the video data by the visual encoding module to obtain image features corresponding to the video data; Encoding the video data by the semantic coding module to obtain semantic features corresponding to the video data, and / or encoding the image features by the semantic coding module to obtain semantic features corresponding to the video data; The image features and semantic features are decoded by the decoding module to obtain a multimodal feature signal corresponding to the image features and semantic features, wherein the multimodal feature signal is used to characterize whether the behavior of personnel in the substation is in an abnormal state or a normal state; Determining the state of human behavior in the substation according to the multimodal characteristic signal; The training steps of the multimodal feature generation model include: Acquire at least one training sample from preset personnel behavior big data, wherein the training sample includes at least one visual image and a language label corresponding to the visual image; Inputting the training sample into the multimodal feature generation model, and updating the parameters of the multimodal feature generation module; The number of the training samples is multiple, and the multiple training samples are used to characterize that the behavior of the personnel in the substation is in the first abnormal state, and the language label of each of the training samples is also used to characterize the abnormality level of the behavior of the personnel in the substation in the first abnormal state; Determining the behavior of personnel in the substation according to the multimodal characteristic signal includes: When the video data is input into the multimodal feature generation model, and the multimodal feature generation model determines that the scene corresponding to the video data includes a first abnormal state, an alarm signal is outputted to indicate that the behavior of personnel in the substation is in an abnormal state, and a level signal is outputted to indicate the abnormal level of the first abnormal state; The training samples include at least one training sample for representing that the personnel in the substation are in an abnormal state and at least one training sample for representing that the personnel in the substation are in a normal state; Determining the behavior of personnel in the substation according to the multimodal characteristic signal includes: In the case where the multimodal characteristic signal does not represent the abnormal state and the normal state, determining that there is a potential abnormal state in the behavior of personnel in the substation, and determining the specific status of the personnel through scene analysis; The personnel behavior detection method further includes: When the multimodal characteristic signal indicates that the behavior of personnel in the substation is abnormal, triggering an alarm mechanism and sending the video data to the cloud; and / or, Generate an event report based on the video data, wherein the event report includes at least one of an image analysis result, a semantic analysis result, and a scene prediction result; After the alarm information is issued, continue to detect the behavior of personnel in the substation until it is confirmed that the abnormal state has been eliminated; When the multimodal characteristic signal indicates that the behavior of the personnel in the substation is in a normal state, the behavior of the personnel in the substation continues to be detected.

2. The personnel behavior detection method according to claim 1, characterized in that: Before inputting the training sample into the multimodal feature generation model, the method further includes: Performing transformation processing on the visual images in the training samples to obtain at least one processed visual image, wherein the transformation processing includes image scaling and / or color adjustment and / or noise processing and / or clarity processing and / or mixed images; According to the language tag corresponding to the visual image, a language tag of the processed visual image is determined.

3. The personnel behavior detection method according to claim 1, characterized in that: The substation is provided with a plurality of photographic components, and the obtaining of the video data captured by the photographic components specifically comprises: Acquire multiple video data captured by the multiple photographic components; The multiple video data are subjected to image processing, wherein the image processing includes noise filtering processing, histogram equalization processing, contrast enhancement processing, and / or frame rate adjustment processing.

4. A main controller, characterized in that: The main controller is used to implement the personnel behavior detection method as described in any one of claims 1 to 3.

5. A personnel behavior detection system, characterized in that: The behavior detection system comprises: A data acquisition module having multiple high-resolution photographic components for capturing video data; A multimodal feature generation module is connected to the data acquisition module, and comprises a visual encoding module, a semantic encoding module and a decoding module. The visual encoding module is used to perform visual encoding on the video data to obtain image features, the semantic encoding module is used to perform semantic encoding on the video data to obtain semantic features, and the decoding module is used to perform decoding processing on the image features and the semantic features, and output a multimodal feature signal; A detection module connected to the multimodal feature generation module, the detection module is used to output a detection signal representing the behavior of the person according to the multimodal feature signal; An alarm module connected to the detection module; when the detection signal indicates that the behavior of the person is in an abnormal state, an alarm signal is output; The main controller as described in claim 4 is connected to the multimodal feature generation module and the detection module, and the main controller is used to realize personnel behavior detection through the data acquisition module, the multimodal feature generation module, the detection module and the alarm module.

6. A substation, characterized in that: The substation comprises: A main controller as claimed in claim 4; or The personnel behavior detection system as described in claim 5.

Images