Abnormal action retrieval positioning method, system, equipment and medium

Abstract

The invention relates to the technical field of personnel action retrieval and positioning in an electric power inspection system, and discloses an abnormal action retrieval and positioning method, system and device and a medium, and the method comprises the steps: obtaining first image data of a target electric power scene, and carrying out the preprocessing, thereby providing a reliable basis for subsequent operation; a situation awareness image encoder is established, personnel attitude information can be efficiently integrated into a visual feature space, and the scene awareness understanding ability is improved. And an identity-based difficult case set is preset, so that the model is helped to identify and process complex conditions, and the accuracy and robustness of abnormal action retrieval and positioning are improved. Image and text embedding fusion is carried out, image and text information is utilized, the cross-modal understanding and retrieval capability of the model is enhanced, and a multi-modal large-scale language model meeting image and text embedding fusion is established. The method has important application value in scenes such as electric power inspection and safety monitoring, abnormal actions are quickly and accurately retrieved and positioned in an automatic and intelligent mode, the working efficiency is improved, and the safety guarantee is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of personnel action retrieval positioning in power inspection systems, and in particular to an abnormal action retrieval positioning method, system, device and medium. BACKGROUND

[0002] In the field of intelligent power system inspection, traditional image-based personnel abnormal action retrieval positioning methods often struggle to meet the needs of practical applications when faced with only verbal event reports and tasks that require fine behavior understanding, such as detecting abnormal activities during inspection. Specifically, these methods have obvious limitations in handling complex scenarios and subtle actions, and are unable to accurately capture and identify key information.

[0003] Existing T-PReID (Temporal-Pose Robust Re-Identification) systems, while improving the accuracy of personnel identification to some extent, are mostly designed for static, identity-centric detection scenarios. These systems typically assume that queries are based on clearly defined visual attributes, and when users provide vague, incomplete, or behavior-centric descriptions, the system's performance is often poor and difficult to effectively address. In addition, T-PReID systems also have limitations in interpreting or distinguishing subtle actions, such as distinguishing between "falling" and "sitting". The system lacks sufficient recognition and differentiation capabilities, leading to potential misjudgments in practical applications.

[0004] At the same time, existing video anomaly detection (VAD) methods also have many problems. These methods usually rely on rough binary labels (i.e., normal and abnormal), lack fine-grained text-level positioning capabilities, and are difficult to accurately describe and locate abnormal behavior. In addition, existing VAD datasets mostly focus on scene-level analysis rather than identity-centric analysis, resulting in small dataset size, sparse annotation information, and relatively narrow application field. These problems severely restrict the application effect of VAD methods in actual intelligent power system inspection, and further improvement and perfection are urgently needed. SUMMARY

[0005] In view of the above existing problems, the present application is proposed.

[0006] Therefore, the present application provides an abnormal action retrieval positioning method, system, device and medium, which can solve the limitations of traditional methods in complex scenarios and subtle action recognition, and achieve more accurate personnel abnormal action retrieval positioning.

[0007] To solve the above technical problems, the present application provides the following technical solutions:

[0008] In a first aspect, the present application provides an abnormal action retrieval positioning method, comprising:

[0009] Obtaining target power scene first image data, and performing first preprocessing on the first image data;

[0010] The first preprocessing is used to obtain personnel posture information in the first image data;

[0011] Establishing a situation awareness image encoder, which is used to integrate the personnel posture information into a visual feature space;

[0012] Predefining an identity-based difficult example set;

[0013] According to the output of the situation awareness image encoder, the identity-based normal set and the identity-based difficult example set, performing image and text embedding fusion;

[0014] Establishing a multi-modal large language model meeting the image and text embedding fusion, and performing abnormal action retrieval positioning according to the multi-modal large language model.

[0015] As a preferred scheme of the abnormal action retrieval positioning method of the present application, wherein: the first preprocessing on the first image data comprises:

[0016] Establishing a key point detector, which comprises a pre-set key point set and a mark for the key point set;

[0017] The key point detector is used to mark key points for the input first image data;

[0018] The first image data is input into the key point detector to obtain a posture heat map, which is the personnel posture information.

[0019] As a preferred scheme of the abnormal action retrieval positioning method of the present application, wherein: the situation awareness image encoder comprises a feature conversion unit, a visual feature extraction unit and an integration unit;

[0020] The feature conversion unit comprises several layers of convolution layers, normalization layers and nonlinear activation layers, and is used to convert the personnel posture information into learnable features;

[0021] The visual feature extraction unit is used to extract visual features from the personnel posture information;

[0022] The integration unit is established through a multi-head cross attention mechanism.

[0023] As a preferred scheme of the abnormal action retrieval positioning method, the preset identity-based difficult example set comprises:

[0024] Obtaining a first frame of a certain individual in the target power scene first image data and a natural language description of the current behavior of the corresponding individual;

[0025] Obtaining a second frame of the same individual in the target power scene first image data under different behavior conditions and a natural language description;

[0026] Mixing the relevant parameters of the two frames obtained by the corresponding individual as a difficult example for the identity of the corresponding individual.

[0027] As a preferred scheme of the abnormal action retrieval positioning method, the image and text embedding fusion according to the output of the situational awareness image encoder, the identity-based normal set and the identity-based difficult example set comprises:

[0028] Establishing a cross-modal encoder, the cross-modal encoder comprising a classification token;

[0029] Using the output of the classification token to predict whether the output of the situational awareness image encoder, the identity-based normal set and the identity-based difficult example set match, that is, calculating a binary classification loss.

[0030] As a preferred scheme of the abnormal action retrieval positioning method, the image and text embedding fusion according to the output of the situational awareness image encoder, the identity-based normal set and the identity-based difficult example set further comprises:

[0031] Randomly masking part of the text in the identity-based normal set and the identity-based difficult example set;

[0032] Using paired images to predict the missing labels through the mask label prediction head.

[0033] This preferred scheme can promote the model to learn more robust cross-modal representations by randomly masking part of the text, and enhance the model's understanding of the correlation between images and text. Using paired images to predict the missing labels through the mask label prediction head not only improves the model's ability to capture image details, but also enhances the consistency between text descriptions and image content. By introducing the mask label prediction mechanism, the accuracy and robustness of image and text embedding fusion are further improved, thereby achieving better performance in the abnormal action retrieval positioning task.

[0034] As a preferred solution of the abnormal action retrieval positioning method, the loss function of the multi-modal large language model meeting the embedding fusion of the image and the text is a weighted loss function of an image-text loss function, a bidirectional contrast loss function and a mask token prediction loss function.

[0035] In a second aspect, the present application provides an abnormal action retrieval positioning system, comprising:

[0036] A data acquisition and processing module is configured to acquire first image data of a target power scene and perform first preprocessing on the first image data.

[0037] The first preprocessing is configured to acquire personnel posture information in the first image data.

[0038] A situation awareness module is configured to establish a situation awareness image encoder, which is configured to integrate the personnel posture information into a visual feature space.

[0039] A set establishment module is configured to predefine an identity-based difficult example set.

[0040] A fusion module is configured to perform embedding fusion of an image and text according to an output of the situation awareness image encoder, an identity-based normal set and an identity-based difficult example set.

[0041] A model establishment module is configured to establish a multi-modal large language model meeting the embedding fusion of the image and the text, and perform abnormal action retrieval positioning according to the multi-modal large language model.

[0042] In a third aspect, the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method as described above when executing the computer program.

[0043] In a fourth aspect, the present application provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the method as described above.

[0044] Compared with the prior art, the beneficial effects of the present application are: the present application proposes an abnormal action retrieval positioning method, obtains target power scene first image data and performs preprocessing, which can ensure the accuracy and integrity of the image data and provide a reliable basis for subsequent steps. A situation awareness image encoder is established, which can efficiently integrate personnel posture information into a visual feature space, improving the perception and understanding ability of the scene. A preset identity-based difficult example set helps the model better identify and handle complex situations, improving the accuracy and robustness of abnormal action retrieval positioning. Image and text embedding fusion is performed, which fully utilizes image and text information and enhances the cross-modal understanding and retrieval ability of the model. A multi-modal large language model that meets the image and text embedding fusion is established, which can realize accurate retrieval and positioning of abnormal actions and improve the intelligent level and practicality of the system. The method of the present application has important application value in power inspection, safety monitoring and other scenes. Through automation and intelligentization, fast and accurate retrieval and positioning of abnormal actions are realized, which not only improves work efficiency, but also enhances safety protection. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0046] Figure 1 A method flow chart of an abnormal action retrieval positioning method provided by an embodiment of the present application.

[0047] Figure 2 An architecture schematic diagram of a "human-computer interaction" posture perception framework for text-based action abnormality retrieval positioning in power inspection of an abnormal action retrieval positioning method provided by an embodiment of the present application.

[0048] Figure 3 A human-computer interaction framework diagram during testing of an abnormal action retrieval positioning method provided by an embodiment of the present application.

[0049] Figure 4 An iteration process schematic diagram of a human-computer interaction mechanism during testing of an abnormal action retrieval positioning method provided by an embodiment of the present application.

[0050] Figure 5 An internal structure diagram of an electronic device of an abnormal action retrieval positioning method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the protection scope of the present application.

[0052] Embodiment 1, refer to Figures 1-4 For the first embodiment of the present application, the embodiment provides an abnormal action retrieval positioning method, comprising:

[0053] In the prior related art, there are some problems. The traditional image-based personnel abnormal action retrieval positioning method has limitations in processing complex scenes and subtle actions, and it is difficult to meet the actual application requirements. And the existing T-PReID system and video anomaly detection method (VAD) also have their own shortcomings, and it is difficult to effectively deal with the abnormal action retrieval positioning task in the power inspection system.

[0054] The present application provides a method that can effectively solve the above-mentioned problems. Next, how to realize the abnormal action retrieval positioning method will be described in detail in combination with multiple embodiments;

[0055] Figure 1 A method flowchart of an abnormal action retrieval positioning method is shown, comprising:

[0056] S101, acquiring target power scene first image data, and performing first preprocessing on the first image data;

[0057] It should be noted that at present, in the field of intelligent inspection of power systems, the technical means for personnel abnormal action retrieval and positioning based on images are generally concentrated on static scenes and detection modes with identity recognition as the core. These technologies usually preset the query condition as the explicit definition of visual attributes, that is, require the user to provide specific and clear visual feature description. However, in actual application, when the description information provided by the user is relatively vague, incomplete, or focuses on the behavior itself rather than the identity feature, the performance of the existing technology is often unsatisfactory. The more prominent problem is that these technical means have obvious shortcomings in explaining or distinguishing some subtle actions. For example, for actions such as "falling down" and "sitting down", which are similar in vision but have completely different actual meanings, the existing system often has difficulty in accurately identifying and distinguishing them, thereby affecting the overall effect and reliability of intelligent inspection.

[0058] Therefore, the application designs a brand-new efficient retrieval and accurate positioning scheme for abnormal actions in the power inspection scene. This scheme not only overcomes the limitations of traditional image processing methods in complex scenes and subtle action recognition, but also significantly improves the accuracy and robustness of abnormal action retrieval and positioning, providing a new technical path for intelligent power system inspection.

[0059] It should be noted that the most important thing in design is to explore some rules and build and optimize models according to the data that can be obtained. Therefore, the application obtains first image data of the target power scene.

[0060] In some specific embodiments, the first image data can include personnel action features, environmental background information, and timestamp information. Personnel action features refer to the posture, action, and other features of personnel in the image. These features can reflect the behavior patterns of personnel and are the key basis for abnormal action retrieval and positioning. Environmental background information includes scenes, objects, and other information in the image. These information can provide context information about the occurrence of the action, which helps to more accurately understand the meaning of the action. The timestamp information records the collection time of the image data, which helps to locate and track the abnormal action in time. Through comprehensive analysis of these information, accurate retrieval and positioning of abnormal actions can be achieved.

[0061] In some specific embodiments, the first image data can be collected by high-definition cameras. These cameras are deployed in key areas of the power system, such as substations, power transmission line perimeters, etc. The cameras have high sensitivity and high resolution, and can capture subtle actions of personnel and subtle changes in environmental background.

[0062] In some specific embodiments, the first image data can also be collected by image acquisition devices carried by unmanned aerial vehicles or inspection robots. These devices can acquire image data in complex or difficult-to-reach power scenes, further expanding the application range of the application method. The collected first image data is then transmitted to the data processing center for subsequent processing and analysis.

[0063] In the embodiment of the application, the first preprocessing is used to obtain the personnel posture information in the first image data;

[0064] In some specific embodiments, the first preprocessing can be implemented by establishing a deep learning model that is trained to automatically identify and extract personnel posture information in the image. This step is the basis for subsequent situation awareness and abnormal action retrieval positioning. Through accurate personnel posture information extraction, the accuracy and effectiveness of the subsequent steps can be ensured. The specific process of the first preprocessing may include image enhancement, noise removal, edge detection, etc. to improve image quality and highlight personnel posture features, thereby providing strong support for the accurate identification of the deep learning model.

[0065] However, the model requires too much computation, so a new preprocessing method needs to be designed to significantly reduce the computation. In the preferred embodiment of the present application, a preprocessing strategy based on a key point detector is adopted.

[0066] In the embodiment of the present application, the first preprocessing of the first image data includes:

[0067] A key point detector is established, which includes a set of preset key points and establishes a mark for the set of key points;

[0068] The key point detector is used to mark the key points of the input first image data;

[0069] The first image data is input into the key point detector to obtain a posture heat map, which is the personnel posture information.

[0070] It should be noted that the key point detector is an algorithm or model that identifies specific body parts (such as elbows, knees, etc.) from an image. These parts are called key points. The key point detector can identify and mark the positions of these points.

[0071] It should be noted that the set of preset key points refers to a pre-defined set of body key points. For example, in human posture estimation, this may include shoulder, elbow, wrist, hip, knee, and ankle positions. This set is set according to the needs of the application scenario.

[0072] It should be noted that the mark for the set of key points refers to assigning each key point an identifier or label so that the system can distinguish different key points. For example, by assigning a specific mark to the elbow key point, the system knows that this point represents the elbow and not other parts.

[0073] It should be noted that the process of marking the key points of the input first image data refers to analyzing the input image using the key point detector and marking all positions belonging to the "set of preset key points" on the image. This means that all pre-defined key points will be marked on the original image.

[0074] It should be noted that the pose heat map is a representation method used to show the probability of each pixel in the image becoming a certain key point. High probability areas are usually represented by warmer colors (such as red), while low probability areas are represented by cooler colors (such as blue). The pose heat map is essentially a visual representation of the human pose, which reflects the possible positions of the key points in the image.

[0075] In an embodiment of the present application, given an input image I, a 2D key point detector is applied to locate anatomical landmarks (such as joints and limbs) and construct a dense pose heat map P. This heat map encodes the spatial layout and relative configuration of the human body parts.

[0076] It should be noted that obtaining the first image data of the target power scene and performing first preprocessing on the first image data can significantly improve the accuracy and efficiency of abnormal action retrieval positioning. First, by obtaining high-quality first image data, subtle movements of personnel in the power scene and subtle changes in the environment background can be captured, providing a rich information base for subsequent analysis. Secondly, the first preprocessing step removes redundant information in the image by extracting personnel posture information, highlighting key features, making subsequent processing and analysis more efficient and accurate. This step not only reduces the computational complexity, but also improves the real-time performance and robustness of the system. Therefore, the method proposed in the present application lays a solid foundation in the preprocessing stage, providing a reliable guarantee for subsequent situation awareness, embedding fusion of images and text, and abnormal action retrieval positioning steps.

[0077] S102, a situation awareness image encoder is established, which is used to integrate personnel posture information into a visual feature space;

[0078] It should be noted that the data after the first preprocessing cannot be directly used in the subsequent steps of the present application, therefore, a method for converting the features of the first preprocessed data into learnable features is needed. The present application achieves this by designing a situation awareness image encoder.

[0079] In an embodiment of the present application, the situation awareness image encoder includes a feature conversion unit, a visual feature extraction unit, and an integration unit;

[0080] The feature conversion unit includes several convolutional layers, normalization layers, and nonlinear activation layers, and is used to convert personnel posture information into learnable features;

[0081] The visual feature extraction unit is used to extract visual features from the personnel posture information;

[0082] The integration unit is established through a multi-head cross-attention mechanism.

[0083] It is important to note that the Pose-aware image encoder is a model or algorithm designed to process and understand image content, with a particular focus on transforming pose information into a data representation that aids in identifying abnormal motions. Its goal is to improve the understanding of the scene and enable more accurate localization and identification of abnormal behaviors.

[0084] It is important to note that the convolutional layer in the feature transformation unit is a commonly used layer type in deep learning, especially when dealing with image data. They detect local features such as edges or textures in the input data by applying a series of filters (also known as kernels). The normalization layer is used to adjust the distribution of data in different batches, making the training process more stable and efficient. Common normalization methods include Batch Normalization, which can be used after each layer to maintain consistency in data scale. In neural networks, activation functions introduce nonlinearity, allowing the network to learn more complex patterns. Common activation functions include ReLU (Rectified Linear Unit), Sigmoid, and Tanh. Nonlinear activation layers are usually placed after each convolutional layer. The learnable feature refers to the data representation obtained after processing by the feature transformation unit, which can be further learned and utilized by machine learning models to complete specific tasks such as classification or regression.

[0085] It is important to note that the visual feature extraction unit is responsible for extracting visual features from the pose information of the person that help understand the scene. Here, visual features may include but are not limited to the pose of the human body, the direction and speed of motion, etc. This process is crucial for capturing individual behavior patterns, especially when distinguishing between normal and abnormal motions.

[0086] In some specific embodiments, in order to convert it into a learnable feature, the invention passes P through a custom PoseConv block composed of stacked convolution, normalization, and nonlinear activation layers. The generated pose embedding f P As an explicit behavior code, it captures structure related to motion, complementing texture and color information in the original image.

[0087] The original image I is processed by Swin-BTransformer, which encodes hierarchical visual features in spatial windows. This generates a set of patch-wise embeddings f I And a global class label Representing aggregated image context. To achieve effective fusion of appearance and pose modalities, the invention employs a multi-head cross-attention mechanism, taking f P As the query, f I As keys and values:

[0088]

[0089] f V =f I +f CA

[0090] The process generates a pose-fused image representation f V that combines appearance-level and structural motion cues. This fusion enhances the model's ability to distinguish visually similar but behaviorally distinct actions, enabling more accurate and contextually relevant detection in behavior-centric search scenarios.

[0091] It should be noted that the establishment of the situation awareness image encoder can effectively combine the personnel pose information with the visual features, and improve the accuracy and robustness of the abnormal action retrieval positioning. In traditional image processing methods, only the appearance features such as texture and color of the image are relied on for abnormal action recognition, and the pose information of the personnel is ignored. However, in actual application, the pose of the personnel often contains rich behavior information, which is of great significance for distinguishing normal and abnormal actions. Through the situation awareness image encoder, the personnel pose information can be integrated into the visual feature space, so that the model can consider both appearance features and pose information when identifying abnormal actions, thereby improving the accuracy of identification.

[0092] S103, preset identity-based difficult example set;

[0093] It should be noted that neither the traditional model training nor the regularity finding method has enough data set to provide the basis for training or regularity finding, so the present application also designs a data set expansion method. That is, the data expansion is realized by establishing an identity-based difficult example set.

[0094] In the embodiment of the present application, the preset identity-based difficult example set comprises:

[0095] Obtain the first frame of a certain individual in the target power scene first image data and the natural language description of the current behavior of the corresponding individual;

[0096] Obtain the second frame of the same individual in the target power scene first image data under different behavior conditions and the natural language description;

[0097] Mix the relevant parameters of the two frames obtained for the corresponding individual as a difficult example for the identity of the corresponding individual.

[0098] In embodiments of the invention, let (I, T) denote a positive example image-text pair, where I is a frame containing a certain individual and T is a natural language description of the individual's current behavior (e.g., walking, sitting, falling). Then sample a second frame I' and its related text description T' for the same person under a different behavior condition. The mashup pair (I, T') shares the same identity semantics but differs in behavior - considered as a hard example. This formulation introduces a form of behavior-sensitive contrastive learning that penalizes incorrect semantic alignment. It forces the model to differentiate fine-grained action descriptions rather than relying on identity priors only. As a result, the learned embedding space is more sensitive to subtle behavior changes, facilitating more accurate detection in contexts where behavior plays a key role, such as anomaly detection, risk monitoring, or public safety surveillance.

[0099] In some specific implementations, first, the invention requires selecting a behavior sample of a specific inspector from a dataset. Suppose the invention has an inspector A, who has multiple behaviors in a day's work:

[0100] Normal behavior: walking, bending to check equipment, standing to review records.

[0101] Abnormal behavior: accidental fall.

[0102] These behaviors are all recorded by video surveillance, and each video has a corresponding text description.

[0103] Second, create positive example image-text pairs;

[0104] Select a normal "walking" behavior from the above behaviors as a positive example:

[0105] Image I: a picture of inspector A walking normally in a corridor.

[0106] Text description T: "Inspector A is walking normally in the corridor."

[0107] This set (I, T) constitutes a positive example image-text pair.

[0108] Third, create hard example samples, the key here is to find samples of the same person (i.e., inspector A) but under different behavior conditions. For example, the invention can select a picture of him falling and the related text description:

[0109] Image I': a picture of inspector A falling at a certain location in the same corridor.

[0110] Text description T': "Inspector A accidentally fell while working."

[0111] Combine these two different behaviors to form a hard example sample:

[0112] Hard example image-text pairs: (I, T') and (I', T), i.e.:

[0113] (I, T'): the image is a picture of the inspector A walking normally, but the text description is "Inspector A accidentally fell while working."

[0114] (I', T): the image is a picture of the inspector A falling, but the text description is "Inspector A is walking normally in the corridor."

[0115] Finally, the behavior-sensitive contrast learning form is introduced. In this stage, the model is trained to distinguish these hard example samples. Specifically:

[0116] The model is required to judge whether the pair matches according to the provided image and text description. Since (I, T') and (I', T) are both mismatched, they are used as hard examples to train the model to recognize this inconsistency. This approach forces the model to not only rely on individual identity features (such as clothing, appearance, etc.), but also pay attention to behavior features (such as posture, action, etc.), thereby improving its sensitivity to subtle behavior changes.

[0117] It should be noted that the preset identity-based hard example set can effectively expand the data set and enhance the generalization ability of the model. In the traditional training method, the model often relies on limited labeled data for learning, which may lead to poor performance of the model when facing new, unseen scenarios. By using the preset identity-based hard example set, we can artificially create more challenging training samples that can simulate various complex situations that may occur in the real world. In this way, the model can be exposed to more diverse data during training, learning more robust feature representations, and making more accurate judgments when encountering new scenarios. In addition, the introduction of the hard example set can also encourage the model to pay more attention to behavior features, improving its sensitivity to subtle behavior changes, which is crucial for tasks such as abnormal action retrieval and positioning.

[0118] S104, according to the output of the situational awareness image encoder, the identity-based normal set and the identity-based hard example set, the image and the text are embedded and fused;

[0119] It should be noted that after the data preparation is completed, the data needs to be analyzed to find patterns and summarize an effective feature representation method. This step is a key link in the abnormal action retrieval and positioning method.

[0120] In the embodiment of the application, according to the output of the situational awareness image encoder, the identity-based normal set and the identity-based hard example set, the image and the text are embedded and fused, including:

[0121] The cross-modal encoder includes a classification token;

[0122] The output of the classification token is used to predict whether the output of the situation awareness image encoder, the identity-based normal set, and the identity-based hard example set match, i.e., to calculate a binary classification loss.

[0123] In embodiments of the present invention, the embedding fusion of images and text based on the output of the situation awareness image encoder, the identity-based normal set, and the identity-based hard example set further includes:

[0124] Randomly mask part of the text in the identity-based normal set and the identity-based hard example set;

[0125] Use the paired images to recover the missing labels through the mask label prediction head.

[0126] It should be noted that the cross-modal encoder is a model or algorithm component that processes different types of input data (such as images and text) and maps them into a common feature space for effective comparison and matching. The cross-modal encoder is used here to combine image information and text descriptions to enhance the understanding of the scene.

[0127] In some specific implementations, in the field of natural language processing (NLP), especially when using architectures like Transformer, the "classification token" (usually labeled as [CLS]) is a special token added to the beginning of the input sequence. The final hidden state of this token can be used as an aggregated representation of the entire input sequence, often used for classification tasks. In this context, the output of the classification token is used to predict the degree of matching between the output of the situation awareness image encoder, the identity-based normal set, and the identity-based hard example set.

[0128] It should be noted that the binary classification loss refers to a loss function that measures the difference between the model's predicted results and the true labels. In this case, it is used to evaluate whether the output of the situation awareness image encoder, the identity-based normal set, and the identity-based hard example set match. By minimizing this loss, the model can learn how to better distinguish between normal and abnormal behavior.

[0129] It should be noted that random masking is a data augmentation technique that randomly masks part of the input data (here, part of the words in the text) during training. This is done to force the model to learn to predict the masked parts based on the context, thereby improving its understanding and generation capabilities.

[0130] In some specific embodiments, the masked token prediction head refers to a component of the model designed specifically to predict the masked tokens based on contextual information. In this way, the model not only learns to understand complete sentences, but also infers missing information, enhancing its ability to handle incomplete information.

[0131] In embodiments of the present application, the image-text matching objective supervises binary classification of aligned and unaligned image-text pairs. After f V and its corresponding text embedding f T are fused, the concatenated features are processed by a cross-modal transformer encoder. The output of the [CLS] token is used to predict:

[0132]

[0133] where p denotes the binary true match label, is the predicted match probability. This objective encourages the model to learn global alignment and joint reasoning over appearance and behavior signals.

[0134] In some specific embodiments, to align the representations in a shared embedding space, the present application applies a bidirectional contrastive loss that pulls matching image-text pairs closer while pushing non-matching pairs apart. The pose-aware image features are aggregated by combining patch-level statistics and global class tokens:

[0135]

[0136] Similarly, for the text features f T . The loss function is:

[0137]

[0138] where S IT and S TI are the soft-max normalized similarities computed using cosine distance and temperature scaling τ. This objective promotes discriminative detection in the embedding space, especially in cases where the appearance is similar but the behavior is different.

[0139] In some specific embodiments, to enhance fine-grained image-to-text understanding, the present application randomly masks tokens in the input text and uses a paired image to recover the missing tokens through a masked token prediction head:

[0140]

[0141] where is the soft-max output over the vocabulary tokens. This task improves the model's ability to reason about missing or uncertain semantic elements.

[0142] It should be noted that the embedding fusion of images and texts based on the output of the situation-aware image encoder, the identity-based normal set and the identity-based difficult example set can significantly improve the precision and robustness of abnormal action retrieval positioning. By fusing the appearance information in the image and the posture information extracted by the self-defined PoseConv block, the model can more comprehensively understand the behavior in the scene. In addition, by introducing the identity-based difficult example set, the model's sensitivity to subtle behavior changes is further enhanced by creating challenging training samples. The application of cross-modal encoders enables image and text information to be effectively compared and matched in a common feature space, further improving the model's performance. This comprehensive approach not only improves the accuracy of abnormal action retrieval positioning, but also enhances the model's generalization ability in complex scenarios.

[0143] S105, a multi-modal large language model satisfying embedding fusion of images and texts is established, and abnormal action retrieval positioning is performed according to the multi-modal large language model.

[0144] In some specific embodiments, the multi-modal large language model satisfying embedding fusion of images and texts can use a Transformers architecture or a pre-trained model such as BERT as a basis and be fine-tuned with knowledge and data in a specific field. Such a model can fully utilize image and text information and achieve understanding of complex scenarios through embedding fusion. In the task of abnormal action retrieval positioning, the multi-modal large language model can capture the appearance features in the image and the posture information extracted by the situation-aware image encoder, while combining the behavior information in the text description to achieve accurate identification of abnormal actions. Through training and fine-tuning on large-scale datasets, the model can learn more robust and generalizable feature representations, thereby improving its performance in real-world applications. Ultimately, using the established multi-modal large language model, we can achieve fast and accurate retrieval and positioning of abnormal actions in surveillance videos, providing strong technical support for security monitoring, risk warning and other fields.

[0145] In the embodiments of the present application, the loss function of the multi-modal large language model satisfying embedding fusion of images and texts is a weighted loss function of the image-text loss function, the bidirectional contrast loss function and the mask token prediction loss function.

[0146] In embodiments of the invention, given an initial textual query T0, the model first performs a coarse retrieval localization step to return the top k candidate images. These results are then examined by the MLLM, which identifies key semantic omissions - such as missing action labels, object functions, or ambiguous pose indicators - that can make the precise retrieval localization more difficult. In response, the MLLM poses targeted follow-up questions aimed at eliciting clarifying information from the user, and once the user provides answers, the system dynamically appends the new information to an updated query T1. The model then restarts the retrieval localization based on this refined input. This interactive loop can continue for multiple rounds, progressively narrowing the semantic gap between the user intent and the model's interpretation - without the need for retraining or parameter updates to the model.

[0147] The total loss function combines all the target losses:

[0148]

[0149] where λ sim , λ ccl , and λ mlm are weights (typically set to 1).

[0150] It is important to note that Figure 2 The architecture of a multi-modal fusion model is shown in the figure, which is used to combine image and text information for understanding and analysis. The following is a detailed explanation of the contents of the figure:

[0151] Image input: A person image is shown on the left, wearing a blue work uniform and a safety hat, possibly working with the right hand at a construction site.

[0152] Text input: A descriptive text is given on the right: "He is wearing a blue work uniform and a safety hat, and he is probably working at a construction site, holding his right hand."

[0153] Pose-aware image encoder (PE): Feature extraction is performed on the input image to generate visual feature vectors fV.

[0154] Feature mapping: The extracted visual features are mapped into a multi-dimensional space to form multiple feature representations fv1, fv2,..., fvn.

[0155] Text encoder: The input text is encoded to generate a text feature vector fT.

[0156] Feature mapping: The text features are mapped into the same space as the image features to form multiple feature representations ft1, ft2,..., ftn.

[0157] Self-attention module: Applied to image features and text features respectively, to enhance the internal information interaction and understanding ability of each other.

[0158] Feedforward layer: further processes the output of the self-attention module to enhance feature representation capabilities.

[0159] Cross-attention module: cross-links image features and text features through a query vector q, enabling information fusion between image and text.

[0160] Feedforward layer: processes the output of the cross-attention module again to further optimize the fused feature representation.

[0161] Masked language modeling head: used to predict the masked part of the text, such as "blue pants", to enhance the model's understanding and generation capabilities of the text.

[0162] Contrastive learning: by calculating the similarity between image features and text features, optimize model parameters to make matching image and text pairs have higher similarity.

[0163] Image-text matching pairing: the final output of whether the image and text match, as shown in the figure, "1" means matching, "0" means not matching.

[0164] Figure 3 The workflow of a multi-modal dialogue system is shown in the figure. Here is a detailed explanation:

[0165] Dialogue scenario C1, text encoder: converts the input text information (such as "a person is directing work") into a vector representation in the multi-modal embedding space.

[0166] Image encoder: converts the input image information into a vector representation in the multi-modal embedding space.

[0167] Multi-modal embedding space, in this space, text and image information are mapped to the same feature space, facilitating comparison and fusion.

[0168] The figure shows the distribution of different information points in this space, by calculating the distance or similarity between these points, their relevance can be judged.

[0169] According to the information in the multi-modal embedding space, extract candidate information related to the current dialogue scenario.

[0170] Scenario reconstruction, reconstructor: based on the extracted candidate information, reconstruct the dialogue scenario to generate a more accurate description (such as "he is directing work with his right hand").

[0171] Questioner, according to the reconstructed scenario, ask temporary questions (such as "what is this person wearing?") to further refine the dialogue content.

[0172] Filterer, based on the user's actual question (e.g., "What is this person wearing?"), filters the most relevant questions from the question generator and provides an answer (e.g., "a blue work uniform").

[0173] User, the user asks a question (e.g., "What is this person wearing?"), and the system generates and returns an answer based on the above process.

[0174] Figure 4 The workflow of a multi-turn dialogue system is demonstrated in the following steps, which is used to gradually confirm and refine the description of the image content. Here is a detailed explanation:

[0175] First round of user questioning: Is this an accurate description of the image?

[0176] Description content: "This person is a male, and this person is wearing a blue work uniform and a safety hat."

[0177] System answer: No.

[0178] Follow-up action: Continue asking questions.

[0179] The second round of user may have asked further questions or requested more detailed descriptions.

[0180] Third round of user questioning: Is this text an accurate description of the image?

[0181] Description content: "A man is wearing a blue work uniform and a safety hat. His blue work uniform covers the torso and legs, with some wrinkles and stains. The safety hat on his head has a chin strap, fixed below his chin."

[0182] System answer: Yes.

[0183] Fourth round of user questioning: Is this person male or female?

[0184] System answer: This person is male.

[0185] Fifth round of user questioning: Does this person have something in their hand?

[0186] System answer: No clear answer, but can be inferred as "no."

[0187] Based on the information from the above K rounds of dialogue, a detailed description is generated: "A man is wearing a blue work uniform and a safety hat. His blue work uniform covers the torso and legs, with some wrinkles and stains. The safety hat on his head has a chin strap, fixed below his chin. He raises his right hand, with his fingers slightly curled, as if making a certain gesture. His arm is stretched upwards, with a straight posture."

[0188] In summary, the present application proposes an abnormal action retrieval positioning method, which obtains target power scene first image data and pre-processes it, can ensure the accuracy and integrity of the image data, and provides a reliable basis for the subsequent steps. The situation awareness image encoder is established, which can efficiently integrate the personnel posture information into the visual feature space, improve the perception and understanding ability of the scene. The preset identity-based difficult example set helps the model to better identify and handle complex situations, and improves the accuracy and robustness of abnormal action retrieval positioning. The embedding fusion of image and text is carried out, which fully utilizes the image and text information, and enhances the cross-modal understanding and retrieval ability of the model. The multi-modal large language model meeting the image and text embedding fusion is established, which can realize accurate retrieval and positioning of abnormal actions, and improve the intelligent level and practicality of the system. The method of the present application has important application value in power inspection, safety monitoring and other scenes. Through the automatic and intelligent way, the rapid and accurate retrieval and positioning of abnormal actions are realized, which not only improves the work efficiency, but also enhances the safety protection.

[0189] In one preferred embodiment, the present application was extensively evaluated on two challenging textual person abnormal action retrieval positioning datasets. The first is the publicly available PAB (Pedestrian Abnormal Action) dataset, which serves as a benchmark for textual abnormal action retrieval positioning. The second is the newly proposed EBPAD (Electricity Business Pedestrian Abnormal Action) dataset of the present application, which is specifically designed to capture pedestrian abnormal actions in the electricity business scene, enriching the diversity and authenticity of the evaluation settings.

[0190] PAB contains 1,015,583 images, covering 2,600 pedestrian identities, and each image is accompanied by a text annotation containing appearance, action, and scene description. The dataset contains 1,600 action types and provides balanced representation of abnormal and normal behavior, with a ratio of 3:2. PAB provides fine-grained frame-by-frame annotations, built from synthetic and real sources. This dataset is particularly suitable for textual abnormal behavior search, significantly expanding behavior diversity and addressing key gaps in existing person action retrieval positioning and video anomaly detection datasets, enabling detailed retrieval and positioning of rare, scene-specific actions.

[0191] EBPAD contains 10,468 images, covering 845 identities, and each image is accompanied by two text descriptions with an average of 58 words. The training set contains 7,842 image-text pairs of 705 identities, while the validation and test sets contain 1,280 and 1,346 image-text pairs of 140 remaining identities, respectively. This dataset focuses on the electricity business scene, highlighting the challenges of cross-modal action retrieval positioning in complex industrial conditions, as well as specific domain visual semantics.

[0192] The resolution of the input image is 720x1280 pixels. During training, a series of augmentations are applied to the images to avoid overfitting and improve generalization ability. Each image is randomly adjusted in the scale range of [0.8, 1.2], randomly offset from the center and cropped to maintain the original size. The camera intrinsic matrix K is adjusted accordingly to maintain the multi-view Figure 1 consistency. To avoid overfitting of the detector, some noise is introduced into the translation vector t of the extrinsic matrix during training. In the training setting, the Adam optimizer is used in combination with a cyclic learning rate scheduler, with a maximum learning rate setting of 1x10^-3. The encoder and decoder networks are initialized using pre-trained weights on ImageNet-1K.

[0193] The model is implemented in PyTorch and trained using the AdamW optimizer for 30 epochs, with a learning rate linearly decaying from 1e-4 to 1e-5 and a weight decay of 0.01. The batch size used by the present application is 22. The image encoder uses Swin-B, the text encoder uses 6-layer BERT, and the cross-encoder uses 6 layers and is initialized from X-VLM. The final model has a parameter size of 230.8M.

[0194] The present application compares HiPAn with several representative baseline models: RaSa, IRRA and APTM. The results are shown in Table 1. RaSa achieves an R@1 of 80.79% in the zero-shot setting, highlighting the difficulty of the task. Extensive experiments on PAB and EBPAD benchmarks show that HiPAn consistently outperforms the powerful baseline models in detection accuracy, achieving an R@1 of 85.27% and an mAP of 90.10% on PAB, and an R@1 of 71.13% and an mAP of 80.49% on EBPAD, while demonstrating strong generalization ability in out-of-distribution scenarios

[0195] Table 1: Comparison of cross-dataset performance. The best results are bolded.

[0196]

[0197] Figure 4An iterative process of the test-time human-machine interaction mechanism of the present application is illustrated by way of example, which is crucial for refining the precise characterization of image content through query. In the first round, the system proposes questions to assess the consistency between the provided text and the image. If the answer is negative, it will continue to ask further questions, conducting an iterative search to obtain more detailed information. As the process proceeds, for example in the kth round, answers including statements such as "the individual is male" and "the individual is wearing a blue overalls and safety helmet" are accumulated. These answers are then combined into a comprehensive query, which describes the attributes of the individual in detail. This includes a detailed description of the overalls, including wrinkles and stains, the chin strap of the safety helmet is securely fastened, and the right hand is raised, with the fingers slightly bent. This iterative process eventually leads to a "new ranking" action retrieval localization result. The graphical representation clearly demonstrates the effectiveness of the test-time human-machine interaction method of the present application. It vividly demonstrates the ability of the model in context-aware dialogue generation, gradually narrowing the semantic gap between the initial query and the actual image semantics. Through human-machine interactive query refinement, the model is able to locate more relevant results, improving the accuracy. This highlights its ability to manage ambiguous text input and perform accurate cross-modal (image-text) matching, which is a key requirement in practical applications.

[0198] To verify the effectiveness of each component in the proposed HiPAn framework of the present application, the present application conducts comprehensive ablation experiments on the PAB and EBPAD datasets, and the results are shown in Table 2. Starting from a baseline without a pose-guided fusion module (PE), identity-based hard negative mining (IHNM), or test-time human-machine interaction (THI), the present application gradually adds each module to evaluate its individual and joint contribution. On the PAB dataset, adding PE increases R@1 from 80.27% to 80.69%, and adding IHNM alone increases R@1 to 80.80%. The addition of THI brings greater improvement, increasing R@1 to 81.17%. When PE and IHNM are used jointly, R@1 reaches 81.47%, and further adding THI (i.e., the complete HiPAn model) achieves the highest R@1 of 82.57% and mAP of 90.10%, demonstrating the complementary advantages of all three modules. Similar trends are observed on the EBPAD dataset. PE and IHNM alone bring modest improvements in action retrieval localization accuracy, while THI again makes a significant contribution (R@1 increases from 68.82% to 69.81%). The complete model achieves the best performance, with R@1 of 71.13% and mAP of 80.49%, indicating that the design of the present application has strong generalization ability and consistent effectiveness on the datasets.

[0199] Table 2: Ablation study on the posture-guided fusion module (PE), the identity-based hard example mining module (IHNM), and the test-time human-in-the-loop interaction module (THI) in both datasets.

[0200]

[0201] The present application proposes HiPAn, which is a framework for pedestrian cross-modal action retrieval and localization in power system security monitoring. The interactive text-to-image personnel anomaly search (T-PAS) task proposed by the present application reflects the real conditions in the power grid environment, in which operating personnel or patrol personnel may exhibit key but subtle behaviors (e.g., falling or lying down) in high-risk scenarios, and only text observations are available. By utilizing a multi-modal large language model (MLLM) for a test-time human-in-the-loop cycle, HiPAn allows users to iteratively refine ambiguous queries. A posture-aware encoder captures motion-related structures to improve action-level discrimination, while identity-based hard example mining introduces structured contrast to learn fine-grained behavior representations. The cross-modal learning objectives of the present application collectively enhance the alignment between visual and language modalities. Experiments on large-scale benchmarks confirm that HiPAn significantly improves retrieval and localization accuracy in both in-domain and out-of-distribution settings. These results emphasize the applicability of the framework in intelligent power patrol systems, where timely retrieval and localization of personnel abnormal actions are crucial for rapid decision-making and risk mitigation.

[0202] Embodiment 3, with reference to Figure 5 The present embodiment also provides an abnormal action retrieval and localization system, comprising:

[0203] A data acquisition and processing module is configured to acquire first image data of a target power scene and perform first preprocessing on the first image data.

[0204] The first preprocessing is configured to obtain personnel posture information in the first image data.

[0205] A situational awareness module is configured to establish a situational awareness image encoder, which is configured to integrate the personnel posture information into a visual feature space.

[0206] A set establishment module is configured to predefine an identity-based hard example set.

[0207] A fusion module is configured to perform embedding fusion of images and text based on the output of the situational awareness image encoder, the identity-based normal set, and the identity-based hard example set.

[0208] A model establishment module is configured to establish a multi-modal large language model that satisfies the embedding fusion of images and text, and perform abnormal action retrieval and localization based on the multi-modal large language model.

[0209] The above-mentioned each unit module can be embedded in or independent of the processor in the electronic device in hardware form, or can be stored in the memory in the electronic device in software form, so as to call and execute the operation corresponding to each module by the processor.

[0210] The embodiment also provides an electronic device, which can be a terminal, and an internal structure diagram of the electronic device can be as shown in the figure. The electronic device comprises a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The computer program is executed by the processor to implement an abnormal action retrieval positioning method. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the electronic device, or an external keyboard, touchpad or mouse, etc. Figure 5

[0211] The embodiment also provides a computer readable storage medium, which stores a computer program. The computer program is executed by the processor to implement the following steps:

[0212] Obtaining target power scene first image data, and performing first preprocessing on the first image data;

[0213] The first preprocessing is used to obtain personnel posture information in the first image data;

[0214] Establishing a situation awareness image encoder, which is used to integrate the personnel posture information into a visual feature space;

[0215] Predefining an identity-based difficult example set;

[0216] According to the output of the situation awareness image encoder, the identity-based normal set and the identity-based difficult example set, performing embedding fusion of images and texts;

[0217] Establishing a multi-modal large language model meeting the embedding fusion of images and texts, and performing abnormal action retrieval positioning according to the multi-modal large language model.

[0218] ​It should be noted that the above examples are only used to illustrate the technical solutions of the present application but not limit the present application, and although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the spirit and scope of the present application, and all modifications and equivalents thereof should be included in the scope of the claims of the present application.

[0219] Although the preferred embodiments of the present application have been described, those skilled in the art who have the benefit of the present disclosure will appreciate that additional modifications and variations to the preferred embodiments can be made without departing from the spirit and scope of the present application. Accordingly, it is intended that the appended claims encompass all such modifications and variations as falling within the scope of the present application.

[0220] Obviously, various modifications and changes can be made to the present application by those skilled in the art without departing from the spirit and scope of the present application. Thus, it is intended that the present application encompass all such modifications and changes as fall within the scope of the claims and their equivalents.

Claims

1. A method for retrieving and locating abnormal actions, characterized in that: include: Acquire first image data of a target power scene, and perform first preprocessing on the first image data; The first preprocessing is used to obtain the posture information of the person in the first image data; Establishing a situation awareness image encoder, wherein the situation awareness image encoder is used to integrate the personnel posture information into a visual feature space; Preset identity-based hard example set; Performing image and text embedding fusion based on the output of the situation awareness image encoder, the identity-based normal set, and the identity-based hard example set; A multimodal large-scale language model that satisfies the embedding fusion of the image and text is established, and abnormal action retrieval and positioning is performed based on the multimodal large-scale language model.

2. The abnormal action retrieval and positioning method according to claim 1, characterized in that: The performing a first preprocessing on the first image data includes: Establishing a key point detector, the key point detector including a preset key point set and establishing a marker for the key point set; The key point detector is used to mark key points on the input first image data; The first image data is input into a key point detector to obtain a posture heat map, which is the posture information of the person.

3. The abnormal action retrieval and positioning method according to claim 2, characterized in that: The situation awareness image encoder includes a feature conversion unit, a visual feature extraction unit and an integration unit; The feature conversion unit includes several convolution layers, normalization layers and nonlinear activation layers, and is used to convert personnel posture information into learnable features; The visual feature extraction unit is used to extract visual features through personnel posture information; The integration unit is established through a multi-head cross-attention mechanism.

4. The abnormal action retrieval and positioning method according to claim 3, characterized in that: The preset identity-based difficult example set includes: Acquire a first frame of an individual in first image data of a target power scene and a natural language description of a current behavior of the corresponding individual; The second frame and natural language description of the same individual under different behavioral conditions in the first image data of the same target power scene; The relevant parameters of the two frames obtained for the corresponding individual are mixed as a difficult example for the identity of the corresponding individual.

5. The abnormal action retrieval and positioning method according to claim 4, characterized in that: The embedding fusion of the image and the text according to the output of the situation awareness image encoder, the identity-based normal set, and the identity-based difficult example set includes: establishing a cross-modal encoder, the cross-modal encoder comprising classification tokens; The output of the classification token is used to predict whether the output of the situation awareness image encoder, the identity-based normal set, and the identity-based hard example set match, that is, to calculate the binary classification loss.

6. The abnormal action retrieval and positioning method according to claim 5, characterized in that: The embedding fusion of the image and the text according to the output of the situation awareness image encoder, the identity-based normal set, and the identity-based difficult example set further includes: Randomly mask parts of the text in the identity-based normal set and the identity-based hard set; The paired images are used to recover the missing landmarks via the masked landmark prediction head.

7. The abnormal action retrieval and positioning method according to claim 6, characterized in that: The loss function for establishing a multimodal large language model that satisfies the embedding fusion of the image and text is a weighted loss function of the image-text loss function, the bidirectional contrast loss function, and the mask mark prediction loss function.

8. An abnormal action retrieval and positioning system, applying the method according to any one of claims 1 to 7, characterized in that: include: a data acquisition and processing module, configured to acquire first image data of a target power scene and perform first preprocessing on the first image data; The first preprocessing is used to obtain the posture information of the person in the first image data; A situation awareness module, configured to establish a situation awareness image encoder, wherein the situation awareness image encoder is configured to integrate the personnel posture information into a visual feature space; A set building module, used to preset identity-based hard example sets; a fusion module for performing embedding fusion of image and text based on the output of the situation awareness image encoder, the identity-based normal set, and the identity-based difficult example set; The model building module is used to build a multimodal large language model that satisfies the embedding fusion of the image and text, and perform abnormal action retrieval and positioning based on the multimodal large language model.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the abnormal action retrieval and positioning method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the abnormal action retrieval and positioning method according to any one of claims 1 to 7 are implemented.

Images