Tracking method and device of substation operation site target and storage medium

Abstract

The invention discloses a tracking method and device for a substation operation site target and a storage medium, and the method comprises the steps: obtaining a multi-view multi-frame continuous image collected for a substation operation site, and recognizing a personnel image in the multi-frame continuous image corresponding to each view; extracting limb key points in the personnel image, determining limb actions of the operating personnel based on the limb key points, determining action feature vectors of the limb actions, and determining limb actions of the target operating personnel based on the action feature vectors; fusing the action feature vectors of the limb actions of the target operating personnel in the different view angle images corresponding to the same acquisition time point to obtain fused action features, and classifying the limb actions of the target operating personnel based on the fused action features to obtain limb actions under different action categories; and sorting the limb actions under each action category according to a time sequence to obtain a tracking action sequence of the target operator under each action category.

Description

Technical Field

[0001] The present invention relates to the technical field of substation operation safety, and in particular to a method, device and storage medium for tracking targets at a substation operation site. Background Art

[0002] Substation near-line work is a crucial component of power equipment inspection, repair, maintenance, and upgrades, ensuring reliable grid operation. However, substation operating environments are complex, and tracking the sequence of actions performed by substation personnel during operations is crucial to ensuring personnel safety and work quality.

[0003] Currently, the movements of workers at substation sites are typically tracked manually. However, this tracking method can be time-consuming and labor-intensive, and can also lead to errors if the substation environment is complex or if there are many workers. Summary of the Invention

[0004] The present invention provides a method, device and storage medium for tracking targets at a substation operation site, which are mainly capable of improving the tracking efficiency and tracking accuracy of the operating actions of operators at the substation operation site.

[0005] According to a first aspect of the present invention, a method for tracking a target at a substation operation site is provided, comprising:

[0006] Acquire multiple frames of continuous images from multiple perspectives collected at the substation operation site, and identify at least one person image of an operator in the multiple frames of continuous images corresponding to each perspective;

[0007] Extracting key points of the operator's limbs from the personnel image, determining the operator's limb movements based on the key points, determining a motion feature vector corresponding to the limb movement, and determining the limb movement of the target operator based on the motion feature vector;

[0008] Fusing motion feature vectors of the target worker's limb movements in images from different perspectives corresponding to the same acquisition time point to obtain fused motion features, and classifying the target worker's limb movements based on the fused motion features to obtain limb movements in different action categories;

[0009] The body movements under each action category are sorted in chronological order to obtain the tracking action sequence of the target operator under each action category.

[0010] Optionally, the method further includes:

[0011] Track the target operator in multiple frames of continuous images from multiple perspectives to obtain the target operator's motion trajectory at each perspective;

[0012] Determining the spatial position relationship and viewing angle ranges of the image acquisition devices corresponding to the multiple viewing angles, and constructing a spatial topological constraint relationship of the image acquisition devices based on the spatial position relationship and the viewing angle ranges;

[0013] Determining a conversion matrix between different image acquisition devices based on the spatial topological constraint relationship, and judging whether the motion state of the target worker under the different image acquisition devices meets preset conditions of a normal state based on the conversion matrix;

[0014] If the motion state meets the preset condition, the motion trajectory of the target operator under each viewing angle is merged to obtain a comprehensive motion trajectory of the target operator.

[0015] Optionally, judging whether the motion state of the target worker under different image acquisition devices meets preset conditions of a normal state based on the conversion matrix includes:

[0016] The pixel coordinates of the target worker in the multiple frames of continuous images are converted into a three-dimensional coordinate system under the corresponding image acquisition device, and the coordinates of the target worker in the three-dimensional coordinates are converted into a world coordinate system based on the conversion matrix to obtain the world coordinates of the target worker;

[0017] Determining the target worker's motion direction vector to be verified under different image acquisition devices based on the target worker's world coordinates in multiple frames of continuous images corresponding to different image acquisition devices;

[0018] Determining an expected motion direction vector of the target worker transferring between different image acquisition devices, and respectively determining an angle between each motion direction vector to be verified and the expected motion direction vector;

[0019] If the angle of each of the motion direction vectors to be verified is less than the preset angle threshold, it is determined that the motion state of the target operator under different image acquisition devices meets the preset conditions; otherwise, it is determined that the motion state of the target operator under different image acquisition devices does not meet the preset conditions.

[0020] Optionally, determining the operator's body movements based on the body key points includes:

[0021] Determining position information of each of the limb key points, and based on the position information, determining a positional relationship between each of the limb key points, wherein the positional relationship includes a distance relationship, an angle relationship, and a direction relationship between each of the limb key points;

[0022] Determine the relationship feature vector corresponding to the positional relationship, and input the relationship feature vector into a preset limb motion prediction model for motion prediction to obtain the limb motion of the operator, wherein the preset limb motion prediction model is pre-trained based on a set of positional relationships of sample limb key points with motion labels.

[0023] Optionally, determining the body movement of the target operator based on the movement feature vector includes:

[0024] Based on the motion feature vector, the body movements of each operator are matched with the standard body movements in a preset body movement library, and the body movements of the target operator are determined based on the matching results, wherein the preset body movement library stores standard body movements with operator identity identification.

[0025] Optionally, the limb movements of the target operator are classified based on the fused motion features to obtain limb movements in different motion categories, including:

[0026] Determining standard action feature vectors corresponding to standard limb movements under different action categories, and calculating the similarity between the operator's limb movement and each of the standard limb movements based on the fused action features and the standard action feature vectors;

[0027] Based on the similarity, the body movements of the target operator are classified to obtain body movements in different action categories.

[0028] Optionally, if the different perspectives include a first perspective, a second perspective, and a third perspective, the motion feature vector of the target worker in the image at the first perspective is used as the first motion feature vector, the motion feature vector of the target worker in the image at the second perspective is used as the second motion feature vector, and the motion feature vector of the target worker in the image at the third perspective is used as the third motion feature vector;

[0029] The step of fusing the motion feature vectors of the target worker's body movements in the images of different perspectives corresponding to the same acquisition time point to obtain the fused motion features includes:

[0030] performing feature-level fusion processing on the first action feature vector, the second action feature vector, and the third action feature vector to obtain a feature fusion vector;

[0031] performing element-level fusion processing on the first action feature vector, the second action feature vector, and the third action feature vector to obtain an element fusion vector;

[0032] performing low-order fusion processing on the first motion feature vector, the second motion feature vector, and the third motion feature vector to obtain a low-order fusion vector;

[0033] The feature fusion vector, the element fusion vector, and the low-order fusion vector are transformed using a preset transformation function to obtain the fusion action feature.

[0034] According to a second aspect of the present invention, there is provided a device for tracking a target at a substation operation site, comprising:

[0035] An acquisition unit is configured to acquire multiple frames of continuous images from multiple perspectives collected at the substation operation site, and to identify at least one image of an operator in the multiple frames of continuous images corresponding to each perspective;

[0036] a determination unit, configured to extract key points of the operator's limbs from the operator image, determine the operator's limb movements based on the key points of the limbs, determine a motion feature vector corresponding to the limb movement, and determine the limb movement of the target operator based on the motion feature vector;

[0037] a classification unit, configured to fuse motion feature vectors of the target worker's limb movements in images from different perspectives corresponding to the same acquisition time point to obtain fused motion features, and classify the target worker's limb movements based on the fused motion features to obtain limb movements in different action categories;

[0038] The sorting unit is used to sort the body movements under each action category in chronological order to obtain the tracking action sequence of the target operator under each action category.

[0039] According to a third aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for tracking substation operation site targets.

[0040] According to a fourth aspect of the present invention, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for tracking substation operation site targets when executing the program.

[0041] According to the present invention, a method, device, and storage medium for tracking a target at a substation operation site are provided. Compared with the current method of manually tracking the movements of workers at the substation operation site, the present invention collects multiple frames of continuous images from multiple perspectives of the substation, identifies the key points of the substation workers' limbs and their corresponding limb movements in the multiple frames of continuous images, determines the target worker based on the limb movement characteristics, fuses the limb movement characteristics of the target worker from different perspectives to obtain fused limb movement characteristics, classifies the limb movements of the target worker based on the fused limb movement, obtains limb movements under different movement categories, and finally sorts the limb movements under different movement categories in chronological order to obtain limb movements under different movement categories. Thus, by analyzing the multiple frames of continuous images from multiple perspectives to extract the key points and limb movements of the worker, and finally correlating the limb movement behaviors of the target worker captured from multiple angles to obtain a limb movement sequence, the present invention can avoid the time and labor consumption caused by manual tracking of movement sequences, and also avoid the problem of tracking errors caused by manual negligence, thereby improving the tracking efficiency and accuracy of the substation worker's limb movement sequence. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0043] Figure 1 A flow chart of a method for tracking a substation operation site target provided by an embodiment of the present invention is shown;

[0044] Figure 2 A flow chart of another method for tracking substation operation site targets provided by an embodiment of the present invention is shown;

[0045] Figure 3 A schematic structural diagram of a device for tracking a target at a substation operation site provided by an embodiment of the present invention is shown;

[0046] Figure 4 A schematic structural diagram of another substation operation site target tracking device provided by an embodiment of the present invention is shown;

[0047] Figure 5 A schematic diagram of the physical structure of a computer device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0048] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of the present application can be combined with each other.

[0049] Currently, manual tracking of workers' movements at substation work sites is time-consuming and labor-intensive, and can lead to tracking errors if the substation environment is complex or there are many workers.

[0050] In order to solve the above problems, an embodiment of the present invention provides a method for tracking targets at a substation operation site, such as Figure 1 As shown, the method includes:

[0051] 101. Acquire multiple frames of continuous images from multiple perspectives collected at a substation operation site, and identify at least one operator's image in the multiple frames of continuous images corresponding to each perspective.

[0052] In an embodiment of the present invention, multiple image acquisition devices, such as high-definition image acquisition devices, are pre-installed at the substation operation site. The installation positions and angles of the image acquisition devices are determined based on the layout and operation process of the substation operation site to ensure that the substation operation site can be photographed from different angles. Each image acquisition device is then used to photograph the substation operation site in real time to capture multiple frames of continuous images from different perspectives. The captured images are denoised using methods such as Gaussian filtering or median filtering to remove random noise and salt and pepper noise from the images and improve image quality. Each frame of the image from each perspective is then sequentially input into a preset personnel detection model for operator detection to obtain an image of the operator in each frame of the image. The image of the operator is present in the form of a labeled box. In an embodiment of the present invention, in order to improve the detection accuracy of the preset personnel detection model, it is first necessary to train and construct the preset personnel detection model. Based on this, the method includes: constructing a preset initial personnel detection model; obtaining a sample data set, wherein the sample data set includes multiple sample substation image data sets with annotation information, and the annotation information includes an annotation box and an annotation type of the target in the annotation box, and the annotation type includes people, power equipment, etc.; dividing the sample data set into training data and test data, using the training data to train the preset initial personnel detection model, and using the test data to test the trained preset initial personnel detection model, and finally using the trained preset initial personnel detection model that meets the test conditions as the preset personnel detection model.

[0053] Specifically, during the model training process, first build a preset initial personnel detection model, and then download the substation image dataset from the network. Make sure that the dataset contains all necessary files, including operating personnel. Convert the annotation file into a format that the preset initial personnel detection model can understand, and finally train and test the model. Specifically, you can first divide the dataset: use random or specific strategies (such as stratified sampling) to divide the sample dataset into training set and test set. Then use the training set to train the model, and use the test set to test the trained model to evaluate its performance on unseen data. Calculate and record indicators such as mCP, precision, and recall on the test set. If the model performance does not meet the requirements, you can return to the training stage for more iterations or adjustments. In this way, a preset personnel detection model that meets the requirements is obtained.

[0054] 102. Extract the key points of the operator's limbs from the personnel image, determine the operator's limb movements based on the key points of the limbs, and determine the action feature vector corresponding to the limb movement, and determine the limb movement of the target operator based on the action feature vector.

[0055] Among them, the key points of the limbs include the characteristic points with significant geometric structures on the operator's head, shoulders, elbows, wrists, hips, knees, ankles and other parts; the limb movements include: switch operation, tool use, equipment installation, raising head, waving, hugging, failure, kicking, fusion of legs, running, turning, bending over, etc.

[0056] For the embodiment of the present invention, in order to determine the limb movements of the operator, it is first necessary to extract the limb key points of the operator from the personnel image. The embodiment of the present invention can adopt a cross-shot skeleton key point re-identification algorithm, such as G-RMI (Google ReseCrch Multi-Person Pose EstimCtion, posture processing estimation algorithm), CFN (ComplementCry FeedforwCrd Network, complementary feature feedforward neural network), etc. Furthermore, the following method can be used to extract limb key points: construct an image pyramid corresponding to each person image; and construct a difference pyramid corresponding to each person image based on the image pyramid; use each difference pyramid layer except the first and last layers of the difference pyramid as a target difference pyramid layer, and use each pixel of the image in the target difference pyramid layer as a target pixel; determine each adjacent pixel within a preset difference neighborhood corresponding to the target pixel in a difference pyramid layer adjacent to the target difference pyramid layer, and determine each pixel within a preset same-layer neighborhood corresponding to the target pixel in the target difference pyramid layer; if the target pixel is the maximum pixel value or the minimum pixel value among the adjacent pixels and the same-layer pixels, then determine the target pixel as a limb key point in the corresponding image.

[0057] Specifically, the preset differential neighborhood and the preset same-layer neighborhood are both set according to actual needs. Specifically, in order to extract the limb key points in each person image, it is first necessary to construct an image pyramid for each image. The specific construction method of the image pyramid is: taking any image C in each image as an example, first use image C as the bottom-level image of the image pyramid. This layer of image has the highest resolution and is the clearest. Then, Gaussian blur processing is performed on the bottom-level image. After Gaussian blur processing, downsampling processing is performed. The downsampling process is such as merging every 2×2 pixels in the bottom-level image into a new pixel. All new pixels constitute the image in the upper pyramid layer corresponding to the bottom level. If the upper pyramid layer corresponding to the bottom level is determined to be the second pyramid layer, the image in the second pyramid layer is continued to be Gaussian blurred. Gaussian blur processing is performed, and then downsampling processing is performed. The downsampling process is such as merging every 2×2 pixels in the image in the second pyramid layer into a new pixel, and all new pixels constitute the image in the previous pyramid layer corresponding to the second pyramid layer, that is, the image in the third pyramid layer. And so on, Gaussian blur processing and downsampling processing are performed on the image in the previous pyramid layer layer by layer to obtain the image in the current pyramid layer until the resolution of the image in the latest pyramid layer meets the requirements. Finally, the image pyramid of image C is composed of various pyramid layers. Similarly, the image pyramid of each image can be constructed according to the above method, and the resolution of the image in each pyramid layer in the image pyramid is different.

[0058] Furthermore, after constructing the image pyramid corresponding to each image, it is also necessary to construct a Gaussian difference pyramid. The specific construction scheme is as follows: taking the two adjacent pyramid layers of the first pyramid layer and the second pyramid layer in the image pyramid as an example, since the resolutions of the images in the first pyramid layer and the second pyramid layer are different, first, a high-resolution image layer and a low-resolution image layer are determined in the first pyramid layer and the second pyramid layer, and the resolutions of the images in the high-resolution image layer and the low-resolution image layer, that is, the number of rows and columns of the images are determined respectively. Based on the resolutions of the images in the high-resolution image layer and the low-resolution image layer, a magnification factor of the image in the low-resolution image layer is determined. According to the magnification factor, the image in the low-resolution image layer is upsampled. The specific upsampling method is such as bilinear interpolation, bicubic interpolation, nearest neighbor method, etc., so that the resolution of the image in the low-resolution image layer is processed to the same resolution as the image in the high-resolution image layer. Therefore, according to the above method, the images in each adjacent pyramid layer in the image pyramid can be processed to images with the same resolution. Next, images of the same resolution in two adjacent pyramid layers of the image pyramid are subtracted to obtain a Gaussian difference pyramid. Starting from the bottom to the top of the Gaussian difference pyramid, they are sequentially recorded as the first difference pyramid layer, the second difference pyramid layer, the third difference pyramid layer, and so on. Starting from the second difference pyramid layer, the adjacent evaluation pixels and the same-layer evaluation pixels of each pixel in the second difference pyramid layer are determined. The specific determination method takes pixel C in the second difference pyramid layer as an example. In the first and third difference pyramid layers, the pixels in the preset difference neighborhood corresponding to pixel C are determined, i.e., the adjacent evaluation pixels. At the same time, in the second difference pyramid layer, the pixels in the preset same-layer neighborhood corresponding to pixel C are determined, i.e., the same-layer evaluation pixels. Then, it is determined whether pixel C is the pixel with the largest pixel value or the smallest pixel value among the adjacent evaluation pixels and the same-layer evaluation pixels. If so, pixel C is determined as the limb keypoint in the corresponding image, thereby determining the limb keypoint in each image. When the embodiment of the present invention uses a Gaussian difference pyramid to extract limb key points, the extracted limb key points are scale-invariant because image information at different scales is taken into account. That is, the same limb key points can be accurately identified at different scales, thereby improving the accuracy of extracting limb key points in the image.

[0059] Furthermore, after extracting the key points of each worker's limbs, the distance between adjacent limb key points is calculated as the limb length based on the positional information of the limb key points. For example, the length of the upper arm is the distance from the shoulder to the elbow, and the length of the forearm is the distance from the elbow to the wrist. Simultaneously, the angles between the limb key points are calculated, such as the angle formed by the shoulder, elbow, and wrist, which can reflect the degree of limb bending. By comparing the changes in limb length and angle under different movements, the limb movement can be accurately judged. Then, the action feature vector corresponding to the limb movement is determined using methods such as word embedding. The action feature vector of each worker is then similarly matched with the standard action feature vector corresponding to standard limb movements with worker identification stored in a preset limb movement library. Based on the similarity matching results, an identity is assigned to each worker. This allows the target worker to be identified within each worker based on the identity of each worker. The preset limb movement library stores standard limb movements corresponding to multiple workers with worker identification. For example, the cosine similarity calculation is performed on the limb movement feature vectors corresponding to the standard limb movement feature vectors corresponding to the standard limb movements of each operator in the preset limb movement library, and the maximum cosine similarity is determined among the cosine similarities, and the identity identifier of the operator corresponding to the maximum cosine similarity is determined in the preset limb movement library, and the identity of operator A is marked using the identity identifier. Thus, according to the above method, the identity of each operator in each personnel image can be marked, and finally, based on the identity marking results, the target operator can be determined from each operator, and the limb movement of the target operator can be determined based on the limb movement of each operator. The embodiment of the present invention can mark the identities of operators in multi-view captured images by presetting the limb movement library, and can directly find the target operator who needs to track the action sequence among multiple operators quickly and accurately.

[0060] 103. The motion feature vectors of the target operator's body movements in images of different perspectives corresponding to the same acquisition time point are fused to obtain fused motion features. Based on the fused motion features, the target operator's body movements are classified to obtain body movements under different motion categories.

[0061] Among them, different action categories include walking, running, jumping, crawling, grabbing, pulling, twisting, etc.

[0062] For the embodiments of the present application, for example, if the different viewing angles include viewing angle a, viewing angle b, viewing angle c, and the collection time points include 8:00, 8:01, 8:02, and 8:03, the limb action feature vectors of the target worker at the viewing angle a, the viewing angle b, and the viewing angle c collected at 8:00 are fused to obtain the fusion action feature corresponding to 8:00; the limb action feature vectors of the target worker at the viewing angle a, the viewing angle b, and the viewing angle c collected at 8:01 are fused to obtain the fusion action feature corresponding to 8:01; the limb action feature vectors of the target worker at the viewing angle a, the viewing angle b, and the viewing angle c collected at 8:02 are fused to obtain the fusion action feature corresponding to 8:02; and the limb action feature vectors of the target worker at the viewing angle a, the viewing angle b, and the viewing angle c collected at 8:03 are fused to obtain the fusion action feature corresponding to 8:03.

[0063] Further, after determining the fusion action features of the target worker at different collection time points, it is also necessary to classify the limb actions of the target worker based on the fusion action features. Based on this, the method comprises: determining the standard action feature vectors corresponding to the standard limb actions under different action categories, and respectively calculating the similarity between the limb actions of the worker and each of the standard limb actions based on the fusion action features and the standard action feature vectors; and classifying the limb actions of the target worker based on the similarity to obtain the limb actions under different action categories.

[0064] Specifically, the preset action category library stores the standard action feature vectors corresponding to the standard limb actions under multiple action categories. For the limb action A of the target worker, the cosine similarity between the limb action A and each of the standard limb actions in the preset action category library is respectively calculated based on the fusion action feature corresponding to the limb action A, and the action category to which the standard limb action corresponding to the maximum cosine similarity belongs is determined as the action category corresponding to the limb action A. In this way, the action category to which each limb action of the target worker corresponds can be determined in the above manner. The embodiments of the present application can convert the limb actions of the worker into quantifiable indexes by classifying the limb actions of the target worker, thereby laying a foundation for subsequent tracking of the action sequence.

[0065] 104. The limb actions under each action category are respectively sorted in time sequence to obtain the tracking action sequence of the target worker under each action category.

[0066] For the embodiment of the present invention, after determining the limb movements of different action categories of the target operator, the limb movements under each action category are sorted based on the time sequence of the acquisition of the personnel image, and the action sequence under each action category is obtained. The action sequence describes the continuous change process of a certain action of the target operator within a certain time period, which emphasizes the continuity and timing of the action. For example, by sorting the various limb movements (leg lifting, stepping, foot landing, etc.) under the leg action category in chronological order, the entire walking process sequence of the target operator can be obtained. Thus, by analyzing multiple frames of continuous images under multiple perspectives to extract the limb key points and limb movements of the operator, and finally correlating the limb movement behaviors of the target operator captured from multiple angles, a limb movement sequence is obtained, which can avoid the time-consuming and labor-intensive problem caused by manual tracking of the action sequence, and can also avoid the problem of tracking errors caused by manual negligence, so that the present invention can improve the tracking efficiency and tracking accuracy of the limb movement sequence of the substation operator.

[0067] According to a method for tracking a target at a substation operation site provided by the present invention, compared with the current method of manually tracking the movements of workers at the substation operation site, the present invention collects multiple frames of continuous images from multiple perspectives of the substation, identifies the key points of the substation workers' limbs and their corresponding limb movements in the multiple frames of continuous images, determines the target worker based on the limb movement features, fuses the limb movement features of the target worker from different perspectives to obtain fused limb movement features, classifies the limb movements of the target worker based on the fused limb movements, obtains limb movements under different movement categories, and finally sorts the limb movements under different movement categories in chronological order to obtain limb movements under different movement categories. Thus, by analyzing the multiple frames of continuous images from multiple perspectives to extract the key points and limb movements of the workers, and finally correlating the limb movement behaviors of the target workers captured from multiple angles to obtain a limb movement sequence, the present invention can avoid the time and labor-consuming problem of manually tracking the movement sequence, and also avoid the problem of tracking errors caused by manual negligence, thereby improving the tracking efficiency and accuracy of the substation worker's limb movement sequence.

[0068] Furthermore, in order to better illustrate the above process of classifying data, as a refinement and extension of the above embodiment, the embodiment of the present invention provides another method for tracking substation operation site targets, such as Figure 2 As shown, the method includes:

[0069] 201. Acquire multiple frames of continuous images from multiple perspectives collected at a substation operation site, and identify at least one operator's image in the multiple frames of continuous images corresponding to each perspective.

[0070] Specifically, models such as YOLO can be used to identify the workers in each frame of the image.

[0071] 202. Extract the operator's limb key points from the personnel image, determine the operator's limb movements based on the limb key points, and determine the action feature vector corresponding to the limb movement, and determine the target operator's limb movement based on the action feature vector.

[0072] For the embodiment of the present invention, after identifying the limb key points of the operator, it is also necessary to determine the limb movements of the operator based on the limb key points. Based on this, step 202 specifically includes: determining the position information of each of the limb key points, and based on the position information, determining the position relationship between each of the limb key points, wherein the position relationship includes the distance relationship, angle relationship, and direction relationship between each of the limb key points; determining the relationship feature vector corresponding to the position relationship, and inputting the relationship feature vector into a preset limb movement prediction model for movement prediction to obtain the limb movement of the operator, wherein the preset limb movement prediction model is pre-trained based on a position relationship set of sample limb key points with movement labels.

[0073] Specifically, in order to improve the prediction accuracy of the preset limb movement prediction model, it is first necessary to train and construct the preset limb movement prediction model. Based on this, the method includes: constructing a preset initial limb movement prediction model, wherein the preset initial limb movement prediction model can be a classification model, such as a support vector machine, a random forest, etc.; obtaining a sample data set, wherein the sample data set includes multiple sample limb position relationship data sets with labeled information, and the labeled information is a limb movement label; dividing the sample data set into training data and test data, using the training data to train the preset initial limb movement prediction model, and using the test data to test the trained preset initial limb movement prediction model, and finally using the trained preset initial limb movement prediction model that meets the test conditions as the preset limb movement prediction model.

[0074] Specifically, during the model training process, first build a preset initial limb motion prediction model, and then download a sample limb position relationship dataset from the network. Make sure that the dataset contains all necessary files, and finally train and test the model. Specifically, you can first divide the dataset: use random or specific strategies (such as stratified sampling) to divide the sample dataset into a training set and a test set. Then use the training set to train the model, and use the test set to test the trained model to evaluate its performance on unseen data. Calculate and record indicators such as mCP, precision, and recall on the test set. If the model performance does not meet the requirements, you can return to the training stage for more iterations or adjustments. In this way, a preset limb motion prediction model that meets the requirements is obtained.

[0075] Furthermore, based on the position information of each limb key point, the Euclidean distance between two adjacent limb key points is calculated, for example, the distance from the wrist key point to the elbow key point is calculated. And the angles between the three key points are calculated. For example, the angles between the shoulder key point, the elbow key point and the wrist key point are calculated to determine the degree of bending of the arm. And the relative direction between the two key points is calculated. For example, the direction of the wrist key point relative to the elbow key point is determined. Based on the positional relationships such as the distance relationship, angle relationship, and direction relationship between the limb key points determined in the above manner, the distance relationship, angle relationship, and direction relationship are combined into a relationship feature vector, and the relationship feature vector is input into a preset limb motion prediction model, which outputs the limb motion of the operator. Such as standing, sitting, lying on your back, walking, bending over, etc.

[0076] Furthermore, if the action sequence of the target operator is to be tracked, it is also necessary to determine the target operator among the various operators. Based on this, the method includes: based on the action feature vector, matching the limb movements of each operator with the standard limb movements in a preset limb movement library, and determining the limb movements of the target operator based on the matching results, wherein the preset limb movement library stores standard limb movements with operator identity identification.

[0077] Specifically, for a certain operator, the similarity between the operator's body movements and the standard body movements in the preset body movement library is calculated respectively, and the operator's identity identifier corresponding to the standard body movement with the maximum similarity is determined as the operator's identity identifier. In this way, the identity identifier of each operator can be determined according to the above method. Finally, based on the identity identifier of each operator, the target operator and his corresponding body movements can be determined among each operator.

[0078] 203. Fuse the motion feature vectors of the target operator's body movements in images from different perspectives corresponding to the same acquisition time point to obtain fused motion features, and classify the target operator's body movements based on the fused motion features to obtain body movements under different motion categories.

[0079] Among them, if the different perspectives include a first perspective, a second perspective and a third perspective, the action feature vector of the target operator in the image under the first perspective is used as the first action feature vector, the action feature vector of the target operator in the image under the second perspective is used as the second action feature vector, and the action feature vector of the target operator in the image under the third perspective is used as the third action feature vector.

[0080] For the embodiment of the present application, in order to make full use of the relationship between various action features, extract more implicit features, for a certain limb action, the action feature vectors of the target workers in the images of different views corresponding to the same collection time point need to be fused, based on which, step 203 specifically includes: performing feature-level fusion processing on the first action feature vector, the second action feature vector and the third action feature vector to obtain a feature fusion vector; performing element-level fusion processing on the first action feature vector, the second action feature vector and the third action feature vector to obtain an element fusion vector; performing low-order fusion processing on the first action feature vector, the second action feature vector and the third action feature vector to obtain a low-order fusion vector; and performing transformation processing on the feature fusion vector, the element fusion vector and the low-order fusion vector by using a preset transformation function to obtain the fused action feature.

[0081] Specifically, if the first action feature vector is (a1, a2), the second action feature vector is (b1, b2), and the third action feature vector is (c1, c2), the specific fusion processing method includes: performing feature-level fusion between different feature vectors, that is, performing Hadamard product on all elements between vectors, then performing convolution transformation under a certain weight w1 to obtain a feature fusion vector f(w1*(a1*b1*c1, a2*b2*c2, a3*b3*c3)); at the same time, performing element-level fusion on all feature vector data, that is, performing Hadamard product on each element between vectors, then assigning different weight values w2, w3 and w4 to the results after each multiplication, and then performing linear transformation to obtain an element fusion vector f(w2*a1*b1*c1, w3*a2*b2*c2, w4*a3*b3*c3); in addition, performing low-order fusion processing on all feature vectors, then assigning a weight coefficient w5 to the result after the fusion processing, and then performing linear transformation to obtain a low-order fusion vector f(w5(a1, a2, b1, b2, c1, c2)); finally, combining the above feature fusion vector, element fusion vector and low-order fusion vector together by using a preset transformation function to obtain the fused action feature, where the preset transformation function can be selected according to actual needs. It should be noted that the above examples are only illustrative and do not limit the embodiments of the present application. Thus, by performing fusion processing on the first action feature vector, the second action feature vector and the third action feature vector, different features can be automatically or explicitly combined to generate new feature combinations, these combined features can contain complex nonlinear relationships between original features, can capture more fine and rich information in the data, that is, can make full use of the relationship between various data, extract more implicit features, and at the same time, take into account high-order and low-order processing, so that the data utilization is more sufficient, the resulting data trend result is more accurate, and the demand of actual application scenarios is met.

[0082] Furthermore, after determining the fused motion features of the target operator at different image acquisition time points, it is necessary to classify the various limb movements of the target operator based on the fused motion features. The specific classification method is the same as that in step 103 and will not be repeated here.

[0083] 204. Sort the body movements under each action category in chronological order to obtain the tracking action sequence of the target operator under each action category.

[0084] Specifically, the limb movements under each action category are sorted according to the image acquisition time sequence, and the sorted limb movements constitute the action sequence of the target operator.

[0085] 205. Track the target operator in multiple frames of continuous images under multiple viewing angles to obtain a motion trajectory of the target operator under each viewing angle.

[0086] Specifically, for any viewpoint, target tracking technology, such as YOLO, Faster R-CNN, or other deep learning target detection algorithms, is used to identify the target worker in the first frame of the multi-frame image corresponding to that viewpoint. The target worker's features in the first frame are extracted, and then, based on these features, a feature matching algorithm is used to match the target worker in the remaining frames. The target worker's position information is determined in each image and converted into a real-world spatial coordinate system, thereby obtaining the target worker's trajectory at that viewpoint. Thus, according to the above method, the target worker's trajectory at each viewpoint can be determined.

[0087] 206. Determine the spatial position relationship and viewing angle ranges of the image acquisition devices corresponding to the multiple viewing angles, and construct a spatial topological constraint relationship of the image acquisition devices based on the spatial position relationship and viewing angle ranges.

[0088] Among them, topological constraint types include coverage constraints, connection constraints, and collaboration constraints between image acquisition devices. Coverage constraints require that the entire substation operation site be fully covered by the viewing angles of all image acquisition devices, without any blind spots. The overlapping viewing angles of adjacent image acquisition devices are set to ensure the reliability and fault tolerance of the image acquisition devices. For example, if the viewing angles of image acquisition devices A and B overlap in a certain area of ​​the substation operation site, the ratio of the overlapping area to the viewing angle of each image acquisition device is measured to ensure that it is within the set overlapping viewing angle range. Connection constraints: In a wired transmission system, the wiring scheme of the image acquisition devices is planned to ensure that each image acquisition device can be connected to the nearest switch or video recorder via a network cable to avoid excessive cable lengths and cross-interference, thereby ensuring stable signal transmission. Collaboration constraints: In scenarios where multiple image acquisition devices need to work together, time synchronization of the image acquisition devices is achieved. For example, using the Network Time Protocol (NTP) to ensure that each image acquisition device captures images at the same time to facilitate subsequent data fusion and processing. Define the collaborative rules of image acquisition devices. For example, when an operator is detected entering the substation operation site, adjacent image acquisition devices automatically adjust their angles and focal lengths to track the target together.

[0089] Specifically, the measurement tool can be used to measure the control position relationship and viewing angle range of the image acquisition device under each viewing angle. Each image acquisition device is represented as a node in the graph, and the edges between the nodes represent the topological constraint relationship between them. For example, if the viewing angle ranges of image acquisition device A and image acquisition device B overlap, an edge is added between their corresponding nodes, and the overlap type and overlap ratio are marked. The overlap type is such as "coverage overlap", and the overlap ratio is such as 15%. Corresponding attributes are assigned to each node and edge. The attributes of the node include spatial position coordinates (x, y, z), viewing angle range (horizontal viewing angle, vertical viewing angle), etc.; the attributes of the edge include coverage range, connection strength, coordination rules, etc. In this way, the spatial topological constraint relationship of the image acquisition device can be formed by each node and edge.

[0090] 207. Based on the spatial topological constraint relationship, determine the conversion matrix between different image acquisition devices, and based on the conversion matrix, judge whether the motion state of the target operator under different image acquisition devices meets the preset conditions of the normal state.

[0091] The transformation matrix describes the spatial transformation relationship from the coordinate system of one image acquisition device to the coordinate system of another image acquisition device. In three-dimensional space, the transformation matrix is ​​usually composed of a rotation matrix and a translation vector.

[0092] In an embodiment of the present invention, multiple points with distinct features that can be clearly seen from the perspectives of multiple image acquisition devices are selected as feature points within the substation operation area, such as the corner points of columns in the substation operation area and the center points of targets. An image processing algorithm (such as SIFT, SURF, etc.) is used to extract feature points from the images captured by each image acquisition device, and feature matching is performed to find corresponding feature point pairs in the images of different image acquisition devices. Based on the spatial topological constraint relationship and the known spatial position information of each feature point in the feature point pair, the coordinates of each feature point in the coordinate system of each image acquisition device are measured. For example, for feature point P, the coordinates in the unified coordinate system of the substation are (x, y, z). Then, based on the spatial position relationship and viewing angle range of image acquisition devices A and B, its coordinates in the coordinate system of image acquisition device A (x1, y1, z1) and the coordinates in the coordinate system of image acquisition device B (x2, y2, z2) are calculated. The coordinate transformation relationship [x2, y2, z2, 1] between image acquisition devices A and B is determined. T =T×[x1,y1,z1,1] T Where T is the transformation matrix, T = [R | t]. Then, based on the coordinate transformation relationship between image acquisition device A and image acquisition device B and the coordinates of multiple feature point pairs in the coordinate system of image acquisition device A and the coordinate system of image acquisition device B, respectively, a coordinate transformation equation can be constructed. The equations for the multiple feature point pairs are combined into a matrix equation. The rotation matrix R and translation vector t of the matrix equation are solved using the least squares method to obtain the transformation matrix T.

[0093] Furthermore, after determining the conversion matrix, it is necessary to determine whether the target worker's motion state under different image acquisition devices is normal based on the conversion matrix. Based on this, step 207 specifically includes: converting the pixel coordinates of the target worker in the multiple frames of continuous images into the three-dimensional coordinate system of the corresponding image acquisition device, and converting the target worker's coordinates in the three-dimensional coordinate system into the world coordinate system based on the conversion matrix to obtain the world coordinates of the target worker; determining the target worker's motion direction vector to be verified under the different image acquisition devices based on the target worker's world coordinates in the multiple frames of continuous images corresponding to the different image acquisition devices; determining the expected motion direction vector of the target worker when transferring between the different image acquisition devices, and respectively determining the angle between each of the motion direction vectors to be verified and the expected motion direction vector; if the angle of each of the motion direction vectors to be verified is less than a preset angle threshold, determining that the target worker's motion state under the different image acquisition devices meets the preset condition; otherwise, determining that the target worker's motion state under the different image acquisition devices does not meet the preset condition.

[0094] wherein the preset angle threshold is set according to actual requirements. Specifically, the intrinsic matrix of the image acquisition device is determined, and the depth of the target operator in the image acquisition device coordinate system is obtained through the depth sensor or estimated through a stereovision method or the like, and finally the coordinates of the target operator in the image acquisition device coordinate system are determined based on the intrinsic matrix, the depth, and the pixel coordinates of the target operator, and then the coordinates of the target operator in the image acquisition device coordinate system are converted into the world coordinate system according to the conversion matrix to obtain the world coordinates of the target operator. Further, for each image acquisition device, the motion direction vector of the target operator between adjacent frames of images is calculated, for example, if the world coordinates of the target operator in the jth frame of image are (x j , y j , z j ), and the world coordinates of the target operator in the j+1th frame of image are (x j+1 , y j+1 , z j+1 ), then the to-be-verified motion direction vector of the target operator in the jth frame and the j+1th frame of image is Thus, in the above manner, the to-be-verified motion direction vector of the target operator in each two frames of images can be determined. Further, the position coordinates of the adjacent two image acquisition devices A and B are determined as (x A , y A , z A ) and (x B , y B , z B ) respectively, then the expected motion direction vector is Thus, in the above manner, the expected motion direction vector corresponding to any adjacent image acquisition device pair can be determined. Then the angle between the to-be-verified motion direction vector and the corresponding expected motion direction vector is calculated, if the angle is less than the preset angle threshold, it is determined that the motion state of the target operator under different image acquisition devices meets the preset condition, i.e. the motion state is normal, if the angle is greater than or equal to the preset angle threshold, it is determined that the motion state of the target operator under different image acquisition devices does not meet the preset condition, i.e. the motion state is abnormal.

[0095] 208、If the motion state meets the preset condition, the motion trajectories of the target operator under each view angle are merged to obtain a comprehensive motion trajectory of the target operator.

[0096] Specifically, the trajectory coordinates of each point in the motion trajectory of the target worker in each view with a normal motion state are determined, and the trajectory coordinates in each view at the same time point are weighted and averaged to obtain the weighted and averaged trajectory coordinates at each time point, and the weighted and averaged trajectory coordinates at each time point are used to form a comprehensive motion trajectory of the target worker. The embodiment of the present application can improve the tracking accuracy of the motion trajectory by using the spatial topological constraints of the image acquisition device and the space-time constraints of the motion target trajectory to track the motion trajectory of the target worker.

[0097] Further, the comprehensive motion trajectory and the action sequence of the target worker are stored in correspondence with the identity. By analyzing the comprehensive motion trajectory and the action sequence of the target worker, the work safety of the target worker can be analyzed, and abnormal work can be warned to remind relevant personnel to take necessary safety measures, so that the work safety of the worker can be ensured.

[0098] According to another substation work site target tracking method provided by the present application, compared with the current way of tracking the actions of the workers in the substation work site by manual, the present application identifies the limb key points and the corresponding limb actions of the substation workers in the acquired multiple frames of continuous images in multiple views of the substation, determines the target worker based on the limb action features, fuses the limb action features of the target worker in different views to obtain the fused limb action features, classifies the limb actions of the target worker based on the fused limb action to obtain the limb actions in different action categories, and finally sorts the limb actions in different action categories in time sequence to obtain the limb actions in different action categories. Thus, the limb key points and the limb actions of the workers are extracted by analyzing the multiple frames of continuous images in multiple views, the target worker limb action behaviors captured from multiple angles are finally associated to obtain the limb action sequence, the time-consuming and labor-consuming problem caused by manual tracking of the action sequence can be avoided, and the problem of tracking errors caused by negligence of manual can also be avoided, so that the present application can improve the tracking efficiency and the tracking accuracy of the substation worker limb action sequence.

[0099] Further, as a specific implementation of Figure 1 , the embodiment of the present application provides a substation work site target tracking device, as shown in Figure 3 , the device comprises an acquisition unit 31, a determination unit 32, a classification unit 33 and a sorting unit 34.

[0100] The acquisition unit 31 can be used to acquire multiple frames of continuous images in multiple views collected for the substation work site, and identify at least one worker image in each view of the multiple frames of continuous images.

[0101] The determination unit 32 can be used to extract the operator's limb key points from the personnel image, determine the operator's limb movements based on the limb key points, and determine the action feature vector corresponding to the limb movement, and determine the target operator's limb movement based on the action feature vector.

[0102] The classification unit 33 can be used to fuse the motion feature vectors of the target worker's limb movements in images of different perspectives corresponding to the same acquisition time point to obtain fused motion features, and classify the target worker's limb movements based on the fused motion features to obtain limb movements under different motion categories.

[0103] The sorting unit 34 may be configured to sort the body movements of each movement category in chronological order to obtain a tracking movement sequence of each movement category of the target operator.

[0104] In specific application scenarios, in order to track the movement trajectory of the target operator, such as Figure 4 As shown, the device further includes a trajectory determination unit 35 .

[0105] The trajectory determination unit 35 can be used to track the target worker in multiple frames of continuous images under multiple perspectives to obtain the motion trajectory of the target worker under each perspective; determine the spatial position relationship and viewing angle range of the image acquisition devices corresponding to the multiple perspectives, and construct a spatial topological constraint relationship of the image acquisition devices based on the spatial position relationship and the viewing angle range; determine the transformation matrix between different image acquisition devices based on the spatial topological constraint relationship, and judge whether the motion state of the target worker under different image acquisition devices meets the preset conditions of the normal state based on the transformation matrix; if the motion state meets the preset conditions, the motion trajectory of the target worker under each perspective is merged to obtain a comprehensive motion trajectory of the target worker.

[0106] In a specific application scenario, in order to determine whether the motion state of the target operator under different image acquisition devices meets the preset conditions of the normal state, the trajectory determination unit 35 includes a conversion module 351, a first determination module 352, and a judgment module 353.

[0107] The conversion module 351 can be used to convert the pixel coordinates of the target operator in multiple frames of continuous images into a three-dimensional coordinate system under the corresponding image acquisition device, and based on the conversion matrix, convert the coordinates of the target operator in the three-dimensional coordinates into a world coordinate system to obtain the world coordinates of the target operator.

[0108] The first determination module 352 may be configured to determine the target worker's motion direction vector to be verified under different image acquisition devices based on the target worker's world coordinates in multiple frames of continuous images corresponding to different image acquisition devices.

[0109] The first determination module 352 may be specifically configured to determine the expected motion direction vector of the target operator when transferring between different image acquisition devices, and to respectively determine the angle between each motion direction vector to be verified and the expected motion direction vector.

[0110] The determination module 353 can be used to determine that the motion state of the target operator under different image acquisition devices meets the preset conditions if the angle of each of the motion direction vectors to be verified is less than a preset angle threshold; otherwise, it is determined that the motion state of the target operator under different image acquisition devices does not meet the preset conditions.

[0111] In a specific application scenario, in order to determine the operator's body movements, the determination unit 32 includes a second determination module 321 and a prediction module 322 .

[0112] The second determination module 321 can be used to determine the position information of each of the limb key points, and based on the position information, determine the position relationship between each of the limb key points, wherein the position relationship includes the distance relationship, angle relationship, and direction relationship between each of the limb key points.

[0113] The prediction module 322 can be used to determine the relationship feature vector corresponding to the position relationship, and input the relationship feature vector into a preset limb motion prediction model for motion prediction to obtain the limb motion of the operator, wherein the preset limb motion prediction model is pre-trained based on a set of positional relationships of sample limb key points with action labels.

[0114] In a specific application scenario, in order to determine the body movements of the target operator, the determination unit 32 further includes a matching module 323 .

[0115] The matching module 323 can be used to match the body movements of each operator with the standard body movements in a preset body movement library based on the motion feature vector, and determine the body movements of the target operator based on the matching results, wherein the preset body movement library stores standard body movements with operator identity identification.

[0116] In a specific application scenario, in order to classify the body movements of the target operator, the classification unit 33 includes a calculation module 331 and a classification module 332 .

[0117] The calculation module 331 can be used to determine the standard action feature vectors corresponding to the standard limb movements under different action categories, and based on the fused action features and the standard action feature vectors, respectively calculate the similarity between the operator's limb movements and each of the standard limb movements.

[0118] The classification module 332 may be configured to classify the body movements of the target operator based on the similarity to obtain body movements in different categories.

[0119] In a specific application scenario, if the different perspectives include a first perspective, a second perspective, and a third perspective, the action feature vector of the target worker in the image under the first perspective is used as the first action feature vector, the action feature vector of the target worker in the image under the second perspective is used as the second action feature vector, and the action feature vector of the target worker in the image under the third perspective is used as the third action feature vector; in order to fuse the action feature vectors of the limb movements of the target worker in images of different perspectives corresponding to the same acquisition time point, the classification unit 33 also includes a fusion module 333 and a transformation module 334.

[0120] The fusion module 333 may be configured to perform feature-level fusion processing on the first motion feature vector, the second motion feature vector, and the third motion feature vector to obtain a feature fusion vector.

[0121] The fusion module 333 may also be configured to perform element-level fusion processing on the first motion feature vector, the second motion feature vector, and the third motion feature vector to obtain an element-fused vector.

[0122] The fusion module 333 may also be configured to perform low-order fusion processing on the first motion feature vector, the second motion feature vector, and the third motion feature vector to obtain a low-order fusion vector.

[0123] The transformation module 334 may be configured to transform the feature fusion vector, the element fusion vector, and the low-order fusion vector using a preset transformation function to obtain the fused action feature.

[0124] It should be noted that for other corresponding descriptions of the functional modules involved in the tracking device for a substation operation site target provided by the embodiment of the present invention, reference can be made to Figure 1 The corresponding description of the method shown will not be repeated here.

[0125] Based on the above Figure 1The method shown, accordingly, an embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, which implements the following steps when executed by a processor: obtaining multiple frames of continuous images from multiple perspectives collected at the substation operation site, and identifying at least one person image of an operator in the multiple frames of continuous images corresponding to each perspective; extracting the operator's limb key points from the personnel image, determining the operator's limb movements based on the limb key points, and determining the action feature vector corresponding to the limb movement, and determining the limb movement of the target operator based on the action feature vector; fusing the action feature vectors of the limb movements of the target operator in images of different perspectives corresponding to the same acquisition time point to obtain fused action features, and classifying the limb movements of the target operator based on the fused action features to obtain limb movements under different action categories; sorting the limb movements under each action category in chronological order to obtain a tracking action sequence under each action category of the target operator.

[0126] Based on the above Figure 1 The method shown and Figure 3 The embodiment of the device shown in the figure, the embodiment of the present invention also provides a physical structure diagram of a computer device, such as Figure 5 As shown, the computer device includes: a processor 41, a memory 42, and a computer program stored in the memory 42 and executable on the processor, wherein the memory 42 and the processor 41 are both arranged on a bus 43. When the processor 41 executes the program, the following steps are implemented: obtaining a plurality of continuous frames of images from multiple perspectives collected at the substation operation site, and identifying at least one person image of an operator in the plurality of continuous frames of images corresponding to each perspective; extracting the operator's limb key points from the personnel images, determining the operator's limb movements based on the limb key points, and determining the action feature vector corresponding to the limb movement, and determining the limb movement of the target operator based on the action feature vector; fusing the action feature vectors of the limb movements of the target operator in images from different perspectives corresponding to the same acquisition time point to obtain fused action features, and classifying the limb movements of the target operator based on the fused action features to obtain limb movements under different action categories; and sorting the limb movements under each action category in chronological order to obtain a tracking action sequence for each action category of the target operator.

[0127] Through the technical solution of the present invention, the present invention collects multiple frames of continuous images from multiple perspectives of the substation, and identifies the key points of the body parts of the substation workers and their corresponding body movements in the multiple frames of continuous images. Based on the body movement features, the target worker is determined, and the body movement features of the target worker from different perspectives are fused to obtain fused body movement features. Based on the fused body movements, the body movements of the target worker are classified to obtain body movements under different action categories, and finally the body movements under different action categories are sorted in chronological order to obtain body movements under different action categories. Thus, by analyzing the multiple frames of continuous images from multiple perspectives to extract the key points and body movements of the workers, the body movement behaviors of the target workers captured from multiple angles are finally associated to obtain a body movement sequence, which can avoid the time-consuming and labor-intensive problem of manually tracking the action sequence, and can also avoid the problem of tracking errors caused by manual negligence, so that the present invention can improve the tracking efficiency and tracking accuracy of the body movement sequence of the substation workers.

[0128] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, centralized on a single computing device, or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. In some cases, the steps shown or described can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0129] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A method for tracking a target at a substation operation site, characterized in that: include: Acquire multiple frames of continuous images from multiple perspectives collected at the substation operation site, and identify at least one person image of an operator in the multiple frames of continuous images corresponding to each perspective; Extracting key points of the operator's limbs from the personnel image, determining the operator's limb movements based on the key points, determining a motion feature vector corresponding to the limb movement, and determining the limb movement of the target operator based on the motion feature vector; Fusing motion feature vectors of the target worker's limb movements in images from different perspectives corresponding to the same acquisition time point to obtain fused motion features, and classifying the target worker's limb movements based on the fused motion features to obtain limb movements in different action categories; The body movements under each action category are sorted in chronological order to obtain the tracking action sequence of the target operator under each action category.

2. The method according to claim 1, characterized in that The method further comprises: Track the target operator in multiple frames of continuous images from multiple perspectives to obtain the target operator's motion trajectory at each perspective; Determining the spatial position relationship and viewing angle ranges of the image acquisition devices corresponding to the multiple viewing angles, and constructing a spatial topological constraint relationship of the image acquisition devices based on the spatial position relationship and the viewing angle ranges; Determining a conversion matrix between different image acquisition devices based on the spatial topological constraint relationship, and judging whether the motion state of the target worker under the different image acquisition devices meets preset conditions of a normal state based on the conversion matrix; If the motion state meets the preset condition, the motion trajectory of the target operator under each viewing angle is merged to obtain a comprehensive motion trajectory of the target operator.

3. The method according to claim 2, characterized in that The determining, based on the conversion matrix, whether the motion state of the target worker under different image acquisition devices meets preset conditions of a normal state includes: The pixel coordinates of the target worker in the multiple frames of continuous images are converted into a three-dimensional coordinate system under the corresponding image acquisition device, and the coordinates of the target worker in the three-dimensional coordinates are converted into a world coordinate system based on the conversion matrix to obtain the world coordinates of the target worker; Determining the target worker's motion direction vector to be verified under different image acquisition devices based on the target worker's world coordinates in multiple frames of continuous images corresponding to different image acquisition devices; Determining an expected motion direction vector of the target worker transferring between different image acquisition devices, and respectively determining an angle between each motion direction vector to be verified and the expected motion direction vector; If the angle of each of the motion direction vectors to be verified is less than the preset angle threshold, it is determined that the motion state of the target operator under different image acquisition devices meets the preset conditions; otherwise, it is determined that the motion state of the target operator under different image acquisition devices does not meet the preset conditions.

4. The method according to claim 1, wherein The determining of the operator's body movements based on the body key points includes: Determining position information of each of the limb key points, and based on the position information, determining a positional relationship between each of the limb key points, wherein the positional relationship includes a distance relationship, an angle relationship, and a direction relationship between each of the limb key points; Determine the relationship feature vector corresponding to the positional relationship, and input the relationship feature vector into a preset limb motion prediction model for motion prediction to obtain the limb motion of the operator, wherein the preset limb motion prediction model is pre-trained based on a set of positional relationships of sample limb key points with motion labels.

5. The method according to claim 1, wherein The determining of the body movements of the target operator based on the movement feature vector includes: Based on the motion feature vector, the body movements of each operator are matched with the standard body movements in a preset body movement library, and the body movements of the target operator are determined based on the matching results, wherein the preset body movement library stores standard body movements with operator identity identification.

6. The method according to claim 1, characterized in that Based on the fused motion features, the limb movements of the target operator are classified to obtain limb movements in different motion categories, including: Determining standard action feature vectors corresponding to standard limb movements under different action categories, and calculating the similarity between the operator's limb movement and each of the standard limb movements based on the fused action features and the standard action feature vectors; Based on the similarity, the body movements of the target operator are classified to obtain body movements in different action categories.

7. The method according to claim 1, characterized in that If the different perspectives include a first perspective, a second perspective, and a third perspective, the motion feature vector of the target worker in the image at the first perspective is used as the first motion feature vector, the motion feature vector of the target worker in the image at the second perspective is used as the second motion feature vector, and the motion feature vector of the target worker in the image at the third perspective is used as the third motion feature vector; The step of fusing the motion feature vectors of the target worker's body movements in the images of different perspectives corresponding to the same acquisition time point to obtain the fused motion features includes: performing feature-level fusion processing on the first action feature vector, the second action feature vector, and the third action feature vector to obtain a feature fusion vector; performing element-level fusion processing on the first action feature vector, the second action feature vector, and the third action feature vector to obtain an element fusion vector; performing low-order fusion processing on the first motion feature vector, the second motion feature vector, and the third motion feature vector to obtain a low-order fusion vector; The feature fusion vector, the element fusion vector, and the low-order fusion vector are transformed using a preset transformation function to obtain the fusion action feature.

8. A tracking device for a target at a substation operation site, characterized in that: include: An acquisition unit is configured to acquire multiple frames of continuous images from multiple perspectives collected at the substation operation site, and to identify at least one image of an operator in the multiple frames of continuous images corresponding to each perspective; a determination unit, configured to extract key points of the operator's limbs from the operator image, determine the operator's limb movements based on the key points of the limbs, determine a motion feature vector corresponding to the limb movement, and determine the limb movement of the target operator based on the motion feature vector; a classification unit, configured to fuse motion feature vectors of the target worker's limb movements in images from different perspectives corresponding to the same acquisition time point to obtain fused motion features, and classify the target worker's limb movements based on the fused motion features to obtain limb movements in different action categories; The sorting unit is used to sort the body movements under each action category in chronological order to obtain the tracking action sequence of the target operator under each action category.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images