Multi-target tracking method and system for power equipment inspection
By predicting the detection bounding boxes and cross-union ratio of power equipment targets and combining them with the Transformer network, the robustness problem of multi-target tracking under occlusion is solved, and effective tracking and feature extraction of power equipment targets are achieved.
Patent Information
- Application Number
- CN202411138428.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-11-29
AI Technical Summary
Existing multi-target tracking methods struggle to effectively track power equipment targets under occlusion conditions, and feature extraction methods based on convolutional neural networks exhibit poor robustness to occlusion and misalignment issues.
By predicting the detection boxes of each target in the next frame of the inspection image, calculating the cross-union ratio between pairs of detection boxes, finding targets that may be occluded, and saving the complete image information of the target before occlusion occurs, a Transformer-based power equipment target tracking feature extraction network is adopted, which integrates a jigsaw puzzle optimization module and an auxiliary information embedding module to improve robustness.
It enables effective tracking of multiple power equipment targets under occlusion conditions, improves the robustness of the feature extraction network to occlusion and misalignment problems, and ensures the integrity and accuracy of the target trajectory.
Smart Images

Figure CN119048555A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of target tracking, and in particular relates to a multi-target tracking method and system for power equipment inspection. Background Art
[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.
[0003] For target inspection in large outdoor substation scenarios, the target types of power equipment are similar and prone to occlusion and other problems. However, existing multi-target tracking methods are difficult to track multiple targets under occlusion.
[0004] In addition, in feature extraction networks, although convolutional neural network-based methods have achieved great success, they only process one local neighborhood at a time, and the convolution and downsampling operators will cause the loss of detail information, and they are less robust to occlusion, misalignment and other problems. Summary of the invention
[0005] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a multi-target tracking method and system for power equipment inspection. In view of the problem that power equipment targets are prone to occlusion, it is innovatively proposed to predict the detection frame of each target in the next frame of inspection image and calculate the intersection and union ratio between each detection frame to find the target that may be occluded, and save the relatively complete image information of the target before occlusion occurs, so as to be used for trajectory tracking of each power equipment target, thereby realizing the tracking of multiple power equipment targets under occlusion.
[0006] In order to achieve the above object, the present invention adopts the following technical solution:
[0007] A first aspect of the present invention provides a multi-target tracking method for power equipment inspection, comprising:
[0008] Obtain the current inspection image, detect the detection frame of each target in the current inspection image, calculate the motion state of each target, and predict the position of each target in the next inspection image;
[0009] Based on the detection frame of each target, the features of each target are obtained through the feature extraction network;
[0010] Based on the detection frame, motion state and features of each target, the similarity between each target and each historical trajectory in the current frame inspection image is calculated, and the target is added to the end of the historical trajectory with the highest similarity. If the similarity between a target and all historical trajectories is lower than the threshold, the target is used as a new trajectory, and the historical trajectory of the unmerged target is used as an unmatched trajectory.
[0011] Based on the position of each target in the next inspection image, the intersection-and-union ratio between two targets is calculated, and the features of each target corresponding to the intersection-and-union ratio exceeding the threshold are added to the occlusion list;
[0012] Based on the target features, the similarity between the newly added trajectory and the terminal target in an unmatched trajectory is calculated. If the target of the unmatched trajectory has been in the occlusion list, the similarity between the newly added trajectory and the target in the occlusion list is calculated. If the similarity between the newly added trajectory and the terminal target or the target in the occlusion list exceeds the threshold, the newly added trajectory is merged with the unmatched trajectory.
[0013] Furthermore, the similarity between the target and each historical trajectory is: a weighted average of the intersection-over-union ratio, center point distance, feature similarity and motion state similarity of the detection frame of the target in the current frame inspection image and the target in the historical trajectory.
[0014] Furthermore, if a detection frame of a certain target already exists in the occlusion list, the features of the target will no longer be added to the occlusion list.
[0015] Furthermore, the method further includes: if the intersection-and-union ratio between any two targets is less than a threshold, then removing the features of each target corresponding to the intersection-and-union ratio less than the threshold from the occlusion list.
[0016] Furthermore, the step of the feature extraction network includes:
[0017] Dividing the area of the inspection image corresponding to the detection frame into a plurality of slices;
[0018] For each slice, the inspection view information is integrated to obtain the output features;
[0019] Combine the output features of all slices to obtain the target feature.
[0020] A second aspect of the present invention provides a multi-target tracking system for power equipment inspection, comprising:
[0021] The target detection module is configured to: obtain the current frame inspection image, detect the detection frame of each target in the current frame inspection image, calculate the motion state of each target, and predict the position of each target in the next frame inspection image;
[0022] A feature extraction module is configured to: obtain the features of each target through a feature extraction network based on the detection frame of each target;
[0023] The trajectory integration module is configured to: calculate the similarity between each target in the current frame inspection image and each historical trajectory based on the detection frame, motion state and features of each target, add the target to the end of the historical trajectory with the highest similarity, and if the similarity between a target and all historical trajectories is lower than a threshold, treat the target as a newly added trajectory, and treat the historical trajectory of the unmerged target as an unmatched trajectory;
[0024] The occlusion prediction processing module is configured to: calculate the intersection-and-union ratio between two targets based on the position of each target in the next inspection image frame, and add the features of each target corresponding to the intersection-and-union ratio exceeding the threshold into the occlusion list;
[0025] The re-matching module is configured to: calculate the similarity between the newly added trajectory and the terminal target in an unmatched trajectory based on the target features, and if the target of the unmatched trajectory has been in the occlusion list, calculate the similarity between the newly added trajectory and the target in the occlusion list; if the similarity between the newly added trajectory and the terminal target or the target in the occlusion list exceeds a threshold, merge the newly added trajectory with the unmatched trajectory.
[0026] Furthermore, the occlusion prediction processing module is further configured to: if the intersection-and-union ratio between any two targets is less than a threshold, remove the features of each target corresponding to the intersection-and-union ratio less than the threshold from the occlusion list.
[0027] Furthermore, the occlusion prediction processing module is further configured to: if a detection frame of a certain target already exists in the occlusion list, then the features of the target are no longer added to the occlusion list.
[0028] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the multi-target tracking method for power equipment inspection as described above.
[0029] The fourth aspect of the present invention provides a computer device, including a computer-readable storage medium, a processor, and a computer program stored on the computer-readable storage medium and executable on the processor, wherein when the processor executes the program, the steps in a multi-target tracking method for power equipment inspection as described above are implemented.
[0030] Compared with the prior art, the present invention has the following beneficial effects:
[0031] In order to solve the problem that power equipment targets are prone to occlusion, the present invention innovatively proposes to predict the detection frame of each target in the next inspection image frame and calculate the intersection and union ratio between each detection frame to find the target that may be occluded, and save the relatively complete image information of the target before the occlusion occurs, so as to track the trajectory of each power equipment target and realize the tracking of multiple power equipment targets in the case of occlusion.
[0032] In order to improve the robustness of the feature extraction network to problems such as occlusion and misalignment, the present invention innovatively proposes a Transformer-based power equipment target tracking feature extraction network. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0034] Figure 1 This is a flow chart of a multi-target tracking method for power equipment inspection according to a first embodiment of the present invention;
[0035] Figure 2 is a structural diagram of a feature extraction network according to a first embodiment of the present invention;
[0036] Figure 3 is a schematic diagram of the historical trajectory of the first embodiment of the present invention;
[0037] Figure 4 It is a framework diagram of a target autonomous servo method for electric power equipment according to the second embodiment of the present invention;
[0038] Figure 5 is a flow chart of a method for autonomous servo of a power equipment target according to a second embodiment of the present invention;
[0039] Figure 6 It is a structural diagram of a computer device according to a fourth embodiment of the present invention. DETAILED DESCRIPTION
[0040] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.
[0041] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.
[0042] Embodiment 1
[0043] This embodiment provides a multi-target tracking method for power equipment inspection.
[0044] Misalignment refers to a mismatch or deviation between the features or target objects in the input data and the data layout expected by the model or used during training. Misalignment is caused by many factors, including but not limited to: (1) Changes in the position of the target object: In image recognition or object detection tasks, the position of the target object may be anywhere in the image, rather than fixed to a specific position or area; if the model does not properly adapt to such changes, it may not be able to correctly recognize or locate misaligned targets; (2) Changes in object shape and size: Even if the position of the object is correct, it may cause misalignment if its shape, size, or orientation is inconsistent with the data used when the model was trained.
[0045] This embodiment provides a multi-target tracking method for power equipment inspection, such as Figure 1 As shown, the following steps are included:
[0046] Step 1: Get the current frame inspection image Ii.
[0047] Step 2: Based on the current frame inspection image Ii, combined with several historical frame inspection images, the target prediction position (target prediction frame) is obtained through the target position prediction module.
[0048] The target position prediction module is mainly responsible for predicting the position of the target in the next inspection image frame (ie, the target predicted position).
[0049] Based on the assumption that the number of video sequence frames is high and the position change of the target between several adjacent frames is small, the position of the target in the next frame is predicted in a uniform linear motion according to the position change of the target in the previous two frames and combined with the differential idea, forming a detection frame at the predicted target position.
[0050] Specifically, the power equipment targets in several historical inspection frames and the current inspection frame are identified, the position change data between the power equipment targets in two consecutive inspection frames are calculated, and the position of the power equipment targets in the next inspection frame is predicted in a uniform linear motion manner (i.e., the target predicted position).
[0051] Step 3: For the current frame inspection image, detect the detection frame of each target in the current frame inspection image (i.e., the position of each target in the current frame inspection image), and based on the detection frame of each power equipment target in the current frame inspection image Ii, obtain the tracking features of each power equipment target in the current frame inspection image through the power equipment target tracking feature extraction network (referred to as feature extraction network).
[0052] In feature extraction networks, although convolutional neural network (CNN) based methods have achieved great success, they only process one local neighborhood at a time and cause detail information loss due to convolution and downsampling operators.
[0053] In order to overcome these limitations and improve the robustness of the feature extraction network to problems such as occlusion and misalignment, a Transformer-based power equipment target tracking feature extraction network is proposed, such as Figure 2 shown.
[0054] Specifically, the power equipment target (target area) in the current frame inspection image is encoded into a series of slices, and a Transformer-based power equipment target tracking feature extraction network is constructed. To further enhance robust feature learning in context, a jigsaw slice optimization module and an auxiliary information embedding module are designed.
[0055] The steps of extracting the tracking features of the power equipment target in the current frame inspection image by using the Transformer-based power equipment target tracking feature extraction network include:
[0056] (1) First, divide the target area (detection box of the power equipment target) into different patches of the same size. Assuming the step size is S and the side length of each patch is P, the number of patches can be calculated as:
[0057]
[0058] Among them, H represents the height of the target area image, W represents the width of the target area image, S represents the step size, and P is the side length of the Patch.
[0059] (2) Then, the power equipment image Embedding layer encodes the features of each patch and configures a learnable embedding to encode the overall features of the patch.
[0060] (3) Then, after feature encoding for each patch, the jigsaw slice optimization module rearranges the slice embeddings through shifting and slice shuffling operations to generate robust features with improved recognition ability and more diverse coverage. Specifically: Assuming there is an input sequence [x1, x2, x3, x4, x5], the following shows how to use shifting (the positions in the positional encoding are usually fixed, representing the positions of each word in the input sequence; for example, for a sequence of length 5, the positional encoding may be [0, 1, 2, 3, 4]), slicing (the positional encoding vector is usually sliced into parts that match the length of the input sequence; for example, if the maximum input length of the model is 10, but the actual input sequence length is 5, then only the first 5 values in the positional encoding are used) and shuffling (the positional encoding itself is not shuffled, but the order of the data may be randomly disrupted in the training data to enhance the generalization ability of the model) to process this sequence:
[0061] Original input embedding: [[e1],[e2],[e3],[e4],[e5]];
[0062] Position encoding vector: [[p1],[p2],[p3],[p4],[p5]];
[0063] Add positional encoding shuffle: [[e4+p4],[e1+p1],[e2+p2],[e3+p3],[e5+p5]].
[0064] (4) Then, the Transformer layer embeds the position information into the features of each patch, and the Transformer layer outputs feature vectors that represent the distinguishable features of each patch. The Transformer layer is extremely good at integrating this type of boundary information.
[0065] Position information embedding: that is, configuring learnable embedding, which comes from the image, slicing the image, and each slice has a position representation information.
[0066] (5) The auxiliary information embedding module is used to add the inspection viewing angle information of the power robot (provided by the sensor of the gimbal camera, where the angle of the gimbal camera is represented by RPY) to the distinguishable features of each patch, obtain the output features of each slice, and allow the network to learn more non-visual information.
[0067] Among them, the inspection viewing angle information of the electric robot (also called auxiliary information or supplementary information embedding) is provided by the sensor of the gimbal camera, which is the posture RPY of the gimbal camera taking the image at this time.
[0068] Similar to position embedding, the auxiliary information embedding module uses learnable position boundary information.
[0069] (6) The output features of all slices are combined and batch normalized to obtain the target features (i.e., visual feature information). Figure 2 In the figure, f represents the output feature, which is the abbreviation of feature. BN stands for batch normalization, which means that f is processed by BN batch normalization.
[0070] The training loss of the feature extraction network separates the classification loss and the metric loss, and adopts the identity loss and the triplet loss.
[0071] ID LOSS (identity loss) is mainly used to distinguish different targets. It is often used in target re-identification tasks to ensure that the model can accurately identify and distinguish different targets. A common form of this loss function is cross-entropy loss, which is used for classification tasks. Specifically, the goal of ID Loss is to assign each target to the correct category (that is, the correct identity). For example, in multi-target tracking, each target has a unique identity ID. ID Loss measures the accuracy of the model in identifying the target identity by calculating the cross entropy between the ID distribution predicted by the model and the actual ID distribution. Formulaically, it can be expressed as:
[0072]
[0073] Among them, yi is the true ID label of the target, and pi is the ID probability distribution predicted by the model.
[0074] Triplet Loss: Triplet Loss is often used in metric learning, and its purpose is to bring similar samples closer together and dissimilar samples farther apart. Triplet Loss relies on sample triplets (anchor, positive, negative), where anchor and positive are different samples belonging to the same target, and negative is a sample of a different target. The goal of Triplet Loss is to make the distance between anchor and positive as small as possible, and the distance between anchor and negative as large as possible. Its loss function can be expressed as:
[0075] Triplet Loss=max(0,d(a,p)-d(a,n)+α)
[0076] Among them: d(a,p) represents the distance between the anchor and the positive; d(a,n) represents the distance between the anchor and the negative; α is a hyperparameter called margin, which ensures that the distance between negative samples is a certain amount larger than that between positive samples.
[0077] By minimizing this loss, the model learns a feature space in which features of samples of the same object are close together, while features of samples of different objects are far apart.
[0078] In the framework of target tracking using Transformer, ID Loss and Triplet Loss are used in combination to improve the tracking performance of the model: ID Loss can ensure that the model can accurately distinguish and identify the identities of different targets and enhance the accuracy of tracking; Triplet Loss can optimize the feature space, making the features of the same target more consistent between different frames and reducing drift during tracking.
[0079] Step 4: Trajectory integration.
[0080] Before the target is matched again, the trajectory integration module screens each target in the current frame inspection image (i.e., the position information of the detection frame of each target in the current frame inspection image, the motion state of each target and the visual feature information of each target), and merges the targets that can match the existing trajectory to achieve the effect of initially counteracting occlusion and reducing the total number of trajectories.
[0081] Among them, Figure 3 As shown in , the existing trajectory refers to the trajectory formed by the targets that have been tracked in the previous frames (frame 0 to frame i-1). That is, a trajectory contains the position information, motion information (such as speed and direction) and visual feature information (abbreviated as feature) of a target in several consecutive frames.
[0082] Specifically, location information refers to the information of the detection frame of each target in the inspection image, including the position and size of the detection frame; motion state refers to the speed and acceleration of each target in the inspection image; visual features: the appearance features of the target, usually extracted by the feature extraction network and extracted as visual features, namely ReID features
[0083] The motion state is obtained based on the position of the target detection frame from the i-3th to the i-th frame and the time interval between two consecutive frames.
[0084] The main steps of merging trajectories include the following aspects:
[0085] a. Trajectory association.
[0086] Trajectory association: By calculating the similarity between each target in the current inspection frame image (the location information of the target's detection frame, the target's motion state and visual feature information) and the existing trajectory (that is, the weighted average of the IoU, center point distance, feature similarity, and motion state similarity between the target in the current inspection frame image and the target in the historical trajectory), the distance matrix cost is obtained to determine which targets in the current inspection frame image may belong to the existing trajectory.
[0087] like Figure 3As shown in the figure, there are two existing tracks track1 and track2. Each triangle in track1 represents the position of the detection box of the triangle target in the current frame image. The positions of the detection boxes of the triangle target in all frames are connected together, which is called track1; Track2 is the same.
[0088] Among them, appearance similarity (ie, feature similarity) is the similarity calculated based on the visual features of the target, such as cosine similarity.
[0089] Among them, IoU and center point distance are calculated based on the position information of the target detection box.
[0090] The motion state similarity is calculated based on the motion state of the target.
[0091] Among them, each element in the distance matrix is the similarity between a target and an existing trajectory.
[0092] b. Merger strategy.
[0093] Once similar trajectories are found, the best match is selected, that is, for each target, the existing trajectory with the highest similarity is selected for merging, that is, the identity (ID) of the existing trajectory with the highest similarity is inherited and the target is added to the end of the existing trajectory.
[0094] Update trajectory information: get the merged trajectory information That is, the information of the target in the current frame image (the location information of the detection frame, the motion state of the target and the visual feature information) is updated to the information of the existing trajectory.
[0095] c. New track screening: If the similarity between the target in the current frame image and any existing track is lower than a certain threshold, it will be treated as a new independent track. For example, when a circle detection frame is suddenly detected at a certain moment, the position of the circle detection frame detected in each subsequent frame is connected to form the circle's track 3, which is a new track.
[0096] In summary, the trajectory integration module traverses the targets in the current frame image, and tries to match them with the existing trajectories that have not been updated and are not considered to have ended, through their intersection-over-union ratio with the detection boxes of the existing trajectories and the similarity of image features, and then performs trajectory integration.
[0097] Step 5: Occlusion prediction processing.
[0098] The occlusion prediction module is mainly responsible for finding targets that may be occluded and saving relatively complete image information of the target before occlusion occurs.
[0099] The occlusion prediction module uses the target prediction position output by the target position prediction module to calculate the intersection and union ratio of each target detection box (i.e., each target prediction position) in the next frame. If the intersection and union ratio of a group of targets exceeds the threshold, their identity IDs and image features in the current frame are stored in the occlusion list until the intersection and union ratio of the target's predicted detection box (target prediction position) and other target prediction positions is less than the threshold. At this time, the algorithm considers that the target occlusion process is over or the possibility of occlusion disappears, and removes the target from the occlusion list.
[0100] In addition, by comparing the image features of each ID stored in the target list with the image features of the target in the current frame, if the image features are similar, the target is considered to be in the occlusion list. If the target is already in the occlusion list, the information of the target is not updated. Because the target may have entered the occlusion process at this time, the image information corresponding to the target is not necessarily more effective than the previously saved features, so this algorithm chooses not to update the features of the target in the occlusion list.
[0101] Step 6: Match again.
[0102] The rematching module is responsible for verifying the output of the model using image features and finding a more suitable detection result for the target if the target fails the verification.
[0103] like Figure 3 As shown in the figure, the original tracks are track1 and track2. After step 4, a new track track3 is added. At this time, the matching module will check whether the newly added track3 is reasonable. It will take the circular feature of track3 and the triangle in track1 and the square in track2 for similarity verification. If the similarity is the same as the triangle, it means that track3 is an extension of track1. Track3 is still track1, not a new track.
[0104] The rematching module verifies the newly added track by comparing it with the last detection frame in the previous frame of a historical track and the features retained by the historical track in the occlusion list. If the verification is passed, the newly added track inherits the identity of the historical track. Specifically:
[0105] Assuming that there is a historical track A that has no newly added target in step 4 (i.e., historical track A is an unmatched track), the rematching module is needed to confirm whether the newly added track (i.e., the unmatched detection) corresponds to the historical track A;
[0106] Similarity check: compare the unmatched detected image features (features of the newly added trajectory) with the last detected features of the historical trajectory A in the previous frame; at the same time, if the historical trajectory A was in the occlusion list, compare the unmatched detected image features with the features of the historical trajectory A stored in the occlusion list;
[0107] Similarity score: Calculate the similarity (cosine similarity). If the similarity exceeds the preset threshold, the unmatched detection is considered to have passed the verification. If it passes the verification, the unmatched detection inherits the identity ID of historical trajectory A and updates the status of historical trajectory A.
[0108] This embodiment provides a multi-target tracking method for power equipment inspection, studies the power equipment target tracking feature extraction network, integrates the Transformer network characteristics for power scenarios, and constructs a robust feature extraction network for power scenarios; studies the multi-target tracking technology in complex scenarios, designs a multi-target tracking strategy based on the power equipment target tracking feature extraction network, and reduces the mismatching and mistracking of targets in complex scenarios.
[0109] This embodiment provides a multi-target tracking method for power equipment inspection. In view of the similarity of power equipment target types and the proneness to occlusion, the method realizes multi-target tracking by matching targets based on a detection model and relying on the IoU of detection frames between adjacent frames and a power equipment target tracking feature extraction network.
[0110] Embodiment 2
[0111] This embodiment provides a target autonomous servo method for electric power equipment.
[0112] The present embodiment provides an autonomous servo method for a power equipment target, which breaks the robot's "stop point-preset position" inspection mode and enables the inspection robot to "patrol while walking", thereby solving the problems of low inspection efficiency and poor timeliness.
[0113] This embodiment provides an autonomous servo method for a power equipment target, which controls the acquisition device to smoothly reach a suitable position based on the image feature information feedback to complete the main information acquisition.
[0114] This embodiment provides an autonomous servo method for a power equipment target, such as Figure 4 and Figure 5 As shown, the following steps are included:
[0115] First, the robot uses the semantic information of the spatial position of the equipment in the three-dimensional semantic map to automatically navigate and plan to the inspection target observation area, and automatically adjusts the robot to aim at the target to be inspected, thereby achieving coarse positioning of the inspection target in the large scene of the outdoor substation.
[0116] Then, through a multi-target tracking method for power equipment inspection provided in Example 1, automatic identification, positioning and precise tracking of the target equipment to be detected in the video are achieved.
[0117] Finally, the semantic information of the spatial position of the equipment in the actively acquired video is used to construct a robot visual semantic servo system, and the robot posture is adjusted dynamically in a closed loop to achieve accurate acquisition of inspection target images in local complex scenes.
[0118] In order to eliminate the impact of robot motion on inspection images, the image motion jitter compensation algorithm of the power equipment inspection robot is used to improve the image acquisition quality of the robot inspection.
[0119] Embodiment 3
[0120] This embodiment provides a multi-target tracking system for power equipment inspection, which specifically includes:
[0121] The target detection module (target position prediction module) is configured to: obtain the current frame inspection image, detect the detection frame of each target in the current frame inspection image, calculate the motion state of each target, and predict the position of each target in the next frame inspection image;
[0122] A feature extraction module is configured to: obtain the features of each target through a feature extraction network based on the detection frame of each target;
[0123] The trajectory integration module is configured to: calculate the similarity between each target in the current frame inspection image and each historical trajectory based on the detection frame, motion state and features of each target, add the target to the end of the historical trajectory with the highest similarity, and if the similarity between a target and all historical trajectories is lower than a threshold, treat the target as a newly added trajectory, and treat the historical trajectory of the unmerged target as an unmatched trajectory;
[0124] The occlusion prediction processing module is configured to: calculate the intersection-and-union ratio between two targets based on the position of each target in the next inspection image frame, and add the features of each target corresponding to the intersection-and-union ratio exceeding the threshold into the occlusion list;
[0125] The re-matching module is configured to: calculate the similarity between the newly added trajectory and the terminal target in an unmatched trajectory based on the target features, and if the target of the unmatched trajectory has been in the occlusion list, calculate the similarity between the newly added trajectory and the target in the occlusion list; if the similarity between the newly added trajectory and the terminal target or the target in the occlusion list exceeds a threshold, merge the newly added trajectory with the unmatched trajectory.
[0126] It should be noted here that each module in this embodiment corresponds to each step in Example 1 one by one, and the specific implementation process is the same, which will not be repeated here.
[0127] Embodiment 3
[0128] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the multi-target tracking method for power equipment inspection as described in the first embodiment above are implemented.
[0129] Embodiment 4
[0130] This embodiment provides a computer device, such as Figure 6 As shown, it includes a computer-readable storage medium 1003, a processor 1001, a communication interface 1002, and a computer program stored on the computer-readable storage medium 1003 and executable on the processor 1001, wherein the processor 1001, the communication interface 1002, and the computer-readable storage medium 1003 can be connected via a bus or other means. The communication interface 1002 is used to receive and send data, and when the processor 1001 executes the program, the steps in the multi-target tracking method for power equipment inspection described in the first embodiment are implemented.
[0131] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0132] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0133] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0134] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A multi-target tracking method for power equipment inspection, characterized in that: include: Obtain the current inspection image, detect the detection frame of each target in the current inspection image, calculate the motion state of each target, and predict the position of each target in the next inspection image; Based on the detection frame of each target, the features of each target are obtained through the feature extraction network; Based on the detection frame, motion state and features of each target, the similarity between each target and each historical trajectory in the current frame inspection image is calculated, and the target is added to the end of the historical trajectory with the highest similarity. If the similarity between a target and all historical trajectories is lower than the threshold, the target is used as a new trajectory, and the historical trajectory of the unmerged target is used as an unmatched trajectory. Based on the position of each target in the next inspection image, the intersection-and-union ratio between two targets is calculated, and the features of each target corresponding to the intersection-and-union ratio exceeding the threshold are added to the occlusion list; Based on the target features, the similarity between the newly added trajectory and the terminal target in an unmatched trajectory is calculated. If the target of the unmatched trajectory has been in the occlusion list, the similarity between the newly added trajectory and the target in the occlusion list is calculated. If the similarity between the newly added trajectory and the terminal target or the target in the occlusion list exceeds the threshold, the newly added trajectory is merged with the unmatched trajectory.
2. A multi-target tracking method for power equipment inspection according to claim 1, characterized in that: The similarity between the target and each historical trajectory is: the weighted average of the intersection-over-union ratio, center point distance, feature similarity and motion state similarity of the detection frame of the target in the current frame inspection image and the target in the historical trajectory.
3. A multi-target tracking method for power equipment inspection according to claim 1, characterized in that: If a detection box of a certain target already exists in the occlusion list, the features of the target will no longer be added to the occlusion list.
4. A multi-target tracking method for power equipment inspection according to claim 1, characterized in that: Also includes: If the intersection-and-union ratio between any two targets is less than a threshold, the features of each target corresponding to the intersection-and-union ratio less than the threshold are removed from the occlusion list.
5. A multi-target tracking method for power equipment inspection according to claim 1, characterized in that: The steps of the feature extraction network include: Dividing the area of the inspection image corresponding to the detection frame into a plurality of slices; For each slice, the inspection view information is integrated to obtain the output features; Combine the output features of all slices to obtain the target feature.
6. A multi-target tracking system for power equipment inspection, characterized in that: include: The target detection module is configured to: obtain the current frame inspection image, detect the detection frame of each target in the current frame inspection image, calculate the motion state of each target, and predict the position of each target in the next frame inspection image; A feature extraction module is configured to: obtain the features of each target through a feature extraction network based on the detection frame of each target; The trajectory integration module is configured to: calculate the similarity between each target in the current frame inspection image and each historical trajectory based on the detection frame, motion state and features of each target, add the target to the end of the historical trajectory with the highest similarity, and if the similarity between a target and all historical trajectories is lower than a threshold, treat the target as a newly added trajectory, and treat the historical trajectory of the unmerged target as an unmatched trajectory; The occlusion prediction processing module is configured to: calculate the intersection-and-union ratio between two targets based on the position of each target in the next inspection image frame, and add the features of each target corresponding to the intersection-and-union ratio exceeding the threshold into the occlusion list; The re-matching module is configured to: calculate the similarity between the newly added trajectory and the terminal target in an unmatched trajectory based on the target features, and if the target of the unmatched trajectory has been in the occlusion list, calculate the similarity between the newly added trajectory and the target in the occlusion list; if the similarity between the newly added trajectory and the terminal target or the target in the occlusion list exceeds a threshold, merge the newly added trajectory with the unmatched trajectory.
7. A multi-target tracking system for power equipment inspection as claimed in claim 6, characterized in that: The occlusion prediction processing module is further configured to: if the intersection-and-union ratio between any two targets is less than a threshold, remove the features of each target corresponding to the intersection-and-union ratio less than the threshold from the occlusion list.
8. A multi-target tracking system for power equipment inspection as claimed in claim 6, characterized in that: The occlusion prediction processing module is further configured to: if a detection frame of a certain target already exists in the occlusion list, then no feature of the target is added to the occlusion list.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of a multi-target tracking method for power equipment inspection as described in any one of claims 1 to 5 are implemented.
10. A computer device comprising a computer-readable storage medium, a processor, and a computer program stored in the computer-readable storage medium and executable on the processor, characterized in that: When the processor executes the program, the steps of a multi-target tracking method for power equipment inspection as described in any one of claims 1-5 are implemented.
Citation Information
Patent Citations
Video target data association method and device and storage medium
CN109829405A
Online multi-target tracking method based on occlusion pairs
CN110853078A
Target tracking method and device, electronic equipment and computer readable storage medium
CN114463368A
Personnel fine-grained tracking method and system based on face recognition in surveillance video
CN116311063A
Multi-target tracking method and system
CN117911454A
Cited By
Commodity identification method and device based on image real-time processing
CN120496014A
