Contour identification tracking system, device and method
By introducing the combination of point cloud clustering and YOLOv8-Seg image processing algorithm in the substation tracking system, the problem of low system flexibility is solved, and flexible tracking and identification of unknown types of lifted objects is realized, thus improving the flexibility of the system and scalability.
Patent Information
- Application Number
- CN202411209800.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-26
AI Technical Summary
In the prior art, the substation tracking system has low flexibility and is difficult to deal with unknown types of lifting objects. Due to the characteristics differences between different target objects, the model needs to be re-established and trained, which reduces the flexibility and scalability of the system. sex.
A contour recognition and tracking system is proposed, including point cloud clustering module, image segmentation module and feature fusion module. Through motion prediction, downsampling, pose prediction and clustering, combined with YOLOv8-Seg image processing algorithm, the mobile target of the substation is achieved. contour recognition and tracking can handle feature differences between different target objects without re-establishing the model.
While ensuring the accuracy of point cloud tracking, it realizes category-free tracking, which improves the flexibility and scalability of the system. It is suitable for the variable operating environment of substations and can effectively track unknown types of lifted objects.
Smart Images

Figure CN119027454A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of substation tracking, and in particular relates to a contour recognition and tracking system, equipment and method. Background Art
[0002] The substation operating environment is complex and has many obstructions. It is necessary to integrate data from multiple sensors such as vision, laser, and radar to build an accurate three-dimensional environmental model. A single sensor is difficult to meet the positioning needs in the complex environment of the substation. It is necessary to comprehensively utilize heterogeneous sensors such as lidar, visual sensors, and IMU to achieve stable and reliable autonomous positioning through data fusion optimization. Specifically, filters such as Kalman filtering and particle filtering are used to fuse and estimate multi-source sensor data, eliminate measurement noise, and improve positioning accuracy.
[0003] At present, point cloud single target tracking algorithms mainly adopt the method of specific object modeling, that is, for different types of targets, such as pedestrians, cars, daily necessities, etc., corresponding mathematical models are constructed and trained separately. The implementation of these methods requires in-depth analysis and extraction of the characteristics of the target object to ensure the effectiveness and accuracy of the model.
[0004] However, this method of modeling specific objects has certain limitations. Due to the large differences in features between different target objects, it is necessary to build and train models for each target category, which increases the complexity and computational cost of the model. In addition, in substation field operations, if it is necessary to track unknown types of hoisted objects, the model needs to be rebuilt and trained, which greatly reduces the flexibility and scalability of the tracking system. Summary of the invention
[0005] The present invention provides a contour recognition and tracking system, device and method to solve the problem of low flexibility of the tracking system in the prior art.
[0006] To achieve the above object, the present invention proposes the following technical solutions:
[0007] A contour recognition and tracking system, which is used for contour tracking of a moving target in a substation, and includes a point cloud clustering module, an image segmentation module, and a feature fusion module;
[0008] The point cloud clustering module includes a motion prediction unit, a downsampling unit, a posture prediction unit and a clustering unit;
[0009] The motion prediction unit selects a substation mobile target, obtains historical 3D bounding box information of the substation mobile target, predicts the 3D bounding box of the substation mobile target, and obtains a motion prediction result;
[0010] The down-sampling unit down-samples the result used for motion prediction to obtain sampled data;
[0011] The posture prediction unit predicts the posture of the substation mobile target based on the sampled data to obtain prediction data;
[0012] The clustering unit is used to cluster the prediction data to obtain point cloud features;
[0013] The image segmentation module is used to perform contour recognition on the mobile target of the substation to obtain image features;
[0014] The feature fusion module is used to fuse and segment point cloud features and image features to complete contour recognition and tracking of mobile targets in substations.
[0015] Preferably, the motion prediction unit takes the timestamp of the current frame as input.
[0016] Preferably, the point cloud clustering module also includes an occlusion judgment unit, which, after obtaining the motion prediction result, judges whether the motion prediction result is occluded. If there is no occlusion or the occlusion is not obvious in the current field of view, the process continues; if there is obvious occlusion in the current field of view, the tracking is exited.
[0017] Preferably, the occlusion judgment unit combines the predicted 3D bounding box information of the current frame, the point cloud of the current frame, the 3D bounding box information of the previous frame, the point cloud of the previous frame, and the laser radar coordinate points to judge whether the current substation moving target is occluded.
[0018] Preferably, the downsampling unit process is:
[0019] Assume that the center coordinate of the motion predicted 3D bounding box is, calculate the distance between the center coordinate and all points of the current frame point cloud; obtain the sampling probability vector of the current frame point cloud according to the distance between the center coordinate and all points of the current frame point cloud; complete downsampling.
[0020] Preferably, the posture prediction unit performs posture refinement after posture prediction;
[0021] The pose prediction unit obtains the predicted 3D bounding box after the 3D bounding box transformation as prediction data, and further refines the prediction data.
[0022] Preferably, the point cloud clustering module further includes a confidence judgment unit, which judges whether the tracking result is credible based on a preset point cloud template and the point cloud prediction result in the clustering unit.
[0023] Preferably, the image segmentation module uses a YOLOv8-Seg image processing algorithm to perform contour recognition on the substation mobile target image to obtain image features.
[0024] A contour recognition and tracking method based on a contour recognition and tracking system comprises the following steps:
[0025] Step 1, obtaining historical 3D bounding box information of the substation moving target, and the motion prediction unit predicts the 3D bounding box of the substation moving target based on the historical 3D bounding box information to obtain a motion prediction result;
[0026] Step 2: Combined with the motion prediction result, the occlusion judgment unit determines the degree of occlusion of the current substation moving target. If the judgment is serious, the tracking is exited. Otherwise, the process goes to step 3.
[0027] Step 3, the downsampling unit downsamples the motion prediction result to obtain sampled data;
[0028] Step 4: The posture prediction unit predicts the posture of the mobile target of the substation based on the sampled data to obtain predicted data, and refines the predicted data. The clustering unit clusters the refined data to obtain point cloud features.
[0029] Step 5, image segmentation: perform contour recognition on the substation moving target image to obtain image features;
[0030] Step 6, feature fusion, fuses and segments the point cloud features obtained in step 4 with the image features obtained in step 5 to obtain the substation moving target.
[0031] An electronic device comprising a memory and a processor;
[0032] Memory for storing computer programs;
[0033] The processor is used to execute the computer program, and the computer program implements the steps of a contour recognition and tracking method when executed by the processor.
[0034] The present invention is beneficial in that:
[0035] The present invention proposes a contour recognition and tracking system, which uses a motion prediction unit to predict the motion state, and performs occlusion judgment and downsampling. After rough prediction and refinement of the posture, the system is clustered into point cloud features and segmented and fused with image features. The process proposed by the present invention can achieve category-free tracking while ensuring the accuracy of point cloud tracking, which greatly improves the flexibility and scalability of the system. In substation on-site operations, it is necessary to track unknown types of hoisted objects. The point cloud clustering module proposed by the present invention can handle the feature differences between different target objects. In the face of unknown types of hoisted objects in substation on-site operations, there is no need to re-establish the model and perform training, which is more suitable for the changeable operating environment of the substation, which greatly improves the flexibility and scalability of the tracking system. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:
[0037] Figure 1 A schematic diagram of a contour recognition and tracking system;
[0038] Figure 2 Schematic diagram of point cloud clustering module;
[0039] Figure 3 It is a schematic diagram of the basic principle of the occlusion judgment unit;
[0040] Figure 4 This is a schematic diagram of the point cloud before sampling;
[0041] Figure 5 This is a schematic diagram of the point cloud after sampling;
[0042] Figure 6 This is a flow chart of the pose prediction subunit;
[0043] Figure 7a and Figure 7b They are the coarse prediction results and the pose refinement results respectively;
[0044] Figure 8 Schematic diagram of clustering results;
[0045] Fig. 9 This is a schematic diagram of the convex hull calculation results;
[0046] Fig.10 It is a schematic diagram of the embodiment target;
[0047] Fig.11 Schematic diagram of feature fusion results;
[0048] Fig.12A schematic diagram of a contour recognition and tracking device. DETAILED DESCRIPTION
[0049] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present invention can be combined with each other without conflict.
[0050] The following detailed description is an exemplary description, which is intended to provide further detailed description of the present invention. Unless otherwise specified, all technical terms used in the present invention have the same meaning as those generally understood by those skilled in the art to which the present invention belongs. The terms used in the present invention are only for describing specific embodiments, and are not intended to limit the exemplary embodiments according to the present invention.
[0051] See also Figure 1 As shown, the present invention provides a contour recognition and tracking system, which can be used for lifting objects of unknown types in substation field operations, and can perform in-depth analysis and extraction of the features of the target object to ensure the effectiveness and accuracy of the model.
[0052] It includes point cloud clustering module, image segmentation module and feature fusion module.
[0053] like Figure 2 As shown, the point cloud clustering module is used to realize point cloud clustering, and can extract point cloud information of target objects such as charged bodies and crane arms, and obtain spatial position, distance and other information of the target objects.
[0054] The image segmentation module is used to realize the recognition of charged objects and boom contours in the image, accurately locate and classify the targets in the image at the pixel level, and obtain image information including information such as the texture of the target object.
[0055] The feature fusion module fuses the point cloud clustering information and the image information to obtain a better segmentation result.
[0056] The point cloud clustering module includes a motion prediction unit, a downsampling unit and a pose prediction unit. Specifically: the motion prediction unit is used to predict the position of the 3D bounding box of the current frame; the occlusion judgment unit is used to judge whether the target is occluded in the current point cloud frame based on the position information of the 3D bounding box of the current frame; the downsampling unit is used to downsample the point cloud to reduce the computational complexity of subsequent modules; the pose prediction unit is used to make a preliminary prediction of the position of the target and obtain an accurate 3D bounding box of the target.
[0057] The motion prediction unit selects a mobile target of the substation, which may be a crane arm, hanging objects, wires, structures and other objects in the substation. The motion prediction unit tracks the position and movement of the mobile target of the substation in real time to provide more accurate monitoring and control for the operation.
[0058] The motion prediction unit predicts the motion state of the template based on the Kalman filter. Since the bounding box information includes 3 coordinate values, 3 sizes and 3 angle values, a total of 9 dimensions, the initial state b_0 of the Kalman filter is a 1×9 matrix, and its initial state transfer matrix A 0 for:
[0059]
[0060] Measuring the initial matrix for:
[0061]
[0062] Initial measurement noise and the initial process noise are all unit matrices.
[0063] In order to represent the high uncertainty of the unobservable initial velocity and covariance matrix, the initial error covariance matrix is a diagonal matrix with all diagonal values being 1000.
[0064] The motion prediction unit converts the timestamp of the current frame As input, and the state transition matrix Updated to:
[0065]
[0066] Then predict the state matrix of the current frame (i.e. 3D bounding box information) and the error covariance matrix .
[0067] In a specific implementation, the motion prediction unit predicts the 3D bounding box of the target using historical 3D bounding box information of the target.
[0068] The point cloud clustering module further includes an occlusion judgment unit, which assumes that the predicted 3D bounding box information of the motion prediction unit Accurate enough, combined with the predicted 3D bounding box information of the current frame , point cloud of the current frame And the 3D bounding box information of the previous frame And the previous frame point cloud 、LiDAR coordinate points And other information to judge whether the current target is blocked.
[0069] First calculate The size ranges of the eight vertices on the XYZ axis are respectively denoted as , , , , , .
[0070] Determine the point cloud of the previous frame Every point Whether the following equation is satisfied, and the points that meet the conditions are retained to form the template point cloud of the current frame :
[0071]
[0072] Calculate template point cloud The centroid coordinates of:
[0073]
[0074] The vector formed by the centroid of the laser radar and the template point cloud for:
[0075]
[0076] Assume that the vector set formed by all points of the laser radar and the template point cloud is , find Zhongyu The maximum acute angle between :
[0077]
[0078]
[0079] Solve the point cloud of the previous frame The number of points that satisfy the following formula :
[0080]
[0081] in, Represents the distance between the closest points between the lidar and the template point cloud.
[0082] According to the point cloud of the current frame 3D bounding box information of the current frame and prediction The target point cloud for motion prediction is calculated as , and calculate its centroid , let the vector formed by the centroid and the laser radar be , calculate the point cloud of the current frame The number of points that satisfy the following formula :
[0083]
[0084] as well as represents the degree to which the target is obscured. and If the following formula is satisfied, it is considered that there is no occlusion or the occlusion is not obvious in the current field of view; otherwise, it is considered that the occlusion is serious and the tracking is exited:
[0085]
[0086] in, represents the maximum acceptable occlusion increase rate, Indicates the maximum acceptable minimum number of occlusion points.
[0087]
[0088] Characterizes the maximum tolerable rate of change when occlusion is severe; It represents the maximum number of occlusion points that can be tolerated when there is no occlusion in the previous frame and occlusion suddenly occurs in the current frame.
[0089] Figure 3 The basic principles of the occlusion judgment unit are demonstrated, including the coordinate points of the lidar, the template point cloud, that is, the point cloud of the tracked target in the previous frame, the point cloud of the tracked target in the current frame predicted by the motion prediction unit, and the points where there is occlusion problem for the target. It can be seen that the number of points reflects the occlusion situation of the target to a certain extent.
[0090] The down sampling unit process is as follows:
[0091] Let the center coordinates of the motion predicted 3D bounding box be ,calculate With the current frame point cloud The distance between all points :
[0092]
[0093] in Representation vector and The Euclidean distance between .
[0094] Let the distance vector The maximum value is , the minimum value is ,right Perform maximum and minimum value standardization to obtain the standardized vector :
[0095]
[0096] in, This ensures that the formula can operate normally when the maximum and minimum values are equal. By normalizing the maximum and minimum values, the vector Each element in The memory required for subsequent operations is reduced and the metrics are unified.
[0097] Let vector The elements and ,Right now:
[0098]
[0099] Then the current frame point cloud The sampling probability vector for:
[0100]
[0101] Figure 4 and Figure 5 The point clouds before and after sampling are shown respectively. The downsampling algorithm uses motion prediction for downsampling, which retains the point clouds of the target and its surrounding environment to a greater extent, providing richer information for feature extraction.
[0102] The posture prediction unit includes a posture prediction subunit, a confidence judgment subunit and a posture refinement subunit. The posture prediction unit predicts the posture of the target through the posture prediction subunit, and then the confidence judgment subunit evaluates the prediction. If it is judged to be invalid, the adaptive parameter selection module is started, the neighborhood radius is reselected, and the posture prediction and confidence judgment are performed again. When it is judged to be invalid again, it is recognized that the prediction has failed, otherwise a rough prediction result is output.
[0103] like Figure 6 As shown, the pose prediction subunit inputs the template point cloud and the downsampled current frame point cloud (search point cloud) into the twin point cloud feature descriptor to generate features of the two point clouds, and calculates the point matching relationship between the two point clouds based on the features. The matching point pair set is then input into the point cloud registration algorithm to find the point cloud in the search point cloud that is most similar to the template point cloud, and outputs the transformation matrix from the template point cloud to the current frame point cloud. The template point cloud and its bounding box are transformed according to the matrix to form the position information of the target in the current frame.
[0104] Let the number of interest points be , template point cloud Search point cloud Random sampling The points are recorded as and , the interest point feature matrix of the two point clouds and They are:
[0105]
[0106]
[0107] in, represents the feature descriptor, and for The matrix of represents the feature dimension generated by the feature descriptor, Represents the neighborhood radius.
[0108] When calculating the matching point pairs of two point clouds, first calculate and The similarity matrix of :
[0109]
[0110] Then for For each feature in , use the similarity matrix to find the one with the greatest similarity features, and form a matching pair matrix ,Right now:
[0111]
[0112] in, express Zhongyu The corresponding point subscript.
[0113] Point cloud registration will match the pair matrix As input, use point cloud registration algorithms such as RANSAC and PointDSC to filter out incorrect matching point pairs and calculate the coarse point cloud rotation matrix and the coarse translation matrix as output.
[0114] Point cloud transformation transforms point cloud transformation matrix and template point cloud and its 3D bounding box As input, the template point cloud is transformed using the point cloud transformation matrix to obtain the predicted target point cloud , the formula is as follows:
[0115]
[0116] Since the 3D bounding box It is represented by coordinates, size and posture, so we need to calculate the coordinates of its 8 vertices, recorded as ,right Transform to get:
[0117]
[0118] use You can get the 3D bounding box Transformed predicted 3D bounding box .
[0119] This paper proposes a new similarity metric. The metric first uses the predicted 3D bounding box Filter Search Point Cloud The predicted target point cloud , let the distance between point p and the nearest point of point cloud q be calculated as:
[0120]
[0121] Calculate the predicted target point cloud Every point Point cloud with template The distance between the closest points Vector, that is:
[0122]
[0123] calculate Each point in The vector formed by the distance between the closest points ,Right now:
[0124]
[0125] statistics , Greater than the hyperparameter The number of values of and are recorded as and ,like , If the following formula is not satisfied, the tracking is considered unreliable:
[0126]
[0127] in The range is The hyperparameter threshold of It represents the total number of similar points in the two point clouds, and the denominator represents the total number of the two point clouds. Therefore, this formula describes the similarity ratio between the two point clouds and solves the problem that there is no upper limit on the chamfer distance.
[0128] In a specific implementation, the confidence judgment subunit also considers the impact of the target point cloud changing too fast on subsequent tracking, that is, and The following formula should be satisfied, otherwise the tracking is considered invalid:
[0129]
[0130] in Indicates the maximum acceptable point change rate, ranging from [0, 1]. When the number of points in the predicted target point cloud decreases by more than the limit, it can be considered that the prediction is wrong or the target is occluded, resulting in serious information loss, and should be discarded.
[0131] LiDAR measures distance and other characteristics by emitting laser beams and receiving the reflected signals. Therefore, the point density of the generated point cloud is different for targets in different distance ranges. Therefore, the point density of the target may change dynamically during its movement.
[0132] The pose prediction unit also includes a pose refinement subunit. The pose refinement subunit assumes that the deformation of the tracked target in the current frame is small and the deviation of the rough prediction result is small. The template point cloud and the search point cloud are precisely aligned using the ICP fine registration algorithm to solve the refined rotation matrix. and the refined translation matrix , which can further reduce the registration error.
[0133] The final rotation transformation matrix and translation matrix are:
[0134]
[0135]
[0136] Will and 3D bounding box applied to the template point cloud The target pose can be refined to obtain the final target 3D bounding box. , and and the time of the current frame The Kalman filter of the motion prediction unit is updated as input.
[0137] Considering that the target is in motion, its scanning surface may change, resulting in changes in the scanned point cloud. Therefore, after the refinement is completed, and search point cloud Segment the target point cloud as the template point cloud for the next frame , repeat the above process to complete the pose refinement and obtain the point cloud information.
[0138] The coarse prediction results and the refined pose results in the pose prediction unit are as follows: Figure 7a and Figure 7bAs shown in the figure, the target pose obtained by the point cloud registration algorithm is not completely accurate. When the registration error is slightly large, the template point cloud information will be partially lost. When the error caused by the coarse registration is iterated over multiple frame sequences, the template point cloud of the target will be unusable, so the coarse prediction results need to be refined.
[0139] The point cloud clustering module also includes a clustering unit, which is used to implement point cloud clustering on the point cloud data results after posture refinement through the DBSCAN algorithm.
[0140] The working principle of DBSCAN is based on the neighborhood density of samples. By defining a neighborhood radius and a minimum number of samples in a neighborhood, DBSCAN is able to identify core points, boundary points, and noise points. A core point is a central point with a sufficient number of samples in the neighborhood, a boundary point is a point that does not have enough samples in the neighborhood to be a core point but is located in the neighborhood of a core point, and a noise point is a point that does not have enough samples in the neighborhood to be a core point and is not in the neighborhood of a core point. By connecting core points and boundary points, DBSCAN is able to form high-quality clusters. It is able to group samples with similar density into the same cluster and identify noise points as isolated data points, thereby providing clean and accurate clustering. The clustering results are as follows: Figure 8 shown.
[0141] In a specific embodiment, the point cloud clustering module also includes a convex hull calculation module and a point distance density estimation module. In three-dimensional space, the convex hull is used to find a minimum convex polygon or convex hull that can surround all given points. This minimum convex polygon or convex hull is the convex hull of the point cloud.
[0142] The convex hull calculation module is used to describe the geometric shape of the point cloud. In point cloud processing, the convex hull algorithm is often used to extract the morphological features of the point cloud. For example, in the extraction of branch morphological features, the convex hull algorithm can be used to calculate the morphological parameters such as the length, curvature, and branch angle of the branch. These morphological parameters can quantitatively describe the branch morphology in the point cloud, providing convenience for subsequent analysis and processing. The convex hull calculation results are shown in Figure 1. Fig. 9 shown.
[0143] The point distance density estimation module uses the point distance density estimation algorithm to update the search neighborhood radius of the point cloud. When the initial prediction of the target fails, the adaptive parameter selection module recalculates the search neighborhood radius to help the feature extraction descriptor obtain better features.
[0144] The convex hull calculation module process is as follows:
[0145] Using Graham-Scan to calculate the convex hull, first find a reference point (usually the point with the smallest y coordinate), then sort the other points according to their relative positions to the reference point, and finally use a stack to maintain the points on the convex hull. The algorithm steps are as follows:
[0146] Step 1: Select the point with the smallest x coordinate as the pole.
[0147] Step 2: Sort the remaining points by polar angle, and compare the distances to the pole when the polar angles are the same, and the point closest to the pole.
[0148] Step 3: Push the two points with the smallest polar angle and polar point into the stack.
[0149] Step 4: Scan each point and use the cross product to check whether the broken line segment formed by the first two elements at the top of the stack and this point "turns" to the right.
[0150] Step 5: If satisfied, pop the top element of the stack and return to step 4 until it is not satisfied.
[0151] Step 6: Push the point into the stack and return to continue scanning the remaining points.
[0152] Step 7: The elements in the final stack are the vertex sequence of the convex hull.
[0153] The point distance density estimation module process is as follows:
[0154] After calculating the convex hull to obtain the set of triangular facets of the template point cloud, the areas of the triangular facets are summed to obtain the estimated area of the template point cloud. , then the estimated point distance density for: .
[0155] The image segmentation module uses the YOLOv8-Seg image processing algorithm to realize contour recognition in the image.
[0156] The YOLOv8-Seg image processing algorithm performs pooling operations of different sizes on the feature map through Spatial Pyramid Pooling and integrates the results. This technology can effectively handle targets of varying sizes, thereby improving the generalization ability and robustness of the neural network.
[0157] The feature fusion module is used to fuse and segment the image features obtained by YOLOV8-seg with the point cloud features. The image features contain information such as object texture, and the point cloud features contain information such as object spatial position and distance. The two complement each other to obtain the result. Fig.10 As shown, the feature fusion results are as follows Fig.11 shown.
[0158] The contour recognition and tracking system proposed in the present invention can be used for lifting objects of unknown types in on-site operations of substations, and can perform in-depth analysis and extraction of the features of the target objects to ensure the effectiveness and accuracy of the model.
[0159] The point cloud clustering module realizes point cloud clustering, can extract point cloud information of target objects such as charged objects and cranes, and obtain information such as the spatial position and distance of the target objects. The image segmentation module realizes the recognition of the contours of charged objects and cranes in the image, accurately locates the targets in the image and classifies them at the pixel level, and obtains image information including information such as the texture of the target objects. The feature fusion module fuses and segments the point cloud clustering information and image information to obtain better segmentation results.
[0160] The point cloud clustering module proposed in the present invention can achieve category-free tracking while ensuring the accuracy of point cloud tracking, thereby greatly improving the flexibility and scalability of the system. In substation on-site operations, it is necessary to track unknown types of hoisted objects. The point cloud clustering module proposed in the present invention can handle the feature differences between different target objects. When faced with unknown types of hoisted objects in substation on-site operations, there is no need to re-establish the model and conduct training. It is more suitable for the changeable operating environment of the substation, which greatly improves the flexibility and scalability of the tracking system.
[0161] The image segmentation module in the present invention adopts the YOLOv8-Seg image processing algorithm. The core of YOLOv8-Seg is a deep convolutional neural network, which gradually extracts features from the image through multiple convolutional layers, pooling layers and fully connected layers. These features represent various visual attributes of the image, such as edges, textures, and shapes. By combining these features with target detection and semantic segmentation tasks, YOLOv8-Seg can simultaneously achieve target positioning and pixel classification, which improves the efficiency of the process of processing substation images and can obtain more accurate substation image information.
[0162] Based on the above information, the present invention further proposes to fuse and segment the image features and point cloud features of the target substation to complete the contour recognition and tracking.
[0163] Example 2
[0164] The present invention provides a contour recognition and tracking method, taking the recognition of charged objects in a substation and the contour of a boom as examples, and specifically comprises the following steps:
[0165] Step 1, obtain the historical 3D bounding box information of the charged object and the boom to predict the 3D bounding box of the target and obtain the motion prediction result;
[0166] Step 2: Combined with the motion prediction results, the current target is blocked to determine the degree of the target being blocked. If the target is blocked, the process is exited and the position of the substation-related equipment is adjusted to re-enter the tracking process. Otherwise, the process goes to step 3.
[0167] Step 3, down-sampling the motion prediction result to obtain sampled data;
[0168] Step 4: Predict and evaluate the target’s posture. If the evaluation result is unqualified, the tracking is terminated. If it is qualified, the sampled data is refined to obtain point cloud features.
[0169] Step 5, image segmentation: use the YOLOv8-Seg image processing algorithm to realize contour recognition in the target substation image and obtain image features;
[0170] Step 6, feature fusion, fuses and segments the point cloud features obtained in step 4 with the image features obtained in step 5 to achieve accurate segmentation and identification and tracking of live objects and boom contours in the substation at the work site.
[0171] In a specific embodiment, the motion prediction result includes the 3D bounding box information of the current frame, the point cloud of the current frame, the 3D bounding box information of the previous frame, the point cloud of the previous frame, the laser radar coordinate point information, etc.
[0172] Embodiment 3:
[0173] See also Fig.12 As shown, the present invention also provides an electronic device 100; the electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on the at least one processor 102, and at least one communication bus 104.
[0174] The memory 101 can be used to store the computer program 103. The processor 102 implements the steps of the method for contour recognition and tracking described in Example 2 by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101. The memory 101 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data (such as audio data) created according to the use of the electronic device 100, etc. In addition, the memory 101 can include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices.
[0175] The at least one processor 102 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 102 may be a microprocessor or any conventional processor, etc. The processor 102 is the control center of the electronic device 100, and uses various interfaces and lines to connect various parts of the entire electronic device 100.
[0176] It is known from common technical knowledge that the present invention can be implemented by other embodiments that do not deviate from its spirit or essential features. Therefore, the above disclosed embodiments are only illustrative in all respects and are not exclusive. All changes within the scope of the present invention or within the scope equivalent to the present invention are included in the present invention.
[0177] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0178] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0179] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0180] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A contour recognition and tracking system, the system is used for contour tracking of a moving target in a substation, characterized in that: Including point cloud clustering module, image segmentation module and feature fusion module; The point cloud clustering module includes a motion prediction unit, a downsampling unit, a posture prediction unit and a clustering unit; The motion prediction unit selects a substation mobile target, obtains historical 3D bounding box information of the substation mobile target, predicts the 3D bounding box of the substation mobile target, and obtains a motion prediction result; The down-sampling unit down-samples the result used for motion prediction to obtain sampled data; The posture prediction unit predicts the posture of the substation mobile target based on the sampled data to obtain prediction data; The clustering unit is used to cluster the prediction data to obtain point cloud features; The image segmentation module is used to perform contour recognition on the mobile target of the substation to obtain image features; The feature fusion module is used to fuse and segment point cloud features and image features to complete contour recognition and tracking of mobile targets in substations.
2. A contour recognition and tracking system as claimed in claim 1, characterized in that: The motion prediction unit takes as input the timestamp of the current frame.
3. A contour recognition and tracking system as claimed in claim 1, characterized in that: The point cloud clustering module also includes an occlusion judgment unit. After obtaining the motion prediction result, the occlusion judgment unit judges whether the motion prediction result is occluded. If there is no occlusion or the occlusion is not obvious in the current field of view, the process continues; if there is obvious occlusion in the current field of view, the tracking is exited.
4. A contour recognition and tracking system as claimed in claim 3, characterized in that: The occlusion judgment unit combines the predicted 3D bounding box information of the current frame, the point cloud of the current frame, the 3D bounding box information of the previous frame, the point cloud of the previous frame, and the laser radar coordinate points to judge whether the current substation moving target is occluded.
5. A contour recognition and tracking system as claimed in claim 1, characterized in that: The down sampling unit process is as follows: Assume that the center coordinate of the motion predicted 3D bounding box is, calculate the distance between the center coordinate and all points of the current frame point cloud; obtain the sampling probability vector of the current frame point cloud according to the distance between the center coordinate and all points of the current frame point cloud; complete downsampling.
6. A contour recognition and tracking system as claimed in claim 1, characterized in that: The posture prediction unit performs posture refinement after posture prediction; The pose prediction unit obtains the predicted 3D bounding box after the 3D bounding box transformation as prediction data, and further refines the prediction data.
7. A contour recognition and tracking system as claimed in claim 1, characterized in that: The point cloud clustering module also includes a confidence judgment unit, which judges whether the tracking result is credible based on the preset point cloud template and the point cloud prediction result in the clustering unit.
8. A contour recognition and tracking system as claimed in claim 1, characterized in that: The image segmentation module uses the YOLOv8-Seg image processing algorithm to perform contour recognition on the substation moving target image to obtain image features.
9. A contour recognition and tracking method based on a contour recognition and tracking system according to any one of claims 1 to 8, characterized in that: The steps include: Step 1, obtaining historical 3D bounding box information of the substation moving target, and the motion prediction unit predicts the 3D bounding box of the substation moving target based on the historical 3D bounding box information to obtain a motion prediction result; Step 2: Combined with the motion prediction result, the occlusion judgment unit determines the degree of occlusion of the current substation moving target. If the judgment is serious, the tracking is exited. Otherwise, the process goes to step 3. Step 3, the downsampling unit downsamples the motion prediction result to obtain sampled data; Step 4: The posture prediction unit predicts the posture of the mobile target of the substation based on the sampled data to obtain predicted data, and refines the predicted data. The clustering unit clusters the refined data to obtain point cloud features. Step 5, image segmentation: perform contour recognition on the substation moving target image to obtain image features; Step 6, feature fusion, fuses and segments the point cloud features obtained in step 4 with the image features obtained in step 5 to obtain the substation moving target.
10. An electronic device, characterized in that: including memory and processor; Memory for storing computer programs; A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the contour recognition and tracking method as claimed in claim 9 are implemented.
