Method and device for detecting moving small targets in panoramic video

Abstract

The application provides a kind of panoramic video moving small target detection method and equipment, it is related to image recognition technical field.The method comprises: obtaining panoramic video and mapping to spherical coordinate system;For current frame in panoramic video, determine at least two different historical frames, and calculate the feature difference of current frame and each historical frame, obtain multi-time scale motion map;According to the fusion weight of the preset semantic region to which the pixel belongs, the corresponding pixel in the multi-time scale motion map is weighted and fused according to the fusion weight, and background residual and periodic background suppression term are added, to obtain the comprehensive motion evidence of each pixel;According to the comprehensive motion evidence, the candidate region of moving small target is determined;The candidate region of adjacent frame is associated to form a space-time candidate tube, local cropping is carried out based on the space-time candidate tube, and moving small target is identified through a preset identification network.The application can improve the detection ability of moving small target in panoramic video, while reducing the overall reasoning overhead.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method and device for detecting small moving targets in panoramic video. Background Technology

[0002] The equipment areas, perimeter walls, and overhead structures of ultra-high voltage substations are exposed to complex natural environments for extended periods. Intrusion of moving foreign objects such as plastic bags, kite strings, balloons, and birds can potentially cause equipment discharge, short circuits, fires, or risks to personnel. While panoramic cameras can cover a 360° scene from a single point, making them suitable for continuous monitoring of large areas of substations, their 6K and 8K videos often contain targets occupying only a small fraction of the entire image, and the target's movement speed may be below the conventional inter-frame difference threshold.

[0003] Current full-image object detection methods typically scale ultra-high-definition panoramic images to a fixed input size for the network. However, the texture and contours of small, distant objects tend to disappear after scaling. Performing deep network inference directly at the original resolution significantly increases computational and memory usage with the image area, making it difficult to meet real-time edge analysis requirements. While fixed mesh tiling can improve the relative size of small objects, it involves repeated inference over a large number of pure background areas and may truncate objects at tile boundaries or 0° / 360° seams. Furthermore, equidistant cylindrical projection panoramic images exhibit significant spherical projection characteristics: the same object is stretched to varying degrees at different latitudes, and the pixel scale near the top and bottom boundaries is inconsistent with the true angular scale; pixels at the beginning and end of the horizontal plane are adjacent on the sphere but separated on the planar image. Directly using ordinary rectangular neighborhood, planar distance, and conventional non-maximum suppression can easily lead to the splitting of cross-seam targets, unreasonable candidate box scales, and duplicate alerts. In addition, at the substation site, the swaying of leaves and grass, cloud shadows and sudden changes in illumination, rain, snow and fog, mechanical movements of equipment, compressed noise and camera micro-shakes can all create pseudo-motion. Frame difference or background subtraction in a single time interval cannot simultaneously take into account both fast and slow targets: if the threshold is too high, it will miss slow small targets, and if the threshold is too low, it will treat periodic vegetation, light and shadow and noise as foreground. Summary of the Invention

[0004] This invention provides a method and device for detecting small moving targets in panoramic videos, thereby improving the detection capability of small moving targets in panoramic videos while reducing overall inference overhead.

[0005] In a first aspect, embodiments of the present invention provide a method for detecting small moving targets in panoramic video, comprising: Acquire panoramic video and map the panoramic video to a spherical coordinate system; For the current frame in the panoramic video, at least two different historical frames are determined, and the feature difference between the current frame and each historical frame is calculated to obtain a multi-timescale motion map. Based on the preset semantic region to which the pixel belongs in the panoramic video, the fusion weight of the pixel is determined. The corresponding pixels in the multi-timescale motion map are weighted and fused according to the fusion weight, and background residual and periodic background suppression terms are added to obtain the comprehensive motion evidence of each pixel. Based on the comprehensive motion evidence of each pixel, candidate pixels of the moving small target are determined, and the candidate pixels are connected and merged to obtain the candidate region of the current frame. Candidate regions of adjacent frames are correlated to form a spatiotemporal candidate tube. Local cropping is then performed based on the spatiotemporal candidate tube to obtain a cropped image. Based on the cropped image, a preset recognition network is used to identify small moving targets.

[0006] In one possible implementation, for the current frame in the panoramic video, determining at least two different historical frames and calculating the feature differences between the current frame and each historical frame to obtain a multi-timescale motion map includes: Based on the current frame and at least two different preset frame intervals, determine at least two different historical frames corresponding to the current frame; Calculate at least one feature difference between the current frame and each historical frame, and perform weighted summation and normalization on the at least one feature difference to obtain the multi-timescale motion map; wherein, the feature difference includes: luminance difference, chrominance difference, local structure difference and sparse motion vector.

[0007] In one possible implementation, determining the pixel fusion weight based on the preset semantic region to which the pixel belongs in the panoramic video includes: A preset baseline weight is determined based on the preset semantic region to which the pixel belongs in the panoramic video. The image quality of the current frame is obtained, and the historical stability of the preset semantic region is calculated based on historical background residuals, foreground frequency, and periodic motion characteristics. The weight adjustment value is determined based on the current frame image quality and the historical stability of the preset semantic region; The preset baseline weights are adjusted according to the weight adjustment value to obtain the pixel fusion weights; wherein the historical stability is positively correlated with the weight adjustment value, and the current frame image quality is positively correlated with the weight adjustment value.

[0008] In one possible implementation, adding background residuals includes: Obtain the fast and slow backgrounds for each preset semantic region; the update rates of the fast backgrounds and slow backgrounds are different for different semantic regions, and the update rate of the fast backgrounds is greater than that of the slow backgrounds for the same semantic region. Based on the preset semantic region to which the pixel belongs, determine the fast background and slow background corresponding to the pixel; Calculate the difference between pixel brightness and the corresponding fast background to obtain the fast background residual; Calculate the difference between pixel brightness and the corresponding slow background to obtain the slow background residual; The absolute values ​​of the fast background residual and the slow background residual are weighted and then added.

[0009] In one possible implementation, determining candidate pixels for the moving small target based on comprehensive motion evidence from each pixel includes: Update the leakage integral slow target accumulation value of the pixel based on the comprehensive motion evidence of the pixel; Calculate the statistical characteristics of historical comprehensive motion evidence for each preset semantic region to obtain the historical motion energy of each preset semantic region, and determine the strong motion threshold of each preset semantic region based on the historical motion energy of each preset semantic region and the current environmental state. Determine the strong motion threshold corresponding to the pixel based on the preset semantic region to which the pixel belongs; If the overall motion evidence of a pixel is greater than the strong motion threshold corresponding to the pixel, then the pixel is determined to be a candidate pixel for a moving small target. If the overall motion evidence of a pixel is less than or equal to the strong motion threshold corresponding to the pixel, and if the pixel's leakage integral slow target cumulative value is greater than the preset slow target cumulative activation threshold, and the corresponding centroid cumulative net displacement is not less than the preset minimum effective displacement and the cumulative response duration is not less than the preset minimum duration, then the pixel is determined to be a candidate pixel for a moving small target.

[0010] In one possible implementation, the step of performing local cropping based on the spatiotemporal candidate tube to obtain the cropped image includes: Based on the properties of each spatiotemporal candidate tube, the computation priority of each spatiotemporal candidate tube is determined, and a preset number of spatiotemporal candidate tubes with high computation priority are selected to obtain the target spatiotemporal candidate tube. A local tangent plane is established at the spherical center of the target spatiotemporal candidate tube, and a tight-scale angle field of view and a wide-scale angle field of view are determined; wherein the field of view angle of the tight-scale angle field of view is smaller than the field of view angle of the wide-scale angle field of view; Based on the tight-scale angle field of view and the wide-scale angle field of view, the current frame is reprojected and sampled using the local tangent plane as the projection plane to obtain the cropped image.

[0011] In one possible implementation, the spatiotemporal candidate tube's attributes include at least one of the following: motion intensity, persistence, risk zone weight, small target priority, and historical interference probability; The determination of the computation priority of each spatiotemporal candidate tube based on its attributes includes: Based on the preset weights of each attribute, the weighted value of each attribute of the spatiotemporal candidate tube is calculated, and the calculation priority of each spatiotemporal candidate tube is determined according to the magnitude of the weighted value.

[0012] In one possible implementation, the step of identifying small moving targets based on the cropped image using a preset recognition network includes: The cropped image, along with at least one of the following: background cropping, multi-timescale motion graph, spatiotemporal candidate tube trajectory mask, and historical keyframes, is input into the preset recognition network to obtain the moving small target type and bounding box output by the preset recognition network.

[0013] In one possible implementation, after identifying the small moving target based on the cropped image using a preset recognition network, the method further includes: Obtain the risk area in the panoramic video; Calculate the spherical overlap rate between the moving target and the risk zone, the shortest angular distance from the moving target to the risk zone, and the velocity component of the moving target toward the risk zone; The overall risk is determined based on the spherical overlap rate, the shortest angular distance, the velocity components, and the duration, type, and confidence level of the moving small target. If the overall risk exceeds a preset risk threshold, a risk alarm will be triggered.

[0014] In a second aspect, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect or any possible implementation thereof.

[0015] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows: This invention constructs a detection architecture that integrates multi-timescale motion change detection, spatiotemporal candidate tubes, spherical local high-resolution cropping, and secondary recognition. First, panoramic video is mapped to a spherical coordinate system for temporal motion analysis. Multiple historical frames are selected to construct multi-timescale motion maps, balancing the extraction of motion features from both fast-moving and slowly moving small targets. Multi-scale motion information is then adaptively weighted and fused with the semantic region to which the pixel belongs. Background residuals and periodic background suppression terms are introduced to suppress abrupt changes in illumination, compressed noise, and irregular flickering, improving the perception of low-contrast, slow-moving foreign objects. Simultaneously, disturbances caused by vegetation swaying, cloud shadows, and changes in illumination are effectively suppressed, resulting in accurate screening. Moving small target candidate regions: A spatiotemporal candidate tube is constructed by associating candidate regions in adjacent frames. Based on the spatiotemporal candidate tube, spherical local high-resolution cropping is performed. Unlike the whole-image reasoning or fixed-slice method, only a small amount of original resolution image after motion evidence screening is sent to the recognition network. While preserving the detailed features of small targets, it avoids the invalid calculation caused by large areas of pure background in ultra-high-definition panoramic images, significantly reducing the memory usage and computing power of edge devices and ensuring the real-time performance of the algorithm. At the same time, the spatiotemporal candidate tube maintains the temporal continuity of candidate targets across frames, reducing false alarms and missed detections caused by fragmented noise, and finally achieving efficient and accurate recognition of moving small targets in complex panoramic scenes. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the implementation of a method for detecting moving small targets in panoramic video according to an embodiment of the present invention. Figure 2 This is a schematic diagram of local clipping and reprojection of the tangent plane for spherical distortion sensing provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a multi-evidence secondary identification network provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a foreign object intrusion state machine and alarm determination provided in an embodiment of the present invention. Figure 5 This is a flowchart illustrating the implementation of a method for detecting small moving targets in panoramic video according to another embodiment of the present invention. Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0017] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0018] To illustrate the technical solution described in this invention, specific embodiments are described below.

[0019] The embodiments of the present invention aim to solve the problems of low pixel ratio of small moving targets, insignificant slow motion, large computational load of full-image inference, target truncation caused by panoramic projection distortion and stitching seams, false alarms caused by dynamic backgrounds, and difficulty in forming stable intrusion alarms from instantaneous detection results in ultra-high-definition panoramic videos.

[0020] Figure 1 This is a schematic diagram illustrating the implementation process of the method for detecting small moving targets in panoramic video provided in an embodiment of the present invention. Figure 1 As shown, it includes: Step S101: Acquire panoramic video and map the panoramic video to a spherical coordinate system.

[0021] Taking a substation as an example, panoramic cameras are installed at fixed monitoring locations to acquire 6K or 8K equidistant cylindrical projection video frames output by the panoramic cameras. The image width W, height H, frame timestamp, and camera installation posture are recorded. A mapping from pixel coordinates to spherical latitude and longitude angles is established, and semantic regions such as equipment body area, energized hazard area, perimeter wall area, passageway area, vegetation area, and sky area are divided in the spherical coordinate system. Each region stores risk level, background update rate, motion threshold, allowed target category, and alarm rules; when the region outline crosses the panoramic stitching seam, it is saved in the form of spherical polygons or periodic coordinates.

[0022] The mapping relationship between pixel coordinates and spherical coordinates is as follows: in, It is the azimuth angle. The pitch angle, u and v These are the horizontal and vertical pixel coordinates of the panoramic image, respectively. W and H These represent the image width and height, respectively.

[0023] Step S102: For the current frame in the panoramic video, determine at least two different historical frames, and calculate the feature difference between the current frame and each historical frame to obtain a multi-timescale motion map.

[0024] This embodiment estimates the global displacement caused by camera micro-shakes and encoding from anchor point regions such as static device edges and building outlines after excluding known dynamic regions and foreground candidates, and performs lateral periodic boundary registration between the current frame and historical frames.

[0025] Then, select at least two different preset frame intervals. Δ k ,For example At that time, the current frame time respectively with 、 、 、 By comparing the registered historical frames, four motion maps are obtained. In this embodiment, one or more feature differences, such as luminance difference, chromaticity difference, local structure difference, or sparse motion vector, are calculated for the registered frames to form multi-timescale motion maps.

[0026] For example: in, For the current frame time The first method adopted k Frame interval Get the pixels of the motion map Values; For pixel position; Brightness; For chromaticity; This represents the local structural similarity between the current frame and historical frames. For the corresponding frame interval k Sparse motion vectors on; 、 、 、 As weight; This indicates that the noise level is normalized based on the regional noise level.

[0027] Step S103: Determine the fusion weight of the pixel based on the preset semantic region to which the pixel belongs in the panoramic video. Perform weighted fusion on the corresponding pixels in the multi-timescale motion map according to the fusion weight, and add background residual and periodic background suppression terms to obtain the comprehensive motion evidence of each pixel.

[0028] In one possible implementation, this embodiment adaptively weights the motion map at multiple time scales according to image quality, region category, and historical stability, and suppresses abrupt changes in illumination, compression noise, and irregular flickering through dual background differences and periodic background suppression terms.

[0029] (1) Adaptive weight determination First, the preset baseline weight of the pixel is determined based on the preset semantic region to which the pixel belongs in the panoramic video.

[0030] Then, the image quality of the current frame is obtained, and the historical stability of the preset semantic region is calculated based on the historical background residual, the frequency of foreground occurrence, and the periodic motion characteristics. Based on the image quality of the current frame and the historical stability of the preset semantic region, the weight adjustment value is determined.

[0031] Historical stability describes whether a pixel or local region has been stable over a long period of time. It can be determined by the following statistics: the mean and variance of the historical background residuals, the frequency of exceeding the motion threshold, whether the motion direction changes repeatedly, the presence of stable periodic spectral peaks, and whether the background model update converges. It can be defined as follows: ∈[0,1], the larger the value, the more stable the position.

[0032] Based on the weight adjustment value, the preset baseline weights are adjusted to obtain the pixel fusion weights. Among them, historical stability is positively correlated with the weight adjustment value, that is, when a new motion appears in a stable region, its weight is increased, and when a long-term periodic motion region appears, its weight is decreased; the current frame image quality is positively correlated with the weight adjustment value, that is, the higher the image quality, the greater the corresponding weight.

[0033] (2) Add background residual In this embodiment, a fast background is maintained separately for each semantic region. and slow background Both background images are the same size as the current frame in terms of spatial dimensions, but they belong to different semantic regions. The pixels, using different update rates , The freeze / suppression rules, such as the update and suppression strategies for vegetation areas, differ from those for equipment and passageway areas. For vegetation areas, a higher rapid background update rate can be adopted in conjunction with periodic motion suppression. Equipment areas and electrified hazard areas should use slower updates, and updates should be paused in candidate areas, confirmed target areas, and short-term occlusion areas. A higher rapid background update rate is used to follow slow changes in illumination and minor cloud shadows, etc., in short-term environmental changes. A lower slow background update rate is used to preserve long-term stable scenes and prevent stationary foreign objects from being absorbed into the background too quickly. Generally, the following conditions are met. .

[0034] In this embodiment, the fast background and slow background corresponding to the pixel are determined according to the preset semantic region to which the pixel belongs, and then the difference between the two backgrounds is calculated.

[0035] The dual background difference refers to the two background residuals obtained by comparing the current frame with both a fast-moving and a slow-moving background. and .

[0036] Reflecting the deviation of the current image from short-term environmental memory, the fast background can quickly follow the gradual illumination, so this residual helps to reduce slow overall brightness changes.

[0037] It reflects the deviation of the current image from a long-term stable scene. Slow background updates are slower and can preserve the difference between the target and the original scene for a longer period of time when the target is stationary or moving slowly.

[0038] (3) Calculation of comprehensive motion evidence in, For historical stability; For a moment Pixel position Comprehensive evidence of movement at the site; For time scale sequence number; The total number of time scales; Based on pixel position semantic region Current frame image quality and historical stability The adaptively determined first Each time scale fusion weight; Brightness; and These are fast backgrounds and slow backgrounds, respectively. and These are the fusion weights for the fast background residual and the slow background residual, respectively. This is a periodic background suppression term calculated based on periodic motion, direction reversal, motion range, and semantic region.

[0039] Step S104: Based on the comprehensive motion evidence of each pixel, determine the candidate pixels of the moving small target, and merge the candidate pixels to obtain the candidate region of the current frame.

[0040] In one possible implementation, this embodiment generates thresholds within each semantic region based on the historical mean and variance of motion energy and weather / day / night conditions. τ r ,right Binarization and morphological cleanup are performed. Simultaneously, a leakage integral-type slow target accumulation map is maintained. A t This allows targets with insufficient motion energy in a single frame but undergoing continuous spatial migration over multiple long time intervals to still be activated; and connected components at the lateral beginning and end boundaries are merged according to spherical adjacency relationships. Detailed steps are as follows: (1) Based on the comprehensive motion evidence of the pixels, update the leakage integral slow target accumulation value of the pixels to obtain the slow target accumulation map: Then, the statistical characteristics of historical comprehensive motion evidence for each preset semantic region are calculated to obtain the historical motion energy of each preset semantic region: historical motion energy refers to the historical motion energy of comprehensive motion evidence before the current moment. Background motion levels obtained through long-term statistical analysis in locations without identified targets or candidate contaminants serve to answer the question of how much motion response and fluctuation typically occur in a given area under normal circumstances. For example, in a region... Inside, regarding the recent A valid historical moment The values ​​are subjected to exponential moving average statistics to obtain the mean. and variance .

[0041] (2) Based on the historical motion energy of each preset semantic region and the current environmental state, determine the strong motion threshold of each preset semantic region: in, This represents the regional correlation sensitivity coefficient. For image quality; Environmental conditions such as day and night, rain, snow, fog, and backlight; This is an environmental compensation item; and These are upper and lower limits. Smaller values ​​can be used in equipment areas and hazardous electrical areas. To improve sensitivity, a larger area can be used in vegetated areas. And it is superimposed with periodic inhibition.

[0042] (3) Determination of candidate pixels In this embodiment, if the overall motion evidence of a pixel is greater than the strong motion threshold corresponding to the pixel, then the pixel is determined to be a candidate pixel of a moving small target; if the overall motion evidence of a pixel is less than or equal to the strong motion threshold corresponding to the pixel, then if the pixel's leakage integral slow target cumulative value is greater than the preset slow target cumulative activation threshold, and the corresponding centroid cumulative net displacement is not less than the preset minimum effective displacement and the cumulative response duration is not less than the preset minimum duration, then the pixel is determined to be a candidate pixel of a moving small target.

[0043] The candidate pixel determination logic can be described as follows: in, Δs t This represents the cumulative displacement of the corresponding centroid within the time window; Tt This indicates the duration or number of valid frames for which the cumulative response has persisted; The semantic region to which the pixel belongs The threshold of strong motion; This represents the cumulative value of the leaking integral slow target. Accumulate activation thresholds for slow targets; The minimum effective displacement required to eliminate in-situ noise and minor vibrations; This is the minimum duration.

[0044] This expression contains two parallel activation paths: The first path is a strong instantaneous motion path, that is... It is activated directly and mainly covers flying birds and fast-moving objects. The second path is the slow target path, meaning the single-frame response may not be high, but... Activation is only granted when the threshold is consistently exceeded, and the target undergoes sufficient displacement for a sufficient duration.

[0045] Increase and The condition is to avoid mistaking brightness flickering, compressed noise, or periodic stationary oscillations at fixed locations for slow targets. After generating the binary foreground map, it still needs to undergo morphological cleansing, connected component analysis, and area filtering before the final candidate regions are generated.

[0046] (4) Connected component analysis The connected domain here is the evidence of integrated motion. Or slow target cumulative chart The foreground pixels are connected after thresholding and morphological processing. Each connected region is further transformed into a candidate region, including attributes such as candidate mask, bounding box, centroid, area, and average motion direction. Therefore, merging connected regions can also be described as merging candidate regions across the stitching seam.

[0047] Specifically, when a real target spans the 0° / 360° seam of a panoramic image, it creates two separate connected regions on the left and right edges of the planar image. These should not be treated as two separate targets; instead, they should be checked to ensure they simultaneously meet the following conditions: located in the neighborhood of the left and right boundaries, the distance between their spherical centers is less than a threshold, their longitudinal ranges overlap or are close, and their direction of movement is consistent with time. If these conditions are met, the two mask segments are merged into a single candidate using periodic coordinates, which can be represented by the starting point plus the period width or a spherically enclosed region.

[0048] Step S105: Associate the candidate regions of adjacent frames to form a spatiotemporal candidate tube, and perform local cropping based on the spatiotemporal candidate tube to obtain the cropped image.

[0049] Here, adjacent candidate frames can be correlated based on factors such as the distance between the centers of the spheres, area changes, direction of motion, and coarse appearance features to form a spatiotemporal candidate tube. Adjacent frames refer to adjacent analysis moments, not necessarily two physically consecutive frames in the video stream. If analyzed frame by frame, then... Frame and Frame; if we take one frame out of every two frames for analysis, then... Frame and The specific association steps can be implemented using existing methods, and will not be detailed in this embodiment.

[0050] Then, the spatiotemporal candidate tubes can be filtered to reduce the computational load. For example, based on the properties of each spatiotemporal candidate tube, the computational priority of each spatiotemporal candidate tube can be determined, and a preset number of spatiotemporal candidate tubes with high computational priority can be selected to obtain the target spatiotemporal candidate tubes.

[0051] The priority calculation formula can be: in, For exercise intensity, For sustainability, As the risk zone weight, Prioritize small goals. This represents the historical interference probability. to This is the fusion coefficient.

[0052] Step S106: Based on the cropped image, identify small moving targets using a preset recognition network.

[0053] like Figure 2 As shown, in this embodiment, a local tangent plane is established at the center of the sphere of the target spatiotemporal candidate tube, and a tight-scale angle field of view and a wide-scale angle field of view are determined. The field of view angle of the tight-scale angle field of view is smaller than that of the wide-scale angle field of view. Based on the tight-scale angle field of view and the wide-scale angle field of view, the current frame is reprojected and sampled using the local tangent plane as the projection surface to obtain the cropped image.

[0054] In one possible implementation, the image cropping step can be accomplished in four steps: scale estimation, motion expansion, dual-scale generation, and tangent plane sampling. (1) Map the candidate box or candidate mask to spherical coordinates to obtain the horizontal angle width. and vertical angle height To avoid pixel stretching in high-latitude regions affecting the cropping scale; (2) Estimate the azimuth angular velocity based on the spherical center of the candidate tube in the most recent few frames. and pitch angular velocity In the predicted time Internally obtained: The cutting center can be set between the current center and the predicted center, or moved forward appropriately along the direction of movement.

[0055] (3) The horizontal and vertical angular fields of view of tight-scale clipping can be expressed as follows: A wide-scale field of view is used to preserve context such as equipment, fences, roads, or vegetation, and can be represented as: (4) Establish a local tangent plane with the selected center of the sphere, back-project the local output pixels to the latitude and longitude angle of the sphere, and then map them to the original panoramic image for bilinear sampling.

[0056] The horizontal axis uses the modulus Periodic index: Therefore, crossing The target of the seam can remain continuous in the same local image.

[0057] In the above steps: ; These are the horizontal angle width and the vertical angle height, respectively. These are the angular velocities for azimuth and elevation, respectively. For predicting time; To predict angular displacement; The expansion coefficient at the target scale; The displacement expansion coefficient is used for motion. A one-sided safety boundary in both horizontal and vertical directions; These are the minimum and maximum values ​​of the angular field of view, respectively. This is the magnification factor of the wide scale relative to the compact scale; The width of the original panoramic image; This is the amplitude limiting function.

[0058] In one possible implementation, a secondary recognition input can be constructed, comprising the current high-resolution cropped image, background cropping, multi-timescale motion maps, spatiotemporal candidate tube trajectory masks, and historical keyframes. The secondary recognition network includes at least: a branch for outputting accurate bounding boxes and target detection for known categories such as plastic bags, birds, small animals, and balloons / kites; a temporal realism branch for determining whether the motion exhibits target consistency and excluding pseudo-motions caused by lighting and vegetation; and an open set branch for outputting scores for unknown objects based on the feature distance between the current target and the background memory, continuous changes, and targetability scores. The results of tight-scale and wide-scale cropping are fused using confidence scores, and duplicate bounding box merging is performed in the spherical coordinate domain.

[0059] The background cropping is from a slow background. A quick background can also be added if necessary. Cropping uses the exact same spherical center, angular field of view, and sampling mapping as the current cropped image, ensuring a one-to-one correspondence between pixel positions and allowing direct comparison of the target's differences from the long-term background. The spatiotemporal candidate tube trajectory mask, derived from the spatiotemporal candidate tubes, renders the current candidate box, historical center trajectory, predicted position, and motion corridor as a binary mask or heatmap, mapping it to a local tangent plane to indicate the location and motion direction the model should focus on. Historical keyframes, derived from the established circular buffer of original frames, select several frames based on the candidate tube's historical time, image sharpness, time interval, or occlusion, aligning them with the candidate tube trajectory before cropping. Their purpose is to help the temporal realism branch determine whether boundaries, appearance, and motion are continuous and consistent.

[0060] For example, a secondary recognition network can be seen in [reference needed]. Figure 3 As shown.

[0061] The total training loss of the secondary recognition model can be expressed as: in, Corresponding to bounding box regression, Corresponding category identification, Corresponding to the target, Corresponding to timing consistency, Corresponding to the open set foreign object detection, Corresponding to the consistency constraint across splice seams. , , , , These are the corresponding weighting coefficients.

[0062] This invention constructs a detection architecture that integrates multi-timescale motion change detection, spatiotemporal candidate tubes, spherical local high-resolution cropping, and secondary recognition. First, panoramic video is mapped to a spherical coordinate system for temporal motion analysis. Multiple historical frames are selected to construct multi-timescale motion maps, balancing the extraction of motion features from both fast-moving and slowly moving small targets. Multi-scale motion information is then adaptively weighted and fused with the semantic region to which the pixel belongs. Background residuals and periodic background suppression terms are introduced to suppress abrupt changes in illumination, compressed noise, and irregular flickering, improving the perception of low-contrast, slow-moving foreign objects. Simultaneously, disturbances caused by vegetation swaying, cloud shadows, and changes in illumination are effectively suppressed, resulting in accurate screening. Moving small target candidate regions: A spatiotemporal candidate tube is constructed by associating candidate regions in adjacent frames. Based on the spatiotemporal candidate tube, spherical local high-resolution cropping is performed. Unlike the whole-image reasoning or fixed-slice method, only a small amount of original resolution image after motion evidence screening is sent to the recognition network. While preserving the detailed features of small targets, it avoids the invalid calculation caused by large areas of pure background in ultra-high-definition panoramic images, significantly reducing the memory usage and computing power of edge devices and ensuring the real-time performance of the algorithm. At the same time, the spatiotemporal candidate tube maintains the temporal continuity of candidate targets across frames, reducing false alarms and missed detections caused by fragmented noise, and finally achieving efficient and accurate recognition of moving small targets in complex panoramic scenes.

[0063] like Figure 4 As shown, in one embodiment, after identifying a small moving target, a foreign object intrusion determination mechanism is also designed to achieve accurate warning of foreign object intrusion in the substation, including: Identify risk areas in panoramic video; Calculate the spherical overlap rate between the moving target and the risk zone, the shortest angular distance from the moving target to the risk zone, and the velocity component of the moving target toward the risk zone; The overall risk is determined based on the spherical overlap rate, the shortest angular distance and velocity components, as well as the duration, type and confidence level of the moving small target; If the overall risk exceeds the preset risk threshold, a risk alarm will be triggered.

[0064] Specifically, the comprehensive risk score can be calculated using the following formula: in, For confidence level, The overlap ratio of the spheres. For velocity components, For duration, For type of risk quantity, The shortest angular distance from the target to the risk zone. , , , , , These are the corresponding weighting coefficients.

[0065] Meanwhile, this embodiment employs a state machine of "candidate appearance - continuous observation - foreign object confirmation - entry into or approach to the risk zone - alarm generation," setting confirmation thresholds, hold thresholds, and release thresholds to allow short-term occlusion while suppressing single-frame jitter. Based on the target category, unknown foreign object score, target trajectory, equipment risk level, and duration, it outputs general concern, important, or emergency alarms.

[0066] This embodiment outputs the target time, panoramic pixel location, spherical azimuth / tilt angle, target bounding box, category, unknown foreign object score, overall confidence level, trajectory, risk level, and alarm snapshot. False alarm regions and new foreign object samples, confirmed manually, are added to a controlled sample library for updating region thresholds, interference templates, or offline incremental training, but do not directly and unsupervisedly modify the core parameters of the deployed model.

[0067] See Figure 5 As shown, in one embodiment, this method inputs a 6K / 8K equidistant cylindrical projection panoramic video stream, and then executes the following sequentially: S1: Acquire ultra-high-definition panoramic video and complete spherical calibration; S2: Image cross-seam registration to construct dual-timescale background memory; S3: Generating motion evidence based on multiple timescales; S4: Semantic partitioning suppresses interference and extracts target candidate regions; S5: Aggregates spatiotemporal candidate boxes and dynamically allocates computing power budget; S6: Combines spherical distortion sensing for high-resolution cropping of local areas; S7: Multi-evidence secondary detection and classification to identify unknown foreign objects; S8: Determine if an intrusion occurs in a risky area, confirm the target status, and output a tiered alarm.

[0068] The final output includes the target location, category, confidence level, spherical trajectory, and hierarchical alarm information.

[0069] Based on the above embodiments, the improvements of the present invention include: (1) A closed-loop architecture is proposed, consisting of multi-timescale motion change detection, spatiotemporal candidate tubes, spherical local high-resolution cropping, and secondary recognition of multiple evidences. Unlike fixed slices or full-image high-resolution inference, the deep network only processes a small number of original resolution regions selected by video motion evidence and maintains cross-frame continuity through candidate tubes.

[0070] (2) A multi-timescale difference and leakage integral accumulation mechanism for slow small targets is proposed. At the same time, fast / slow dual background, semantic partition threshold and periodic motion suppression are used to weaken vegetation swaying, cloud shadow and illumination change while retaining weak motion such as slow plastic bags and climbing small animals.

[0071] (3) A candidate merging and clipping method based on spherical geometry perception is proposed. The merging of candidate centers, distances, trajectories and repeated boxes is all processed in the spherical domain; the local clipping adaptively determines the angular field of view based on latitudinal distortion, spherical target scale and predicted motion, and performs periodic sampling and tangent plane reprojection on the 0° / 360° splicing seam.

[0072] (4) A candidate priority scheduling mechanism based on computing power budget is proposed, which uses the persistence of the target, the risk area, the target scale, the historical interference probability and the motion intensity to determine the secondary identification order, and prioritizes the processing of more dangerous, smaller and more trustworthy candidates under the limited edge computing power.

[0073] (5) Propose a multi-evidence secondary recognition method based on current cropped images, background cropping, motion difference and candidate tube mask, and add an open set foreign object branch, so that in addition to recognizing predefined categories such as plastic bags, birds, and small animals, the system can also give unknown foreign object alarms for moving foreign objects with continuous target and significant background differences that have not been trained.

[0074] (6) A hysteresis state machine based on spherical trajectory and risk area is proposed. Instead of using single frame overlap as the sole alarm basis, it integrates approach speed, duration, category / unknown score and regional risk level to reduce instantaneous jitter, short-term occlusion and repeated alarms.

[0075] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0076] Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Figure 6 As shown, the electronic device 6 of this embodiment includes a processor 60 and a memory 61. The memory 61 stores a computer program 62. When the processor 60 executes the computer program 62, it implements the steps in the various method embodiments described above.

[0077] For example, computer program 62 may be divided into one or more modules / units, which are stored in memory 61 and executed by processor 60 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 62 in electronic device 6.

[0078] Electronic device 6 may include, but is not limited to, processor 60 and memory 61. Those skilled in the art will understand that... Figure 6 This is merely an example of electronic device 6 and does not constitute a limitation on electronic device 6. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 6 may also include input / output devices, network access devices, buses, etc.

[0079] For the sake of simplicity and clarity, only the above-described functional modules / units are used as examples. In practical applications, the functions described above can be assigned to different functional modules / units as needed. These modules / units can be implemented in hardware, software, or a combination of both.

[0080] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not detailed or described in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Unless otherwise specified or in conflict with logic, the terminology and / or descriptions between different embodiments are consistent and can be referenced interchangeably. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0081] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for detecting moving small targets in panoramic video, characterized in that, include: Acquire panoramic video and map the panoramic video to a spherical coordinate system; For the current frame in the panoramic video, at least two different historical frames are determined, and the feature difference between the current frame and each historical frame is calculated to obtain a multi-timescale motion map. Based on the preset semantic region to which the pixel belongs in the panoramic video, the fusion weight of the pixel is determined. The corresponding pixels in the multi-timescale motion map are weighted and fused according to the fusion weight, and background residual and periodic background suppression terms are added to obtain the comprehensive motion evidence of each pixel. Based on the comprehensive motion evidence of each pixel, candidate pixels of the moving small target are determined, and the candidate pixels are connected and merged to obtain the candidate region of the current frame. Candidate regions of adjacent frames are correlated to form a spatiotemporal candidate tube. Local cropping is then performed based on the spatiotemporal candidate tube to obtain a cropped image. Based on the cropped image, a preset recognition network is used to identify small moving targets.

2. The method for detecting moving small targets in panoramic video according to claim 1, characterized in that, For the current frame in the panoramic video, at least two different historical frames are determined, and the feature differences between the current frame and each historical frame are calculated to obtain a multi-timescale motion map, including: Based on the current frame and at least two different preset frame intervals, determine at least two different historical frames corresponding to the current frame; Calculate at least one feature difference between the current frame and each historical frame, and perform weighted summation and normalization on the at least one feature difference to obtain the multi-timescale motion map; wherein, the feature difference includes: luminance difference, chrominance difference, local structure difference and sparse motion vector.

3. The method for detecting moving small targets in panoramic video according to claim 1, characterized in that, The step of determining the fusion weight of a pixel based on the preset semantic region to which it belongs in the panoramic video includes: A preset baseline weight is determined based on the preset semantic region to which the pixel belongs in the panoramic video. The image quality of the current frame is obtained, and the historical stability of the preset semantic region is calculated based on historical background residuals, foreground frequency, and periodic motion characteristics. The weight adjustment value is determined based on the current frame image quality and the historical stability of the preset semantic region; The preset baseline weights are adjusted according to the weight adjustment value to obtain the pixel fusion weights; wherein the historical stability is positively correlated with the weight adjustment value, and the current frame image quality is positively correlated with the weight adjustment value.

4. The method for detecting moving small targets in panoramic video according to claim 1, characterized in that, The addition of background residuals includes: Obtain the fast and slow backgrounds for each preset semantic region; the update rates of the fast backgrounds and slow backgrounds are different for different semantic regions, and the update rate of the fast backgrounds is greater than that of the slow backgrounds for the same semantic region. Based on the preset semantic region to which the pixel belongs, determine the fast background and slow background corresponding to the pixel; Calculate the difference between pixel brightness and the corresponding fast background to obtain the fast background residual; Calculate the difference between pixel brightness and the corresponding slow background to obtain the slow background residual; The absolute values ​​of the fast background residual and the slow background residual are weighted and then added.

5. The method for detecting moving small targets in panoramic video according to claim 1, characterized in that, The step of determining candidate pixels for a moving small target based on comprehensive motion evidence from each pixel includes: Update the leakage integral slow target accumulation value of the pixel based on the comprehensive motion evidence of the pixel; Calculate the statistical characteristics of historical comprehensive motion evidence for each preset semantic region to obtain the historical motion energy of each preset semantic region, and determine the strong motion threshold of each preset semantic region based on the historical motion energy of each preset semantic region and the current environmental state. Determine the strong motion threshold corresponding to the pixel based on the preset semantic region to which the pixel belongs; If the overall motion evidence of a pixel is greater than the strong motion threshold corresponding to the pixel, then the pixel is determined to be a candidate pixel for a moving small target. If the overall motion evidence of a pixel is less than or equal to the strong motion threshold corresponding to the pixel, and if the pixel's leakage integral slow target cumulative value is greater than the preset slow target cumulative activation threshold, and the corresponding centroid cumulative net displacement is not less than the preset minimum effective displacement and the cumulative response duration is not less than the preset minimum duration, then the pixel is determined to be a candidate pixel for a moving small target.

6. The method for detecting moving small targets in panoramic video according to claim 1, characterized in that, The step of performing local cropping based on the spatiotemporal candidate tube to obtain the cropped image includes: Based on the properties of each spatiotemporal candidate tube, the computation priority of each spatiotemporal candidate tube is determined, and a preset number of spatiotemporal candidate tubes with high computation priority are selected to obtain the target spatiotemporal candidate tube. A local tangent plane is established at the spherical center of the target spatiotemporal candidate tube, and a tight-scale angle field of view and a wide-scale angle field of view are determined; wherein the field of view angle of the tight-scale angle field of view is smaller than the field of view angle of the wide-scale angle field of view; Based on the tight-scale angle field of view and the wide-scale angle field of view, the current frame is reprojected and sampled using the local tangent plane as the projection plane to obtain the cropped image.

7. The method for detecting moving small targets in panoramic video according to claim 6, characterized in that, The attributes of the spatiotemporal candidate tube include at least one of the following: motion intensity, persistence, risk zone weight, small target priority, and historical interference probability; The determination of the computation priority of each spatiotemporal candidate tube based on its attributes includes: Based on the preset weights of each attribute, the weighted value of each attribute of the spatiotemporal candidate tube is calculated, and the calculation priority of each spatiotemporal candidate tube is determined according to the magnitude of the weighted value.

8. The method for detecting moving small targets in panoramic video according to claim 1, characterized in that, The step of identifying small moving targets based on the cropped image using a preset recognition network includes: The cropped image, along with at least one of the following: background cropping, multi-timescale motion graph, spatiotemporal candidate tube trajectory mask, and historical keyframes, is input into the preset recognition network to obtain the moving small target type and bounding box output by the preset recognition network.

9. The method for detecting moving small targets in panoramic video according to any one of claims 1 to 8, characterized in that, After identifying the small moving target based on the cropped image using a preset recognition network, the method further includes: Obtain the risk area in the panoramic video; Calculate the spherical overlap rate between the moving target and the risk zone, the shortest angular distance from the moving target to the risk zone, and the velocity component of the moving target toward the risk zone; The overall risk is determined based on the spherical overlap rate, the shortest angular distance, the velocity components, and the duration, type, and confidence level of the moving small target. If the overall risk exceeds a preset risk threshold, a risk alarm will be triggered.

10. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 9.

Images