A focal target detection method based on air-ground collaborative perspective

Abstract

The present invention discloses a focal target detection method based on air-ground collaborative perspective. The method is based on a high-altitude camera carried by an unmanned aerial vehicle with a high-altitude perspective and a first-person perspective of wearable cameras worn by multiple pedestrians on the ground. The method uses the air-ground collaborative perspective to detect, associate and fuse the visual attention of multiple people in the scene to achieve the positioning of the focal target; step S1: input N+1 time-synchronized videos with a duration of step S2: use the sliding window technology to consistently divide all the videos obtained in step S1 into short segments with a duration of T; step S3: use a human-based object detection algorithm and a multi-perspective data fusion method to obtain candidate focal targets from each frame of the current segment, etc. The present invention captures the surrounding personnel and environmental details at the same position through multiple first-person perspective cameras, and uses the above-mentioned air-ground collaborative perspective to achieve the detection of focal targets in the scene.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a focus target detection method based on empty collaborative perspective. Background Art

[0002] The background technologies involved in the present invention are:

[0003] (1) Video Co-segmentation and Saliency Detection: Video co-segmentation is to simultaneously segment objects of a common category from two or more videos. In practice, almost all existing video co-segmentation methods are based on the appearance information of the object. Some methods also exploit complementary motion features and consider temporal information to jointly segment the target (references). Unlike this method, video co-segmentation searches for all common objects, including human subjects, in different videos without considering whether the detected subject attracts the interest of other surrounding subjects.

[0004] (2) Video Saliency Detection: Video saliency detection aims to identify objects or regions with the highest perceptual saliency. Many machine learning-based methods, such as sparse reconstruction, low-rank consistency, and deep learning-based models, have been developed for video saliency detection. Most of them use appearance and motion features to separate salient objects and non-salient regions, and usually target a single view. Saliency detection is a general method for identifying perceptually salient objects or regions, which identifies the focal target through the shared visual attention of surrounding people.

[0005] (3) Visual attention via gaze estimation: Many existing methods aim to simulate the human visual attention mechanism to identify salient regions / objects in natural or social environments. However, existing methods mostly process scenes close to the camera with only a few objects in its field of view. This method addresses this limitation by combining a top-view video and multiple first-person view videos.

[0006] (4) Multi-view video analysis: Collaborative analysis of multi-view videos is of great significance. However, the existing collaborative analysis of multi-view videos has a limited field of view, and the need to wear wearable cameras affects its usability in video surveillance systems. Currently, a combination of top and first-person camera settings has been applied to collaborative video analysis. This method combines the aerial view and the first-person view to achieve practical applications in surveillance systems, namely, important person detection in video surveillance. Summary of the invention

[0007] In order to solve the existing technology, the purpose of the present invention is to propose a new camera placement method and develop a new video analysis algorithm to better detect the focal target in the scene, so as to complete the abnormal target behavior detection based on multi-view video surveillance. Specifically, the present invention adopts a high-altitude camera carried by a drone (providing a high-altitude perspective) and multiple wearable cameras worn by pedestrians on the ground (providing a first-person perspective), and uses the perspective of air-ground collaboration to detect, associate and integrate the visual attention (observation target and observation range) of multiple people in the scene to complete the positioning of the focal target (person) in the scene. The present invention can process and analyze the high-altitude view taken by the drone and the first-person perspective view captured by the horizontal camera, so as to achieve rapid detection and accurate identification of the focal target of the scene, and complete the group activity analysis based on video surveillance.

[0008] The present invention is implemented by adopting the following technical scheme:

[0009] A focal target detection method based on air-ground collaborative perspective, the method is based on a high-altitude camera carried by a drone with a high-altitude perspective and a first-person perspective of wearable cameras worn by multiple pedestrians on the ground, and utilizes the air-ground collaborative perspective to detect, associate and fuse the visual attention of multiple people in the scene to achieve the positioning of the focal target; the method comprises the following steps:

[0010] Step S1: Input N+1 time-synchronized videos with a duration of

[0011] Step S2: Using the sliding window technique, all the videos obtained in step S1 are consistently divided into short segments of duration T;

[0012] Step S3: using a person detection algorithm and a multi-view data fusion method to obtain a candidate focus target from each frame of the current segment;

[0013] Step S4: Using energy function Calculate the energy value of each candidate focus target in each perspective, and combine the multi-perspective energy values ​​to calculate the energy value of the current frame

[0014]

[0015] Where: E1 and E2 represent the intra-frame energy term and inter-frame energy term respectively, P t , P s represents the focus target candidates of the tth and sth frames;

[0016] Step S5: When all the segments have not been processed, return to step S3 to identify the next segment; when all the segments have been processed, proceed to step S6;

[0017] Step S6: The focus target energy value detection results of all segments in time are analyzed together to obtain the detection result.

[0018] Further, the process of obtaining the candidate focus target in each frame of the current segment in step S3 is as follows:

[0019] Denote the kth clip in the nth first-person view as Denote the kth clip in the aerial view as For each window W k , using multi-view video clipping, i.e. and To detect the focus target in the top view, And the corresponding object in the first-person view When there is no confusion, the symbol Simplified to The symbol Simplified to Each clip consists of T frames, and They are the tth frame in the aerial view and the nth first-person view clips, respectively;

[0020] In each frame, all detected objects are regarded as a set of focus target candidates, each frame is regarded as a node, and the focus target candidates in the frame are regarded as its state;

[0021] A multi-view target association algorithm is used to unify the objects detected in different videos, that is, to identify and match the same person in different videos; the detected target groups are uniformly represented as As the candidate set of the focal target, P t is a candidate for the focal target; a conditional random field model is used for focal target detection, where each frame is regarded as a node in the conditional random field model, and each candidate node is regarded as a state of the node; a candidate focal target is found in each frame t through the conditional random field model;

[0022] Using the energy function E(P t ,P s |F t ,F s ) to estimate P t ,P s As a focal target, it reflects the credibility of a candidate focal target. The energy function is specifically expressed as

[0023]

[0024] Among them, E1 and E2 represent the intra-frame and inter-frame energies, respectively, which characterize the probability of the focus target being in the same frame and in different frames;

[0025] Based on the built-in confidence E1 of the frame, the view deviation level of the candidate object is defined according to its position in the field of view of other objects; the energy in the coordinate system is constructed by calculating the focus ratio level of the candidate object, that is, the ratio of the number of subjects whose field of view covers the candidate object to the total number of subjects in the scene:

[0026] E1(P t )=-Sig(V(P t )+F(P t ))

[0027] Where Sig represents the sigmoid function, V(·) and P(·) respectively reflect the candidate P t The two energies of viewing angle deviation and focusing ratio level; V(·) is defined as

[0028]

[0029] in The total number of objects detected at frame t is |P t |, represents the viewing direction of object P, Represents the transition from object P to candidate P t vector, P traverses except object P t External All objects in , brackets <·> represent the angle between two vectors; use and The angle between them is used to estimate the deviation level of the point in the view;

[0030] Modify using damping function and define F(·) as

[0031]

[0032] Where Ind(·) is the index function. This energy term is used to calculate the field of view containing the candidate P t The ratio of the target, where 2θ is the field of view of the target;

[0033] Since the inter-frame energy E2 depends on the change of the focus target confidence and the temporal consistency across frames, the inter-frame energy is constructed as

[0034] E2(P t ,P s )=-Sig(ΔV(P t ,P s )+ΔF(P t ,P s )+O(P t ,P s )),

[0035] Among them, ΔV(P t ,P s )=V(P s )-V(P t ) represents the view deviation level from P t To P s The change of ΔF(P t ,P s )=ΔF(P s )-F(P t ) represents the change of focusing ratio level; O(P t ,P s ) indicates P t With P s The overlap between them can ensure the spatial consistency from frame t to frame s.

[0036] Beneficial Effects

[0037] 1. The present invention can process and analyze the high-altitude views taken by drones and the first-person perspective views captured by horizontal cameras, thereby realizing rapid detection and accurate identification of scene focus targets and completing group activity analysis based on video surveillance.

[0038] 2. This paper adopts a simple and efficient strategy to develop and design a method for detecting focal objects in multiple synchronized videos based on complementary and time-varying views, that is, using complementary top-view and first-person videos shot by a mobile camera to identify people of common interest. Experimental results on the complementary view video dataset show that this method can effectively identify focal objects by capturing the global position in the top view and the local appearance details in the first-person view.

[0039] 3. The present invention belongs to the field of computer vision, and studies the detection of focal targets in multiple synchronized videos with temporal changes. Fast and accurate focal target recognition plays an important role in social scene monitoring and analysis, but current research mainly focuses on focal target detection in single-view videos. The present invention obtains the entire scene and ground targets of a global view by using a high-altitude camera installed on a drone to fly at high altitude; the surrounding personnel and environmental details at the same location are captured by multiple first-person cameras, and the above-mentioned air-ground collaborative perspective is used to realize the detection of focal targets in the scene. The method proposes a spatiotemporal potential energy function, while taking into account the built-in confidence of the frame and the consistency between frames, thereby obtaining an effective conditional random field formula. The complementary view video dataset provided by the invention provides a benchmark for the study of multi-view focal target detection, and a large number of experimental results have proved the effectiveness and superiority of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of a focus target detection method based on air-ground collaborative perspective of the present invention. DETAILED DESCRIPTION

[0041] The present invention is described in detail below in conjunction with the accompanying drawings:

[0042] like Figure 1 As shown, the present invention is a focus target detection method based on air-ground collaborative perspective, and its execution steps are as follows:

[0043] Step S1: Input N+1 time-synchronized videos (N first-person perspective videos and 1 bird's-eye view video), with a duration of

[0044] Step S2: Use a sliding window to consistently divide all the videos obtained in step S1 into short segments of length T;

[0045] Step S3: using a person detection algorithm and a multi-view data fusion method to obtain a candidate focus target from each frame of the current segment;

[0046] Step S4: Using energy function Calculate the energy value of each candidate focus target in each perspective, and combine the multi-perspective energy values ​​to calculate the current frame energy value E

[0047]

[0048] Step S5: When all the segments have not been processed, return to step S3 to identify the next segment; when all the segments have been processed, proceed to step S6;

[0049] Step S6: The focus target energy value detection results of all segments in time are analyzed together to obtain the detection result.

[0050] The present invention is provided with a mobile camera with a synchronized clock, and a new algorithm is developed to improve the level of focus target detection. It is mainly explained from two aspects: one is the solution to the challenging problem in algorithm development, and the other is the implementation process of focus target detection based on the above method.

[0051] (1) Solutions based on challenging problems: The present invention describes the challenging problems faced by the present invention in developing new algorithms to improve the level of focus target detection, and proposes corresponding solutions.

[0052] For algorithm development, the following three challenging problems need to be solved:

[0053] a) Person recognition across multiple first-person and aerial views.

[0054] b) Estimation of the viewing direction of all objects, including the mobile camera wearer, in both first-person and overhead views.

[0055] c) Build a unified model for focal object detection by fusing all objects appearing from all viewpoints.

[0056] To address the above challenges, the present invention adopts a new approach: by associating the same target in multiple first-person view and aerial view videos, a multi-view fusion method is proposed to estimate the gaze direction of each target in the aerial view.

[0057] To detect the focus object in multiple videos, we capture N+1 images of length θ captured by N first-person view wearable cameras and top-view drone-mounted cameras. First, a sliding window technique is used to consistently divide all N+1 videos into non-overlapping short-time clips of length T. Then, a human detection algorithm and a multi-view data fusion strategy are used to obtain the focus target candidate image of each frame.

[0058] Denote the kth clip in the nth first-person view as Denote the kth clip in the aerial view as For each window W k , using multi-view video clipping, i.e. and To detect the focus target in the top view, And the corresponding object in the first-person view When there is no confusion, the symbol Simplified to The symbol Simplified to Each clip consists of T frames, and They are the tth frame in the aerial view and the nth first-person view clip respectively.

[0059] In each frame, all detected objects are regarded as a set of focus target candidates, each frame is regarded as a node, and the focus target candidates in the frame are regarded as its state. First, human body detection is performed on each frame of N+1 videos. A multi-view target association algorithm is used to unify the objects detected in different videos, that is, to identify and match the same person in different videos. As the candidate set of the focal target, P t Candidates for focus.

[0060] Next, a conditional random field model is introduced for focus target detection, where each frame is regarded as a node in the conditional random field model and each candidate node is regarded as a state of the node. Through the conditional random field model, a candidate focus target is found in each frame t.

[0061] Estimate the gaze direction of each character. Since the top-view video provides a global view of the entire scene from a high altitude, the present invention can map characters in different first-person perspectives to the high-altitude view, and identify a group of targets in the high-altitude view video. The unified detection of the target is performed and the gaze direction is converted into an aerial view for fusion.

[0062] In order to implement the above method, three problems need to be solved:

[0063] i. Object matching between the aerial view and the first-person view.

[0064] ii. Estimation of the viewing direction of the wearer of the first-person camera when looking down.

[0065] iii. Observe the orientation estimates of all other objects.

[0066] The solution extends the cross-view topic association algorithm to solve problems i and ii. The algorithm is applied to associate objects in each first-person view with the aerial view, and then the aerial view is used as a bridge to build a unified detection of objects across the aerial view and multiple first-person views.

[0067] For problem iii, three direction vectors will be defined:

[0068]

[0069] in Indicates from arrive The angle β and γ can be calculated from the high altitude view. The angle α can be obtained from the given vector That is, the viewing direction of the subject P (parallel to the ground plane) is derived. In this way, It can be converted into the gaze direction estimation in the first-person perspective view. Through the plane geometric transformation, the present invention calculates the viewing angle of the subject P in the top view as

[0070]

[0071] For each target, the method fuses his / her gaze direction estimated from multiple first-person views into the final gaze direction in the top view. Specifically, in order to uniformly detect a target in multiple first-person views, the view with the highest face detection confidence of the target is selected to estimate his / her gaze direction.

[0072] c) In the conditional random field model, based on the estimated gaze direction information, an intra-frame energy is defined to reflect the confidence of the candidate focus target in each frame, and an inter-frame energy is defined to reflect the spatiotemporal consistency between frames. In each frame of each video, the minimum comprehensive energy will produce a focus target.

[0073] Then, the proposed algorithm calculates the energy value of each candidate using the energy function Calculate the energy value of each candidate focus target in each perspective, and combine the multi-perspective energy values ​​to calculate the energy value of the current frame

[0074]

[0075] Where E1 and E2 represent the intra-frame energy term and the inter-frame energy term, respectively, t , P s represents the focus target candidates at frames t and s.

[0076] Based on the built-in confidence E1 of the frame, the view deviation level of the candidate object is defined according to its position in the field of view of other objects. The energy in the coordinate system is constructed by calculating the focus ratio level of the candidate object, that is, the ratio of the number of subjects whose field of view covers the candidate object to the total number of subjects in the scene.

[0077] E1(P t )=-Sig(V(P t )+F(P t )),

[0078] Where Sig represents the sigmoid function, V(·) and P(·) respectively reflect the candidate P t The two energies of viewing angle deviation and focusing ratio level. V(·) is defined as

[0079]

[0080] in The total number of objects detected at frame t is |P t |, represents the viewing direction of object P, Represents the transition from object P to candidate P t vector, P traverses except object P t External In all objects, brackets <·> represent the angle between two vectors. and The angle between is used to estimate the deviation level of the point in the view. We further modify it using the damping function and define F(·) as

[0081]

[0082] Where Ind(·) is the index function. This energy term is used to calculate the field of view containing the candidate P t The ratio of the target, where 2θ is the field of view of the target.

[0083] Since the inter-frame energy E2 depends on the change of the focus target confidence and the temporal consistency across frames, the inter-frame energy is constructed as

[0084] E2(P t ,P s )=-Sig(ΔV(P t ,P s )+ΔF(P t ,P s )+O(P t ,P s ))

[0085] Among them, ΔV(P t ,P s )=V(P s )-V(P t ) represents the view deviation level from P t To P s The change of ΔF(P t ,P s )=ΔF(P s )-F(P t ) represents the change in the focus ratio level. t ,P s ) indicates P t With P s The overlap between them can ensure the spatial consistency from frame t to frame s.

[0086] d) Finally, the present invention merges the focus object detection results of all segments to achieve focus object detection for all frames over time.

Claims

1. A focal target detection method based on air-ground collaborative perspective, characterized by: The method is based on the first-person perspective of a high-altitude camera carried by a drone with a high-altitude perspective and wearable cameras worn by multiple pedestrians on the ground, and utilizes the perspective of air-ground collaboration to detect, associate and fuse the visual attention of multiple people in the scene to achieve the location of the focus target; it includes the following steps: Step S1: Input N+1 time-synchronized videos with a duration of Step S2: Using the sliding window technique, all the videos obtained in step S1 are consistently divided into short segments of duration T; Step S3: using a person detection algorithm and a multi-view data fusion method to obtain a candidate focus target from each frame of the current segment; Step S4: Using energy function Calculate the energy value of each candidate focus target in each perspective, and combine the multi-perspective energy values ​​to calculate the energy value of the current frame Where: E1 and E2 represent the intra-frame energy term and inter-frame energy term respectively, P t , P s represents the focus target candidates of the tth and sth frames; Step S5: When all the segments have not been processed, return to step S3 to identify the next segment; when all the segments have been processed, proceed to step S6; Step S6: The focus target energy value detection results of all segments in time are analyzed together to obtain the detection result.

2. The method for detecting a focal target based on an air-ground collaborative perspective according to claim 1, characterized in that: The process of obtaining the candidate focus target in each frame of the current segment in step S3 is as follows: Denote the kth clip in the nth first-person view as Denote the kth clip in the aerial view as For each window W k , using multi-view video clipping, i.e. and To detect the focus target in the top view, And the corresponding object in the first-person view When there is no confusion, the symbol Simplified to The symbol Simplified to Each clip consists of T frames, and They are the tth frame in the aerial view and the nth first-person view clips, respectively; In each frame, all detected objects are regarded as a set of focus target candidates, each frame is regarded as a node, and the focus target candidates in the frame are regarded as its state; A multi-view target association algorithm is used to unify the objects detected in different videos, that is, to identify and match the same person in different videos; the detected target groups are uniformly represented as As the candidate set of the focal target, P t A conditional random field model is used for focus target detection, where each frame is regarded as a node in the conditional random field model and each candidate node is regarded as a state of the node. Through the conditional random field model, a candidate focus target is found in each frame t; Using the energy function E(P t ,P s |F t ,F s ) to estimate P t ,P s As a focal target, it reflects the credibility of a candidate focal target. The energy function is specifically expressed as Among them, E1 and E2 represent the intra-frame and inter-frame energies, respectively, which characterize the probability of the focus target being in the same frame and in different frames; Based on the built-in confidence E1 of the frame, the view deviation level of the candidate object is defined according to its position in the field of view of other objects; the energy in the coordinate system is constructed by calculating the focus ratio level of the candidate object, that is, the ratio of the number of subjects whose field of view covers the candidate object to the total number of subjects in the scene: E1(P t )=-Sig(V(P t )+F(P t )), Where Sig represents the sigmoid function, V(·) and P(·) respectively reflect the candidate P t The two energies of viewing angle deviation and focusing ratio level; V(·) is defined as in The total number of objects detected at frame t is |P t |, represents the viewing direction of object P, Represents the transition from object P to candidate P t vector, P traverses except object P t External All objects in , brackets <·> represent the angle between two vectors; use and The angle between them is used to estimate the deviation level of the point in the view; Modify using damping function and define F(·) as Where Ind(·) is the index function; this energy term is used to calculate the field of view containing the candidate P t The ratio of the target, θ is the field of view of the target; Since the inter-frame energy E2 depends on the change of the focus target confidence and the temporal consistency across frames, the inter-frame energy is constructed as E2(P t ,P s )=-Sig(ΔV(P t ,P s )+ΔF(P t ,P s )+O(P t ,P s )), Among them, ΔV(P t ,P s )=V(P s )-V(P t ) represents the view deviation level from P t To P s The change of ΔF(P t ,P s )=ΔF(P s )-F(P t ) represents the change of focusing ratio level; O(P t ,P s ) indicates P t With P s The overlap between them can ensure the spatial consistency from frame t to frame s.

Images