High-altitude camera and wearable camera video content joint analysis method

Abstract

The application discloses a high-altitude camera and wearable camera video content joint analysis method, which comprises the following steps: S1, using a target detector to obtain all pedestrian targets in a high-altitude view and an overhead view; S2, calculating a feature vector set a1 representing the spatial position relationship of the targets in the high-altitude view; S3, calculating a feature vector set a2 representing the spatial position distribution of the targets in a side view; S4, establishing a dimensional distance matrix E ij Comparing the feature vector sets a1 and a2 to realize the matching of the targets in the two views; S5, according to the matching result, selecting the optimal camera position and viewing direction of the view according to the following formula; S6, outputting the optimal camera position, viewing direction and the matching result of the targets in the two views corresponding to each frame, and the application uses the images captured by the high-altitude view and the first view to complete the image content joint analysis with large view angle difference.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, and relates to a new type of technology based on cross-view camera content joint analysis between the top view of an aerial camera and the first view of a wearable camera, which can be used in the problem of correlating and jointly analyzing the content between different views when a UAV and a mobile camera simultaneously shoot the same scene. BACKGROUND

[0002] The background technology involved in the present application is as follows:

[0003] (1) Pedestrian target detector (see document [1]): As an important part of data preprocessing, the pedestrian target detector has been widely applied to various fields such as target tracking, pedestrian repositioning and action recognition. The commonly used pedestrian detector can be divided into two categories: background modeling based method and statistical learning based method. The former must adapt to the changes of the environment, and is limited by the picture shaking caused by camera shaking. The latter is the commonly used method for pedestrian detection, which constructs a large number of samples to construct a pedestrian detection classifier. At present, the role of deep learning in pedestrian detection cannot be ignored. Among them, the YOLO detector is used in the present application to provide the corresponding target detection frame.

[0004] (2) Aerial camera and wearable camera: The progress of mobile camera technology provides a new solution for video monitoring. A UAV can provide a high-altitude overhead image of a scene on the ground. A wearable camera such as a mobile phone can provide a first-view view of the same scene. The present application proposes a new aerial camera and wearable camera content joint analysis method, in which there is an aerial overhead camera in the air and a first-view camera on the ground. Since the two cameras can complement each other well, it can provide wider coverage and flexibility for outdoor monitoring. The joint analysis of the above two cameras can be effectively applied to tasks such as human tracking and activity recognition. SUMMARY

[0005] The purpose of the present application is to propose an effective camera joint analysis technology for cross-view video, which uses images shot by a high-altitude view and a first view (at the same time) of the same scene to complete the joint analysis of image content with large view angle difference, specifically including three subtasks: ① determining the position of the first-view camera in the aerial camera, ② estimating the observation direction of the first-view camera in the aerial camera view, and ③ completing the multi-pedestrian target matching of the high-altitude view and the first-view view.

[0006] To this end, the technical solution adopted by the present application is as follows:

[0007] An aerial camera and wearable camera video content joint analysis method, comprising the following steps:

[0008] S1, obtaining all pedestrian targets in the high-altitude view and the overhead view by using a target detector;

[0009] S2, calculating a feature vector set a1=(x1, y1) representing the spatial position relationship of the targets in the high-altitude view;

[0010] S3, calculating a feature vector set a2=(x2, y2) representing the spatial position distribution of the targets in the side view;

[0011] S4, establishing a dimensional distance matrix E ij comparing the feature vector sets a1 and a2 to realize the matching of the targets in the two views;

[0012]

[0013] wherein E ij represents the distance between vectors and , i.e., the distance between the i-th target in the high-altitude view and the vector represented by the j-th target in the side view, and λ represents a preset parameter. Based on the above distance measurement, a one-to-one matching relationship of the targets in the two views is established by using a dynamic programming method, and the obtained matching vector sets are respectively denoted as

[0014] S5, selecting the optimal camera position and viewing direction according to the following formula based on the matching result:

[0015]

[0016] wherein K represents the larger value of the number of targets in the two views, n is the number of the final matching pairs, ρ is a preset parameter, ||·||1 represents the 1-norm of a vector, and A1 and A2 represent the sets of the feature vectors a1 and a2 of all targets in the images, respectively.

[0017] S6, outputting the optimal camera position, viewing direction, and the matching result of the multiple targets in the two views corresponding to each frame.

[0018] 2. The method according to claim 1, wherein the feature vector set a1 representing the spatial position relationship of the targets in the high-altitude view is calculated in step S2.

[0019] 201, the position A and the viewing direction θ of the camera in the overhead view are known, and the field of view range of the high-altitude view and the positions of all pedestrian targets relative to the camera are calculated as follows:

[0020] the left and right boundaries of the field of view range of the side view camera are obtained given A and θ and For a target D in the view angle: the relative position of the vector is represented as:

[0021]

[0022] Wherein: f is the built-in parameter of the camera, is the right direction of the camera (indicating the direction vector perpendicular to the right of the camera viewing direction θ);

[0023] 202, the range of x is estimated by the left and right boundaries of the camera view angle:

[0024]

[0025] Wherein: α represents the viewing range of the first view angle camera, usually 90 degrees, and respectively represent the direction vectors of the left and right field of view boundaries of the first view angle camera;

[0026] 203, the normalized processing obtains the coordinate value corresponding to each target as:

[0027]

[0028] 204, according to the value of x1 from small to large, the spatial feature vector set corresponding to all targets in the overhead view angle is obtained:

[0029]

[0030] Wherein: |S| represents the number of targets, respectively represent the set of x1, y1 corresponding to all targets, and correspondingly, A1 represents the set of all target feature vectors a1.

[0031] 3. The method according to claim 1, wherein the feature vector set a2 representing the spatial position distribution of the side view angle target in step S3 is calculated as:

[0032] 301, for the image of the first view angle, the position of the upper left corner of the image is taken as the coordinate origin, and b represents the width of the picture. For a target in the field of view, represents the position of the target, and h represents the height of the target;

[0033] 302, by normalizing the horizontal dimension, the coordinates corresponding to each target are:

[0034]

[0035] 303, according to the value of x2 from small to large, the spatial feature vector set corresponding to all targets in the side view angle can be obtained:

[0036]

[0037] Wherein: M is the number of targets; Respectively, A2 represents the set of all target feature vectors a2.

[0038] Beneficial effects

[0039] 1. The application can be used for the correlation and joint analysis of the content between different angles in the case of simultaneous shooting of the same scene by the unmanned aerial vehicle and the mobile camera through the new technology of joint analysis of the cross-angle camera content between the top view of the high-altitude camera and the first view of the wearable camera.

[0040] 2. The application cooperatively analyzes the data of different angles, and is applied to target tracking, pedestrian matching, action recognition and other tasks; the application comprises two parts, the first part is to estimate the spatial position relationship of the target under the side view angle, detect all targets, estimate the depth and spatial position relationship of the target by using the size and position of the detected target, and establish a feature vector; the second part is to estimate the spatial position relationship in the corresponding high-altitude view. On the corresponding high-altitude view image, the position of the side view camera and its observation angle are located by traversing each detected object and possible direction; for each traversed position and direction, the spatial position relationship of all visible objects is estimated, and a feature vector is established; the targets in the two angles are matched by defining a matching function of the spatial position relationship between the targets, and then the position and angle of the horizontal camera are obtained.

[0041] 3. The effectiveness of the application is verified by the experimental results of the joint analysis of the target correlation and camera pose estimation of the high-altitude view and the first view image.

[0042] 4. On the basis of the existing computer vision problem research, the application selects the basis of target tracking, pedestrian matching and other problems for research, and independently designs a cross-angle multi-target matching method based on spatial position relationship, effectively solves the joint analysis problem between the high-altitude view and the side view angle, and achieves good results. Specifically, the application has the following advantages:

[0043] a. High applicability. The application first studies the correlation problem between the picture of the nearly vertically downward high-altitude view and the picture of the side view angle, and can be effectively applied to the system of the multi-angle camera system, and provides a basis for target tracking, pedestrian matching, action recognition and other work.

[0044] b. Better matching effect. The application solves the cross-view angle correlation problem caused by the fact that the appearance features and motion features are not obvious under the condition of large view angle difference by vectorizing the spatial distribution of the target, and better matching effect is achieved.

[0045] c. Higher operation efficiency. The algorithms involved in the application are simple low-dimensional vector representation, which has low requirements for the running environment and can run in ordinary CPU environment, and the algorithm speed is fast, which can achieve real-time (> 30fps) running speed. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 The high-altitude camera and the content joint analysis operation flowchart of the wearable camera involved in the application are shown in the figure.

[0047] Figure 2 The cross-view angle target matching schematic diagram under large view angle difference in the application is shown in the figure. DETAILED DESCRIPTION

[0048] The cross-view angle matching problem studied in the application is the research on high-altitude view angle and side view angle video, and the specific operation process is as shown in the figure. Figure 1 The process includes five parts.

[0049] Step S1: target detection. The method of deep learning which is widely used at present is used to complete this operation. Specifically, YOLOv3 target detector is used to detect all pedestrian targets under two view angles at the same time.

[0050] Step S2: calculate the feature vector set representing the spatial position distribution of the high-altitude view angle target.

[0051] Step S3: calculate the feature vector set representing the spatial position distribution of the side view angle target.

[0052] Step S4: match the targets of the two view angles by comparing the two feature vector sets.

[0053] Step S5: select the optimal camera position and view angle observation direction.

[0054] Finally, the optimal camera position, view angle direction and matching result of the two view angle multi-target corresponding to each frame are obtained.

[0055] The application specifically includes two aspects of content, one is the definition of the problem, and the other is the target matching method based on the above definition.

[0056] (1) Problem definition: given the high-altitude view angle (top view angle) picture and the first view angle (side view angle) picture taken by the unmanned aerial vehicle and the wearable camera at the same time, Figure 2As shown, first, all person targets in the image are detected by a pedestrian detector. The goal of the present patent is to match the targets in two views while completing the estimation of the first view camera position and viewing direction. Since the two views are very different, resulting in poor matching of appearance features, motion features, etc., the present invention uses the spatial position relationship of the target to extract features for target matching. In short, according to the spatial position of each detected target, a feature vector based on the mutual spatial relationship of multiple targets is constructed, and the targets are matched by comparing the vectors.

[0057] (2) Target re-identification method: the target matching method of the present invention matches the targets of the same identity in two views by exploring the spatial position relationship of the targets. The flowchart of the cross-view matching method based on spatial position distribution is as follows Figure 2 The operation steps are as follows:

[0058] Step S1: target detection. The present patent uses the deep learning method widely used at present to complete this operation. Specifically, YOLO v3 target detector is used to detect all pedestrian targets in two views. Respectively represented as and The former represents a set of N targets in the high-altitude view picture, and the latter represents a set of M targets in the side view picture.

[0059] Step S2: calculate the feature vector set a1 of the spatial position relationship of the high-altitude view target. First, assume that the position A and viewing direction θ of the first view camera in the overhead view are known. From this, the high-altitude view field range and the position of all pedestrian targets relative to the camera are calculated, as follows.

[0060] Given A and θ, the left and right boundaries of the side view camera field of view can be easily obtained and For a target D (take the center point of the target detection rectangular frame) in the view, its relative position vector is represented as:

[0061]

[0062] where f is the built-in parameter of the camera, is the right direction of the camera (indicating the direction vector perpendicular to the right of the camera viewing direction θ). The range of x can be estimated by the left and right boundaries of the camera view as:

[0063]

[0064] where α represents the viewing range of the first view camera, usually 90 degrees, and respectively represent the direction vectors of the left and right field of view boundary of the first perspective camera. In order to better compare, first normalize the processing, and the coordinate value corresponding to each target after processing is:

[0065]

[0066] According to the value of x 1 , the spatial feature vector set corresponding to all targets of the high-altitude perspective can be obtained by sorting from small to large:

[0067]

[0068] Where |S| represents the number of targets, respectively represent the set of x1, y1 corresponding to all targets.

[0069] Step S3: Calculate the feature vector set a2 representing the spatial position distribution of the target of the side view perspective. For an image of the first perspective, the method uses the position of the upper left corner of the image as the coordinate origin, and b represents the width of the picture. For a target in the field of view, represents the position of the target, and h represents the height of the target. At the same time, considering the normalization of the horizontal dimension, the coordinates corresponding to each target are:

[0070]

[0071] According to the value of x2 from small to large, the spatial feature vector set corresponding to all targets of the side view perspective can be obtained:

[0072]

[0073] Where M is the number of targets. respectively represent the set of x2, y2 corresponding to all targets.

[0074] Step S4: Match the targets of the two perspectives by comparing the feature vector sets a1 and a2. First, establish a distance measure E with a dimension of |S|×M.

[0075]

[0076] E ij represents the distance between vectors and . By calculating the vector corresponding to a target of a high-altitude perspective or a first perspective matched to multiple vectors in another perspective, the minimum distance is selected as the matching item. In order to solve this matching problem, the patent uses dynamic programming to obtain the optimal matching between the feature vectors and of all targets. Through this method, it is guaranteed that the matching result is one-to-one.​

[0077] Step S5: The first perspective camera is assumed to be in the high-altitude view position A and the perspective viewing direction θ is known. Assuming that the number of target matching in two perspectives is n, the one-to-one result of two perspective target matching (n pairs) obtained by step S4 is respectively The comprehensive matching function is defined as:

[0078]

[0079] Wherein λ>1 is a preset parameter. The optimal camera position A and perspective range θ are obtained by the energy function. The strategy is to loop through all possible positions and directions, and select the optimal camera position A and perspective range θ according to the value of the energy function.

[0080] On the basis of existing computer vision problem research, the present application selects the basis of target tracking, pedestrian matching and other problems for research, and independently designs a cross-perspective multi-target matching method based on spatial position relationship, effectively solves the joint analysis problem between high-altitude perspective and side view perspective, and achieves good results. Specifically, the present application has the following advantages:

[0081] a. High applicability. The present application first studies the correlation problem between the approximately vertically downward high-altitude perspective picture and the side view perspective picture, can be effectively applied in the multi-perspective camera system, and provides a basis for target tracking, pedestrian matching, action recognition and other work.

[0082] b. Good matching effect. The present application solves the cross-perspective correlation problem in the case that the appearance feature and motion feature are not obvious due to large perspective difference by vectorizing the spatial division vector of the target, and achieves good matching effect.

[0083] c. High running efficiency. The algorithm involved in the present application is a simple low-dimensional vector representation, which has low requirements for the running environment and can run in ordinary CPU environment, and the algorithm speed is fast, which can achieve real-time (>30fps) running speed.

Claims

1. A method for joint analysis of video content from a high-altitude camera and a wearable camera, characterized in that: Includes the following steps: S1 uses a target detector to acquire all pedestrian targets in both aerial and overhead views; S2 calculates the set of feature vectors representing the spatial positional relationship of targets from a high-altitude perspective. ; S3 calculates and obtains the set of feature vectors representing the spatial position distribution of the target from the side-view perspective. ; S4 establishes a dimensional distance matrix. Compare and The feature vector set enables the matching of targets from two perspectives; ； in Representing vectors =( , )and The distance between them, i.e., the distance from the high-altitude perspective. i The first target and the side view perspective j The distance between the vectors representing the targets. Let represent a preset parameter; based on the above distance metric, we use dynamic programming to establish a one-to-one matching relationship between targets from two perspectives, and the resulting matching vector sets are denoted as . , ; S5 selects the optimal camera position and viewing direction based on the matching results according to the following formula; ; in: This represents the larger value of the number of targets in the two viewpoints. It is the number of the final matched pairs. These are preset parameters. Represents the 1-norm of a vector; , These represent all target feature vectors in the image. A set; S6 outputs the optimal camera position, viewpoint direction, and multi-target matching results for each frame.

2. The method for joint analysis of video content from a high-altitude camera and a wearable camera according to claim 1, characterized in that: The feature vector set of the target spatial position relationship from the high-altitude perspective in step S2 Calculation process:

201. The position of the first-person view camera in the top view is known. and perspective of observation Therefore, the field of view from the high altitude and the positions of all pedestrian targets relative to the camera are calculated as follows: Given and Obtain the left and right boundaries of the field of view of the side-view camera. and For a certain target in the viewpoint The vector representation of their relative positions is: ； in: These are the camera's built-in parameters. To the right of the camera; 202 Estimation of left and right boundaries from the camera's perspective The scope is: ； in: This indicates the field of view of a first-person perspective camera, typically 90 degrees. The coordinate values ​​for each target are obtained by normalization (203). ； 204 According to The values ​​are sorted from smallest to largest to obtain the set of spatial feature vectors corresponding to all targets from the high-altitude perspective: ； in: Indicates the number of targets. Representing all targets respectively The set, accordingly, Represents all target feature vectors A set of.

3. The method for joint analysis of video content from a high-altitude camera and a wearable camera according to claim 1, characterized in that: The set of feature vectors representing the spatial location distribution of the target from the side-view perspective in step S3 Calculation process: 301 For first-person perspective images, the origin of the coordinate system is the top left corner of the image. This represents the width of the image relative to a specific target within the field of view. This indicates the location of the target. Indicates the height of the target; 302. After normalizing the horizontal dimension, the coordinates of each target are: ； 303 According to Sorting the values ​​in ascending order yields the set of spatial feature vectors corresponding to all targets from the side-view perspective: ； in: The number of targets; Representing all targets respectively The set, accordingly, Represents all target feature vectors A set of.

Images