A camera patching method for global tracking

Abstract

The application discloses a camera supplementing method for global tracking, comprising the following steps: constructing a three-dimensional model of a region and projecting to generate a two-dimensional base map grid coordinate system; solving a homography matrix of an existing camera according to a mapping relationship between acquisition data of the existing camera and the three-dimensional model; calculating an accurate sensing area of the existing camera, and combining lattice areas with continuous relationship to form a closed sensing area; reversely deducing a plurality of blind areas to be supplemented which need to be monitored based on the obtained closed sensing area; calculating an average sensing range of the existing camera, calculating a minimum number of camera supplementing points for each blind area to be supplemented, and conducting control. The application has the beneficial effects that the relationship between the camera visual angle and the space range can be effectively restored in the case of lacking detailed external parameters, and then the blind area range is judged; and the original monitoring system can complete the judgment and supplement of the visual dead angle without replacing the hardware.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent transportation, in particular to a camera supplementing method for global tracking. BACKGROUND

[0002] In a semi-closed scene, such as a highway service area scene, as a key node in the traffic system, it undertakes multiple functions such as temporary parking of vehicles, energy supply and emergency management. In order to realize the daily management and safety protection of the service area, a large number of monitoring cameras are usually laid out during the design and construction stage. However, these cameras often aim at regional monitoring, and the demand for vehicle "global tracking" is not systematically considered.

[0003] With the increasing demand for vehicle dynamic management, path reconstruction and global perception at all times, relying only on existing cameras cannot meet the requirements of spatio-temporal continuity and spatial accuracy in actual scenes. Specifically, the existing camera layout has the following problems:

[0004] Discontinuous spatial coverage: Due to the lack of initial planning for full coverage monitoring in some service area regions, there are monitoring blind spots, and the vehicle entry and exit trajectory is broken.

[0005] Inaccurate frame mapping: Although some areas are visually covered by cameras, due to factors such as imaging distortion, angle deviation or excessive height, the image content is difficult to accurately map to the real world coordinates, affecting three-dimensional reconstruction and multi-camera tracking.

[0006] Weak tracking ability: In the absence of camera internal and external parameters, it is difficult to accurately match image frames and physical space, directly restricting the continuous tracking ability of targets across angles and devices. SUMMARY

[0007] One of the purposes of the present application is to provide a camera supplementing method for global tracking that can solve at least one of the defects in the background art.

[0008] To achieve the above at least one purpose, the technical solution adopted by the present application is: a camera supplementing method for global tracking, comprising the following steps:

[0009] S100: constructing a three-dimensional model of the region and projecting to generate a two-dimensional base map, performing grid division on the two-dimensional base map and marking to obtain a grid coordinate system identifying each grid region;

[0010] S200: according to the mapping relationship between the acquisition data of the existing cameras in the region and the three-dimensional model, solving a homography matrix for estimating the external parameters of the existing cameras in the region;

[0011] S300: Calculate the accurate perception area based on the existing camera in the grid coordinate system according to the obtained homography matrix, and combine the grid areas with continuous relationship in the accurate perception area to form a closed perception area.

[0012] S400: Reverse the un-accurately covered area in the grid coordinate system based on the obtained closed perception area; classify the un-accurately covered area to obtain a plurality of to-be-supplemented blind areas that need to be monitored.

[0013] S500: Calculate the average perception range of the existing camera, calculate the minimum number of camera supplement points through the horizontal and vertical span of each to-be-supplemented blind area, and perform control.

[0014] Preferably, step S200 includes the following process: selecting a static picture with empty space in a frame picture of data collected by each existing camera; selecting a plurality of representative feature point image points in the static picture, and determining a plurality of plane points corresponding to the representative feature points in the two-dimensional base map; under the condition that the image point coordinates and the plane point coordinates are known, calculating a matrix with the smallest error sum of squares of the image points and the plane points as the homography matrix H through the least square method, and the specific expression is as follows:

[0015] ;

[0016] wherein, represents the homogeneous coordinates of the image points, represents the homogeneous coordinates of the plane points, represents the third component of the matrix multiplication result, and n represents the number of selected representative feature points.

[0017] Preferably, step S300 includes the following process: selecting a plurality of groups of image points and corresponding three-dimensional points as verification points from the static picture and the three-dimensional model for the grid area corresponding to the perception area of each existing camera; calculating the Euclidean distance error ε j between the image points converted through the homography matrix H and the three-dimensional points in each group of verification points.

[0018] ;

[0019] If the Euclidean distance error ε j of the plurality of groups of verification points corresponding to the grid area is less than a set threshold value, the grid area is marked as an accurate perception area; wherein P j represents the coordinates of the three-dimensional points in the verification points.

[0020] Preferably, the step S400 comprises the following process: according to the usage function of different positions in the region, dividing the region not accurately covered into a region needing to be accurately monitored and a region not needing to be accurately monitored; identifying the grid region corresponding to the region needing to be accurately monitored; and if it is identified that there is a continuous grid region not accurately covered, aggregating the continuous grid region not accurately covered to form a region to be supplemented.

[0021] Preferably, in the step S500, the calculation of the average sensing range of the existing camera comprises the following process: according to the mapping relationship between the collection data of the existing camera and the three-dimensional model, solving the projection matrix of each existing camera; estimating the approximate position of the existing camera in the space coordinate system from the projection matrix to obtain the approximate space center point of the existing camera; calculating the maximum horizontal distance from the approximate space center point to the boundary of the grid region farthest from the sensing range in the grid coordinate system as the horizontal effective sensing radius of the current existing camera; and taking the average value of the horizontal effective sensing radii of all the existing cameras as the radius of the average sensing range of the existing camera.

[0022] Preferably, the solving of the projection matrix comprises the following process: selecting N groups of mutually corresponding image points and three-dimensional points from the collection data and the three-dimensional model, and constructing the expression of the projection matrix V by the direct linear transformation method as follows:

[0023] ;

[0024] By minimizing the projection error:

[0025] ;

[0026] Using the SVD decomposition method to fit the multiple groups of points to solve the projection matrix ;

[0027] wherein, (u i , v i ) represents the image point coordinates in the collection data of the existing camera, (X j , Y j , Z j ) represents the three-dimensional point coordinates of the corresponding image points in the three-dimensional model, s represents a scale factor, 、 and respectively represent the first, second and third components of the multiplication result of the projection matrix V and the homogeneous coordinates of the three-dimensional point , M represents a rotation and internal parameter coupling matrix, and p4 represents translation information.

[0028] Preferably, the process of calculating and deploying the minimum number of camera supplement points in step S500 is as follows: the horizontal span and vertical span of the blind area to be supplemented are obtained, and the ratio N of the average perception range of the existing camera is calculated kw and N kh ; for the blind area to be supplemented, N kw and N kh are less than or equal to 1, a single-point simulation deployment method is used for supplement; for the blind area to be supplemented, N kw and / or N kh are greater than 1, a multi-point cooperative deployment method is used for supplement.

[0029] Preferably, the specific method of single-point simulation deployment is as follows: each peripheral edge grid of the blind area to be supplemented is selected as a candidate installation point; according to the theoretical effective perception range of the existing camera, the point in the candidate installation point whose theoretical effective perception range can completely cover the blind area to be supplemented and whose overlapping area with the surrounding accurate perception area is the largest is selected as the final deployment point.

[0030] Preferably, the theoretical effective perception range of the existing camera is calculated by the intrinsic parameters, installation height and pitch angle range; the acquisition of the pitch angle range of the existing camera includes the following process: after the calculation of the accurate perception area in step S300, all pixel point sets corresponding to the accurate perception area are extracted from the image collected by the existing camera; according to the resolution information of the existing camera, the distribution position of the pixel point set in the image is estimated to obtain the proportion and angle range of the pixel point set in the image, and then the pitch angle range of the existing camera is calculated.

[0031] Preferably, the specific method of multi-point cooperative deployment is as follows: the main deployment direction is determined according to the proportional relationship between the horizontal span and the vertical span of the blind area to be supplemented; starting from the boundary of the two ends of the main direction, the camera combination deployment is simulated to obtain a plurality of deployment candidate point sets; according to the theoretical effective perception range of the existing camera, the coverage range of each deployment candidate point set is obtained; the deployment candidate point set that can completely cover the blind area to be supplemented and has the largest overlapping area with the surrounding accurate perception area is selected as the final deployment point.

[0032] Compared with the prior art, the application has the following advantages:

[0033] The application provides a mechanism for calculating the installation parameters of the existing camera and reconstructing the visual angle mapping area, so that in the absence of detailed external parameters, the relationship between the camera visual angle and the space range can also be effectively restored, and then the blind area range is determined. This mechanism enables the original monitoring system to not need to replace the hardware, and only needs to supplement modeling and analysis on the image side to complete the judgment and completion of the visual dead angle. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 The whole working procedure of the present application is schematically shown. DETAILED DESCRIPTION

[0035] In the description of the present application, it should be noted that the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. The illustrative description of the above terms in the present specification should not be understood as necessarily referring to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in the present specification.

[0036] In the description of the present application, it should be noted that for the orientation words such as the terms "center", "transverse", "longitudinal", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise" and the like indicate the orientation and positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and cannot be understood as limiting the specific protection scope of the present application.

[0037] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence.

[0038] In the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connection", "fixing" and the like should be understood in a broad sense, for example, it can be connected, or detachable, or integrated; it can be mechanical connection, or electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, it can be the internal communication of two elements or the interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0039] In the present application, unless specifically defined and limited otherwise, the first feature is "on" or "under" the second feature can include that the first and second features are in direct contact, or that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, the first feature is "on", "above" and "over" the second feature includes that the first feature is directly above and obliquely above the second feature, or only indicates that the first feature is higher in horizontal height than the second feature. The first feature is "under", "below" and "underneath" the second feature includes that the first feature is directly below and obliquely below the second feature, or only indicates that the first feature is lower in horizontal height than the second feature.

[0040] The terms "comprising" and "having" and any variations thereof in the specification and claims of the present application are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or apparatus that includes a list of steps or units not necessarily limited to those clearly listed, but can include other steps or units not clearly listed or inherent to such processes, methods, products or apparatuses.

[0041] One of the preferred embodiments of the present application, as shown in Figure 1 A camera filling method for global tracking, comprising the following steps:

[0042] S100: Construct a three-dimensional model of the region and project it to generate a two-dimensional base map, divide the two-dimensional base map into grids and mark it to obtain a grid coordinate system identifying each grid region.

[0043] S200: According to the mapping relationship between the existing camera acquisition data in the region and the three-dimensional model, the homography matrix for estimating the extrinsic parameter of the existing camera in the region is solved.

[0044] It can be understood that since the existing cameras in the region are mostly deployed early, there are problems such as non-uniform model and installation position offset, so the original intrinsic parameter matrix K and extrinsic parameter matrix R may not be accurately recovered; therefore, a reverse method is needed to reconstruct the spatial extrinsic parameter of the existing camera.

[0045] S300: According to the obtained homography matrix, the accurate perception region based on the existing camera in the grid coordinate system is calculated, and the grid regions with continuous relationship in the accurate perception region are combined to form a closed perception region.

[0046] It can be understood that, according to the existing camera external parameter obtained by reverse deduction, the current camera can accurately identify the region to obtain the accurate perception region, and there are three cases; one case is that the accurate perception region corresponds to the grid region which is continuous; another case is that the accurate perception region corresponds to the grid region which is partially continuous and partially discrete; and the third case is that the accurate perception region corresponds to the grid region which is discrete. For the grid region with discrete relationship, after the vehicle passes through the region, the trajectory loss is easy to occur; therefore, only the grid region with continuous relationship is combined to form a closed perception region, and the grid region with discrete relationship can be perceived by subsequent point supplement.

[0047] For the determination of the continuous relationship between the grid regions, the interval distance can be used for judgment. Specifically, if the interval distance between the centers of any two grid regions in the accurate perception region is less than the set distance threshold, it can be determined that there is a continuous relationship between the two grid regions. The specific value of the distance threshold is preferably 5m to 15m, which can be determined by the actual needs of those skilled in the art. In order to facilitate understanding, a specific example will be described below.

[0048] Taking a service area as an example, the specific value of the distance threshold is 10m, the size of the grid region is 3m x 3m, and the length of an ordinary car is generally 2.8m. Therefore, when the target vehicle passes through the two grid regions with continuous relationship, the distance length that is not recognized by the camera is 4.8m. It can be assumed that the driving speed of the target vehicle in the service area is 15km / h, so the time when the vehicle is not recognized by the camera is about 1.15s. Generally, in the recognition range of a single camera, if the vehicle disappears for more than 1.5s, it will be judged that the vehicle trajectory is lost; that is, in this example, when the target vehicle passes through the two grid regions with continuous relationship, the vehicle trajectory can be recognized, and the trajectory of the un-recognized region can be automatically completed by interpolation or other methods. The specific trajectory completion method is known to those skilled in the art, and therefore will not be described in detail here.

[0049] S400: Based on the obtained closed perception region, the un-accurately covered region in the grid coordinate system is deduced; the un-accurately covered region is classified to obtain a plurality of to-be-supplemented blind areas that need to be monitored.

[0050] It can be understood that different positions in the region correspond to different functions. For example, the internal region of the service area can be divided into parking area, road area, green area and building area, etc. Among them, the parking area and the road area belong to the area that needs to be accurately monitored, and the green area and the building area are suitable for the area that does not need to be accurately monitored; therefore, when calculating the to-be-supplemented blind area, only the parking area and the road area need to be considered.

[0051] S500: Calculate the average perception range of the existing camera, calculate the minimum number of camera supplement points through the horizontal and vertical span of each to-be-supplemented blind area, and perform control.

[0052] It can be understood that based on the above technical solution, the application provides a mechanism for calculating the installation parameters of the existing camera and reconstructing the visual angle mapping area, so that in the case where there is a lack of detailed external parameters of the existing camera, the relationship between the existing camera and the space range can also be effectively restored, and then the blind area range in the region can be judged. This mechanism makes the original monitoring system not need to replace the hardware, and only needs to supplement modeling and analysis on the image side, so as to complete the judgment and completion of the visual dead angle, which is a very cost-effective system expansion scheme.

[0053] By mapping the accurate perception area confirmed in the image to the BEV visual angle in the real world, an accurate connection from the image plane to the spatial coordinate system can be established. This process provides reliable coordinate basis for subsequent camera supplement, radar connection, point cloud registration, etc., effectively improving the spatial consistency of multi-sensor collaborative perception and the accuracy of target completion. According to the mapping between the pixel accurate perception area and its BEV space projection range, information extraction and matching are limited within the accurate perception area, avoiding incorrect association caused by inaccurate pixels, and improving the accuracy of multi-source sensor data fusion.

[0054] In this embodiment, the construction of the three-dimensional model in step S100 can be realized by photogrammetry technology combined with GNSS control point data measured on site; common photogrammetry technologies include multi-view stereo reconstruction and SfM, etc., which can be selected by the person skilled in the art according to the actual needs. Among them, the three-dimensional world coordinates (X w , Y w and Z w ) provided by the GNSS control point can be used to convert the original three-dimensional model from the model coordinate system to the real geographic space coordinate system of the region. The obtained three-dimensional model can be projected in the BEV visual angle to generate a two-dimensional base map.

[0055] For the grid division of the two-dimensional base map, the size of the unit grid can be selected by the person skilled in the art according to actual needs, for example, the size of the unit grid can be 2m*2m or 3m*3m. The divided grid can be encoded in the form of a combination of letters and numbers. For example, the horizontal grid can be numbered according to letters A to Z. If the number of horizontal grids exceeds the number of single letters, the letters can be numbered in a repeated form, such as AA or AB, etc. The vertical grid can be numbered according to numbers 1 to N, and finally a grid coordinate system with a unique identifier such as A1 and B3 is formed for each grid area.

[0056] In this embodiment, the specific process of reconstructing the existing camera extrinsic parameter by the reverse method in step S200 is as follows: selecting a static picture in a frame of picture collected by each existing camera, taking a service area as an example, the static picture is a picture under the condition of no vehicle or less vehicle occlusion. In the static picture, image points of a plurality of representative feature points are selected, and plane points corresponding to the representative feature points in the two-dimensional base map are determined. For the representative feature points, corner points or edge points can be selected. In the case that the coordinates of the image points and the coordinates of the plane points are known, the matrix with the minimum sum of error squares of the image points and the plane points is calculated as the homography matrix H by the least square method, and the specific expression is as follows: and the plane point coordinates

[0057] .

[0058] It can be understood that the meaning of the above expression is that, among all possible matrices H, find a matrix that can make the sum of error squares of the projection points and the true image points minimum, and this optimal matrix is the required homography matrix H. Wherein, represents the homogeneous coordinates of the image points, represents the homogeneous coordinates of the plane points, represents the third component of the matrix multiplication result, and n represents the number of selected representative feature points.

[0059] It should be noted that, in order to ensure the stability and accuracy of the obtained homography matrix H, a plurality of new image points and corresponding three-dimensional points can be additionally selected from the collected data of each existing camera to evaluate the error of the solution result of the homography matrix H. Specifically, the image points selected additionally can be mapped to the three-dimensional space coordinate system by the solved homography matrix H to obtain the mapping point positions of the image points in the three-dimensional space coordinate system. The distance error between the obtained mapping point positions and the actual corresponding three-dimensional points is analyzed. If the error is less than a set threshold, it means that the accuracy of the solved homography matrix H meets the requirements, otherwise the solution result of the homography matrix H is updated by more groups of image points and corresponding three-dimensional points. ​

[0060] In this embodiment, after obtaining the more accurate homography matrix H, the accurate sensing range of the existing camera can be identified by the accuracy of the homography matrix H. The specific calculation process of the accurate sensing area corresponding to each existing camera in step S300 is as follows: selecting multiple groups of image points and corresponding three-dimensional points as verification points from the grid area corresponding to the sensing area of each existing camera in the static picture and the three-dimensional model; calculating the Euclidean distance error ε j between the image points in each group of verification points after conversion by the homography matrix H and the three-dimensional points. j The expression of the Euclidean distance error ε

[0061] ; wherein P j represents the coordinates of the three-dimensional points in the verification points.

[0062] According to the above expression, if the Euclidean distance error ε j of the multiple groups of verification points corresponding to the grid area is less than the set threshold value, the grid area is marked as an accurate sensing area.

[0063] For the convenience of understanding, the following can be explained in detail through specific examples.

[0064] Suppose the grid area corresponding to the sensing area of a certain existing camera is A1 to A3, B1 to B3, and C1 to C3, i.e. the sensing area of the camera can cover 9 grid areas. Then multiple groups of image points and corresponding three-dimensional points as verification points can be selected from each grid area, the image points come from the camera's acquisition data, and the three-dimensional points come from the three-dimensional model; the number of groups of verification points corresponding to each grid area can be selected by the actual needs of those skilled in the art, for example, 20 groups or 30 groups can be selected.

[0065] For each group of verification point data, the image points can be converted from two-dimensional coordinate points to three-dimensional space coordinate points by the homography matrix H, and then the obtained three-dimensional space coordinate points are calculated with the three-dimensional points in the verification points. The Euclidean distance error ε j . The specific value of the threshold value for judging the Euclidean distance error ε j can be selected by the actual needs of those skilled in the art, such as the threshold value of 1m. Therefore, when the Euclidean distance error ε j corresponding to all groups of verification points in the grid area is less than 1m, it can be considered that the grid area is an accurate sensing area; if the corresponding Euclidean distance error ε j of multiple groups of verification points in the grid area is greater than or equal to 1m, the grid area can be marked as a conversion failure area, and the corresponding picture area is not included in the effective sensing range.

[0066] After the identification of the accurate perception range of all the grid areas of the plurality of existing cameras in the region is completed, a closed perception area can be obtained according to the continuity of the grid areas. The continuity of the grid areas can be determined by a convex hull analysis method. The obtained closed perception area has the following properties: it can continuously cover most of the accurate perception area; its shape and orientation can reflect the main perception direction under the current existing camera pose; and it can be used as a basic reference for high-quality coverage areas in subsequent point optimization.

[0067] Specifically, according to the obtained closed perception area, the frame area of each existing camera is labeled and classified in the grid coordinate system under the BEV perspective. Finally, the entire grid coordinate system is divided into two types of areas. One type is the closed perception area in which each existing camera can accurately identify and the grid areas are continuous. The other type includes the accurately identified areas that are in a discrete state, the conversion failure areas corresponding to each existing camera, and the areas that are not covered by all existing cameras. This type of area can be marked as an accurately uncovered area.

[0068] In this embodiment, according to the use functions of different positions in the region during step S400, the accurately uncovered area can be divided into an accurately monitored area and an accurately monitored area. The grid area corresponding to the accurately monitored area is identified. If it is identified that there is a continuous accurately uncovered grid area, the continuous accurately uncovered grid area is aggregated to form a to-be-supplemented blind area.

[0069] It can be understood that the discrete accurately uncovered grid area in the accurately monitored area is generally located near the closed perception area, that is, the grid area belongs to the transition area of the adjacent closed perception area. Therefore, the missing range of the grid area is small, and the time of the vehicle passing through is short. That is, after the vehicle passes through the discrete accurately uncovered grid area, it can be quickly captured by the surrounding cameras and the trajectory can be completed in time. Therefore, it is not necessary to perform perception completion on the grid area. More simply, for a discrete single accurately uncovered grid area, its area is very small relative to the perception range of the entire camera, and the area ratio is generally less than 1%. In the case that the surrounding cameras can quickly identify and complete the trajectory, the cost performance of setting an additional camera for perception coverage is very low.

[0070] It should be known that when the camera is supplemented, the camera used for supplementing can be a new model camera or a backup camera of the same model as the original camera. For supplementing with a new model camera, its intrinsic and extrinsic parameters are clear, and its perception coverage range can be accurately known. However, due to the difference in model, the new model camera and the original camera in the region may have differences in intrinsic parameters, imaging, time alignment and the like, thereby affecting the subsequent data fusion process of the regional monitoring system. Supplementing with a backup camera of the same model as the original camera can ensure the stability of the subsequent data fusion of the regional monitoring system. At the same time, according to the foregoing content, the effective perception range of the existing camera in the region can be deduced, and in this embodiment, a backup camera of the same model as the original camera is preferably used for supplementing.

[0071] In this embodiment, the calculation of the average perception range of the existing camera in step S500 includes the following process: according to the mapping relationship between the collection data of the existing camera and the three-dimensional model, the projection matrix of each existing camera is solved. The approximate position of the existing camera in the space coordinate system is estimated from the projection matrix to obtain the approximate space center point of the existing camera; the maximum horizontal distance from the approximate space center point to the boundary of the grid region with the farthest perception range in the grid coordinate system is calculated as the horizontal effective perception radius of the current existing camera. The average value of the horizontal effective perception radius of all existing cameras is taken as the radius of the average perception range of the existing camera.

[0072] Specifically, the solving of the projection matrix includes the following process:

[0073] Based on the correspondence between the image points and the three-dimensional points in the world coordinate system, the expression of the projection matrix V is constructed by the direct linear transformation method (DLT) as follows:

[0074] .

[0075] Where (u i , v i ) represents the image point coordinates in the collection data of the existing camera, (X j , Y j , Z j ) represents the three-dimensional point coordinates of the corresponding image points in the three-dimensional model, and s represents the scale factor.

[0076] By minimizing the projection error:

[0077] .

[0078] The SVD decomposition method is used to fit multiple groups of points to solve the projection matrix .

[0079] wherein, , and respectively represent the first, second and third components of the multiplication result of the projection matrix V and the three-dimensional point homogeneous coordinates , represent a rotation and intrinsic coupling matrix, represent translation information, and R represents a rotation matrix of the extrinsic parameter.

[0080] Specifically, after the solution of the projection matrix is completed, the approximate position of the existing camera in the world coordinate system of the three-dimensional model, i.e., the center point O of the existing camera, can be estimated according to the solved projection matrix V. It should be noted that due to the influence of image point selection error, intrinsic parameter missing, etc., the center point O is only an approximate estimation result, but it can still be used as a basic point for subsequent spatial analysis.

[0081] After obtaining the approximate spatial center points O of all existing cameras, the geometric coverage capability of the existing camera, i.e., the current perception range of the existing camera, can be further analyzed according to the accurately labeled perception area. It can be assumed that the boundary point of the grid area farthest from the center point O in the accurate perception area of the existing camera is G j , and the three-dimensional coordinates of this point G j in the three-dimensional model are (X j , Y j , Z j ); then the Euclidean distance D j from the center point O (X o , Y o , Z o ) of the existing camera to the point G j is:

[0082] .

[0083] In order to simplify the analysis of the perception range of the existing camera, the maximum horizontal distance R max from the center point O to the point G j can be regarded as the horizontal effective perception radius of the current existing camera by directly calculating through the grid coordinate system under the BEV view angle.

[0084] .

[0085] It should be known that due to the different installation heights and current pitch angles of the existing cameras, the effective perception ranges of the existing cameras are different, so when simulating the supplement points, the average value of the effective perception ranges of all the existing cameras can be used as a standard value for simulation.

[0086] In this embodiment, the specific process of calculating and deploying the minimum number of camera supplement points in step S500 is as follows: the horizontal span and vertical span of the blind area to be supplemented are obtained, and the ratio N kw and N kh of the average perception range of the existing camera are calculated respectively; for the blind area to be supplemented that satisfies N kw and N kh are less than or equal to 1, a single-point simulation deployment method is used for supplement; for the blind area to be supplemented that satisfies N kw and / or N kh is greater than 1, a multi-point cooperative deployment method is used for supplement.

[0087] It can be understood that for the blind area to be supplemented that satisfies N kw and N kh are less than or equal to 1, the effective perception range of a single camera can achieve complete coverage, so only one camera needs to be added. For the blind area to be supplemented that satisfies N kw and / or N kh is greater than 1, the effective perception range of a single camera cannot achieve complete coverage, so multiple camera deployment points need to be added. In order to enhance the tracking ability of the entire monitoring system in the region as much as possible, when supplementing cameras, whether it is single-point simulation deployment or multi-point cooperative deployment, in addition to meeting the full coverage of the blind area to be supplemented, it should also cover other areas as much as possible.

[0088] Specifically, the specific method of single-point simulation deployment is as follows: each peripheral edge grid of the blind area to be supplemented is selected as a candidate installation point; according to the theoretical effective perception range of the existing camera, the point position with the largest overlapping area with the surrounding accurate perception area in the candidate installation point is selected as the final deployment position, which can completely cover the blind area to be supplemented.

[0089] The specific method of multi-point cooperative deployment is as follows: the main deployment direction is determined according to the proportional relationship between the horizontal span and the vertical span of the blind area to be supplemented; starting from the boundaries at both ends of the main direction, the camera combination deployment is simulated to obtain multiple sets of deployment candidate point sets; according to the theoretical effective perception range of the existing camera, the coverage range of each set of deployment candidate point sets is obtained; the set of deployment candidate point sets that can completely cover the blind area to be supplemented and has the largest overlapping area with the surrounding accurate perception area is selected as the final deployment position.

[0090] It can be understood that the sensing range of the existing camera calculated in the foregoing is based on the current fixed pitch angle, however, the pitch angle of the camera is a range value, and when the camera is supplemented, the pitch angle can be adjusted arbitrarily within the pitch angle range according to actual needs; therefore, before the camera is supplemented, the theoretical effective sensing range needs to be calculated according to the intrinsic parameters of the existing camera, the installation height, and the pitch angle range. Among them, since the model of the supplemented camera is consistent with the original existing camera, the intrinsic parameters do not change, and the installation height can be set by itself according to actual needs; for the calculation of the pitch angle range, the acquisition data of the existing camera can be used to obtain the pitch angle range. In order to facilitate understanding, the following will be specifically described.

[0091] Specifically, the acquisition of the pitch angle range of the existing camera includes the following process: after the calculation of the accurate sensing area in step S300 is completed, all pixel point sets corresponding to the accurate sensing area in the image collected by the existing camera are extracted; according to the resolution information of the existing camera, the distribution position of the pixel point set in the image is estimated, the proportion and angle range of the pixel point set in the image are obtained, and then the pitch angle range of the existing camera is calculated.

[0092] In simple terms, if the pixel points corresponding to the accurate sensing area in the acquisition data of the existing camera are concentrated in the lower half of the image, it indicates that the pitch angle of the existing camera is large, which can be regarded as the upper limit of the pitch angle range; then the lower limit of the pitch angle range can be obtained, and the pitch angle range of the entire existing camera can be obtained. Then when supplementing, the installation height can be selected according to the actual supplement position, and then the theoretical effective sensing range of the camera can be obtained through the pitch angle range, and the pitch angle of the camera when supplementing is determined according to the theoretical effective sensing range that can cover the to-be-supplemented blind area and has the largest overlapping area with the surrounding accurate sensing area.

[0093] The above describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above embodiments, and the above embodiments and descriptions in the specification are only the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection claimed by the present application is defined by the appended claims and their equivalents.

Claims

1. A camera point-filling method for global tracking, characterized in that, Includes the following steps: S100: Construct a 3D model of the region and project it to generate a 2D base map. Divide the 2D base map into a grid and mark it to obtain a grid coordinate system that identifies each grid region. S200: Based on the mapping relationship between the data collected by existing cameras in the area and the 3D model, solve the homography matrix used to estimate the extrinsic parameters of existing cameras in the area; S3 00: Calculate the accurate perception area based on the existing camera in the grid coordinate system according to the obtained homography matrix, and combine the grid areas with continuous relationship in the accurate perception area to form a closed perception area; S400: Based on the obtained closed sensing area, the area not accurately covered in the grid coordinate system is deduced; the area not accurately covered is classified to obtain multiple blind spots that need to be monitored; S500: Calculates the average sensing range of existing cameras, calculates the minimum number of cameras needed to fill blind spots based on the horizontal and vertical span of each blind spot area, and deploys them accordingly; Step S300 includes the following process: For each existing camera's sensing area corresponding to a grid area, select multiple sets of image points and corresponding 3D points from static images and 3D models as verification points; Calculate the Euclidean distance error between the image points in each set of verification points and the 3D points after transformation by the homography matrix H. ; ； Among them, P j This represents the coordinates of a 3D point in the verification point. Represents the homogeneous coordinates of image points. Represents the result of matrix multiplication The third component; If the Euclidean distance error of the multiple sets of verification points corresponding to the grid area If all values ​​are less than the set threshold, the grid area is marked as an accurate sensing area.

2. The camera point-filling method for global tracking as described in claim 1, characterized in that, Step S200 includes the following process: Select a single, empty, static frame from the data collected by each existing camera; Select image points of several representative feature points in the static image, and at the same time determine the plane points corresponding to each representative feature point in the two-dimensional base map; Given the coordinates of the image points and the plane points, the matrix that minimizes the sum of squared errors between the image points and the plane points is used as the homography matrix H, calculated using the least squares method. The specific expression is as follows: ； in, Represents the homogeneous coordinates of image points. Represents the homogeneous coordinates of a point in a plane. Represents the result of matrix multiplication The third component, n, represents the number of representative feature points selected.

3. The camera point-filling method for global tracking as described in claim 1, characterized in that, Step S400 includes the following process: Based on the usage functions of different locations within the area, the areas that are not accurately covered are divided into areas that need to be accurately monitored and areas that do not need to be accurately monitored. The grid areas corresponding to the areas that need to be accurately monitored are identified. If consecutive grid areas that are not accurately covered are identified, they are aggregated to form blind spots.

4. The camera point supplementation method for global tracking as described in claim 1, characterized in that, In step S500, the calculation of the average sensing range of the existing camera includes the following process: Based on the mapping relationship between the data collected by the existing cameras and the 3D model, solve the projection matrix of each existing camera; The approximate position of the existing camera in the spatial coordinate system is estimated from the projection matrix, and the approximate spatial center point of the existing camera is obtained. The maximum horizontal distance from the approximate spatial center point to the boundary of the grid region with the farthest sensing range in the grid coordinate system is calculated as the effective horizontal sensing radius of the existing camera. The average horizontal effective sensing radius of all existing cameras is taken as the radius of the average sensing range of the existing cameras.

5. The camera point-filling method for global tracking as described in claim 4, characterized in that, Solving for the projection matrix involves the following process: From the collected data and the 3D model, select N sets of corresponding image points and 3D points, and construct the projection matrix V using the direct linear transformation method as follows: ； By minimizing the projection error: ； The SVD decomposition method is used to fit multiple sets of points, and the projection matrix is ​​solved. ; Among them, (u i v i (X) represents the coordinates of an image point in the existing camera's captured data. j Y j Z j ) represents the 3D coordinates of the corresponding image point in the 3D model, and s represents the scaling factor. , as well as Represent the projection matrix V and the homogeneous coordinates of the three-dimensional point, respectively. The first, second, and third components of the multiplication result, M represents the rotation and intrinsic parameter coupling matrix, and p4 represents the translation information.

6. The camera point-filling method for global tracking as described in any one of claims 1-5, characterized in that, The process of calculating and deploying the minimum number of camera replacement points in step S500 is as follows: Obtain the lateral and longitudinal spans of the area to be filled, and calculate the ratio N between these spans and the average sensing range of the existing cameras. kw and N kh ; For satisfying N kw and N kh For blind spots where all values ​​are less than or equal to 1, a single-point simulation deployment method is used to fill the blind spots. For satisfying N kw and / or N kh For blind spots greater than 1, a multi-point collaborative deployment method is used to fill the blind spots.

7. The camera point-filling method for global tracking as described in claim 6, characterized in that, The specific method for single-point simulated deployment is as follows: Select each outer edge grid of the area to be filled as a candidate installation point; Based on the theoretical effective sensing range of existing cameras, the points selected from the candidate installation points that can completely cover the blind area to be filled and have the largest overlap with the surrounding accurate sensing area are selected as the final deployment points.

8. The camera point-filling method for global tracking as described in claim 7, characterized in that, The theoretical effective sensing range of existing cameras is calculated using intrinsic parameters, installation height, and pitch angle range. Obtaining the pitch angle range involves the following process: After completing the calculation of the accurate perception area in step S300, extract the set of all pixels corresponding to the accurate perception area from the images captured by the existing camera. Based on the resolution information of the existing camera, the distribution position of the pixel set in the image is estimated, the proportion and angle range of the pixel set in the image are obtained, and then the pitch angle range of the existing camera is calculated.

9. The camera point supplementation method for global tracking as described in claim 6, characterized in that, The specific methods for multi-point coordinated deployment are as follows: The main control direction is determined based on the ratio of the horizontal and vertical spans of the area to be filled. Starting from the boundaries at both ends of the main direction, simulate the combined deployment of cameras to obtain multiple sets of candidate deployment points; Based on the theoretical effective sensing range of existing cameras, the coverage range of each set of candidate points is obtained. The final control points are selected from a set of candidate points that can completely cover the blind spots and have the largest overlap with the surrounding accurate sensing areas.

Images