Monocular vision indoor positioning method of support point search

Abstract

The present application belongs to the technical field of monocular vision indoor positioning, and particularly relates to a monocular vision positioning method for support point search. The present application is mainly aimed at the scene where an ordinary monocular camera and a positioning environment are relatively fixed. The present application can realize high-precision two-dimensional positioning of a target in a visible environment. Specifically, the present application can be divided into two stages, i.e. an offline stage and an online stage. In the offline stage, the monocular camera is calibrated to obtain an intrinsic matrix and distortion coefficients. An image of the positioning environment is collected to construct a reference surface positioning coordinate system. Then, a homography matrix is solved by selecting pixel coordinates and world coordinate point pairs. In the online stage, first, a target detection network is used to detect a positioning target in a visible region, and a background image is updated according to the output result. Then, the output target category is further searched in the target detection frame to estimate the positioning pixel point of the target on the reference surface. Finally, the positioning pixel point is corrected for distortion and projected and mapped to estimate the two-dimensional coordinates of the target in the real environment.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of monocular vision positioning, and particularly relates to a monocular vision indoor positioning method for support point search. BACKGROUND

[0002] With the rapid development of mobile networks, cloud computing and Internet of Things technologies, location-based services are attracting more and more attention. In the current big data technology background, all aspects of human life will generate a large amount of data, however, if the obtained data does not have corresponding location information, the usability of the data will be greatly reduced. For open outdoor environments, the Global Positioning System (GPS) and Beidou navigation system combined with communication base station positioning methods have been widely used, however, in indoor environments, satellite signals are difficult to penetrate buildings, and therefore cannot assist devices in positioning in indoor environments.

[0003] With the development of computer vision technology and the widespread use of surveillance cameras, great market and opportunities have been created for the application and development of vision-based positioning technology. Vision-based positioning technology can be widely used in public entertainment security, home security monitoring, crime prevention, accident monitoring and early warning, and other scenarios. Current mainstream cameras are divided into three categories, namely monocular cameras that can only obtain one two-dimensional image at a time, binocular cameras that can simultaneously obtain two two-dimensional images with differences from two similar angles, and depth cameras that can obtain target distances. Monocular cameras are simple in structure, low in cost, and widely used. In the positioning scene of monocular cameras, there are usually two categories, one is the vision positioning method that needs to build a library, that is, multiple images are captured in the offline stage, an offline image library is established, and corresponding position labels are labeled. In the online stage, the motion target captures an image, which is matched with the images in the offline library, and then a mapping relationship between the two images is established through a feature point matching method, and finally the position of the online image is output. The other method is that the monocular camera is fixed, and the mapping relationship between the image coordinate system and the positioning reference coordinate system is solved in the offline stage. In the online stage, the target in the captured image is directly extracted, the corresponding positioning pixel point is solved, and then the mapping is performed to output the position of the target in the real world.

[0004] In the second type of monocular visual positioning method, the traditional method "Vu H, Nguyen V G, Pham A T, et al. Pedestrian Localization and Trajectory Reconstruction in a Surveillance Camera Network [C] / / Proceedings of the Eighth International Symposium on Information and Communication Technology. 2017: 393-400." needs to model the background to solve, and then find the target area by the absolute change of pixel value, and then select the corresponding pixel point as the positioning pixel point output. This kind of method is sensitive to non-target background change and light change, and the search strategy of the positioning pixel point completely depends on the accuracy of the extracted target area, and there is no further search for the support pixel point of the target on the ground. Therefore, the traditional method has limited positioning ability and is easily disturbed by environmental noise and background change, resulting in that the positioning algorithm is not robust enough. SUMMARY

[0005] In the traditional foreground extraction method, multiple frames of images need to be collected to model the background, and then the foreground target area is estimated by using the probability or classification method. This method has large amount of calculation, depends on accurate background modeling, and is sensitive to light change. In view of the problem, the application proposes a background modeling method based on target detection result, which updates the last background image according to the target detection result. This method is not sensitive to noise in the background, and is also not sensitive to the change of the background area of the non-positioning target. For light interference, the application proposes a method of fusing relative change features and absolute change features to estimate the foreground target. The pixel value distribution changes little before and after the brightness change of the background area, and the light interference is suppressed based on this principle. In addition, the traditional method takes the point on the lower frame of the foreground rectangular area as the positioning pixel point, and the positioning accuracy of this strategy depends on the volume and posture of the target. In view of the problem, the target support point search method is proposed in the application, which directly searches the contact (support) pixel point between the positioning target and the ground. The target positioning accuracy is improved.

[0006] In order to achieve the purpose of the application, the technical scheme adopted is as follows:

[0007] Offline stage: build monocular vision positioning scene, solve the mapping relationship between pixel plane and positioning coordinate plane. Pre-train target recognition network for target to be positioned. In online stage, camera inputs the picture to target detection network, updates background according to ID, region of interest pixel coordinates and other information in detection result, and segments region of interest. Then, calculate absolute change quantity and KL divergence of target image and background image, that is, distribution difference. After scale normalization, fuse two kinds of features to obtain change feature. Finally, solve target contour in vertical direction, establish support point search target function, and solve support point set. Finally, map support point to two-dimensional positioning plane. Specifically, the following steps are included:

[0008] Step 1. Build a visual positioning scene.

[0009] Step 2. Offline stage parameter calculation:

[0010] Step 2-1. Monocular camera calibration.

[0011] Step 2-2. Collect positioning environment image and build positioning reference coordinate system.

[0012] Step 2-3. Select pixel coordinates and world coordinate point pairs to solve homography matrix.

[0013] Step 2-4. Train target recognition network.

[0014] Step 3. Online target positioning:

[0015] Step 3-1. Extract target detection result.

[0016] Step 3-2. Background image update.

[0017] Step 3-3. Region of interest segmentation and processing.

[0018] Step 3-4. Foreground feature extraction.

[0019] Step 3-5. Target support point estimation.

[0020] Step 3-6. Positioning result output.

[0021] The application proposes a target extraction and background modeling method combining traditional methods with deep learning, improves the anti-interference ability to background noise and the real-time effectiveness of the background image, and makes the algorithm pay more attention to positioning targets; a method based on the fusion of absolute change characteristics and relative change characteristics suppresses the light and shadow interference in the target neighborhood; a positioning target support point search algorithm is proposed to reduce the influence of the posture of the positioning target on the positioning accuracy. In addition, the positioning target in the proposed positioning method is limited to pedestrians, and the positioning target depends on the training category of the target detection network. Therefore, the monocular visual indoor positioning method with support point search proposed by the application is a monocular visual positioning algorithm with high positioning accuracy, adaptive positioning target and robustness. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a flowchart of the application.

[0023] Figure 2 is the average positioning error under different positioning methods.

[0024] Figure 3 is the cumulative probability distribution of the positioning error under different positioning methods. DETAILED DESCRIPTION

[0025] 1. Constructing a visual positioning scene

[0026] The experimental environment is a rest area in an office building, and a Raspberry Pi 4B and a Raspberry official camera (Raspberry Pi Camera Module 2) are used to monitor the target area in real time. The device is installed on a tripod 2.3 m high from the reference plane, and the included angle between the imaging plane and the positioning reference plane is an acute angle. The video capture frame rate is 10 frames per second (fps). The resolution of each frame image is 1280x720. The positioning target is a pedestrian moving in the area.

[0027] 2. Offline stage parameter calculation

[0028] 2-1. Monocular camera calibration. Use the monitoring camera to shoot several calibration checkerboard images. Use the camera calibration application in the Matlab image processing and computer vision toolkit to read and process the shot images, and solve the calibration parameters. Extract the camera intrinsic matrix M i , and the radial distortion coefficients k1, k2, k3 and the tangential distortion coefficients p1, p2. The intrinsic matrix is specifically shown in the following formula (1), where f c is the focal length of the camera, and a x , c x , a y , c y are the scaling factor and fixed pixel value of the imaging plane and pixel coordinate system in the horizontal and vertical directions, respectively.

[0029]

[0030] 2-2. Acquire positioning environment images and construct a positioning reference coordinate system. Acquire background image b0 and perform distortion correction on the image according to the distortion imaging relationship. The imaging relationship formula is shown in (2), where [x,y] T For normalized pixel plane coordinates, [x′, y′] T These are the distorted coordinates. [u,v] T These are coordinates in the pixel coordinate system.

[0031]

[0032] Steps 2-3. Select pixel coordinate and world coordinate point pairs and solve for the homography matrix. Mark the positioning reference plane coordinate system in the distortion-corrected image, and select the reference point set P1. With the corresponding pixel reference point set P2, The mapping relationship between the pixel plane and the positioning reference plane, i.e., the homography matrix H, is solved according to formula (3). Since H has 8 degrees of freedom and N1≥4.

[0033]

[0034] Steps 2-4. Training the target recognition network. In this experiment, a pre-trained YOLOv5 lightweight target recognition network is used.

[0035] Step 3. Online target location.

[0036] Step 3-1. Extract the target detection results. Extract the i-th frame image f acquired by the camera. i Input the object detection network and output the result y. i As shown in equation (4), a total of K targets were detected. Among them, Let be the ID of the j-th target detected in the i-th frame of the image. The corresponding detection confidence level, These are the two-dimensional pixel coordinates of the top-left and bottom-right corners of the rectangular pixel region where the target is located.

[0037]

[0038] In order to completely include the target within the target detection bounding box, Adjust according to equation (5). Where α s u max v max These are the adjustment coefficient, the maximum pixel coordinate in the horizontal direction, and the maximum pixel coordinate in the vertical direction, respectively.

[0039]

[0040] Step 3-2. Background image update. The background image b i Based on b i-1 , f i , y i Update jointly, the specific update formula is shown in equation (6), where p uv is the pixel point. b i It is insensitive to the changes of non-target background in the environment, such as the changes of door and window states.

[0041]

[0042] Step 3-3. Segmentation and processing of the region of interest. According to f i and b i are segmented into corresponding K sub-images is the target pixel information, is the background pixel information of the corresponding region. The corresponding image is converted into a gray scale image, and the image is filtered by using a median filtering algorithm.

[0043] Step 3-4. Foreground feature extraction. First, the relative change feature The KL divergence is used to extract the distribution change feature of the pixel region. A rectangular sliding window is used to divide the two images into multiple sub-image regions, and the KL divergence values of the corresponding regions of the background image and the current image are calculated. Wherein, N p is the number of pixel points in the sliding window, is the pixel value at the pixel point . The specific calculation formula is shown in equation (7).

[0044]

[0045] Then, the absolute change feature in the region is extracted The specific extraction method is shown in equation (8):

[0046]

[0047] The two kinds of change features are fused. Since and are not dimensionally consistent, scale normalization is needed. The fused feature is shown in equation (9).

[0048]

[0049] Step 3-5. Target support point estimation.

[0050] 3-5-1. Constructing threshold candidate set. In order to reflect the change degree of the variable pixel points in different sub-regions, a threshold candidate set vector t is constructed i,j , and N t thresholds are selected from t i,j to divide d i,j into different change levels. The specific threshold acquisition method is as follows.

[0051] The change matrix d i,j is reduced in dimension and sorted in descending order to obtain the change feature d i,j .

[0052] d i,j = [d1 i,j , d2 L , …, d i,j T , L = (u'2-u'1)·(v'2-v'1) (10)

[0053] The change rate of adjacent values is calculated

[0054]

[0055] Save all data before the point with the maximum change rate to construct the threshold candidate set t i,j .

[0056] t i,j = [d1 i,j , d2 i,j , …, d m i,j T (12)

[0057] Select n1 thresholds at equal intervals to construct the threshold vector

[0058]

[0059] 3-5-2. Extracting target blur profile according to threshold vector . Under different thresholds , search for the lower edge profile (maximum vertical coordinate) of the target. The search results are shown in the following formula (14). And calculate the first and second order statistics μ s , σ s of n search results. The first order statistics represent the mean of the distribution of the lower edge longitudinal coordinate, and the second order statistics represent the stability of the lower edge longitudinal coordinate in the search results.

[0060]

[0061] 3-5-3. Establishing support point search target equation. In μ​s In the search of the support points of the localization target in the image, the target ID is known, and the corresponding V support points are searched according to the prior information. For example, V = 2 for a pedestrian, representing two feet. In the two-dimensional orthographic projection curve of each object contour, the support points are theoretically the first V minimum value points. However, due to the influence of noise, feature extraction capability, camera pose, etc., the support points degenerate into the mean curve segments corresponding to the first V convex sets. Use the iterative method to search the corresponding V support points in turn. First, establish the target solving equation shown in equation (15), where ξ is a relaxation parameter set according to the prior knowledge ID.

[0062]

[0063] 3-5-4. Search for solving support points. It can be divided into three steps to solve the optimal segmentation point First, calculate the second difference value of the first-order statistics According to the parameter ξ, extract the convex set segmentation point vector c, which is specifically shown in equation (16).

[0064]

[0065] Then calculate the edge point loss corresponding to each convex set segment, and the calculation method is shown in equation (17).

[0066]

[0067] Finally, estimate the support points, that is, locate the pixel points. In line with the threshold candidate set construction method, sort the loss vector p in reverse order, take the first M candidate losses with the maximum loss change rate, and record the original data index vector a, and finally output the estimated support points The specific calculation method is shown in equation (18).

[0068]

[0069] Step 3-6. Output the localization result. Solve the corresponding correct pixel coordinates without distortion, and then map the pixel coordinates to the localization reference system through the homography matrix. Output the estimated two-dimensional position.

[0070] To verify the effectiveness of the present application, two groups of positioning scenarios are set. The first group of monitoring cameras is located between the light source and the positioning area, i.e. the positioning scenario with weak light interference. The second group of positioning scenarios is that the positioning area is located between the light source and the monitoring camera, i.e. the positioning scenario with strong light interference. Both positioning scenarios are indoor office areas, and the entire positioning area is located within the visual area of the monitoring camera. The positioning target is a pedestrian walking normally according to the specified trajectory, and finally the error between the estimated trajectory points and the true trajectory is counted. Through experiments, the average positioning error of the present application method in the two scenarios is 5.04 cm and 7.46 cm. The average positioning of the background technology method in the two positioning scenarios is 7.62 cm and 14.91 cm. In addition, the cumulative error probability distribution is counted, and the positioning error of 80% of the positioning points of the present application method in the two scenarios is less than 8.15 cm and 12.21 cm, respectively. The positioning error of 80% of the positioning points of the background technology method in the two scenarios is less than 12.32 cm and 21.40 cm, respectively. The experimental results show that in the set positioning scenarios, the suppression ability of the present application method to light interference is better than that of the background technology method. In addition, the overall positioning performance of the present application method is better than that of the background technology method. Therefore, the experiment proves that the monocular vision indoor positioning method of the present application is a robust and high-precision indoor positioning method.

Claims

1. A monocular vision indoor positioning method of support point search, characterized in that, The method comprises an offline stage and an online stage; The offline stage comprises constructing a monocular vision positioning scene, obtaining offline stage parameters and pre-training a target recognition network, wherein the method for obtaining the offline stage parameters is: For the constructed monocular visual positioning scene, the monocular camera is calibrated to obtain the calibration parameters: obtain the camera intrinsic matrix , and the radial distortion coefficient and the tangential distortion coefficient , the intrinsic matrix is specifically: ， wherein is the camera focal length, are the scaling factors and fixed translation pixel values in horizontal and vertical directions of the imaging plane and pixel coordinate system, respectively; Collecting positioning environment information: collecting background graphics According to the distortion imaging relationship, the background graphics are corrected, and the imaging relationship is: ， wherein is the normalized pixel plane coordinate, is the distorted coordinate, is the coordinate in the pixel coordinate system; Selecting pixel coordinate and world coordinate point pairs to solve homography matrix : Marking out the positioning reference plane coordinate system in the image after distortion correction, selecting the reference point set and the corresponding pixel reference point set , and solving the mapping relationship between the pixel plane and the positioning reference plane according to the following formula: ， 8 degrees of freedom, ; The online stage is to locate the target according to the target detection result, specifically: input the first frame image obtained by the camera into a target detection network to output a result frame image input target detection network, output result : ， Set a total of detected targets, wherein, is the ID of the th target detected in the frame image, is the detection confidence corresponding to the th target, is the two-dimensional pixel coordinates of the top-left corner and the bottom-right corner of the rectangular pixel region where the corresponding target is located. To adjust :​ ， wherein are respectively an adjustment coefficient, a maximum pixel coordinate in the horizontal direction, a maximum pixel coordinate in the vertical direction; to the background image based on , , updating jointly: ， wherein is a pixel point; According to Will And Divided into corresponding Sub-image Where For the target pixel information, The background pixel information of the corresponding area is converted into a gray image, and the image is filtered by a median filtering algorithm; Extracting foreground features: extracting relative change features , using KL divergence value to extract the distribution change feature of pixel region, using rectangular sliding window to divide two images into multiple sub-image regions, calculating the KL divergence value of the corresponding region of the background pixel information and the target pixel information of the current image ， wherein, is the number of pixel points in the sliding window, is the pixel value at the pixel point in the region; and the absolute change feature in the region is extracted : ， The two change features are fused, and since With The dimensions are inconsistent, and scale normalization is needed. The fused feature is: ， Estimating target support points: constructing a threshold candidate set vector From which a threshold is selected The number of thresholds will be Divided into different levels of change, the specific threshold acquisition method is: characterizing the changes dimension reduction and descending ordering: ， Computing the rate of change of adjacent values : ， constructing a threshold candidate set from all data before the maximum point of the preservation change rate : ， equally spaced threshold value construction threshold vector : ， At different thresholds Next, the lower edge profile of the search target is searched for: ， and calculate first and second order statistics of the search results The first order statistics represent the mean of the distribution of the lower edge ordinate, and the second order statistics represent the stability of the lower edge ordinate in the search results. Establishing the search target equation of support points: in searching the support points of the positioning target in the image, the target ID is known, and according to prior information, the corresponding support points are searched, and the target solving equation is established as: ， wherein is a relaxation parameter set according to the prior knowledge ID; Solving optimal segmentation points : first compute the second difference of the first order statistics , according to the parameters extracting convex set segmentation point vectors : ， Recalculating the edge point loss corresponding to each convex set segment: ， on the loss vector Sort in reverse order, take the top candidate loss with the largest loss change rate, and record the original data index vector , and finally output the estimated support point : ， Solve The corresponding correct pixel coordinates without distortion, and then map the pixel coordinates to the positioning reference system through the homography matrix, and output the estimated two-dimensional position.

Images