Indoor single person positioning method based on binocular camera and ultra-wideband fusion

Abstract

The indoor single-person positioning method based on the fusion of binocular camera and ultra-wideband first deploys an ultra-wideband base station and places a binocular camera in an indoor environment. Then, the human body posture estimation is performed on the left and right images taken by the calibrated binocular camera, and the human body parts are matched after being obtained. Then, the pixel disparity value of the matching part is calculated, and the two-dimensional coordinate position of the person in the binocular camera coordinate system is calculated using the disparity model. Finally, the camera positioning data and the ultra-wideband positioning data are fused to obtain the final two-dimensional coordinate position of the person. The present invention obtains the pixel points of the part through human body posture estimation, and then matches and locates. Compared with the positioning algorithm that detects the human frame and matches the feature points, the influence of the background feature point matching is reduced, and the accuracy of visual positioning is improved. In addition, the ultra-wideband positioning data is integrated to achieve extremely high positioning accuracy when the person is close to the camera indoors, and higher positioning accuracy when the person is far away from the camera.

Description

Technical Field

[0001] The invention relates to an indoor single-person positioning method based on binocular cameras and ultra-wideband fusion. Background Art

[0002] On average, people spend more than 70% of their time indoors, so indoor positioning technology has always received extensive attention and research. Many scholars have conducted research on sensor positioning technologies such as Bluetooth positioning, geomagnetic positioning, and ultra-wideband positioning. With the development of computer vision technology, some scholars have also paid attention to and studied indoor positioning technology based on computer vision. Monocular vision positioning and binocular vision positioning technologies have made certain progress. Research on indoor positioning has a wide range of applications in real life, such as accurately determining the location of people in an emergency and providing timely rescue.

[0003] In terms of sensor positioning, some scholars have proposed using ultra-wideband for indoor positioning to achieve more accurate positioning. However, due to the complex indoor environment, ultra-wideband signals are easily affected by multipath effects and non-line-of-sight propagation, which greatly affects the positioning accuracy. In terms of visual positioning, some scholars have proposed using target detection models such as YOLO to detect the human frame, and then using traditional feature point matching algorithms to solve the parallax of feature points on the left and right images taken by the binocular camera for positioning. However, since the detection frame size of the target detection model is related to the human posture, a detection frame that is too large will result in more background feature points being used for matching and calculation, making the positioning error larger. Summary of the invention

[0004] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide a method for indoor single-person positioning based on binocular camera positioning based on human posture estimation and integrated with ultra-wideband positioning.

[0005] The indoor single-person positioning method based on binocular camera and ultra-wideband fusion of the present invention comprises the following steps:

[0006] S1. First, deploy an ultra-wideband base station and place a binocular camera in an indoor environment.

[0007] S2. Then, human body posture estimation is performed on the left and right images taken by the calibrated binocular camera, and human body parts are matched after being obtained.

[0008] S3. Then calculate the pixel disparity value of the matching part, and use the disparity model to calculate the two-dimensional coordinate position of the person in the binocular camera coordinate system.

[0009] S4. Finally, the camera positioning data and the ultra-wideband positioning data are fused to obtain the final two-dimensional coordinate position of the person.

[0010] The advantages of the present invention are: by estimating the human body posture to obtain the pixel points of the parts, and then matching and positioning, compared with the positioning algorithm that detects the human frame and matches the feature points, the influence of the background feature point matching is reduced, and the accuracy of visual positioning is improved. In addition, the ultra-wideband positioning data is integrated to achieve extremely high positioning accuracy when the person is close to the camera indoors, and higher positioning accuracy when the person is far away from the camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0012] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0013] The indoor single-person positioning method based on binocular camera and ultra-wideband fusion includes the following steps:

[0014] 1. Deployment of ultra-wideband base stations and placement of binocular cameras in indoor environments;

[0015] In an indoor environment, an ultra-wideband base station is deployed near each of the four corners, so that the four ultra-wideband base stations can form a rectangular area. An ultra-wideband base station is deployed in the middle of any long side to improve the ultra-wideband positioning accuracy. A binocular camera is placed in the middle of any short side of the rectangular area to ensure that the camera field of view can cover the left and right boundaries of the rectangular area as much as possible.

[0016] 2Human body parts positioning and matching based on binocular cameras;

[0017] In order to reduce the impact of reduced accuracy caused by background feature point matching, it is necessary to make the matching feature points as close to the person as possible. Therefore, a human posture estimation model is introduced, which focuses on the parts of the human body. By matching the parts as feature points, the background impact can be reduced, thereby improving positioning accuracy. Human posture estimation is performed on the two images to obtain the pixel coordinates of the parts. The main parts and auxiliary parts on the two images are matched to obtain the pixel coordinates of the matching parts. Specifically, it includes:

[0018] 11) Obtain the pixel coordinates of the human body parts in the left and right binocular images. Run the human body posture estimation model on the two images taken by the calibrated binocular camera to obtain the pixel coordinates of multiple parts of the human body. If the human body posture estimation model has no output on at least one image, skip the corresponding two images.

[0019] 12) Part matching of left and right binocular images. The neck, left and right shoulders, left and right hips, and lumbar sacral area are selected as the main parts. These parts will not change with the change of human posture, which is helpful to improve the positioning accuracy; other parts such as eyes, ears, nose, left and right elbows, left and right hands, left and right knees, left and right ankles are selected as auxiliary parts. These parts will change with the change of human posture and have some impact on the positioning accuracy; traverse the same part on the two images at the same time. If the confidence of at least one part is less than the threshold of 0.2, the part matching fails; otherwise, the part matching is successful, and the pixel coordinates of the matching part are recorded.

[0020] 3. Parallax model locates the real-world position of the person;

[0021] The disparity model can convert pixel coordinates into real-world coordinates. The weighted average method is used to calculate the pixel disparity of the main matching part and the auxiliary matching part and their pixel values ​​that deviate from the left camera. With the left camera as the origin, the line connecting the left and right cameras is the x-axis, and the line perpendicular to the x-axis and parallel to the front of the camera is the y-axis. The disparity model can be used to calculate the distance of a person from the binocular camera, that is, the offset on the y-axis, and the distance from the left camera, that is, the offset on the x-axis, to obtain the two-dimensional coordinates of the person. Specifically, it includes:

[0022] 21) Calculate the disparity of the pixels of the matching part and the pixels that deviate from the left camera. After the part is successfully matched, the weighted average method is used to calculate the disparity of the pixels of the main matching part and the auxiliary matching part and the pixels that deviate from the left camera. If the part is the main matching part, the weight coefficient is 0.8; if the part is the auxiliary matching part, the weight coefficient is 0.2. In the pixel coordinate system, let the two-dimensional coordinates of the pixel point of the main matching part in the left image be (x ml ,y ml ), the two-dimensional coordinates of the pixel points of the corresponding main matching parts in the right image are (x mr ,y mr ), then the pixel disparity of the main matching part in the left and right images is x ml -x mr , the pixel deviation of the main matching part from the left camera is x ml ; Assume that the two-dimensional coordinates of the pixel point of the auxiliary matching part in the left image are (x al ,y al ), the two-dimensional coordinates of the pixel point of the corresponding auxiliary matching part in the right image are (x ar ,y ar ), then the pixel disparity of the auxiliary matching part in the left and right images is x al -x ar , the auxiliary matching part deviates from the left camera by x pixels al .

[0023] Then the weighted average pixel disparity P wavg and weighted average position deviation from left camera pixel X wavg The calculation formula is as follows:

[0024]

[0025] Where i represents the i-th main matching part, n represents the total number of main matching parts, j represents the j-th auxiliary matching part, and m represents the total number of auxiliary matching parts.

[0026] represents the sum of the pixel disparity of the main matching part in the left and right images,

[0027] represents the sum of the pixel disparity of the auxiliary matching part in the left and right images,

[0028] Indicates the sum of the pixels where the main matching part deviates from the left camera. Indicates the sum of the pixels of the auxiliary matching part that deviates from the left camera;

[0029] 22) Calculate the two-dimensional coordinates of the person in the camera coordinate system. The principle of the disparity model is similar triangles. The two-dimensional coordinates of the person are (X, Y), the focal length in the camera internal reference is f, and the baseline distance between the binocular cameras is D meters. Then the calculation formulas for X and Y are as follows:

[0030]

[0031] Where X represents the distance of the person from the left camera in the x-axis direction, and Y represents the distance of the person from the binocular camera in the y-axis direction, thereby obtaining the two-dimensional coordinate position of the person under the camera.

[0032] 4. Fusion of ultra-wideband to locate the person’s position;

[0033] One disadvantage of binocular camera positioning is that the farther a person is from the camera, the lower the positioning accuracy. Since the accuracy of ultra-wideband positioning is independent of the person's position and is only related to the multipath effect and non-line-of-sight propagation, by integrating ultra-wideband positioning, when the person is closer to the camera, the weight of visual positioning is increased to obtain extremely high positioning accuracy, and when the person is farther away from the camera, the weight of ultra-wideband positioning is increased to obtain higher positioning accuracy. A person carries an ultra-wideband positioning tag, and the ultra-wideband base station can obtain the two-dimensional positioning data of the tag and convert it into two-dimensional positioning data in the camera coordinate system. The camera's two-dimensional positioning coordinates and the converted ultra-wideband tag's two-dimensional positioning coordinates are then fused through the variable weighted average method to obtain the person's final two-dimensional coordinates. Specifically including: 31) The tag's two-dimensional positioning data is converted into two-dimensional positioning data in the camera coordinate system. Suppose the two-dimensional coordinate position of the left camera in the ultra-wideband coordinate system is (Xo ,Y o ), the coordinates of the marker point in the ultra-wideband coordinate system are (X uwb ,Y uwb ), the coordinates of the marker point in the camera coordinate system are (X vision ,Y vision ), the angle θ formed by the camera coordinate system and the positive direction of the y-axis of the ultra-wideband coordinate system is calculated as follows:

[0034]

[0035] For a certain point (X origin ,Y origin ), and convert it to the coordinates in the camera coordinate system (X transform ,Y transform ):

[0036]

[0037] 32) The variable weighted average method is used to fuse the camera 2D positioning data and the converted UWB tag 2D positioning data. As the distance between a person and the camera in the positive direction of the y-axis increases, the positioning accuracy of the camera in the y-axis direction gradually decreases. The variable weighted average method is used to make the weight coefficient of the camera positioning gradually decrease as the distance between a person and the camera increases. Assume that the long side of the rectangular area formed by the UWB base station is A meters, the short side is B meters, and The positioning weight coefficients of the camera and the ultra-wideband are both 0.5 at 1.5 meters, and the positioning weight coefficients of the camera at other locations are where y vision is the distance of the person in the positive direction of the camera's y-axis. The ultra-wideband positioning weight coefficient is 1-W.

[0038] The final two-dimensional coordinates of the person (X fusion ,Y fusion ) is as follows:

[0039]

[0040] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. An indoor single-person positioning method based on binocular camera and ultra-wideband fusion includes the following steps: 1S1.Ultra-wideband base station deployment and binocular camera placement in indoor environments; In an indoor environment, an ultra-wideband base station is deployed near each of the four corners, so that the four ultra-wideband base stations can form a rectangular area. An ultra-wideband base station is deployed in the middle of any long side to improve the ultra-wideband positioning accuracy. A binocular camera is placed in the middle of any short side of the rectangular area to ensure that the camera field of view can cover the left and right boundaries of the rectangular area as much as possible. 2S2. Human body parts positioning and matching based on binocular camera; In order to reduce the impact of reduced accuracy caused by background feature point matching, it is necessary to make the matched feature points as much as possible on the person. Therefore, a human posture estimation model is introduced, which focuses on the parts of the human body. By matching the parts as feature points, the background impact is reduced, thereby improving the positioning accuracy; human posture estimation is performed on the two images to obtain the pixel coordinates of the parts; the main parts and auxiliary parts on the two images are matched to obtain the pixel coordinates of the matching parts; specifically including: S21. Obtain pixel coordinates of human body parts in the left and right binocular images; Run the human pose estimation model on the two images taken by the calibrated binocular camera to obtain the pixel coordinates of multiple parts of the human body; If the human pose estimation model has no output on at least one image, skip the corresponding two images; S22. Match the parts of the left and right binocular images; select the neck, left and right shoulders, left and right hips, and lumbar sacral region as the main parts, which will not change with the change of human posture and will help improve the positioning accuracy; other parts such as eyes, ears, nose, left and right elbows, left and right hands, left and right knees, and left and right ankles are used as auxiliary parts, which will change with the change of human posture and have some impact on the positioning accuracy; traverse the same part on the two images at the same time, if the confidence of at least one part is less than the threshold, the part matching fails; otherwise, the part matching is successful, and the pixel coordinates of the matching part are recorded; 3S3. Parallax model locates the real world position of a person; The disparity model converts pixel coordinates into real-world coordinates; the weighted average method is used to calculate the pixel disparity of the main matching part and the auxiliary matching part and their pixel values ​​that deviate from the left camera; the left camera is used as the origin, the line connecting the left and right cameras is the x-axis, and the line perpendicular to the x-axis and parallel to the front of the camera is the y-axis; the disparity model is used to calculate the distance of the person from the binocular camera, that is, the offset on the y-axis, and the distance from the left camera, that is, the offset on the x-axis, so as to obtain the two-dimensional coordinates of the person; specifically, it includes: S31. Calculate the disparity of the matching part pixels and the deviation from the left camera pixels; after the part matching is successful, use the weighted average method to calculate the disparity of the pixels of the main matching part and the auxiliary matching part, and the deviation from the left camera pixels; in the pixel coordinate system, let the two-dimensional coordinates of the pixel point of the main matching part in the left image be (x ml ,y ml ), the two-dimensional coordinates of the pixel points of the corresponding main matching parts in the right image are (x mr ,y mr ), then the pixel disparity of the main matching part in the left and right images is x ml -x mr , the pixel that the main matching part deviates from the left camera is x ml ; Assume that the two-dimensional coordinates of the pixel point of the auxiliary matching part in the left image are (x al ,y al ), the two-dimensional coordinates of the pixel point of the corresponding auxiliary matching part in the right image are (x ar ,y ar ), then the pixel disparity of the auxiliary matching part in the left and right images is x al -x ar , the auxiliary matching part deviates from the left camera by x pixels al ; Then the weighted average pixel disparity P wavg and weighted average position deviation from left camera pixel X wavg The calculation formula is as follows: Where i represents the i-th main matching part, n represents the total number of main matching parts, j represents the jth auxiliary matching part, m represents the total number of auxiliary matching parts, represents the sum of the pixel disparity of the main matching part in the left and right images, represents the sum of the pixel disparity of the auxiliary matching part in the left and right images, Indicates the sum of the pixels where the main matching part deviates from the left camera. surface The sum of the pixels of the auxiliary matching part deviating from the left camera; S32. Calculate the two-dimensional coordinates of the person in the camera coordinate system; the principle of the disparity model is similar triangles, the two-dimensional coordinates of the person are (X, Y), the focal length in the camera intrinsic parameter is f, and the baseline distance between the binocular cameras is D meters, then the calculation formulas for X and Y are as follows: Where X represents the distance of the person from the left camera in the x-axis direction, and Y represents the distance of the person from the binocular camera in the y-axis direction, thereby obtaining the two-dimensional coordinate position of the person under the camera; 4S4.Ultra-wideband fusion to locate the person’s position; One disadvantage of binocular camera positioning is that the farther a person is from the camera, the lower the positioning accuracy. Since the ultra-wideband positioning accuracy is independent of the person's position and is only related to the multipath effect and non-line-of-sight propagation, by integrating ultra-wideband positioning, when the person is closer to the camera, the weight of visual positioning is increased to obtain extremely high positioning accuracy, and when the person is farther away from the camera, the weight of ultra-wideband positioning is increased to obtain higher positioning accuracy. The person carries an ultra-wideband positioning tag, and the ultra-wideband base station obtains the two-dimensional positioning data of the tag and converts it into two-dimensional positioning data in the camera coordinate system. Then, the variable weighted average method is used to fuse the camera's two-dimensional positioning coordinates and the converted ultra-wideband tag's two-dimensional positioning coordinates to obtain the person's final two-dimensional coordinates. Specifically, it includes: S41. The tag two-dimensional positioning data is converted into two-dimensional positioning data in the camera coordinate system; let the two-dimensional coordinate position of the left camera in the ultra-wideband coordinate system be (X o ,Y o ), the coordinates of the marker point in the ultra-wideband coordinate system are (X uwb ,Y uwb ), the coordinates of the marker point in the camera coordinate system are (X vision ,Y vision ), the angle θ formed by the camera coordinate system and the positive direction of the y-axis of the ultra-wideband coordinate system is calculated as follows: For a certain point (X origin ,Y origin ), and convert it to the coordinates in the camera coordinate system (X transform ,Y transform ): S42. The variable weighted average method is used to fuse the two-dimensional positioning data of the camera and the converted two-dimensional positioning data of the ultra-wideband tag; as the distance between the person and the camera in the positive direction of the y-axis increases, the positioning accuracy of the camera in the y-axis direction gradually decreases. The variable weighted average method is used to make the weight coefficient of the camera positioning gradually decrease as the distance between the person and the camera increases; assuming that the long side of the rectangular area formed by the ultra-wideband base station is A meters, the short side is B meters, and the position camera positioning weight coefficient is where y vision is the distance of the person in the positive direction of the camera's y axis, and the ultra-wideband positioning weight coefficient is 1-W; the person's final two-dimensional coordinate (X fusion ,Y fusion ) is as follows:

2. The indoor single-person positioning method based on binocular camera and ultra-wideband fusion as claimed in claim 1, characterized in that: The threshold value in step S22 is 0.

2.

3. The indoor single-person positioning method based on binocular camera and ultra-wideband fusion as claimed in claim 1, characterized in that: In the weighted average method described in step S31, if the part is a main matching part, the weight coefficient is 0.8; if the part is an auxiliary matching part, the weight coefficient is 0.

2.

4. The indoor single-person positioning method based on binocular camera and ultra-wideband fusion as claimed in claim 1, characterized in that: In step S42, At a distance of 1 meter, the positioning weight coefficient of the camera and the positioning weight coefficient of the ultra-wideband are both 0.5.

Images