Multi-camera human body recognition and positioning method and system, computer equipment, medium and program product
Through the multi-camera human body recognition and positioning method, the BP neural network and YOLOv8 model can be used to achieve accurate personnel positioning without calibration, solving the problems of complex calibration and low positioning accuracy in the existing technology, and realizing dynamic adjustment of the lighting environment and energy saving .
Patent Information
- Application Number
- CN202411212815.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-08
AI Technical Summary
The existing multi-camera human body positioning technology has complex calibration process and low positioning accuracy in complex environments, making it difficult to achieve refined lighting control and energy-saving goals.
The multi-camera human body recognition and positioning method is adopted, and the pixel coordinates of multiple cameras are converted into world coordinates using BP neural network. The YOLOv8 model is used for human body recognition and Hungarian algorithms to match human body, achieving accurate personnel positioning without camera calibration.
It improves the accuracy and efficiency of personnel positioning, and realizes dynamic adjustment of lights according to personnel distribution in the lighting environment, satisfying visual comfort and achieving energy savings.
Smart Images

Figure CN118918613A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of lighting control, and specifically relates to a multi-camera human body recognition and positioning method, system, computer equipment, storage medium, and program product. Background Art
[0002] In public places such as offices, classrooms, libraries, and reading rooms, there are high requirements for the lighting environment for work and study, and the number and location of people in these places often change. In order to meet the visual comfort needs of people in real time in a dynamic environment while achieving energy saving, the information provided by existing network cameras is fully utilized to achieve personnel identification and positioning, and obtain the position coordinates of people in real time, so that the power output of the lighting group can be controlled according to the distribution of people, and the brightness of the lighting fixtures can be dynamically adjusted to meet everyone's visual comfort needs and achieve energy saving overall.
[0003] In large office environments, there are usually dozens to hundreds of people involved. Due to the frequent flow of personnel, some workstations may be unoccupied for a long time, but the lamps continue to illuminate, resulting in a large amount of power waste; in college classrooms, because it is an open learning environment, students randomly choose seats during evening self-study, and the lights in the classroom are often kept on, but only a few students are studying, and a small number of people occupy a large amount of lighting resources. Similar situations also exist in libraries and reading rooms. Therefore, it is necessary to use intelligent control systems to solve these problems based on the distribution of people flow, so as to achieve the goals of on-demand lighting, comfortable lighting and effective use of energy.
[0004] Advocating low-carbon economy, lighting on demand, energy conservation and emission reduction, applying artificial intelligence, computer technology and electronic technology to the lighting control of the above environments, making full use of the images provided by network cameras used for monitoring public places, identifying and locating people, and providing information support for lighting optimization control, have become the key to the intelligentization of public place lighting.
[0005] At present, infrared pyroelectric sensors and microwave sensors can be used for indoor human positioning. However, the method of using the above sensors to locate people has the following defects:
[0006] First, infrared pyroelectric sensors can detect infrared radiation generated by the human body. When someone enters the sensor's detection range, it will sense the changes in infrared radiation in the surrounding environment and generate corresponding electrical signals to determine the presence, movement direction and relative position of the human body. However, infrared pyroelectric sensors can only detect whether there are people in a certain area, and cannot determine the exact location and number of people.
[0007] Second, microwave sensors detect the presence and movement of people by sending microwave signals and measuring the reflection of the signals. They have strong penetration and good privacy, but are easily affected by electromagnetic interference, resulting in relatively poor position detection accuracy and high system cost.
[0008] In public places, video surveillance technology is widely used, making full use of existing multiple network cameras for personnel detection and positioning, which can realize value-added services. Lighting control based on the location and distribution of personnel can not only meet the comfortable lighting needs of personnel in different positions, but also achieve the purpose of energy saving.
[0009] Traditional multi-camera human body positioning technology is usually based on the dual-camera epipolar constraint method, which realizes human body positioning by calibrating the internal and external parameters of the camera. However, in actual engineering applications, due to the complexity of the environment, changes in lighting, and the camera installation angle, the calibration process becomes complicated, especially when the camera is installed in a position that is difficult to operate. Therefore, in engineering applications, it is difficult to implement personnel positioning using calibration methods. Since the camera uses a vertical projection model, when the person is close to the camera, the lens will be distorted, and this systematic error will affect the positioning accuracy. In order to improve the positioning accuracy, multiple cameras can be used for personnel positioning. However, the traditional multi-camera human body positioning method mainly determines the three-dimensional position through dual-camera parallax calculation, and fuses the positioning results of multiple cameras. This process is relatively cumbersome.
[0010] In response to the problems of complex calibration and tedious calculation of the above-mentioned cameras, the present invention proposes a method for personnel positioning using multiple cameras. In combination with the BP neural network, the pixel coordinates of personnel in multiple cameras are used as the input of the BP neural network, thereby outputting the world coordinates of the personnel. Personnel positioning is achieved without camera calibration, thereby achieving the purpose of comfortable lighting and energy saving. Summary of the invention
[0011] The purpose of the present invention is to provide a multi-camera human body recognition and positioning method, system, computer equipment, and storage medium to solve the problem that in lighting control, due to the poor perception ability of personnel, it is impossible to perform refined lighting control according to the distribution of personnel, meet people's visual comfort and achieve the goal of lighting energy-saving control.
[0012] The present invention is achieved through the following technical solutions:
[0013] A multi-camera human body recognition and positioning method comprises the following steps:
[0014] S1, obtaining images taken by cameras distributed at different locations and having a common imaging space;
[0015] S2, perform human body recognition on the images obtained in S1 to obtain the bounding box of the human head;
[0016] S3, calculating the pixel coordinates of the center point of each bounding box based on the head bounding box, and obtaining the position coordinates of multiple human bodies in each camera in the same pixel coordinate system;
[0017] S4. Using the Hungarian algorithm to obtain the mapping relationship of the same human body in different images, achieve human body matching, and obtain the position coordinates of the same human body in different pixel coordinate systems;
[0018] S5. Use the position coordinates of the same human body in different pixel coordinate systems as the input of the human body positioning model, and output the position coordinates of the human body in the world coordinate system.
[0019] Furthermore, in S2, the trained YOLOv8 model is used to perform human recognition on images captured by multiple cameras.
[0020] Furthermore, the human head dataset is used to train the YOLOv8 model. By identifying the human head, the human head bounding box is obtained, and the center point of the human head bounding box is used as the position coordinate of the human body in the pixel coordinate system.
[0021] The source of the human body recognition dataset is as follows: human body images are collected through a camera under a lighting environment, the collected images are annotated using the LabelImg tool and stored in the YOLO format to produce a human body recognition dataset in a lighting environment.
[0022] Furthermore, in S3, the position coordinates of multiple human bodies in each camera in the same pixel coordinate system are recorded as:
[0023]
[0024] in is the i-th person identified by camera C1, is the jth person identified by camera C2, is the kth person identified by camera C3, and n is the total number of people identified by the camera.
[0025] Further, S4 is specifically:
[0026] Combining position similarity and feature similarity, the human body matching problem of three cameras is transformed into a bipartite graph matching problem to solve the minimum weight, and the associated weight is obtained:
[0027]
[0028] In the formula α is the weight factor, The pixel coordinates of the center point of the bounding box detected by camera C1 after homography transformation The coordinates of camera C2 The Euclidean distance between them, S(a,b) is the two arrays composed of all feature vectors of camera C1 and camera C2 and The cosine similarity matrix between ;
[0029] Calculate W for all bounding boxes in C1 and C2 ij Weight, using the Hungarian algorithm for human matching;
[0030] Similarly, match C2 and C3;
[0031] Finally, we get the corresponding relationship between the coordinates of the same person in the C1, C2 and C3 pixel coordinate systems, and the human body coordinate set M in the three images is:
[0032]
[0033] Where R i is the i-th person in the space.
[0034] Furthermore, in S5, the human body positioning model adopts a BP neural network model, and the training process of the BP neural network model is as follows:
[0035] First, initialize the weights and biases of the BP neural network model;
[0036] Second, the position coordinates of the same human body in different pixel coordinate systems are input into the BP neural network model as training data, and the world coordinates of the predicted person are calculated through forward propagation as the predicted value;
[0037] Third, the error between the predicted value and the actual spatial coordinates of the human body in the lighting space is calculated, and the weights and biases of each layer are adjusted to minimize the error through error back propagation;
[0038] Perform multiple iterations until the maximum number of iterations is reached or the error is reduced to a predetermined target value, and finally obtain a trained BP neural network model.
[0039] The present invention also discloses a multi-camera human body recognition and positioning system, comprising:
[0040] An image acquisition module, used to acquire images taken by cameras distributed at different locations and having a common imaging space;
[0041] A human body recognition module is used to perform human body recognition on the acquired images and obtain a human head bounding box;
[0042] The pixel coordinate calculation module calculates the pixel coordinates of the center point of each bounding box based on the head bounding box, and obtains the position coordinates of multiple human bodies in each camera in the same pixel coordinate system;
[0043] The human body matching module is used to obtain the mapping relationship of the same human body in different images by using the Hungarian algorithm, realize human body matching, and obtain the position coordinates of the same human body in different pixel coordinate systems;
[0044] The human body positioning module is used to take the position coordinates of the same human body in different pixel coordinate systems as the input of the human body positioning model and output the position coordinates of the human body in the world coordinate system.
[0045] The present invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the multi-camera human body recognition and positioning method are implemented.
[0046] The present invention also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the multi-camera human body recognition and positioning method are implemented.
[0047] The present invention also discloses a computer program product, including a computer program / instruction, which implements the steps of the multi-camera human body recognition and positioning method when executed by a processor.
[0048] Compared with the prior art, the present invention has the following beneficial technical effects:
[0049] The present invention provides a human body recognition and positioning system applied to a lighting control system, which makes full use of existing security or monitoring network cameras in public places, uses multiple cameras to simultaneously shoot the indoor lighting environment, performs human body recognition on the captured images, generates a human head bounding box and outputs the pixel coordinates of the center point of the bounding box; uses the Hungarian algorithm to achieve human body matching, and obtains the position coordinates of the same human body in different pixel coordinate systems; uses the position coordinates of the same human body in different pixel coordinate systems as input, inputs them into a trained human body positioning model, and outputs the coordinates of the human body in the world coordinate system, thereby realizing personnel recognition and positioning in the lighting space. The positioning results are sent to the lighting controller to realize dimming control of the light group according to the personnel distribution information in the lighting environment. The present invention solves the calibration process that must be carried out when using cameras for personnel positioning at present, and can achieve visual comfort and energy saving according to the corresponding control algorithm based on the human body recognition and positioning results.
[0050] Furthermore, the YOLOv8 model is specifically adopted as a human recognition model, which has high accuracy and high speed.
[0051] Furthermore, the human head dataset is used to train the YOLOv8 model. By identifying the human head in the image and using the center point of the head bounding box as the position coordinate of the human body in the pixel coordinate system, the human head information is effectively used to describe the approximate spatial position of the human body on the ground, thereby achieving accurate positioning of the human body.
[0052] Furthermore, the human body positioning model adopts BP neural network to fit the complex relationship between pixel coordinates and world coordinates. The pixel coordinates of the human body in the three cameras are used as input values and the world coordinates corresponding to the human body are used as output values. The BP neural network is trained to realize the positioning of people in the lighting space. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 It is the structure diagram of BP neural network;
[0054] Figure 2 It is a schematic diagram of epipolar constraint;
[0055] Figure 3 A schematic diagram of the positioning of personnel for three cameras;
[0056] Figure 4 It is the system structure diagram;
[0057] Figure 5 This is the human detection result of the YOLOv8 model in the verification example; Figure a shows the pixel coordinates of the center points of the bounding boxes corresponding to multiple heads in camera C1, Figure b shows the pixel coordinates of the center points of the bounding boxes corresponding to multiple heads in camera C2, and Figure c shows the pixel coordinates of the center points of the bounding boxes corresponding to multiple heads in camera C3;
[0058] Figure 6 A flowchart of a multi-camera human body recognition and positioning method of the present invention;
[0059] Figure 7 The present invention is a principle block diagram of a multi-camera human body recognition and positioning system. DETAILED DESCRIPTION
[0060] The present invention is further described below in conjunction with specific embodiments.
[0061] Example 1
[0062] like Figure 6 As shown, the present invention discloses a multi-camera human body recognition and positioning method, comprising the following steps:
[0063] S1, obtaining images taken by cameras distributed at different locations and having a common imaging space;
[0064] S2, perform human body recognition on the images obtained in S1 respectively to obtain the head bounding box, calculate the pixel coordinates of the center point of each bounding box based on the head bounding box, and obtain the position coordinates of multiple human bodies in each camera in the same pixel coordinate system;
[0065] S3, using the Hungarian algorithm to obtain the mapping relationship of the same human body in different images, realize human body matching, and obtain the position coordinates of the same human body in different pixel coordinate systems;
[0066] S4. The position coordinates of the same human body in different pixel coordinate systems are used as the input of the human body positioning model, and the position coordinates of the human body in the world coordinate system are output to achieve accurate human body spatial positioning.
[0067] Example 2
[0068] In S2, the YOLOv8 model is used for human body recognition. This embodiment mainly introduces the process of establishing the human body recognition model.
[0069] (1) Using YOLOv8 as a model for human recognition
[0070] At present, deep learning algorithms dominate the field of human detection, including two-stage detection algorithms based on candidate boxes and one-stage detection algorithms based on regression. Although two-stage algorithms such as the R-CNN series have high precision, they have large parameter calculations and often cannot meet real-time requirements. One-stage algorithms such as YOLO directly regress the target position and classification probability, with high precision and fast speed. Therefore, the present invention selects the end-to-end target detection algorithm YOLOv8 for personnel detection to meet the needs of real-time detection in indoor environments.
[0071] Since the application scenario of the present invention is indoors, considering that the human body is easily blocked, the head recognition dataset is mainly used as the human body recognition dataset in the present invention;
[0072] (2) Create a human recognition dataset under lighting conditions
[0073] In the currently public human recognition data set, we select the annotated images of human heads suitable for this application scenario, and annotate the images collected in the lighting office space and store them in YOLO format to expand the data set, create a data set covering the application scenario of the present invention, and divide it into a training data set and a test data set. It includes a large number of human images of different sitting and standing postures and occluded human images, fully considering the human recognition in the case of sitting posture, occlusion, etc., and providing training samples for learning.
[0074] In the lighting space, in order to pay attention to the coordinates of the person on the ground, the human body is approximately regarded as a cylinder, which can clearly determine the position of the person in the image. The Z-axis direction perpendicular to the ground reflects the vertical height of the person, simplifies the tracking process of the human body, and reduces the complexity of the calculation.
[0075] By identifying the human head and using the pixel coordinates of the center point of the head bounding box as the position coordinates of the human body in the pixel coordinate system, the approximate spatial position of the human body on the ground is effectively described, thereby achieving accurate positioning of the human body.
[0076] (3) Conduct model training and testing to determine the model structure and network parameters
[0077] The YOLOv8 model is used for training on the above datasets, and appropriate loss functions and optimizers are selected. It is then iterated repeatedly to optimize the network weights and adjust hyperparameters, learning rates, and other parameters. In the testing phase, the model is inferred on the test dataset and its performance is evaluated.
[0078] Example 3
[0079] This embodiment mainly introduces the process of constructing a human body positioning model.
[0080] In S4, the process of constructing the human body positioning model specifically includes the following steps:
[0081] S4.1. Perform human body recognition on images captured by multiple cameras respectively to obtain a head bounding box, and use the coordinates of the center point of the head bounding box as an approximate method of the human body position coordinates.
[0082] By calculating the coordinates of the center point of the head bounding box in the pixel coordinate system, the coordinates of the person recognized by multiple cameras in the corresponding image are determined, such as Figure 3 As shown, taking three cameras as an example, they are respectively recorded as:
[0083]
[0084] in is the pixel coordinate of the center point of the head bounding box of the i-th person identified by camera C1, is the pixel coordinate of the center point of the head bounding box of the jth person identified by camera C2, is the pixel coordinate of the center point of the head bounding box of the kth person recognized by camera C3, and n is the total number of people recognized by the camera.
[0085] S4.2. For the overlapping area of the three cameras, the Hungarian algorithm is used to match the human bodies recognized by the three cameras. Combining position similarity and feature similarity, human body matching is transformed into a bipartite graph matching problem to solve the minimum weight.
[0086]
[0087] S4.3, using BP neural network in Matlab environment to fit the complex relationship between pixel coordinates and world coordinates, can effectively approximate the mapping relationship between them. BP neural network structure diagram is shown in Figure 1 shown.
[0088] Establish a BP neural network model, the model input is the coordinates of the human body identified in the three camera images in the pixel coordinate system The output is the coordinates of the human body in the world coordinate system (X W ,Y W ,Z W ).
[0089] The human body coordinates at 50 different positions were collected to establish a BP neural network data set, which was divided into a training data set and a test data set in a ratio of 8:2. The BP neural network model was then trained and tested.
[0090] The BP neural network input layer to the hidden layer is expressed as:
[0091] h i =f(x1·w 1i +x2·w 2i +x3·w 3i -β i ) (5)
[0092] Where h i is the output result of the i-th node in the hidden layer, x1=(u1,v1), x2=(u2,v2), x3=(u3,v3) are the three pixel coordinates of the input, w 1i 、w 2i 、w 3i is the weight of the three nodes in the input layer to the i-th node in the hidden layer, β i is the hidden layer bias, f is the sigmoid activation function and
[0093] The hidden layer to the output layer is expressed as:
[0094]
[0095] In the formula is the network output layer result, h1, h2, h3 are the output results of the three nodes in the network hidden layer, v1, v2, v3 are the weights from the three nodes in the hidden layer to the output layer, and λ is the output layer bias.
[0096] The training and testing process of BP neural network is as follows:
[0097] First, initialize the network's weights and biases;
[0098] The initial w, v weights and β, λ biases of a neural network are usually set manually.
[0099] Secondly, the three pixel coordinates of the same person obtained by the Hungarian algorithm are matched As training data, the network is input and the output value is calculated through forward propagation, that is, the predicted world coordinates of the person.
[0100] Then, the error between the output value and the actual value (i.e., the actual spatial coordinates of the person in the lighting space) is calculated, and the weights and biases of each layer are adjusted to minimize the error through error back propagation; this process is repeated for multiple iterations until the maximum number of iterations is reached or the error is reduced to the predetermined target value, and the BP neural network training is qualified.
[0101] The BP neural network transfers errors through back propagation, that is, by continuously adjusting all parameters in the network to minimize the loss function, the neural network model can achieve higher prediction accuracy.
[0102]
[0103] In the formula, is the predicted value of the i-th sample of the network, y i is the true value of the i-th sample.
[0104] Finally, the BP neural network is tested using the test data set to verify that the test is qualified.
[0105] Example 4
[0106] This paper mainly introduces how to use the Hungarian algorithm in S3 to achieve human body matching by combining position similarity and feature similarity.
[0107] (1) Position similarity
[0108] In multi-view geometry, the homography constraint describes the one-to-one correspondence between image points under two perspectives. For each point (x D1 ,y D1 ) in image D2 is (x D2 ,y D2 ), which can be expressed as: The matrix H is a third-order homography matrix with 8 degrees of freedom and rank 3. When the coordinates of at least four sets of corresponding points are given, the homography matrix H can be solved by a linear method. Using the homography matrix H, the bounding box in the image D1 is homographically transformed to establish the association between the two cameras.
[0109] Assume that the coordinate set of the center point of the bounding box detected by camera C1 is: The coordinate set after homography transformation is: The coordinate set of the center point of the bounding box detected by camera C2 is: Calculate the Euclidean distance between the coordinates of camera C1 after homography transformation and the coordinates of camera C2. The smaller the d value, the closer the positions of the two bounding boxes are.
[0110]
[0111] (2) Feature similarity
[0112] The head bounding box detected by YOLOv8 is resized to 64×128, and the HOG features of the bounding box are extracted. Each detected bounding box generates a feature vector, and all feature vectors of cameras C1 and C2 form two arrays. and Calculate the cosine similarity between two arrays to construct a similarity matrix. The cosine similarity formula is:
[0113]
[0114] The final association weight is obtained by combining position similarity with feature similarity. The calculation formula is:
[0115]
[0116] In the formula α is the weight factor. Calculate W for all bounding boxes in C1 and C2 ij Weight, based on the Hungarian algorithm matching mathematical model can be described as:
[0117]
[0118] where z ij is the decision variable, and defines the matching matrix Z n×n satisfy:
[0119]
[0120] The same applies to matching C2 and C3.
[0121] Furthermore, the specific steps of using the Hungarian algorithm for head matching in S4 include: first, subtract the minimum value from each row and column in the square matrix, and then cover all zero values with the least number of lines. If the number of lines is equal to the size of the square matrix, the optimal allocation is found; otherwise, the minimum value is subtracted from the uncovered elements, the minimum value is added to the elements on the crossing lines, and this process is repeated until the optimal allocation is found. Finally, the corresponding relationship between the coordinates of the same person in the C1, C2 and C3 pixel coordinate systems is obtained, that is, the human body coordinate set M in the two images is:
[0122]
[0123] Where R i is the i-th person in the space.
[0124] Example 5
[0125] This paper mainly introduces the basis and advantages of using BP neural network in S4 compared with the traditional multi-camera human body positioning method.
[0126] Traditional multi-camera human body positioning technology is usually based on the dual-camera epipolar constraint method. By calibrating the internal and external parameters of the camera, the spatial position of the human body is determined based on the matching points in the image pair and the geometric relationship between the cameras, and the positioning results of multiple cameras are fused. This method can be divided into four processes: camera calibration, feature extraction and matching, calculation of the basic matrix and the essential matrix, and positioning.
[0127] (1) Camera calibration
[0128] Firstly, two frames of images captured by a camera at a fixed position are obtained at the same time, and the camera is calibrated using the Zhang Zhengyou calibration method to obtain the camera's intrinsic parameter matrix and distortion parameters, thereby correcting the image distortion.
[0129] (2) Feature extraction and matching
[0130] The ORB algorithm is used to detect feature points and extract descriptors from the two images. Finally, the BFMatcher brute force matcher is used to match the feature points extracted from the two images to find the corresponding relationship between them.
[0131] (3) The schematic diagram of solving the epipolar constraints of the basic matrix and the essential matrix is as follows Figure 2 As shown, C1 and C2 represent the optical centers of the two cameras, P is a point in space, and the projections of P on the corresponding imaging planes of C1 and C2 are p1 and p2 respectively. The intersection points e1 and e2 of the line connecting C1 and C2 and the image planes π1 and π2 are called poles, and the plane formed by the three points C1, C2, and P is the polar plane. The intersection lines l1 and l2 of the polar plane and π1 and π2 are called polar lines.
[0132] Let P = [X, Y, Z] T According to the pinhole camera model, the projection points of P on the two image planes are:
[0133]
[0134] Where d1 and d2 are the Z-axis coordinates in the camera coordinate system, K is the camera intrinsic matrix, R is the rotation matrix, and T is the translation matrix. After simplification, we have:
[0135]
[0136] Let F = K -T T×R -1 , then the pixel points p1 and p2 on the imaging plane have epipolar constraints:
[0137]
[0138] Where F is the 3×3 basic matrix, let x = K -1 p, then we have
[0139]
[0140] Where E is a 3×3 essential matrix and E=T×R, which is determined by the external parameters R and T. Formula (17) can be written as This means that x2 is on the line l2 = Ex1. That is, once the position of p1 is known, the matching point can be found by searching along the epipolar line. The RANSAC method is usually used to solve the basic matrix, and the intrinsic matrix is calculated in combination with the camera's intrinsic matrix. After performing singular value decomposition on the essential matrix, the relative positions R and T of the camera can be obtained.
[0141] (4) Positioning
[0142] The triangulation method is used to convert the matched feature points into coordinates in three-dimensional space. Taking the position of the C1 camera as the origin of the world coordinate system, let the projection matrices of the two cameras be
[0143] M1=K·[I 3×3 0 3×1 ],M1∈R 3×4 (18)
[0144] M2=K·[R 3×3 T 3×1 ],M2∈R 3×4 (19)
[0145] Since the cross product of a vector with itself is a zero vector, the cross product is used to eliminate the homogeneous factors:
[0146]
[0147] in are the pixel coordinates of the two cameras respectively The homogeneous coordinates of For space point The homogeneous coordinates of
[0148]
[0149] The least squares method is used to solve the linear equations, and the singular value decomposition of the coefficient matrix A is calculated. The smallest singular vector is selected as the solution of the three-dimensional coordinate P. However, the world coordinates obtained at this time are relative coordinates relative to the C1 camera, rather than absolute coordinates in space. Therefore, the external parameters of the C1 camera need to be calibrated to achieve accurate positioning.
[0150] The epipolar constraint positioning method requires the calibration of the internal and external parameters of the camera. However, in actual engineering applications, the calibration process becomes complicated due to factors such as environmental complexity, lighting changes, and camera installation angles, especially when the camera is installed in a difficult-to-operate position. Therefore, in engineering applications, it is often difficult to implement the calibration method for personnel positioning.
[0151] It can be seen from formula (22) that when the camera position is fixed, there is a definite mapping relationship between the pixel coordinates and the world coordinates. Therefore, the present invention uses a BP neural network to fit the mapping relationship, eliminating the complex calibration process of the camera and realizing personnel positioning quickly and accurately.
[0152] Verification Example
[0153] First, the camera system is arranged, which consists of three cameras and a computer. The system structure is as follows Figure 4 As shown, three cameras are arranged in the measured environment area and connected through a network. A computer obtains images of the cameras at the same time through the network. A human body recognition model and a positioning model are installed in the computer.
[0154] The specific identification and positioning process is as follows:
[0155] Step 1: The computer issues instructions to collect images from cameras C1, C2, and C3 simultaneously;
[0156] Step 2: Run the human body recognition model to perform head recognition on the images obtained by the three cameras, generate a head bounding box, and calculate the pixel coordinates [u, v] of the center of the head bounding box, such as Figure 5 As shown, Figure a shows the pixel coordinates of the center points of the bounding boxes corresponding to multiple heads in camera C1, Figure b shows the pixel coordinates of the center points of the bounding boxes corresponding to multiple heads in camera C2, and Figure c shows the pixel coordinates of the center points of the bounding boxes corresponding to multiple heads in camera C3.
[0157] Determine the pixel coordinates of the center of the bounding boxes of multiple heads identified in each camera, and record them as:
[0158]
[0159] Step 3: For the overlapping areas of the three cameras, the Hungarian algorithm is used to match the human bodies identified by the three cameras in combination with position similarity and feature similarity, and the corresponding coordinates of the same person in the C1, C2 and C3 pixel coordinate systems are obtained, as shown in Table 1. That is, the human body coordinate set M in the three images is:
[0160] Where R i is the i-th person in the space.
[0161] Table 1 Image matching results
[0162] personnel <![CDATA[C1 pixel coordinates]]> <![CDATA[C2 pixel coordinates]]> <![CDATA[C3 pixel coordinates]]> 1 (547,177) (951,151) (1323,66) 2 (1182,668) (473,369) (487,178) 3 (1463,124) (1749,367) (1028,492) 4 (903,174) (1296,259) (1524,250) 5 (878,266) (1109,290) (1193,247) 6 (1796,476) (957,519) (255,281)
[0163] Step 4: Run the human body positioning model. The model input is the coordinates of the human body in the pixel coordinate system identified in the three-camera image. The output is the coordinates of the human body in the world coordinate system This enables the positioning of the human body in three-dimensional space.
[0164] The positioning results of the human body positioning model are shown in Table 2.
[0165] Table 2 BP neural network human body positioning results (unit: cm)
[0166]
[0167]
[0168] The actual spatial coordinates of the human body are (X' W ,Y' W ,Z' W ), the prediction coordinates of the BP network are (X W ,Y W ,Z W ), the Euclidean distance is used to represent the positioning error. The specific calculation formula is:
[0169]
[0170] After statistics and calculations, the average positioning error of the present invention is 9.7 centimeters. It can accurately locate the position of a person without camera calibration, and can meet the needs of human body positioning in indoor lighting.
[0171] like Figure 7As shown, the present invention also discloses a multi-camera human body recognition and positioning system, comprising:
[0172] An image acquisition module, used to acquire images taken by cameras distributed at different locations and having a common imaging space;
[0173] A human body recognition module is used to perform human body recognition on the acquired images and obtain a human head bounding box;
[0174] The pixel coordinate calculation module calculates the pixel coordinates of the center point of each bounding box based on the head bounding box, and obtains the position coordinates of multiple human bodies in each camera in the same pixel coordinate system;
[0175] The human body matching module is used to obtain the mapping relationship of the same human body in different images by using the Hungarian algorithm, realize human body matching, and obtain the position coordinates of the same human body in different pixel coordinate systems;
[0176] The human body positioning module is used to take the position coordinates of the same human body in different pixel coordinate systems as the input of the human body positioning model and output the position coordinates of the human body in the world coordinate system.
[0177] The multi-camera human body recognition and positioning method of the present invention can be implemented in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention can be implemented in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0178] If the multi-camera human body recognition and positioning method of the present invention is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. Computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals. Among them, the computer storage medium can be any available medium or data storage device that can be accessed by the computer, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drive (SSD)), etc.
[0179] In an exemplary embodiment, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the multi-camera human body recognition and positioning method when executing the computer program. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc.
[0180] The present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the multi-camera human body recognition and positioning method when executed by a processor.
[0181] The content of the present invention is not limited to the above embodiments, and various modifications or variations that can be made by those skilled in the art without creative work are all covered by the claims of the present invention.
Claims
1. A multi-camera human body recognition and positioning method, characterized in that: The following steps are involved: S1, obtaining images taken by cameras distributed at different locations and having a common imaging space; S2, perform human body recognition on the images obtained in S1 to obtain the bounding box of the human head; S3, calculating the pixel coordinates of the center point of each bounding box based on the head bounding box, and obtaining the position coordinates of multiple human bodies in each camera in the same pixel coordinate system; S4. Using the Hungarian algorithm to obtain the mapping relationship of the same human body in different images, achieve human body matching, and obtain the position coordinates of the same human body in different pixel coordinate systems; S5. Use the position coordinates of the same human body in different pixel coordinate systems as the input of the human body positioning model, and output the position coordinates of the human body in the world coordinate system.
2. A multi-camera human body recognition and positioning method according to claim 1, characterized in that: In S2, the trained YOLOv8 model is used to perform human recognition on images taken by multiple cameras.
3. A multi-camera human body recognition and positioning method according to claim 2, characterized in that: The human head dataset is used to train the YOLOv8 model. By identifying the human head, the human head bounding box is obtained, and the center point of the human head bounding box is used as the position coordinate of the human body in the pixel coordinate system. The source of the human body recognition dataset is as follows: human body images are collected through a camera under a lighting environment, the collected images are annotated using the LabelImg tool and stored in the YOLO format to produce a human body recognition dataset in a lighting environment.
4. The multi-camera human body recognition and positioning method according to claim 1, characterized in that: In S3, the position coordinates of multiple human bodies in each camera in the same pixel coordinate system are recorded as: in is the i-th person identified by camera C1, is the jth person identified by camera C2, is the kth person identified by camera C3, and n is the total number of people identified by the camera.
5. A multi-camera human body recognition and positioning method according to claim 4, characterized in that: S4 is specifically: Combining position similarity and feature similarity, the human body matching problem of three cameras is transformed into a bipartite graph matching problem to solve the minimum weight, and the associated weight is obtained: In the formula α is the weight factor, The pixel coordinates of the center point of the bounding box detected by camera C1 after homography transformation The coordinates of camera C2 The Euclidean distance between them, S(a,b) is the two arrays composed of all feature vectors of camera C1 and camera C2 and The cosine similarity matrix between ; Calculate W for all bounding boxes in C1 and C2 ij Weight, using the Hungarian algorithm for human matching; Similarly, match C2 and C3; Finally, we get the corresponding relationship between the coordinates of the same person in the C1, C2 and C3 pixel coordinate systems, and the human body coordinate set M in the three images is: Where R i is the i-th person in the space.
6. The multi-camera human body recognition and positioning system according to claim 1, characterized in that: In S5, the human body positioning model adopts the BP neural network model. The training process of the BP neural network model is as follows: First, initialize the weights and biases of the BP neural network model; Second, the position coordinates of the same human body in different pixel coordinate systems are input into the BP neural network model as training data, and the world coordinates of the predicted person are calculated through forward propagation as the predicted value; Third, the error between the predicted value and the actual spatial coordinates of the human body in the lighting space is calculated, and the weights and biases of each layer are adjusted to minimize the error through error back propagation; Perform multiple iterations until the maximum number of iterations is reached or the error is reduced to a predetermined target value, and finally obtain a trained BP neural network model.
7. A multi-camera human body recognition and positioning system, characterized in that: include: An image acquisition module, used to acquire images taken by cameras distributed at different locations and having a common imaging space; A human body recognition module is used to perform human body recognition on the acquired images and obtain a human head bounding box; The pixel coordinate calculation module calculates the pixel coordinates of the center point of each bounding box based on the head bounding box, and obtains the position coordinates of multiple human bodies in each camera in the same pixel coordinate system; The human body matching module is used to obtain the mapping relationship of the same human body in different images by using the Hungarian algorithm, realize human body matching, and obtain the position coordinates of the same human body in different pixel coordinate systems; The human body positioning module is used to take the position coordinates of the same human body in different pixel coordinate systems as the input of the human body positioning model and output the position coordinates of the human body in the world coordinate system.
8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the multi-camera human body recognition and positioning method as described in any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the multi-camera human body recognition and positioning method as described in any one of claims 1 to 6 are implemented.
10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the multi-camera human body recognition and positioning method described in any one of claims 1-6 are implemented.
Citation Information
Patent Citations
Personnel-oriented air supply adjustment and control method and device based on image recognition
CN111932630A
Aircraft multi-target tracking method based on improved YOLOV5 algorithm
CN114529799A
Insulator space positioning method based on binocular vision
CN115965687A
Space target pose resolving method based on deep learning and computer program product
CN116363205A
Calibration-free single-camera personnel positioning method and system, computer equipment and storage medium
CN118537540A
Cited By
Passenger monitoring method, device, equipment, storage medium and program product
CN121999471A
