A method for locating and tracking target personnel based on image processing
Through image processing and convolutional neural network technology, the accuracy of target personnel positioning and trajectory tracking is improved, the problem of low accuracy in the existing methods is solved, and faster and more accurate positioning and tracking is achieved.
Patent Information
- Application Number
- CN202410285392.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-13
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-03-13
AI Technical Summary
The existing methods have low accuracy in target personnel positioning and trajectory tracking, resulting in long data processing time and limited positioning accuracy, affecting the accuracy of subsequent data acquisition.
Using an image-based processing method, multiple cameras are used to synchronize images, perform grayscale processing and LBP feature extraction, and extract features through convolutional neural networks, calculate face feature similarity to accurately locate the target person's position, and adjust the camera according to the position. Acquire images, reduce data volume and improve trajectory tracking accuracy.
It significantly improves the accuracy of target personnel positioning and the accuracy of trajectory tracking, reduces data processing volume and time, and improves positioning speed.
Smart Images

Figure CN118196682B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of target personnel positioning and trajectory tracking, and in particular relates to a target personnel positioning and trajectory tracking method based on image processing. Background Art
[0002] With the rapid development of Internet of Things technology, network monitoring data with video surveillance technology as the main carrier has shown explosive growth, and video surveillance technology has been widely used in security warning, combating crimes and other fields. However, the initial data processing methods for video surveillance systems were mainly manual, and the disadvantages of manual processing are low work efficiency, low information utilization and high error rate. However, with the continuous development of deep learning technology, machines can have a deeper understanding of images and extract deeper features, making the effect of neural networks in the field of image recognition exceed that of manual processing. In the fight against crimes, it is very necessary to accurately locate and track the target personnel. Due to the huge advantages of neural networks in the field of image recognition, video surveillance technology based on neural networks has developed rapidly.
[0003] However, the existing methods use all monitoring data from multiple perspectives to obtain the location information of the target person. This method of using all monitoring data from multiple perspectives for positioning requires a large amount of data to be processed, so the data processing takes a long time, resulting in a long time required for positioning. In addition, the existing methods do not focus on key images in the process of using monitoring data to locate the target, so the positioning accuracy is still relatively limited. The level of positioning accuracy directly affects the accuracy of the data acquisition that needs to be processed at the next moment. Therefore, the positioning accuracy also directly affects the accuracy of tracking the trajectory of the target person.
[0004] In summary, the accuracy of existing methods for positioning and tracking target personnel is still low, and it is necessary to propose a new method to solve the above problems. Summary of the invention
[0005] The purpose of the present invention is to solve the problem of low accuracy of positioning and tracking the trajectory of a target person in existing methods, and to propose a method for positioning and tracking the trajectory of a target person based on image processing.
[0006] The technical solution adopted by the present invention to solve the above technical problems is:
[0007] A method for locating and tracking a target person based on image processing, the method specifically comprising the following steps:
[0008] Step 1: at the current time l, multiple cameras arranged in space are used to synchronously collect the original image of the area where the target person is located, and then grayscale processing is performed on each frame of the collected original image to obtain each frame of grayscale image;
[0009] Then, each frame of grayscale image is processed respectively to obtain the position of the moving person in each frame of grayscale image and the position of the moving person in each frame of original image; and a sub-image of the position of each moving person is cut out from each frame of original image and grayscale image respectively to obtain each sub-image pair;
[0010] Step 2: Obtain the face image of the target person stored in the system, and use the obtained face image as a sample image; then perform grayscale processing on the obtained sample image to obtain a sample grayscale image, and then convert the sample grayscale image into a sample LBP feature image;
[0011] Step 3: convert the grayscale image in each sub-image pair obtained in step 1 into an LBP feature image, and obtain a face region image pair to be matched based on the converted image;
[0012] Step 4: Use the first convolutional neural network to extract features from the sample image, and use the second convolutional neural network to extract features from the sample LBP feature image to obtain facial features in the sample image;
[0013] For any pair of face region images to be matched, the first convolutional neural network is used to extract features from the original image in the face region image pair to be matched, and the second convolutional neural network is used to extract features from the LBP feature image in the face region image pair to be matched, so as to obtain the face features in the face region image pair to be matched; similarly, the face features of each face region image pair to be matched are obtained respectively;
[0014] Then, the similarity between the facial features in the sample image and the facial features of each to-be-matched face region image pair is calculated respectively, and it is determined whether there is a to-be-matched face region image pair whose similarity is greater than a threshold value;
[0015] If it exists, then select the to-be-matched face region image pair corresponding to the similarity greater than the threshold, determine the target person's position based on the selected to-be-matched face region image pair, and then execute step five;
[0016] If it does not exist, set time l=l+1 and return to step 1. At this time, there is no need to re-determine which cameras need to be started;
[0017] Step 5: Determine the camera that needs to be activated at the next moment according to the position of the target person obtained in step 4, set l=l+1, and then use the activated camera to return to step 1.
[0018] Furthermore, the grayscale images of each frame are processed respectively to obtain the position of the moving person in each grayscale image frame and the position of the moving person in each original image frame; specifically:
[0019] Step 11: For a certain camera, the grayscale image corresponding to the original image collected by the camera at the current time l is recorded as I l , the grayscale image corresponding to the original image collected by the camera at the previous time l-1 is recorded as I l-1 , then the change d of the gray value of the pixel (x, y) in the two gray images before and after l (x,y) is:
[0020] d l (x,y)=|I l (x,y)-I l-1 (x,y)|
[0021] Among them, I l (x,y) is the grayscale image I l The gray value of the pixel (x, y), I l-1 (x,y) is the grayscale image I l-1 The grayscale value of the pixel (x, y), |·| represents the absolute value;
[0022] Step 12: Grayscale image I l Medium change d l The pixel points whose (x, y) is greater than or equal to the threshold TH are taken as foreground points, and the set of foreground points obtained is recorded as O1;
[0023] Step 13: Grayscale image I l The maximum gray value of the pixel is recorded as z max , the grayscale image I l The minimum gray value of the pixel is recorded as z min , then the initial segmentation threshold Q 0 for:
[0024]
[0025] Step 14, initialize the number of iterations k = 0;
[0026] Step 15: Grayscale image I l The gray value is less than or equal to the segmentation threshold Q k The pixel point is taken as the foreground point, and then the grayscale average value z0 of the foreground point is calculated; the grayscale image I l The gray value is greater than the segmentation threshold Q k The pixel point is taken as the background point, and then the gray value z of the background point is calculated. B ;
[0027] Step 16: According to z0 and z B Calculate the segmentation threshold Q k+1 :
[0028]
[0029] Step 17: Determine whether Q is satisfied k+1 =Q k ;
[0030] If Q is satisfied k+1 =Q k , then use the grayscale image I l The gray value is less than or equal to the segmentation threshold Q k+1 The pixel points of the foreground point set O2 are formed, and then step 18 is executed;
[0031] If Q is not satisfied k+1 =Q k , then set k=k+1 and return to step 15;
[0032] Step 18: Find the union of sets O1 and O2, and use the pixels in the union as the grayscale image I l All the foreground points in the grayscale image I are used, and other pixels are used as background points. Morphological filtering and connectivity analysis are then performed to obtain the grayscale image I of each moving person. l The position of the moving person in the grayscale image I l The position of the moving person in the original image is obtained by using the position of the moving person in the original image;
[0033] Step 19: Use the methods of steps 11 to 18 to process the grayscale images corresponding to each frame of the original image collected at the current moment.
[0034] Furthermore, the sample LBP feature image is obtained by converting the sample grayscale image through an LBP operator.
[0035] Furthermore, the first convolutional neural network and the second convolutional neural network have the same structure.
[0036] Furthermore, the first convolutional neural network is used to extract features from the sample image, and the second convolutional neural network is used to extract features from the sample LBP feature image to obtain facial features in the sample image; specifically:
[0037] The first convolutional neural network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a seventh convolutional layer, an eighth convolutional layer, a ninth convolutional layer, a tenth convolutional layer, an eleventh convolutional layer, a twelfth convolutional layer, a thirteenth convolutional layer, a fourteenth convolutional layer, a fifteenth convolutional layer, a sixteenth convolutional layer, a seventeenth convolutional layer, an eighteenth convolutional layer, a nineteenth convolutional layer and a twentieth convolutional layer;
[0038] For the sample image:
[0039] The sample image is used as the input of the first convolutional layer. The output of the first convolutional layer passes through the second and third convolutional layers in sequence. The output of the first convolutional layer is then concatenated with the output of the third convolutional layer to obtain the concatenated result a. The concatenated result a is used as the input of the fourth convolutional layer.
[0040] The output of the fourth convolutional layer then passes through the fifth and sixth convolutional layers in sequence. The output of the fourth convolutional layer is concatenated with the output of the sixth convolutional layer to obtain a concatenated result b. The concatenated result b then passes through the seventh and eighth convolutional layers in sequence. The output of the eighth convolutional layer is concatenated with the concatenated result b to obtain a concatenated result c.
[0041] The concatenated result c is used as the input of the ninth convolutional layer. The output of the ninth convolutional layer then passes through the tenth and eleventh convolutional layers in sequence. The output of the ninth convolutional layer is concatenated with the output of the eleventh convolutional layer to obtain the concatenated result d.
[0042] The splicing result d then passes through the twelfth convolutional layer and the thirteenth convolutional layer in sequence, and the splicing result d is spliced with the output of the thirteenth convolutional layer to obtain the splicing result e;
[0043] The splicing result e passes through the fourteenth convolutional layer and the fifteenth convolutional layer in sequence, and the splicing result e is spliced with the output of the fifteenth convolutional layer to obtain the splicing result f;
[0044] The splicing result f passes through the sixteenth convolutional layer and the seventeenth convolutional layer in sequence, and the splicing result f is spliced with the output of the seventeenth convolutional layer to obtain the splicing result g;
[0045] The concatenated result g is used as the input of the 18th convolutional layer. The output of the 18th convolutional layer passes through the 19th and 20th convolutional layers in sequence. The outputs of the 18th and 20th convolutional layers are then concatenated to obtain the concatenated result h, which is used as feature A.
[0046] Similarly, the sample LBP feature image is used as the input of the second convolutional neural network, and the feature B of the sample LBP feature image is output through the second convolutional neural network;
[0047] Feature A and feature B are fused, and the feature fusion result is used as the input of the pooling layer. The output of the pooling layer passes through the fully connected layer, and the facial features in the sample image are output through the fully connected layer.
[0048] Furthermore, the target person's position is determined based on the selected face area image pairs to be matched. If the number of selected face area image pairs to be matched is 1, the original image corresponding to the selected face area image pairs to be matched in step one is obtained, and the target person's position is determined based on the obtained original image.
[0049] Furthermore, the position of the target person is determined based on the selected to-be-matched face region image pairs. If the number of the selected to-be-matched face region image pairs is greater than or equal to 2, the method for determining the position of the target person is:
[0050] Step 41: Select the to-be-matched face region image pairs corresponding to the two largest similarities, and obtain the sub-image pairs corresponding to the selected to-be-matched face region image pairs in step 1;
[0051] Step 42: The two sub-image pairs obtained in step 41 are respectively taken as the first sub-image pair and the second sub-image pair; the camera that captures the first sub-image pair is recorded as C1, and the camera that captures the second sub-image pair is recorded as C2;
[0052] The center pixel of the target person in the first sub-image is denoted as A1, and the coordinates of pixel A1 in the world coordinate system are (x1, y1, z1). The center pixel of the target person in the second sub-image is denoted as A2, and the coordinates of pixel A2 in the world coordinate system are (x2, y2, z2). The optical center of camera C1 is denoted as o1, and the optical center of camera C2 is denoted as o2. Then, the line connecting pixel A1 and optical center o1 is straight line l1, and the line connecting pixel A2 and optical center o2 is straight line l2. The direction vectors of straight line l1 and straight line l2 are:
[0053]
[0054]
[0055] Among them, m1, n1, p1 are the elements in the direction vector of the line l1, and m2, n2, p2 are the elements in the direction vector of the line l2;
[0056] Then the equations of line l1 and line l2 are:
[0057]
[0058]
[0059] Step 43: Calculate the vector based on the direction vectors of line l1 and line l2
[0060]
[0061] Draw a line through line l1 parallel to the vector The plane D1, through the line l2 parallel to the vector plane D2, and then get the intersection of plane D1 and plane D2, record the intersection point of the intersection line and straight line l1 as p1, record the intersection point of the intersection line and straight line l2 as p2, and then get the midpoint coordinates of the line segment p1p2, and use the obtained midpoint coordinates as the position of the target person.
[0062] Furthermore, the position of the target person is determined based on the selected to-be-matched face region image pairs. If the number of the selected to-be-matched face region image pairs is greater than or equal to 2, the method for determining the position of the target person is:
[0063] Step 41: Select the to-be-matched face region image pairs corresponding to the two largest similarities, and obtain the sub-image pairs corresponding to the selected to-be-matched face region image pairs in step 1;
[0064] Step 42: The two sub-image pairs obtained in step 41 are respectively taken as the first sub-image pair and the second sub-image pair; the camera that captures the first sub-image pair is recorded as C1, and the camera that captures the second sub-image pair is recorded as C2;
[0065] The center pixel of the target person in the first sub-image is denoted as A1, and the coordinates of pixel A1 in the world coordinate system are (x1, y1, z1). The center pixel of the target person in the second sub-image is denoted as A2, and the coordinates of pixel A2 in the world coordinate system are (x2, y2, z2). The optical center of camera C1 is denoted as o1, and the optical center of camera C2 is denoted as o2. Then, the line connecting pixel A1 and optical center o1 is straight line l1, and the line connecting pixel A2 and optical center o2 is straight line l2. The direction vectors of straight line l1 and straight line l2 are:
[0066]
[0067]
[0068] Then the equations of line l1 and line l2 are:
[0069]
[0070]
[0071] Step 43: Calculate the vector based on the direction vectors of line l1 and line l2
[0072]
[0073] Draw a line through line l1 parallel to the vector The plane D1, through the line l2 parallel to the vector The plane D2 is obtained, and then the intersection line of plane D1 and plane D2 is obtained. The intersection point of the intersection line and the straight line l1 is recorded as p1, and the intersection point of the intersection line and the straight line l2 is recorded as p2. Then the midpoint coordinates of the line segment p1p2 are obtained, and the obtained midpoint coordinates are used as the position 1 of the target person;
[0074] Step 4: Let the plane equation of plane D be z=z a , let the intersection of line l1 and plane D be a1, let the intersection of line l2 and plane D be a2, find the z that minimizes the distance between the intersection a1 and the intersection a2 a , when the distance between the intersection point a1 and the intersection point a2 is the smallest, the midpoint of the intersection point a1 and the intersection point a2 is calculated, and the midpoint of the intersection point a1 and the intersection point a2 is taken as the position 2 of the target person;
[0075] Step 4 and 5: Calculate the midpoint between the target person's position 1 and the target person's position 2 to obtain the final target person's position.
[0076] It should be noted that the central pixel point of the target person may be determined in the following manner: a minimum circumscribed rectangle of the area where the target person is located is determined, and the center of the determined rectangle is used as the central pixel point of the target person.
[0077] The beneficial effects of the present invention are:
[0078] The present invention uses multiple cameras arranged in space to synchronously collect the original images of the area where the target person is located, and then uses the built convolutional neural network to extract the features of the collected images and sample images respectively, and then selects the matching face area image according to the feature similarity. The selected face area image is used to locate the target person, which can significantly improve the positioning accuracy. By improving the positioning accuracy, the camera that needs to collect images for target positioning at the next moment can be accurately started, reducing the amount of data while improving the accuracy of trajectory tracking. BRIEF DESCRIPTION OF THE DRAWINGS
[0079] Figure 1 The present invention is a flow chart of a method for locating and tracking a target person based on image processing. DETAILED DESCRIPTION
[0080] Specific implementation method 1: Combination Figure 1 The present embodiment describes a method for locating and tracking a target person based on image processing, and the method specifically comprises the following steps:
[0081] Step 1: At the current moment l, multiple cameras arranged in space are used to synchronously collect the original image of the area where the target person is located (for the first moment, since the exact location of the target person at that moment cannot be known before image processing, for the possible location of the target person, each camera covering the location is used to collect images, and each camera collects one frame of image. Since the exact location cannot be obtained, the number of cameras started is relatively large, and it is necessary to ensure that all possible locations are covered. At the next moment, the camera continues to collect images. For each frame of the collected image, the moving person in the image can be determined, and then the face image of the moving person is compared with the face image of the target person. Similarity of image pairs. When a face image pair with a similarity greater than a threshold is collected, it means that the target person has been detected. After the position of the target person is determined, the camera can be started in a targeted manner at this moment. For example, the cameras in front, behind, left and right of the position of the target person are started. However, due to the uncertainty of the moving direction, not all the started cameras can collect the image of the target person. However, for each frame of the collected image, the moving person can be determined, and then it is judged whether there is the target person among the moving persons. If the target person moves, the positioning process is executed, otherwise, the image collection at the next moment is started directly). Then, each frame of the collected original image is gray-scaled to obtain each frame of gray-scale image.
[0082] Then, each frame of grayscale image is processed respectively to obtain the position of the moving person in each frame of grayscale image and the position of the moving person in each frame of original image; and a sub-image of the position of each moving person is cut out from each frame of original image and grayscale image to obtain each sub-image pair (for one frame of original image, the sub-image of the position of the moving person in the frame of original image is cut out, and then the sub-image of the position of the moving person is cut out from the grayscale image corresponding to the frame of original image, and the two cut out sub-images form an image pair);
[0083] Each frame of grayscale image is processed separately to obtain the position of the moving person in each frame of grayscale image and the position of the moving person in each frame of original image; specifically:
[0084] Step 11: For a certain camera, the grayscale image corresponding to the original image collected by the camera at the current time l is recorded as I l , the grayscale image corresponding to the original image collected by the camera at the previous time l-1 is recorded as I l-1 , then the change d of the gray value of the pixel (x, y) in the two gray images before and after l (x,y) is:
[0085] d l (x,y)=|Il (x,y)-I l-1 (x,y)|
[0086] Among them, I l (x,y) is the grayscale image I l The gray value of the pixel (x, y), I l-1 (x,y) is the grayscale image I l-1 The grayscale value of the pixel (x, y), |·| represents the absolute value;
[0087] Step 12: Grayscale image I l Medium change d l The pixel points whose (x, y) is greater than or equal to the threshold TH (which can be set according to actual conditions) are taken as foreground points, and the set of foreground points obtained is recorded as O1;
[0088] Step 13: Grayscale image I l The maximum gray value of the pixel is recorded as z max , the grayscale image I l The minimum gray value of the pixel is recorded as z min , then the initial segmentation threshold Q 0 for:
[0089]
[0090] Step 14, initialize the number of iterations k = 0;
[0091] Step 15: Grayscale image I l The gray value is less than or equal to the segmentation threshold Q k The pixel point is taken as the foreground point, and then the grayscale average value z0 of the foreground point is calculated; the grayscale image I l The gray value is greater than the segmentation threshold Q k The pixel point is taken as the background point, and then the gray value z of the background point is calculated. B ;
[0092] Step 16: According to z0 and z B Calculate the segmentation threshold Q k+1 :
[0093]
[0094] Step 17: Determine whether Q is satisfied k+1 =Q k ;
[0095] If Q is satisfied k+1 =Q k , then use the grayscale image I l The gray value is less than or equal to the segmentation threshold Q k+1The pixel points of the foreground point set O2 are formed, and then step 18 is executed;
[0096] If Q is not satisfied k+1 =Q k , then set k=k+1 and return to step 15;
[0097] Step 18: Find the union of sets O1 and O2, and use the pixels in the union as the grayscale image I l All the foreground points in the grayscale image I are used, and other pixels are used as background points. Morphological filtering and connectivity analysis are then performed to obtain the grayscale image I of each moving person. l According to the location of the moving person in the grayscale image I l The position of the moving person in the original image is obtained by using the position of the moving person in the original image;
[0098] According to the position correspondence between the original image and the grayscale image, the position of the moving person in the original image is obtained;
[0099] Step 19: Use the methods of steps 11 to 18 to process the grayscale images corresponding to each frame of the original image collected at the current moment.
[0100] Step 2: Obtain the face image (color image here) of the target person stored in the system, and use the obtained face image as the sample image; then grayscale process the obtained sample image to obtain a sample grayscale image, and then convert the sample grayscale image into a sample LBP feature image (the sample LBP feature image is obtained by converting the sample grayscale image through the LBP operator);
[0101] Step 3: Convert the grayscale image in each sub-image pair obtained in step 1 into an LBP feature image, and obtain the face region image pair to be matched according to the converted image (since the converted sub-image pair includes the entire pedestrian, the face region image pair to be matched can be cut out from it);
[0102] Step 4: Build a first convolutional neural network and a second convolutional neural network with the same structure, use the first convolutional neural network to extract features from the sample image, and use the second convolutional neural network to extract features from the sample LBP feature image to obtain facial features in the sample image;
[0103] For any pair of face area images to be matched, the first convolutional neural network is used to extract features from the original image in the face area image pair to be matched, and the second convolutional neural network is used to extract features from the LBP feature image in the face area image pair to be matched, so as to obtain the face features in the face area image pair to be matched;
[0104] Similarly, the facial features of each pair of face region images to be matched are obtained respectively;
[0105] Then, the similarity between the facial features in the sample image and the facial features of each to-be-matched face region image pair is calculated respectively, and it is determined whether there is a to-be-matched face region image pair whose similarity is greater than a threshold value;
[0106] If it exists (indicating that the position of the target person has moved), then select the to-be-matched face region image pair corresponding to the similarity greater than the threshold (which can be set according to the actual situation), determine the position of the target person based on the selected to-be-matched face region image pair, and then execute step five;
[0107] If not (indicating that the position of the target person has not moved significantly), set time l=l+1 and return to step one.
[0108] 1. Feature extraction process
[0109] The first convolutional neural network is used to extract features from the sample image, and the second convolutional neural network is used to extract features from the sample LBP feature image to obtain the facial features in the sample image; specifically:
[0110] The first convolutional neural network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a seventh convolutional layer, an eighth convolutional layer, a ninth convolutional layer, a tenth convolutional layer, an eleventh convolutional layer, a twelfth convolutional layer, a thirteenth convolutional layer, a fourteenth convolutional layer, a fifteenth convolutional layer, a sixteenth convolutional layer, a seventeenth convolutional layer, an eighteenth convolutional layer, a nineteenth convolutional layer and a twentieth convolutional layer;
[0111] For the sample image:
[0112] The sample image is used as the input of the first convolutional layer. The output of the first convolutional layer passes through the second and third convolutional layers in sequence. The output of the first convolutional layer is then concatenated with the output of the third convolutional layer to obtain the concatenated result a. The concatenated result a is used as the input of the fourth convolutional layer.
[0113] The output of the fourth convolutional layer then passes through the fifth and sixth convolutional layers in sequence. The output of the fourth convolutional layer is concatenated with the output of the sixth convolutional layer to obtain a concatenated result b. The concatenated result b then passes through the seventh and eighth convolutional layers in sequence. The output of the eighth convolutional layer is concatenated with the concatenated result b to obtain a concatenated result c.
[0114] The concatenated result c is used as the input of the ninth convolutional layer. The output of the ninth convolutional layer then passes through the tenth and eleventh convolutional layers in sequence. The output of the ninth convolutional layer is concatenated with the output of the eleventh convolutional layer to obtain the concatenated result d.
[0115] The splicing result d then passes through the twelfth convolutional layer and the thirteenth convolutional layer in sequence, and the splicing result d is spliced with the output of the thirteenth convolutional layer to obtain the splicing result e;
[0116] The splicing result e passes through the fourteenth convolutional layer and the fifteenth convolutional layer in sequence, and the splicing result e is spliced with the output of the fifteenth convolutional layer to obtain the splicing result f;
[0117] The splicing result f passes through the sixteenth convolutional layer and the seventeenth convolutional layer in sequence, and the splicing result f is spliced with the output of the seventeenth convolutional layer to obtain the splicing result g;
[0118] The concatenated result g is used as the input of the 18th convolutional layer. The output of the 18th convolutional layer passes through the 19th and 20th convolutional layers in sequence. The outputs of the 18th and 20th convolutional layers are then concatenated to obtain the concatenated result h, which is used as feature A.
[0119] Similarly, the sample LBP feature image is used as the input of the second convolutional neural network, and the feature B of the sample LBP feature image is output through the second convolutional neural network;
[0120] Feature A and feature B are fused, and the feature fusion result is used as the input of the pooling layer. The output of the pooling layer passes through the fully connected layer, and the facial features in the sample image are output through the fully connected layer.
[0121] The LBP feature image is rotationally invariant and insensitive to illumination. Therefore, by combining the original image with the LBP feature image to extract features, we can avoid the interference caused by uneven illumination while retaining the real information, thereby improving the matching accuracy.
[0122] The higher the similarity of the image pairs, the more important they are to positioning. Therefore, after screening based on similarity, the interference caused by error information can be avoided, the positioning accuracy can be effectively improved, and the amount of data that needs to be processed can be reduced, thereby increasing the positioning speed.
[0123] 2. Positioning process
[0124] 1. If the number of selected face region image pairs to be matched is 1, the original image corresponding to the selected face region image pairs to be matched in step 1 is obtained, and the position of the target person is determined based on the obtained original image.
[0125] First, the original image in the sub-image pair is obtained, and then the coordinates of the central pixel point of the target person in the original image are used as the position of the target person.
[0126] 2. If the number of selected face region image pairs to be matched is greater than or equal to 2, the following method (1) or method (2) is used for positioning.
[0127] The specific process of method (1) is as follows:
[0128] Step 41: Select the to-be-matched face region image pairs corresponding to the two largest similarities, and obtain the sub-image pairs corresponding to the selected to-be-matched face region image pairs in step 1;
[0129] Step 42: The two sub-image pairs obtained in step 41 are respectively taken as the first sub-image pair and the second sub-image pair; the camera that captures the first sub-image pair is recorded as C1, and the camera that captures the second sub-image pair is recorded as C2;
[0130] The center pixel point of the target person in the first sub-image pair is denoted as A1 (since the original image and the grayscale image in the sub-image pair correspond to each other, the original image or the grayscale image can be used to calculate the center pixel point. The method for determining the center pixel point is: determine the minimum circumscribed rectangle of the target person's position, and then use the center of the determined rectangle as the target person's position). The coordinates of pixel point A1 in the world coordinate system are (x1, y1, z1). The center pixel point of the target person in the second sub-image pair is denoted as A2. The coordinates of pixel point A2 in the world coordinate system are (x2, y2, z2). The optical center of camera C1 is denoted as o1, and the optical center of camera C2 is denoted as o2. The line connecting pixel point A1 and optical center o1 is straight line l1, and the line connecting pixel point A2 and optical center o2 is straight line l2. The direction vectors of straight line l1 and straight line l2 are respectively:
[0131]
[0132]
[0133] Among them, m1, n1, p1 are the elements in the direction vector of the line l1, and m2, n2, p2 are the elements in the direction vector of the line l2;
[0134] Then the equations of line l1 and line l2 are:
[0135]
[0136]
[0137] Step 43: Calculate the vector based on the direction vectors of line l1 and line l2
[0138]
[0139] Draw a line through line l1 parallel to the vector The plane D1, through the line l2 parallel to the vector plane D2, and then get the intersection of plane D1 and plane D2, record the intersection point of the intersection line and straight line l1 as p1, record the intersection point of the intersection line and straight line l2 as p2, and then get the midpoint coordinates of the line segment p1p2, and use the obtained midpoint coordinates as the position of the target person.
[0140] The specific process of method (2) is as follows:
[0141] Step 41: Select the to-be-matched face region image pairs corresponding to the two largest similarities, and obtain the sub-image pairs corresponding to the selected to-be-matched face region image pairs in step 1;
[0142] Step 42: The two sub-image pairs obtained in step 41 are respectively taken as the first sub-image pair and the second sub-image pair; the camera that captures the first sub-image pair is recorded as C1, and the camera that captures the second sub-image pair is recorded as C2;
[0143] The center pixel of the target person in the first sub-image is denoted as A1, and the coordinates of pixel A1 in the world coordinate system are (x1, y1, z1). The center pixel of the target person in the second sub-image is denoted as A2, and the coordinates of pixel A2 in the world coordinate system are (x2, y2, z2). The optical center of camera C1 is denoted as o1, and the optical center of camera C2 is denoted as o2. Then, the line connecting pixel A1 and optical center o1 is straight line l1, and the line connecting pixel A2 and optical center o2 is straight line l2. The direction vectors of straight line l1 and straight line l2 are:
[0144]
[0145]
[0146] Then the equations of line l1 and line l2 are:
[0147]
[0148]
[0149] Step 43: Calculate the vector based on the direction vectors of line l1 and line l2
[0150]
[0151] Draw a line through line l1 parallel to the vector The plane D1, through the line l2 parallel to the vector The plane D2 is obtained, and then the intersection line of plane D1 and plane D2 is obtained. The intersection point of the intersection line and the straight line l1 is recorded as p1, and the intersection point of the intersection line and the straight line l2 is recorded as p2. Then the midpoint coordinates of the line segment p1p2 are obtained, and the obtained midpoint coordinates are used as the position 1 of the target person;
[0152] Step 4: Let the plane equation of plane D be z=z a , let the intersection of line l1 and plane D be a1, let the intersection of line l2 and plane D be a2, find the z that minimizes the distance between the intersection a1 and the intersection a2 a , when the distance between the intersection point a1 and the intersection point a2 is the smallest, the midpoint of the intersection point a1 and the intersection point a2 is calculated, and the midpoint of the intersection point a1 and the intersection point a2 is taken as the position 2 of the target person;
[0153] Step 4 and 5: Calculate the midpoint between the target person's position 1 and the target person's position 2 to obtain the final target person's position.
[0154] Step 5: determine the camera to be activated at the next moment according to the position of the target person obtained in step 4, and then use the activated camera to collect images and then return to step 1;
[0155] After obtaining the position of the target person at each moment, the trajectory tracking of the target person is realized.
[0156] In the present invention, according to the location of the target person at the current moment, the camera adjacent to the current position can be started to avoid processing the images collected by all cameras, thereby reducing the amount of data processing and improving the processing speed. Moreover, the present invention uses images corresponding to similarities greater than a threshold value when performing positioning, avoiding the errors introduced by using the information of all images for positioning, so that the positioning speed can be improved while improving the positioning accuracy. Since the method of the present invention can improve the positioning accuracy, it is possible to more clearly determine the image that needs to be processed at the next moment, further improving the accuracy and speed of trajectory tracking.
[0157] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.
Claims
1. A method for positioning and tracking a target person based on image processing, characterized in that: The method specifically comprises the following steps: Step 1: at the current time l, multiple cameras arranged in space are used to synchronously collect the original image of the area where the target person is located, and then grayscale processing is performed on each frame of the collected original image to obtain each frame of grayscale image; Then, each frame of grayscale image is processed respectively to obtain the position of the moving person in each frame of grayscale image and the position of the moving person in each frame of original image; and a sub-image of the position of each moving person is cut out from each frame of original image and grayscale image respectively to obtain each sub-image pair; Step 11: For a certain camera, the grayscale image corresponding to the original image collected by the camera at the current time l is recorded as I l , the grayscale image corresponding to the original image collected by the camera at the previous time l-1 is recorded as I l-1 , then the change d of the gray value of the pixel (x, y) in the two gray images before and after l (x,y) is: d l (x,y)=|I l (x,y)-I l-1 (x,y)| Among them, I l (x,y) is the grayscale image I l The gray value of the pixel (x, y), I l-1 (x,y) is the grayscale image I l-1 The grayscale value of the pixel (x, y), |·| represents the absolute value; Step 12: Grayscale image I l Medium change d l The pixel points whose (x, y) is greater than or equal to the threshold TH are taken as foreground points, and the set of foreground points obtained is recorded as O1; Step 13: Grayscale image I l The maximum gray value of the pixel is recorded as z max , the grayscale image I l The minimum gray value of the pixel is recorded as z min , then the initial segmentation threshold Q 0 for: Step 14, initialize the number of iterations k = 0; Step 15: Grayscale image I l The gray value is less than or equal to the segmentation threshold Q k The pixel point is taken as the foreground point, and then the grayscale average value z0 of the foreground point is calculated; the grayscale image I l The gray value is greater than the segmentation threshold Q k The pixel point is taken as the background point, and then the gray value z of the background point is calculated. B ; Step 16: According to z0 and z B Calculate the segmentation threshold Q k+1 : Step 17: Determine whether Q is satisfied k+1 =Q k ; If Q is satisfied k+1 =Q k , then use the grayscale image I l The gray value is less than or equal to the segmentation threshold Q k+1 The pixel points of the foreground point set O2 are formed, and then step 18 is executed; If Q is not satisfied k+1 =Q k , then set k=k+1 and return to step 15; Step 18: Find the union of sets O1 and O2, and use the pixels in the union as the grayscale image I l All the foreground points in the grayscale image I are used, and other pixels are used as background points. Morphological filtering and connectivity analysis are then performed to obtain the grayscale image I of each moving person. l According to the location of the moving person in the grayscale image I l The position of the moving person in the original image is obtained by using the position of the moving person in the original image; Step 19: using the methods of steps 11 to 18 to process the grayscale images corresponding to each frame of the original image collected at the current moment; Step 2: Obtain the face image of the target person stored in the system, and use the obtained face image as a sample image; then perform grayscale processing on the obtained sample image to obtain a sample grayscale image, and then convert the sample grayscale image into a sample LBP feature image; Step 3: convert the grayscale image in each sub-image pair obtained in step 1 into an LBP feature image, and obtain a face region image pair to be matched based on the converted image; Step 4: Use the first convolutional neural network to extract features from the sample image, and use the second convolutional neural network to extract features from the sample LBP feature image to obtain facial features in the sample image; For any pair of face region images to be matched, the first convolutional neural network is used to extract features from the original image in the face region image pair to be matched, and the second convolutional neural network is used to extract features from the LBP feature image in the face region image pair to be matched, so as to obtain the face features in the face region image pair to be matched; similarly, the face features of each face region image pair to be matched are obtained respectively; Then, the similarity between the facial features in the sample image and the facial features of each to-be-matched face region image pair is calculated respectively, and it is determined whether there is a to-be-matched face region image pair whose similarity is greater than a threshold value; If it exists, then select the to-be-matched face region image pair corresponding to the similarity greater than the threshold, determine the target person's position based on the selected to-be-matched face region image pair, and then execute step five; If it does not exist, set time l = l + 1 and return to step 1; Step 5: Determine the camera that needs to be activated at the next moment according to the position of the target person obtained in step 4, set l=l+1, and return to execute step 1.
2. The method for positioning and tracking a target person based on image processing according to claim 1, characterized in that: The sample LBP feature image is obtained by converting the sample grayscale image through the LBP operator.
3. The method for positioning and tracking a target person based on image processing according to claim 2, characterized in that: The first convolutional neural network and the second convolutional neural network have the same structure.
4. The method for positioning and tracking a target person based on image processing according to claim 3, characterized in that: The first convolutional neural network is used to extract features from the sample image, and the second convolutional neural network is used to extract features from the sample LBP feature image to obtain facial features in the sample image; specifically: The first convolutional neural network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a seventh convolutional layer, an eighth convolutional layer, a ninth convolutional layer, a tenth convolutional layer, an eleventh convolutional layer, a twelfth convolutional layer, a thirteenth convolutional layer, a fourteenth convolutional layer, a fifteenth convolutional layer, a sixteenth convolutional layer, a seventeenth convolutional layer, an eighteenth convolutional layer, a nineteenth convolutional layer and a twentieth convolutional layer; For the sample image: The sample image is used as the input of the first convolutional layer. The output of the first convolutional layer passes through the second and third convolutional layers in sequence. The output of the first convolutional layer is then concatenated with the output of the third convolutional layer to obtain the concatenated result a. The concatenated result a is used as the input of the fourth convolutional layer. The output of the fourth convolutional layer then passes through the fifth and sixth convolutional layers in sequence. The output of the fourth convolutional layer is concatenated with the output of the sixth convolutional layer to obtain a concatenated result b. The concatenated result b then passes through the seventh and eighth convolutional layers in sequence. The output of the eighth convolutional layer is concatenated with the concatenated result b to obtain a concatenated result c. The concatenated result c is used as the input of the ninth convolutional layer. The output of the ninth convolutional layer then passes through the tenth and eleventh convolutional layers in sequence. The output of the ninth convolutional layer is concatenated with the output of the eleventh convolutional layer to obtain the concatenated result d. The splicing result d then passes through the twelfth convolutional layer and the thirteenth convolutional layer in sequence, and the splicing result d is spliced with the output of the thirteenth convolutional layer to obtain the splicing result e; The splicing result e passes through the fourteenth convolutional layer and the fifteenth convolutional layer in sequence, and the splicing result e is spliced with the output of the fifteenth convolutional layer to obtain the splicing result f; The splicing result f passes through the sixteenth convolutional layer and the seventeenth convolutional layer in sequence, and the splicing result f is spliced with the output of the seventeenth convolutional layer to obtain the splicing result g; The concatenated result g is used as the input of the 18th convolutional layer. The output of the 18th convolutional layer passes through the 19th and 20th convolutional layers in sequence. The outputs of the 18th and 20th convolutional layers are then concatenated to obtain the concatenated result h, which is used as feature A. Similarly, the sample LBP feature image is used as the input of the second convolutional neural network, and the feature B of the sample LBP feature image is output through the second convolutional neural network; Feature A and feature B are fused, and the feature fusion result is used as the input of the pooling layer. The output of the pooling layer passes through the fully connected layer, and the facial features in the sample image are output through the fully connected layer.
5. The method for positioning and tracking a target person based on image processing according to claim 4, characterized in that: The target person's position is determined based on the selected face area image pairs to be matched. If the number of selected face area image pairs to be matched is 1, the original image corresponding to the selected face area image pairs to be matched in step one is obtained, and the target person's position is determined based on the obtained original image.
6. The method for positioning and tracking a target person based on image processing according to claim 5, characterized in that: The method of determining the position of the target person according to the selected to-be-matched face region image pairs is as follows: Step 41: Select the to-be-matched face region image pairs corresponding to the two largest similarities, and obtain the sub-image pairs corresponding to the selected to-be-matched face region image pairs in step 1; Step 42: The two sub-image pairs obtained in step 41 are respectively used as the first sub-image pair and the second sub-image pair; the camera that captures the first sub-image pair is recorded as C1, and the camera that captures the second sub-image pair is recorded as C2; The central pixel point of the target person in the first sub-image pair is denoted as A1, and the coordinates of pixel point A1 in the world coordinate system are (x1, y1, z1). The central pixel point of the target person in the second sub-image pair is denoted as A2, and the coordinates of pixel point A2 in the world coordinate system are (x2, y2, z2). The optical center of camera C1 is denoted as o1, and the optical center of camera C2 is denoted as o2. Then, the line connecting pixel point A1 and optical center o1 is straight line l1, and the line connecting pixel point A2 and optical center o2 is straight line l2. The direction vectors of straight line l1 and straight line l2 are: Among them, m1, n1, p1 are the elements in the direction vector of the line l1, and m2, n2, p2 are the elements in the direction vector of the line l2; Then the equations of line l1 and line l2 are: Step 43: Calculate the vector based on the direction vectors of line l1 and line l2 Draw a line through line l1 parallel to the vector The plane D1, through the line l2 parallel to the vector plane D2, and then get the intersection of plane D1 and plane D2, record the intersection point of the intersection line and straight line l1 as p1, record the intersection point of the intersection line and straight line l2 as p2, and then get the midpoint coordinates of the line segment p1p2, and use the obtained midpoint coordinates as the position of the target person.
7. The method for positioning and tracking a target person based on image processing according to claim 5, characterized in that: The method of determining the position of the target person according to the selected to-be-matched face region image pairs is as follows: Step 41: Select the to-be-matched face region image pairs corresponding to the two largest similarities, and obtain the sub-image pairs corresponding to the selected to-be-matched face region image pairs in step 1; Step 42: The two sub-image pairs obtained in step 41 are respectively used as the first sub-image pair and the second sub-image pair; the camera that captures the first sub-image pair is recorded as C1, and the camera that captures the second sub-image pair is recorded as C2; The central pixel point of the target person in the first sub-image pair is denoted as A1, and the coordinates of pixel point A1 in the world coordinate system are (x1, y1, z1). The central pixel point of the target person in the second sub-image pair is denoted as A2, and the coordinates of pixel point A2 in the world coordinate system are (x2, y2, z2). The optical center of camera C1 is denoted as o1, and the optical center of camera C2 is denoted as o2. Then, the line connecting pixel point A1 and optical center o1 is straight line l1, and the line connecting pixel point A2 and optical center o2 is straight line l2. The direction vectors of straight line l1 and straight line l2 are: Then the equations of line l1 and line l2 are: Step 43: Calculate the vector based on the direction vectors of line l1 and line l2 Draw a line through line l1 parallel to the vector The plane D1, through the line l2 parallel to the vector The plane D2 is obtained, and then the intersection line of plane D1 and plane D2 is obtained. The intersection point of the intersection line and the straight line l1 is recorded as p1, and the intersection point of the intersection line and the straight line l2 is recorded as p2. Then the midpoint coordinates of the line segment p1p2 are obtained, and the obtained midpoint coordinates are used as the position 1 of the target person; Step 4: Let the plane equation of plane D be z=z a , let the intersection of line l1 and plane D be a1, let the intersection of line l2 and plane D be a2, find the z that minimizes the distance between the intersection a1 and the intersection a2 a , when the distance between the intersection point a1 and the intersection point a2 is the smallest, the midpoint of the intersection point a1 and the intersection point a2 is calculated, and the midpoint of the intersection point a1 and the intersection point a2 is taken as the position 2 of the target person; Step 4 and 5: Calculate the midpoint between the target person's position 1 and the target person's position 2 to obtain the final target person's position.
