A complex scene personnel detection and identification method and system

Abstract

The application discloses a complex scene personnel detection and recognition method and system, and belongs to the technical field of machine vision, and comprises the following steps: collecting video data of multiple angles of the same complex scene; fusing the video data of multiple angles to obtain a panoramic image of the complex scene; constructing a double-layer intelligent detection model; detecting the panoramic image based on the double-layer intelligent detection model to obtain a multi-target detection result; establishing a scene personnel label library; and performing personnel recognition based on the personnel label library by using a face recognition algorithm to obtain a personnel detection and recognition result. The method can greatly reduce the calculation cost, save the calculation resources, and better adapt to complex scenes under the condition of improving the detection and recognition precision.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of machine vision, and particularly relates to a complex scene personnel detection and identification method and system. BACKGROUND

[0002] At present, the supervision of public places usually depends on video monitoring technology. Since the traditional video monitoring technology can only play a monitoring and storage function, when a dispute or improper behavior occurs in a public place, video monitoring can only be retrieved and consulted according to the time range. However, the introduction of a complex scene real-time video multi-person behavior recognition method and system can monitor people's behavior in a complex scene in real time and give an early warning, so that the management personnel of the public place can handle the abnormal situation in time, thereby ensuring the order and safety of the public place, and having a profound practical significance and application value.

[0003] In addition to the lack of real-time of the existing detection technology, personnel detection and identification in a complex scene is still a challenging task, and still has some shortcomings, including: 1. Occlusion problem: there may be a large amount of occlusion in a complex scene, such as crossing in a crowd, partial occlusion, etc. 2. Posture change: people may appear in various postures, such as standing, walking, bending, etc. 3. Complex background: the background in a complex scene can be very cluttered and diverse, including buildings, vehicles, trees, etc. 4. Light change: changes in lighting conditions will affect the appearance features of personnel, such as shadows, strong light, etc.

[0004] With the rapid development of computer technology, the existing technology for video action behavior detection and personnel identification mainly includes traditional computer vision methods, deep learning methods and optical flow analysis methods. However, the traditional computer vision method and the deep learning method have low detection and identification accuracy in a complex scene due to low image resolution. The optical flow analysis method detects and tracks motion in a video by capturing pixel displacement between consecutive frames in the video, thereby detecting action behavior and identifying personnel, but this method occupies a huge amount of computing resources, which is not conducive to large-scale deployment in actual production environments. Therefore, the present application proposes a complex scene personnel detection and identification method and system with real-time, high accuracy and small computing overhead. SUMMARY

[0005] To solve the above technical problems, the present application proposes a complex scene personnel detection and identification method and system to solve the problems existing in the above-mentioned prior art.

[0006] To achieve the above-mentioned purpose, the present application provides a complex scene personnel detection and identification method, comprising the following steps:

[0007] Collecting video data of the same complex scene from multiple angles; fusing the video data from multiple angles to obtain panoramic images of the complex scene;

[0008] Building a double-layer intelligent detection model; detecting the panoramic images based on the double-layer intelligent detection model to obtain multi-target detection results;

[0009] Establishing a scene personnel label library; performing personnel recognition based on the personnel label library using a face recognition algorithm to obtain personnel detection and recognition results.

[0010] Optionally, the video data from multiple angles is fused based on a multi-view fusion technology to obtain panoramic images of the complex scene; wherein the process of obtaining panoramic images includes:

[0011] Decoding the video data to obtain video streams of different collection angles;

[0012] Frame the video streams based on time series to obtain video images of different angles;

[0013] After processing the video images of different angles, using a Gaussian mixture model separation algorithm for target tracking to obtain a plurality of foreground images;

[0014] Calculating the area overlap of the foreground images corresponding to any two video streams at the same time;

[0015] Setting an area overlap threshold;

[0016] Determining an image stitching line based on the area overlap threshold and the area overlap of the foreground images;

[0017] Stitching the video images at the same time using the image stitching line to obtain panoramic images.

[0018] Optionally, the process of calculating the area overlap of the foreground images corresponding to any two video streams at the same time includes:

[0019] Extracting image features of the foreground images based on a SIFT algorithm;

[0020] Matching the image features of two foreground images based on a feature matching algorithm to obtain foreground images after feature matching;

[0021] Calculating the similarity of the two foreground images after feature matching using the Euclidean distance method;

[0022] Comparing the similarity of any two foreground images to obtain the area overlap.

[0023] Optionally, the process of determining an image stitching line based on the area overlap threshold and the area overlap of the foreground images includes:

[0024] stitching when the region overlap degree of the foreground image is greater than the region overlap degree threshold value;

[0025] calculating image gradients of the current two foreground images to determine an image stitching line.

[0026] Optionally, a double-layer intelligent detection model is constructed based on a double-flow method, the double-layer intelligent detection model including an upper-layer model aiming to minimize calculation overhead and a lower-layer model aiming to maximize image quality.

[0027] The upper-layer model aiming to minimize calculation overhead is:

[0028]

[0029] In the formula, F represents a total value of calculation overhead, represents an overhead unit value of the jth resolution to the ith computer overhead in the occlusion condition, represents a demand amount of the jth resolution to the ith computer overhead in the occlusion condition, represents whether occlusion exists, when = 1, it represents that occlusion exists, and when = 0, it represents that no occlusion exists, represents an overhead unit value of the jth resolution to the pth computer overhead in different illumination conditions, represents a demand amount of the jth resolution to the pth computer overhead in different illumination conditions, represents illumination, represents an estimated value of a pose indicated by optical flow information, represents an overhead unit value of the jth resolution to the kth computer overhead in different poses, represents a demand amount of the jth resolution to the kth computer overhead in different poses.

[0030] The lower-layer model aiming to maximize image quality is:

[0031]

[0032] In the formula, T represents image quality.

[0033] The application further provides a complex scene personnel detection and recognition system, which comprises video acquisition modules, a video preprocessing module, a double-layer intelligent detection module and a personnel recognition module connected in sequence.

[0034] The video acquisition module is configured to acquire video monitoring data of different angles in the same scene.

[0035] The video preprocessing module is configured to perform panorama fusion on the video monitoring data to obtain a panorama image in a complex scene.

[0036] The double-layer intelligent detection module is used for constructing a double-layer model to perform personnel detection on the panoramic image to obtain a multi-target detection result.

[0037] The personnel identification module is used for identifying the multi-target detection result based on a face recognition technology to obtain a personnel detection identification result.

[0038] Optionally, the video preprocessing module comprises a video decoding unit, an image processing unit and an image stitching unit.

[0039] The video decoding unit is used for decoding the video monitoring data to obtain video streams of different collection angles.

[0040] The image processing unit is used for performing frame processing on the video streams to obtain foreground images.

[0041] The image stitching unit is used for performing image stitching on the foreground images to obtain panoramic images.

[0042] Optionally, the double-layer intelligent detection module comprises an upper-layer model constructing unit, a lower-layer model constructing unit and a model solving unit.

[0043] The upper-layer model constructing unit is used for constructing an upper-layer model with the minimum calculation overhead value.

[0044] The lower-layer model constructing unit is used for constructing a lower-layer model with the maximum image quality.

[0045] The model solving unit is used for simultaneously solving the upper-layer model and the lower-layer model based on an ant colony algorithm to obtain a multi-target detection result.

[0046] Optionally, the personnel identification module comprises a personnel label library, a tracking identification unit and an abnormality alarm unit.

[0047] The personnel label library is used for storing personnel information.

[0048] The tracking identification unit is used for performing personnel comparison and identification based on the personnel information of the personnel label library to obtain target personnel information.

[0049] The abnormality alarm unit is used for performing real-time alarm when a target personnel has an abnormal behavior.

[0050] Compared with the prior art, the present application has the following advantages and technical effects:

[0051] The present invention discloses a method for detecting and identifying people in complex scenes, comprising the following steps: collecting video data from multiple angles of the same complex scene; fusing the video data from multiple angles to obtain a panoramic image of the complex scene; constructing a two-layer intelligent detection model; detecting the panoramic image based on the two-layer intelligent detection model to obtain a multi-target detection result; establishing a scene person label library; and performing person recognition using a face recognition algorithm based on the person label library to obtain a person detection and recognition result.

[0052] This method fuses video data from multiple angles to provide more comprehensive and diverse information, thereby improving the accuracy of person recognition. Data from different angles can compensate for the viewing angle limitations or occlusions that may exist from a single angle, enabling the recognition algorithm to more accurately capture the characteristics of the target object.

[0053] The present invention can solve the defect of the current technology that cannot realize detection by constructing a two-layer intelligent detection model. In addition, this method can greatly reduce the computational overhead and save computing resources while improving the detection and recognition accuracy, and can better adapt to complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0055] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0056] Figure 2 2 is a system structure diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0057] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0058] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0059] Example 1

[0060] like Figure 1 As shown, this embodiment provides a method for detecting and identifying people in complex scenes, including the following steps:

[0061] Step one, collecting multi-angle video data of the same complex scene; fusing the multi-angle video data to obtain a panoramic image of the complex scene; the specific implementation process includes:

[0062] By obtaining the optical center coordinate position, distortion coefficient and other parameters of the video acquisition device in the complex scene, the parameters of the video acquisition device are calibrated to provide data parameters for video stitching and fusion.

[0063] Based on the calibrated video acquisition device, multi-angle video data of the same complex scene is collected, and based on multi-view fusion technology, the multi-angle video data is fused to obtain a panoramic image of the complex scene. The process includes: decoding the video data to obtain video streams of different collection angles; first, read the video data from the video file or stream. These data are usually encoded and compressed in a specific video encoding format. The decoder is responsible for decoding the compressed video data into the original video frame sequence. Based on the original video frame sequence, the metadata of the video stream is parsed, including the frame rate, resolution, encoding format and other information. Select the collection angle: select a specific collection angle according to the demand. Finally, demultiplex to separate multiple tracks for processing each video track separately.

[0064] Frame the video stream based on time sequence to obtain video images of different angles; after processing the video images of different angles, use the Gaussian mixture model separation algorithm for target tracking to obtain several foreground images; calculate the region overlap of the foreground images corresponding to any two video streams at the same time; the process includes: extracting the image features of the foreground images based on the SIFT algorithm; matching the feature characteristics of the two foreground images based on the feature matching algorithm to obtain the foreground images after feature matching; calculating the similarity of the two foreground images after feature matching using the Euclidean distance method; comparing the similarity of any two foreground images to obtain the region overlap. Calculate the image gradient of the current two foreground images to determine the image stitching line.

[0065] Set the region overlap threshold; determine the image stitching line based on the region overlap threshold and the region overlap of the foreground image; when the region overlap of the foreground image is greater than the region overlap threshold, perform image stitching; use the image stitching line to stitch the video images at the same time to obtain a panoramic image.

[0066] Step two, constructing a double-layer intelligent detection model; detecting the panoramic image based on the double-layer intelligent detection model to obtain multi-target detection results; the specific implementation process includes:

[0067] Determine the target: construct an upper model with the minimum calculation overhead as the target and a lower model with the maximum image quality as the target;

[0068] Determination of decision variables: the variables affecting the value of the calculation overhead include: occlusion degree, background complexity and pose estimation; the variables affecting the image quality include the allocation of computer resources, including: CPU calculation overhead, memory overhead, network overhead, time complexity overhead and space complexity overhead;

[0069] The process of pose estimation based on the double-flow method includes:

[0070] The depth information of different points in the scene is inferred using the geometric relationship between the parallax information and the camera. The specific implementation process is: the parallax is calculated with parameters such as baseline and focal length; the object pose is estimated based on the depth information of different points in the scene.

[0071] The mathematical expression of the constructed double-layer intelligent detection model is:

[0072]

[0073]

[0074]

[0075] In the formula, F represents the total value of the calculation overhead, represents the overhead unit value of the jth resolution to the ith computer overhead under the occlusion condition, represents the demand amount of the jth resolution to the ith computer overhead under the occlusion condition, represents whether it is occluded, when =1, it represents that there is occlusion, when =0, it represents that there is no occlusion, represents the overhead unit value of the jth resolution to the pth computer overhead under different illumination conditions, represents the demand amount of the jth resolution to the pth computer overhead under different illumination conditions, represents the illumination, represents the pose estimation value represented by the optical flow information, represents the overhead unit value of the jth resolution to the kth computer overhead under different poses, represents the demand amount of the jth resolution to the kth computer overhead under different poses, and T represents the image quality.

[0076] Model solving: the model is solved based on the ant colony algorithm.

[0077] Step three, establishing a scene personnel tag library; personnel recognition is performed based on the personnel tag library using a face recognition algorithm to obtain personnel detection and recognition results; the specific implementation process includes:

[0078] Collect face image data containing different lighting conditions, expressions, postures, ages, genders, occlusions, wearing glasses or masks, etc.; add label information to each face image, including the identity, age, gender, expression, posture, etc. of the face;

[0079] Perform data augmentation on the collected data to obtain a large amount of training data; construct a lightweight SlowFast network, and perform face recognition training on the lightweight SlowFast network based on the augmented image data to establish a personnel label library.

[0080] Based on the personnel label library, the personnel in the complex scene are identified, the captured face information is compared with the face information in the face library, if the captured face information does not exist in the face library, it is marked as a key and tracked, and if it has abnormal behavior, an alarm is triggered, if the captured face information exists in the face library, it is marked as a secondary key, and if it has abnormal behavior, an alarm is triggered;

[0081] The process of constructing the lightweight SlowFast network includes:

[0082] Constructing a slow path, wherein the slow path includes a plurality of convolutional layers and pooling layers for extracting static features in the image;

[0083] Constructing a fast path, wherein the fast path only includes a single convolutional layer;

[0084] Constructing a fusion layer, wherein the fusion layer fuses the path features of the slow path and the fast path based on weighted summation;

[0085] Constructing a fully connected layer and an output layer, wherein the fully connected layer inputs the fused features, and the output layer obtains the face recognition result by using a softmax activation function.

[0086] Embodiment Two

[0087] As shown in Figure 2 , the embodiment provides a complex scene personnel detection and recognition system, which includes a video acquisition module, a video preprocessing module, a double-layer intelligent detection module and a personnel recognition module connected in sequence.

[0088] The video acquisition module acquires video monitoring data of different angles of the same scene based on a plurality of video acquisition devices; the specific implementation process includes: selecting a certain brand of seagull scene camera considering factors such as resolution, frame rate, field of view angle, image quality, etc., arranging the camera according to the target monitoring scene, ensuring the diversity of coverage range and angle, connecting each video acquisition device to the system of the acquisition module, and performing related configuration, including network setting, resolution setting, frame rate setting, etc., starting the video acquisition device, and transmitting the collected video stream to the server or storage device of the acquisition module through the network.

[0089] The video preprocessing module is used for panorama fusion of the video monitoring data to obtain a panoramic image in a complex scene; the specific implementation process includes: the video preprocessing module includes a video decoding unit, an image processing unit, and an image stitching unit; the video decoding unit is used for decoding the video monitoring data to obtain video streams of different acquisition angles; the image processing unit is used for frame taking processing of the video streams from the plurality of video acquisition devices to obtain foreground images; and the image stitching unit is used for image stitching of the foreground images to obtain the panoramic image.

[0090] The double-layer intelligent detection module is used for constructing a double-layer model to detect personnel in the panoramic image to obtain a multi-target detection result; the specific implementation process includes: the double-layer intelligent detection module includes an upper-layer model construction unit, a lower-layer model construction unit, and a model solving unit; the upper-layer model construction unit is used for constructing an upper-layer model with the minimum calculation overhead; the lower-layer model construction unit is used for constructing a lower-layer model with the maximum image quality; and the model solving unit is used for simultaneously solving the upper-layer model and the lower-layer model based on an ant colony algorithm to obtain the multi-target detection result.

[0091] The process of solving by the ant colony algorithm includes:

[0092] Initialize the ant colony: create an ant colony, where each ant represents a potential solution.

[0093] Ant colony search: upper-layer problem search: ants search in the solution space of the upper-layer problem according to heuristic information and local pheromone information, and select solutions of the lower-layer problem. Lower-layer problem search: under the selection of each upper-layer problem solution, ants search in the solution space of the lower-layer problem according to heuristic information and local pheromone information, and update pheromone.

[0094] Pheromone update: update pheromone according to the search results of ants.

[0095] Evaluate the quality of the solution: for each solution of the upper-layer problem, evaluate the quality of the corresponding solution of the lower-layer problem.

[0096] Update the optimal solution: update the global optimal solution according to the quality of the evaluated solution.

[0097] Convergence detection: Check if the algorithm meets the convergence condition. If the condition is met, stop the search; otherwise, continue the search.

[0098] Output result: Output the optimal solution found.

[0099] The personnel identification module is used to identify the multi-target detection result based on face recognition technology to obtain a personnel detection and identification result. The specific implementation process includes: the personnel identification module includes a personnel label library, a tracking identification unit, and an abnormal alarm unit; the personnel label library is used to store personnel information; the tracking identification unit is used to compare and identify personnel based on the personnel information of the personnel label library to obtain target personnel information; the abnormal alarm unit is used to alarm in real time when the target personnel has abnormal behavior.

[0100] Collect face image data containing different light conditions, expressions, postures, ages, genders, occlusions, wearing glasses or masks, etc. Add label information to each face image, including the identity, age, gender, expression, posture, etc. of the face;

[0101] Data augmentation is performed on the collected data to obtain a large amount of training data; a lightweight SlowFast network is constructed, and the lightweight SlowFast network is trained for face recognition based on the enhanced image data to establish a personnel label library.

[0102] Based on the personnel label library, the personnel in the complex scene are identified, the captured face information is compared with the face information in the face library, if the captured face information does not exist in the face library, it is marked and tracked, and when it has abnormal behavior, an alarm is triggered, if the captured face information exists in the face library, it is marked and tracked, and when it has abnormal behavior, an alarm is triggered.

[0103] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for detecting and identifying people in complex scenes, characterized in that: The following steps are involved: Collect video data from multiple angles of the same complex scene; The multi-angle video data is integrated to obtain a panoramic image of a complex scene. The process of obtaining the panoramic image includes: decoding the video data to obtain video streams with different acquisition angles; dividing the video streams into frames based on the time series to obtain video images with different angles; processing the video images with different angles and using the Gaussian mixture model separation algorithm to track the target to obtain several foreground images; calculating the area overlap of the foreground images corresponding to any two video streams at the same time, including: extracting the image features of the foreground image based on the SIFT algorithm, matching the influencing features of the two foreground images based on the feature matching algorithm to obtain the foreground image after feature matching, calculating the similarity of the two foreground images after feature matching, and comparing any two foreground images. Obtaining similarity to obtain regional overlap; setting a regional overlap threshold; determining an image stitching line based on the regional overlap threshold and the regional overlap of the foreground image, including: performing image stitching when the regional overlap of the foreground image is greater than the regional overlap threshold, calculating the image gradient of the current two foreground images to determine the image stitching line; stitching video images of the same time using the image stitching line to obtain a panoramic image; constructing a two-layer intelligent detection model; detecting the panoramic image based on the two-layer intelligent detection model to obtain a multi-target detection result; constructing a two-layer intelligent detection model based on a two-stream method, the two-layer intelligent detection model including an upper model with the goal of minimizing computational overhead and a lower model with the goal of maximizing image quality; the upper model with the goal of minimizing computational overhead is: Where F represents the total computational cost, a ij Indicates the cost unit value of the j-th resolution in the i-th computer cost under occlusion conditions, x ij represents the demand for the jth resolution to the ith type of computer cost under occlusion conditions, f m Indicates whether it is blocked, when f m =1, indicating that there is occlusion. When f m = 0, indicating no occlusion, b pj Indicates the cost unit value of j resolution in the pth type of computer cost under different lighting conditions, y pj represents the demand for the jth resolution in the pth computer cost under different lighting conditions, β represents the illumination, I(u,v) represents the pose estimation value represented by the optical flow information, c kj Indicates the cost unit value of the j-th resolution in the k-th computer cost at different postures, z kj The j-th resolution accounts for the k-th computer cost requirement when representing different postures; The lower model with the highest image quality is: , where T represents the image quality; a scene personnel label library is established; based on the personnel label library, a face recognition algorithm is used to perform personnel recognition on the multi-target detection results to obtain the personnel detection and recognition results.

Images