Smart park video monitoring risk intelligent association method and identification system

Abstract

The invention relates to the technical field of video monitoring association identification, and discloses a smart park video monitoring risk intelligent association method and identification system, and the method comprises the steps: collecting a preprocessed video frame sequence of distributed monitoring equipment, and carrying out the target detection, and obtaining a target person contour image sequence; and carrying out feature extraction on the contour image sequence of the target personnel, forming a target personnel embedding vector of the distributed monitoring equipment by using a feature extraction result, and carrying out cross-video stream association matching and monitoring identification. According to the method, the feature similarity of different target person images in the same real-time video stream is calculated, target person embedding vectors of distributed monitoring equipment are obtained, and target persons corresponding to two groups of target person embedding vectors of which the cross-video stream association relationship is higher than a preset association threshold value are embedded into the target persons corresponding to the target person embedding vectors; and the result is used as a monitoring identification result of the same target person in different key areas in the smart park, so that risk investigation and afterwards tracing are carried out on the target person with potential risk behaviors.

Description

Technical Field

[0001] The present invention relates to the field of video surveillance association identification, and in particular to a smart park video surveillance risk intelligent association method and identification system. Background Art

[0002] With the rapid development of modern information technology, smart parks, as an important part of smart city construction, have become a key means to improve park management efficiency and service levels. In smart parks, video surveillance, as one of the infrastructures, not only bears the responsibility of security prevention and control, but also plays an important role in park management, personnel flow analysis, and facility usage monitoring. However, with the expansion of video surveillance coverage in smart parks and the exponential growth of surveillance data, traditional video surveillance systems face many challenges such as high storage pressure, low video analysis efficiency, and insufficient risk warning capabilities. In particular, how to associate video content from different locations in the park to form a fusion data analysis basis for risk events has become an important issue that needs to be solved urgently. Summary of the invention

[0003] In view of this, the present invention provides a smart park video surveillance risk intelligent association method, which integrates multi-source data and introduces artificial intelligence and big data analysis technology to perform real-time perception, early warning and post-event tracing of potential risks in surveillance videos, and explore the correlation between different risk events.

[0004] To achieve the above object, the present invention provides a smart park video surveillance risk intelligent association method, comprising the following steps: S1: Collect real-time video streams from multiple distributed monitoring devices in the smart park, extract and preprocess video frames, and obtain preprocessed video frame sequences of distributed monitoring devices. The preprocessing includes video frame filtering, color contrast enhancement, and resolution unification.

[0005] S2: Input the preprocessed video frame sequence into the target detection model for target detection to obtain a target person contour image sequence.

[0006] S3: Extract features from the target person's contour image sequence, and use the feature extraction results to form the target person embedding vector of the distributed monitoring device.

[0007] S4: Perform cross-video stream association matching on the target person embedding vectors of different distributed monitoring devices to obtain the cross-video stream association relationship of the target person embedding vectors, and perform monitoring and identification of the target person.

[0008] As a further improvement method of the present invention: Optionally, in step S1, collecting real-time video streams from multiple distributed monitoring devices in the smart park includes: Distributed monitoring equipment is deployed in key areas of the smart park. The distributed monitoring equipment collects regional videos of the key areas in real time to obtain real-time video streams. The key areas include entrances and exits, parking lots, public activity areas, and the surrounding areas of park buildings. Each distributed monitoring device is connected to the monitoring management system through a monitoring network. The monitoring network adopts a wireless network to transmit the real-time video streams in the distributed monitoring equipment to the monitoring management system.

[0009] Collect real-time video streams from multiple distributed monitoring devices in the smart park, compress and encode the real-time video streams, and transmit them to the monitoring management system.

[0010] Optionally, video frame extraction and preprocessing are performed on the real-time video stream, including: The monitoring management system decodes the received real-time video stream and divides the decoded real-time video stream into a number of video frames.

[0011] The preprocessing of the video frames includes filtering, color contrast enhancement and resolution unification, and the preprocessing method is: The noise components in any video frame are filtered, where the noise components of the video frame are the rapidly changing fine textures or random pixel value fluctuations in the video frame. The main source of the noise components is the video flow distortion introduced during the real-time video stream compression encoding process. The filtering method is used to suppress the noise components in the video frame and improve the smoothness of the video frame.

[0012] The color contrast of the filtered video frame is enhanced to enhance the contrast and visual effect of the video frame. The color contrast enhancement is a histogram equalization process.

[0013] All video frames after filtering and color contrast enhancement are adjusted to a uniform resolution to obtain preprocessed video frames, and the video frames are sorted in timestamp order to obtain video frame sequences collected by different distributed monitoring devices.

[0014] Optionally, the S2 step includes: The target detection model is used to extract a target person contour image sequence in a preprocessed video frame sequence through an improved YOLOv8 model.

[0015] The improved YOLOv8 model includes a conv convolutional layer, a C2f module, a bottleneck module and a Head layer. The conv convolutional layer is used to perform deformable convolution processing on the video frames in the preprocessed video frame sequence to obtain video frame features.

[0016] The C2f module is used to divide the video features into two parts. One part uses a 1×1 pixel convolution kernel to achieve cross-channel information fusion to obtain a convolution feature map, which reduces the number of channels and reduces the amount of calculation; the other part is processed by a series of bottleneck modules to refine and enhance the features to obtain an enhanced feature map. The bottleneck module uses two 3×3 pixel convolution kernels to further extract the input features and halve the number of channels. Finally, the concat process is used to fuse the convolution feature map and the enhanced feature map to obtain a fused feature map of the video frame.

[0017] The Head layer is used to divide the fused feature map into multiple target candidate regions and generate bounding box parameters of the target candidate regions, where the bounding box parameters include the center coordinates, width, and height offsets of the target candidate regions. The center coordinates, width, and height offsets are used as the 1st to 4th bounding box parameters in turn, and the confidence score of each target candidate region is calculated. The bounding box parameters of the target candidate regions with confidence scores higher than the preset threshold are retained.

[0018] A bounding box is generated in the video frame using the retained bounding box parameters as the target person contour marking box of the video frame. The video frame area within the target person contour marking box is the target person image in the video frame, and the confidence score of the target candidate area corresponding to the target person image is used as the confidence score of the target person image. The video frame with all target person contour marking boxes is marked, which is the target person contour image corresponding to the video frame.

[0019] The deformable convolution processing adds an offset to the sampling position of the ordinary convolution, expands the convolution range of the convolution kernel, and can capture richer feature information, especially when processing objects with complex shapes or irregular boundaries, which helps to improve the detection accuracy of the target detection model.

[0020] Optionally, the target detection model is trained by a loss function, and the trained target detection model is used to output a target person contour marking frame for each video frame in the preprocessed video frame sequence, and the video frame is marked based on the target person contour marking frame as a target person contour image, and the target person contour images of the continuous video frames are used as a target person contour image sequence of the distributed monitoring device. The expression of the loss function is: : L us (θ)= ∑ n=1 N ∑ i∈{1,2,3,4} 1 2 [log(2π ( s i n (i)) 2 )+ ( Y n (i)- m i n (i)) 2 ( s i n (i)) 2 ] .

[0021] in: Represents the model parameters of the target detection model, including the weights of the convolution kernels in the deformable convolution process, the weights of the convolution kernels in the C2f module and the bottleneck module, and the bounding box regression parameters of the Head layer; represents the i-th true bounding box parameter of the n-th training sample, Represents model parameters Under training, the average of the i-th predicted bounding box parameters of the n-th training sample during multiple iterations, Represents model parameters Under training, the standard deviation of the i-th predicted bounding box parameters of the n-th training sample during multiple iterations, where N is the number of training samples.

[0022] The true bounding box parameters are the bounding box parameters of the manually marked target person in the training sample, and the predicted bounding box parameters are the bounding box parameters of the target person in the training sample predicted by using the target detection model.

[0023] Optionally, in step S3, feature extraction is performed on the target person profile image sequence, including: Target person outline images are extracted from the target person outline image sequence in sequence, and the target person image within the target person outline mark frame in the target person outline image is extracted, and feature extraction is performed on the target person image. The features to be extracted of the target person image include appearance features, motion features and spatiotemporal features of the target person image.

[0024] The appearance features of the target person image include color distribution features and texture features of the target person image, wherein the color distribution features include the color moment of the target person image and the color histogram of the target person image in the RGB color space, and the texture features are binary comparison results of the grayscale values ​​of a pixel and its neighboring pixels.

[0025] The motion characteristics of the target person image include the motion direction and motion speed of the target person image. By graying the target person image, the target person image will also be detected as the target person image in adjacent video frames, and the grayscale of the pixels in the target person image is the same, thereby constructing the basic optical flow equation of the pixels in the target person image.

[0026] The basic optical flow equation of the pixel is solved to obtain the pixel's movement direction and movement speed, and the average movement direction and movement speed of all pixels in the target person image are combined as the movement direction and movement speed of the target person image.

[0027] The spatiotemporal features of the target person image include the timestamp information of the video frame corresponding to the target person image and the location information of the corresponding distributed monitoring device.

[0028] Optionally, forming a target person embedding vector of a distributed monitoring device with the feature extraction result includes: Based on the appearance features, motion features, and spatiotemporal features of the target person's image, the feature similarity between different target person images in the same real-time video stream is calculated, and the target person image sequence with feature similarity higher than the preset threshold is used as the image sequence of the same target person in the real-time video stream, and the target person embedding vector of the target person in the real-time video stream corresponding to the current distributed monitoring device is constructed, where the jth target person in the real-time video stream corresponding to the mth distributed monitoring device is The target person embedding vector is : F( g m )=[ I m j ,f( I m j ), f 1 ( g m ), f 2 ( g m )] .

[0029] in: Indicates the target person In the target person image sequence in the real-time video stream corresponding to the mth distributed monitoring device, the target person image with the highest confidence score, express Appearance characteristics; Indicates the target person The feature extraction result corresponding to the target person image with the smallest timestamp information in the target person image sequence in the real-time video stream corresponding to the mth distributed monitoring device; Indicates the target person In the target person image sequence in the real-time video stream corresponding to the m-th distributed monitoring device, the feature extraction result corresponding to the target person image with the largest timestamp information.

[0030] Then, a target person embedding vector set of M distributed monitoring devices is constructed, where the target person embedding vector set of the mth distributed monitoring device is: {F( g m j )|j∈[1, Sum m ]} , Indicates the jth target person in the real-time video stream corresponding to the mth distributed monitoring device The target person embedding vector, represents the number of target personnel embedding vectors of the mth distributed monitoring device, m ∈[1,M] , M represents the total number of distributed monitoring devices.

[0031] Optionally, in step S4, cross-video stream association matching is performed on target person embedding vectors of different distributed monitoring devices, including: The target persons corresponding to the two groups of target persons whose cross-video stream correlation is higher than the preset correlation threshold are embedded in the vectors as the monitoring and identification results of the same target person in different key areas in the smart park. If the target person has potential risk behavior, the monitoring and identification results are used to conduct risk screening and post-event tracing of the potential risk behavior.

[0032] The target person embedding vector in the mth distributed monitoring device , and the target person embedding vector in the qth distributed monitoring device The cross-video stream association matching process is as follows: S41: Extract target person embedding vector and Appearance features in , calculate the appearance features The similarity of and Appearance relevance , the similarity calculation method of the appearance features is cosine similarity, where q ∈ [1,M],q ≠ m , p∈[1, Sum q ] .

[0033] S42: Extract target person embedding vector and The feature extraction result corresponding to the target person image with the smallest timestamp information and the feature extraction result corresponding to the target person image with the largest timestamp information are calculated using the timestamp information and the distance between the mth distributed monitoring device and the qth distributed monitoring device to obtain the target person embedding vector and Time constraint relevance .

[0034] S43: Embedding vector for target person and Appearance relevance , time constraint association Perform weighting and use the weighted result as the target personnel and If the cross-video stream correlation is higher than the preset correlation threshold, it means the target person and For the same target person, monitoring and identification of the target person in different key areas in the smart park is achieved.

[0035] In order to solve the above problems, the present invention provides a smart park video surveillance risk intelligent association identification system, the system comprising: The video stream processing module is used to collect real-time video streams from multiple distributed monitoring devices in the smart park, and extract and preprocess video frames to obtain the preprocessed video frame sequence of the distributed monitoring device. The preprocessed video frame sequence is input into the target detection model for target detection to obtain the target person contour image sequence.

[0036] The feature extraction module is used to extract features from the target person's contour image sequence and use the feature extraction results to form the target person embedding vector of the distributed monitoring device.

[0037] The association recognition device is used to perform cross-video stream association matching on the target person embedding vectors of different distributed monitoring devices to obtain the cross-video stream association relationship and recognition result of the target person.

[0038] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising: A memory storing at least one instruction; Communication interface, to realize communication between electronic devices; The processor executes the instructions stored in the memory to implement the above-mentioned smart park video surveillance risk intelligent association method.

[0039] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is executed by a processor in an electronic device to implement the above-mentioned smart park video surveillance risk intelligent association method.

[0040] Compared with the prior art, the present invention proposes a smart park video surveillance risk intelligent association method, which has the following advantages: First, this scheme proposes a target detection method. By deploying distributed monitoring equipment in the smart park, real-time video streams of different key areas are collected, and the noise information generated during the compression and transmission of the video stream is filtered by video frame filtering. The brightness channel of the pixels in the video frame is individually histogram equalized, which can expand the dynamic range of the image and make the dark and bright details clearer. This helps to provide better visual effects in high-contrast scenes without affecting the hue and saturation of the image and reducing color distortion. The target detection model is used to detect the target personnel in the video frame, generate a target personnel contour marker box containing the target personnel, and obtain the target personnel images of different real-time video streams. In the parameter training process of the target detection model, the bounding box parameters are modeled as a Gaussian distribution, and then the mean and standard deviation of the predicted bounding box parameters are provided to quantify the uncertainty of the predicted bounding box parameters, and enhance the robustness of the target detection model in detecting targets in complex scenes. Compared with directly regressing the bounding box coordinates, the use of Gaussian distribution can provide a smoother loss function. This smoothness helps to improve the regression performance of the model, making the model more stable and accurate during the training process.

[0041] At the same time, this scheme proposes a multi-video stream association monitoring and identification method, which extracts the appearance features, motion features and spatiotemporal features of the target person images in the real-time video stream, and calculates the feature similarity of different target person images under the same real-time video stream. The target person image sequence with feature similarity higher than the preset threshold is used as the image sequence of the same target person in the real-time video stream, and the target person embedding vector of the target person in the real-time video stream corresponding to the current distributed monitoring device is constructed. The cross-video stream association relationship of the target person embedding vector is calculated by using the appearance correlation and time constraint correlation of the target person embedding vector between different distributed monitoring devices. The target person corresponding to the two groups of target person embedding vectors with cross-video stream association higher than the preset association threshold is used as the monitoring and identification result of the same target person in different key areas in the smart park. If the target person has potential risk behavior, the monitoring and identification result is used to conduct risk investigation and post-tracing of the potential risk behavior, so as to realize the associated monitoring of target persons in different key areas in the smart park. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of a smart park video surveillance risk intelligent association method provided by one embodiment of the present invention.

[0043] Figure 2 A functional module diagram of a smart park video surveillance risk intelligent association identification system provided in one embodiment of the present invention.

[0044] Figure 2In: 100 smart park video surveillance risk intelligent correlation identification system, 101 video stream processing module, 102 feature extraction module, 103 correlation identification device.

[0045] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0047] The embodiment of the present application provides a method for intelligently associating video surveillance risks in a smart park. The execution subject of the method for intelligently associating video surveillance risks in a smart park includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the method provided in the embodiment of the present application. In other words, the method for intelligently associating video surveillance risks in a smart park can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc.

[0048] Embodiment 1: S1: Collect real-time video streams from multiple distributed monitoring devices in the smart park, extract and preprocess video frames, and obtain preprocessed video frame sequences of distributed monitoring devices.

[0049] In step S1, real-time video streams are collected from multiple distributed monitoring devices in the smart park, including: Distributed monitoring equipment is deployed in key areas of the smart park. The distributed monitoring equipment collects regional videos of the key areas in real time to obtain real-time video streams. The key areas include entrances and exits, parking lots, public activity areas, and the surrounding areas of park buildings. Each distributed monitoring device is connected to the monitoring management system through a monitoring network. The monitoring network adopts a wireless network to transmit the real-time video streams in the distributed monitoring equipment to the monitoring management system.

[0050] Collect real-time video streams from multiple distributed monitoring devices in the smart park, compress and encode the real-time video streams, and transmit them to the monitoring management system.

[0051] Perform video frame extraction and preprocessing on real-time video streams, including: The monitoring and management system decodes the received real-time video stream and divides the decoded real-time video stream into a number of video frames; specifically, an embodiment of the present invention uses OpenCV to decode the video stream, and uses the FFmpeg tool to set the number of frames extracted per second in the video stream, extracts video frames from the real-time video stream, and saves the extracted video frames in JPG image format.

[0052] The preprocessing of the video frames includes filtering, color contrast enhancement and resolution unification, and the preprocessing method is: The noise components in any video frame are filtered, wherein the noise components of the video frame are the rapidly changing fine textures or random pixel value fluctuations in the video frame. The main source of the noise components is the video flow distortion introduced in the real-time video stream compression encoding process. The noise components in the video frame are suppressed by filtering to improve the smoothness of the video frame. Specifically, the video frame can be filtered by a Gaussian filter, wherein the Gaussian filter performs weighted averaging on the pixel values, the weight of the central pixel is the largest, and as the distance increases, the weight gradually decreases, thereby smoothing the differences between pixels, improving the smoothness of the pixel values ​​in the neighboring pixel area of ​​the central pixel, and suppressing the noise components.

[0053] The color contrast of the filtered video frame is enhanced to enhance the contrast and visual effect of the video frame. The color contrast enhancement is a histogram equalization process.

[0054] As a preferred embodiment of the present invention, since brightness is separated from color information (hue, saturation) in the HSV color space, the brightness can be processed separately without affecting the hue and saturation of the image, thereby enhancing the image contrast more naturally and reducing color distortion. Therefore, the present invention converts the pixel value of the video frame from the RGB color space to the HSV color space, and calculates the brightness histogram of all pixels of the video frame in the HSV color space, where the pixel In the HSV color space, the brightness is V( ), brightness V( ) is a pixel The maximum value in the RGB color space; according to the brightness histogram of the video frame, the brightness cumulative distribution function of the video frame is calculated, and the brightness cumulative distribution function is normalized so that the brightness cumulative distribution function of different brightness levels is between 0 and 1, where the range of the brightness level is 0-255, and the brightness cumulative distribution function of the brightness level is the sum of the frequencies of the brightness level and all brightness levels below it in the brightness histogram; the normalized brightness cumulative distribution function is used to map the pixel to the new brightness, where the pixel The new brightness is : .

[0055] in: Indicates brightness The corresponding normalized brightness cumulative distribution function; Indicates rounding.

[0056] The brightness of the pixel is replaced with the new brightness and mapped to the RGB color space to obtain the RGB color value of the pixel after color contrast enhancement; by performing histogram equalization on the brightness channel, the dynamic range of the image can be expanded, making the dark and bright details clearer, which helps to provide better visual effects in high-contrast scenes.

[0057] All video frames after filtering and color contrast enhancement are adjusted to a uniform resolution to obtain pre-processed video frames, and the video frames are sorted in the order of timestamps to obtain video frame sequences collected by different distributed monitoring devices. Specifically, the FFmpeg tool can be used to adjust the video frames to a specified resolution.

[0058] S2: Input the preprocessed video frame sequence into the target detection model for target detection to obtain a target person contour image sequence.

[0059] The target detection model is used to extract a target person contour image sequence in a preprocessed video frame sequence through an improved YOLOv8 model.

[0060] The improved YOLOv8 model includes a conv convolutional layer, a C2f module, a bottleneck module and a Head layer. The conv convolutional layer is used to perform deformable convolution processing on the video frames in the preprocessed video frame sequence to obtain video frame features.

[0061] The C2f module is used to divide the video features into two parts. One part uses a 1×1 pixel convolution kernel to achieve cross-channel information fusion to obtain a convolution feature map, which reduces the number of channels and reduces the amount of calculation; the other part is processed by a series of bottleneck modules to refine and enhance the features to obtain an enhanced feature map. The bottleneck module uses two 3×3 pixel convolution kernels to further extract the input features and halve the number of channels. Finally, the concat process is used to fuse the convolution feature map and the enhanced feature map to obtain a fused feature map of the video frame.

[0062] The Head layer is used to divide the fused feature map into multiple target candidate regions and generate bounding box parameters of the target candidate regions, where the bounding box parameters include the center coordinates, width, and height offsets of the target candidate regions. The center coordinates, width, and height offsets are used as the 1st to 4th bounding box parameters in turn, and the confidence score of each target candidate region is calculated. The bounding box parameters of the target candidate regions with confidence scores higher than the preset threshold are retained.

[0063] A bounding box is generated in the video frame using the retained bounding box parameters as the target person contour marking box of the video frame. The video frame area within the target person contour marking box is the target person image in the video frame, and the confidence score of the target candidate area corresponding to the target person image is used as the confidence score of the target person image. The video frame with all target person contour marking boxes is marked, which is the target person contour image corresponding to the video frame.

[0064] The deformable convolution processing adds an offset to the sampling position of the ordinary convolution, expands the convolution range of the convolution kernel, and can capture richer feature information, especially when processing objects with complex shapes or irregular boundaries, which helps to improve the detection accuracy of the target detection model.

[0065] The target detection model is trained by the loss function, and the trained target detection model is used to output the target person contour marking frame of each video frame in the preprocessed video frame sequence. The video frame is marked based on the target person contour marking frame as the target person contour image, and the target person contour images of the continuous video frames are used as the target person contour image sequence of the distributed monitoring device. The expression of the loss function is: :

[0066] L us (θ)= ∑ n=1 N ∑ i∈{1,2,3,4} 1 2 [log(2π ( s i n (i)) 2 )+ ( Y n (i)- m i n (i)) 2 ( s i n (i)) 2 ] .

[0067] in: Represents the model parameters of the target detection model, including the weights of the convolution kernels in the deformable convolution process, the weights of the convolution kernels in the C2f module and the bottleneck module, and the bounding box regression parameters of the Head layer; represents the i-th true bounding box parameter of the n-th training sample, Represents model parameters Under training, the average of the i-th predicted bounding box parameters of the n-th training sample during multiple iterations, Represents model parameters Under training, the standard deviation of the i-th predicted bounding box parameters of the n-th training sample during multiple iterations, where N is the number of training samples.

[0068] The true bounding box parameters are the bounding box parameters of the manually marked target person in the training sample, and the predicted bounding box parameters are the bounding box parameters of the target person in the training sample predicted by using the target detection model.

[0069] Specifically, the ADAM optimization algorithm can be used to train the loss function to obtain the model parameters of the target detection model.

[0070] As a preferred embodiment of the present invention, during the parameter training process of the target detection model, the bounding box parameters are modeled as a Gaussian distribution, thereby providing the mean and standard deviation of the predicted bounding box parameters, quantifying the uncertainty of the predicted bounding box parameters, and enhancing the robustness of the target detection model when detecting targets in complex scenes. Compared with directly regressing the bounding box coordinates, using a Gaussian distribution can provide a smoother loss function. This smoothness helps to improve the regression performance of the model, making the model more stable and accurate during the training process.

[0071] In the process of training the model parameters of the target detection model, public data sets can be used, such as the PASCAL VOC2012 target detection data set, the Cityscapes data set, the ImageNet data set, the PASCAL VOC2007 target detection data set, etc.; this embodiment uses the PASCAL VOC2012 target detection data set as the training set and the PASCAL VOC2007 target detection data set as the test set to train the model parameters of the target detection model.

[0072] S3: Extract features from the target person's contour image sequence, and use the feature extraction results to form the target person embedding vector of the distributed monitoring device.

[0073] In the step S3, feature extraction is performed on the target person profile image sequence, including: Target person outline images are extracted from the target person outline image sequence in sequence, and the target person image within the target person outline mark frame in the target person outline image is extracted, and feature extraction is performed on the target person image. The features to be extracted of the target person image include appearance features, motion features and spatiotemporal features of the target person image.

[0074] The appearance features of the target person image include color distribution features and texture features of the target person image, wherein the color distribution features include the color moment of the target person image and the color histogram of the target person image in the RGB color space, and the texture features are binary comparison results of the grayscale values ​​of a pixel and its neighboring pixels.

[0075] The motion features of the target person image include the motion direction and motion speed of the target person image. By graying the target person image, the target person image will also be detected as the target person image in adjacent video frames, and the grayscale of the pixels in the target person image is the same, thereby constructing the basic optical flow equation of the pixels in the target person image: .

[0076] in: represents the direction of pixel movement, Indicates the speed of pixel movement; Represents the target person image The gray value of the pixel in the horizontal direction The partial derivative of Represents the target person image The gray value of the pixel in the vertical direction The partial derivative of Represents the target person image The partial derivative of the pixel gray value in the time frame direction t.

[0077] The basic optical flow equation of the pixel is solved to obtain the movement direction and speed of the pixel, and the mean movement direction and the mean movement speed of all pixels in the target person image are combined as the movement direction and the movement speed of the target person image; specifically, the Horn-Schunck method can be used to introduce a smoothness constraint, that is, the optical flow field is smooth, and the movement direction and the movement speed of the pixel are solved by minimizing the global energy.

[0078] The spatiotemporal features of the target person image include the timestamp information of the video frame corresponding to the target person image and the location information of the corresponding distributed monitoring device.

[0079] The step of forming a target person embedding vector of a distributed monitoring device with the feature extraction result includes: Based on the appearance features, motion features, and spatiotemporal features of the target person's image, the feature similarity between different target person images in the same real-time video stream is calculated, and the target person image sequence with feature similarity higher than the preset threshold is used as the image sequence of the same target person in the real-time video stream, and the target person embedding vector of the target person in the real-time video stream corresponding to the current distributed monitoring device is constructed, where the jth target person in the real-time video stream corresponding to the mth distributed monitoring device is The target person embedding vector is : F( g m )=[ I m j ,f( I m j ), f 1 ( g m ), f 2 ( g m )] .

[0080] in: Indicates the target person In the target person image sequence in the real-time video stream corresponding to the mth distributed monitoring device, the target person image with the highest confidence score, express Appearance characteristics; Indicates the target person The feature extraction result corresponding to the target person image with the smallest timestamp information in the target person image sequence in the real-time video stream corresponding to the mth distributed monitoring device; Indicates the target person In the target person image sequence in the real-time video stream corresponding to the m-th distributed monitoring device, the feature extraction result corresponding to the target person image with the largest timestamp information.

[0081] Then, a target person embedding vector set of M distributed monitoring devices is constructed, where the target person embedding vector set of the mth distributed monitoring device is: {F( g m j )|j∈[1, Sum m ]} , Indicates the jth target person in the real-time video stream corresponding to the mth distributed monitoring device The target person embedding vector, represents the number of target personnel embedding vectors of the mth distributed monitoring device, m ∈[1,M] , M represents the total number of distributed monitoring devices.

[0082] When calculating the feature similarity between different target person images, one or more combinations of Euclidean distance, Manhattan distance, Hamming distance, cosine similarity and Jaccard coefficient can be used for calculation. Specifically, the following formula can be used to calculate the feature similarity between different target person images: Calculate feature similarity: .

[0083] in: Represents the target person image Similarity of geographical features; represents an exponential function with a natural constant as base; represents the L2 norm; The target person images are appearance, motion, and spatiotemporal characteristics; The target person images are appearance characteristics, motion characteristics, and spatiotemporal characteristics.

[0084] S4: Perform cross-video stream association matching on the target person embedding vectors of different distributed monitoring devices to obtain the cross-video stream association relationship of the target person embedding vectors, and perform monitoring and identification of the target person.

[0085] In the step S4, cross-video stream association matching is performed on target person embedding vectors of different distributed monitoring devices, including: The target persons corresponding to the two groups of target persons whose cross-video stream correlation is higher than the preset correlation threshold are embedded in the vectors as the monitoring and identification results of the same target person in different key areas in the smart park. If the target person has potential risk behavior, the monitoring and identification results are used to conduct risk screening and post-event tracing of the potential risk behavior.

[0086] The target person embedding vector in the mth distributed monitoring device , and the target person embedding vector in the qth distributed monitoring device The cross-video stream association matching process is as follows: S41: Extract target person embedding vector and Appearance features in , calculate the appearance features The similarity of and Appearance relevance , the similarity calculation method of the appearance features is cosine similarity, where q ∈ [1,M],q ≠ m , p∈[1, Sum q ] .

[0087] S42: Extract target person embedding vector and The feature extraction result corresponding to the target person image with the smallest timestamp information and the feature extraction result corresponding to the target person image with the largest timestamp information are calculated using the timestamp information and the distance between the mth distributed monitoring device and the qth distributed monitoring device to obtain the target person embedding vector and Time constraint relevance ; Specifically, the time constraint correlation can be expressed as follows: Perform the calculation: .

[0088] in: represents the distance between the mth distributed monitoring device and the qth distributed monitoring device; Represents the embedding vectors from the target person and The difference between the two closest timestamp information of , where the target person is embedded in the vector and Contains a maximum timestamp information and a minimum timestamp information respectively; represents the distance required to walk from the mth distributed monitoring device to the qth distributed monitoring device. The human walking speed is about 1.5 m / s. The maximum is 1, The larger the value, the more targeted the person. The greater the probability of directly reaching the key area where the qth distributed monitoring device is located after leaving the key area where the mth distributed monitoring device is located, the more likely the target person is to be and The greater the temporal continuity between them.

[0089] S43: Embedding vector for target person and Appearance relevance , time constraint association Perform weighting and use the weighted result as the target personnel and If the cross-video stream correlation is higher than the preset correlation threshold, it means the target person and For the same target person, monitoring and identification of the target person in different key areas in the smart park is realized. Specifically, the sum of the weights of the two associations is 1.

[0090] Embodiment 2: like Figure 2 As shown, it is a functional module diagram of a smart park video surveillance risk intelligent association identification system provided by an embodiment of the present invention, which can implement the smart park video surveillance risk intelligent association method in Example 1.

[0091] The smart park video surveillance risk intelligent association identification system 100 of the present invention can be installed in an electronic device. According to the functions to be implemented, the smart park video surveillance risk intelligent association identification system can include a video stream processing module 101, a feature extraction module 102 and an association identification device 103. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0092] The video stream processing module 101 is used to collect real-time video streams from multiple distributed monitoring devices in the smart park, and extract and preprocess video frames to obtain a preprocessed video frame sequence of the distributed monitoring device, and input the preprocessed video frame sequence into the target detection model for target detection to obtain a target person contour image sequence.

[0093] The feature extraction module 102 is used to extract features from the target person outline image sequence and form a target person embedding vector of the distributed monitoring device with the feature extraction result.

[0094] The association recognition device 103 is used to perform cross-video stream association matching on the target person embedding vectors of different distributed monitoring devices to obtain the cross-video stream association relationship and recognition result of the target person.

[0095] In detail, the modules in the smart park video surveillance risk intelligent association identification system 100 described in the embodiment of the present invention are used in the same manner as described above. Figure 1 The smart park video surveillance risk intelligent association method described in the invention has the same technical means and can produce the same technical effects, so it will not be repeated here.

[0096] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0097] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. And the terms "including", "comprising" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0098] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0099] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A smart park video surveillance risk intelligent association method, characterized in that: The method comprises: S1: Collect real-time video streams from multiple distributed monitoring devices in the smart park, extract and preprocess video frames, and obtain preprocessed video frame sequences of distributed monitoring devices; S2: Input the preprocessed video frame sequence into the target detection model for target detection to obtain a target person contour image sequence; S3: Extract features from the target person's contour image sequence, and use the feature extraction results to form the target person embedding vector of the distributed monitoring device; S4: Perform cross-video stream association matching on the target person embedding vectors of different distributed monitoring devices to obtain the cross-video stream association relationship of the target person embedding vectors, and perform monitoring and identification of the target person.

2. The method for intelligently associating video surveillance risks in a smart park according to claim 1, characterized in that: In step S1, real-time video streams are collected from multiple distributed monitoring devices in the smart park, including: Distributed monitoring equipment is deployed in key areas of the smart park. The distributed monitoring equipment collects regional videos of the key areas in real time to obtain real-time video streams. The key areas include entrances and exits, parking lots, public activity areas, and the periphery of park buildings. Each distributed monitoring equipment is connected to the monitoring management system through a monitoring network. The monitoring network adopts a wireless network to transmit the real-time video streams in the distributed monitoring equipment to the monitoring management system. Collect real-time video streams from multiple distributed monitoring devices in the smart park, compress and encode the real-time video streams, and transmit them to the monitoring management system.

3. A smart park video surveillance risk intelligent association method as described in claim 2, characterized in that: Perform video frame extraction and preprocessing on real-time video streams, including: The monitoring management system decodes the received real-time video stream and divides the decoded real-time video stream into several video frames; The preprocessing of the video frames includes filtering, color contrast enhancement and resolution unification, and the preprocessing method is: Filter the noise components in any video frame, where the noise components in the video frame are the rapidly changing fine textures or random pixel value fluctuations in the video frame. The main source of the noise components is the video flow distortion introduced during the real-time video stream compression encoding process. The noise components in the video frame are suppressed by filtering to improve the smoothness of the video frame. Performing color contrast enhancement on the filtered video frame to enhance the contrast and visual effect of the video frame, wherein the color contrast enhancement is a histogram equalization process; All video frames after filtering and color contrast enhancement are adjusted to a uniform resolution to obtain preprocessed video frames, and the video frames are sorted in timestamp order to obtain video frame sequences collected by different distributed monitoring devices.

4. The method for intelligently associating video surveillance risks in a smart park according to claim 1, characterized in that: The S2 step includes: The target detection model is used to extract a target person contour image sequence in a preprocessed video frame sequence through an improved YOLOv8 model; The improved YOLOv8 model includes a conv convolution layer, a C2f module, a bottleneck module and a Head layer, wherein the conv convolution layer is used to perform deformable convolution processing on the video frames in the preprocessed video frame sequence to obtain video frame features; The C2f module is used to divide the video features into two parts. One part uses a 1×1 pixel convolution kernel to achieve cross-channel information fusion to obtain a convolution feature map, which reduces the number of channels and reduces the amount of calculation. The other part is processed by a series of bottleneck modules to refine and enhance the features to obtain an enhanced feature map. The bottleneck module uses two 3×3 pixel convolution kernels to further extract the input features and halve the number of channels. Finally, the concat process is used to fuse the convolution feature map and the enhanced feature map to obtain a fused feature map of the video frame. The Head layer is used to divide the fused feature map into multiple target candidate regions and generate bounding box parameters of the target candidate regions, where the bounding box parameters include the center coordinates, width, and height offsets of the target candidate regions. The center coordinates, width, and height offsets are used as the 1st to 4th bounding box parameters in turn, and the confidence score of each target candidate region is calculated. The bounding box parameters of the target candidate regions with confidence scores higher than the preset threshold are retained. A bounding box is generated in the video frame using the retained bounding box parameters as the target person contour marking box of the video frame. The video frame area within the target person contour marking box is the target person image in the video frame, and the confidence score of the target candidate area corresponding to the target person image is used as the confidence score of the target person image. The video frame with all target person contour marking boxes is marked, which is the target person contour image corresponding to the video frame.

5. A smart park video surveillance risk intelligent association method as described in claim 4, characterized in that: The target detection model is trained by the loss function, and the trained target detection model is used to output the target person contour marking frame of each video frame in the preprocessed video frame sequence. The video frame is marked based on the target person contour marking frame as the target person contour image, and the target person contour images of the continuous video frames are used as the target person contour image sequence of the distributed monitoring device. The expression of the loss function is: : ； in: Represents the model parameters of the target detection model, including the weights of the convolution kernels in the deformable convolution process, the weights of the convolution kernels in the C2f module and the bottleneck module, and the bounding box regression parameters of the Head layer; represents the i-th true bounding box parameter of the n-th training sample, Represents model parameters Under training, the average of the i-th predicted bounding box parameters of the n-th training sample during multiple iterations, Represents model parameters Under training, the standard deviation of the i-th predicted bounding box parameters of the n-th training sample during multiple iterations, where N is the number of training samples; The true bounding box parameters are the bounding box parameters of the manually marked target person in the training sample, and the predicted bounding box parameters are the bounding box parameters of the target person in the training sample predicted by using the target detection model.

6. The method for intelligently associating video surveillance risks in a smart park according to claim 1, characterized in that: In the step S3, feature extraction is performed on the target person profile image sequence, including: Extracting target person outline images from the target person outline image sequence in sequence, and extracting the target person image within the target person outline mark frame in the target person outline image, and performing feature extraction on the target person image, wherein the features to be extracted of the target person image include appearance features, motion features, and spatiotemporal features of the target person image; The appearance features of the target person image include color distribution features and texture features of the target person image, wherein the color distribution features include the color moment of the target person image and the color histogram of the target person image in the RGB color space, and the texture features are binary comparison results of the grayscale values ​​of a pixel and its neighboring pixels; The motion features of the target person image include the motion direction and motion speed of the target person image. By graying the target person image, the target person image will also be detected as the target person image in adjacent video frames, and the grayscale of the pixels in the target person image is the same, thereby constructing a basic optical flow equation for the pixels in the target person image. Solve the basic optical flow equation of the pixel to obtain the pixel's movement direction and movement speed, and combine the average movement direction and movement speed of all pixels in the target person's image as the movement direction and movement speed of the target person's image; The spatiotemporal features of the target person image include the timestamp information of the video frame corresponding to the target person image and the location information of the corresponding distributed monitoring device.

7. A smart park video surveillance risk intelligent association method as claimed in claim 6, characterized in that: The step of forming a target person embedding vector of a distributed monitoring device with the feature extraction result includes: Based on the appearance features, motion features, and spatiotemporal features of the target person's image, the feature similarity between different target person images in the same real-time video stream is calculated, and the target person image sequence with feature similarity higher than the preset threshold is used as the image sequence of the same target person in the real-time video stream, and the target person embedding vector of the target person in the real-time video stream corresponding to the current distributed monitoring device is constructed, where the jth target person in the real-time video stream corresponding to the mth distributed monitoring device is The target person embedding vector is : ； in: Indicates the target person In the target person image sequence in the real-time video stream corresponding to the mth distributed monitoring device, the target person image with the highest confidence score, express Appearance characteristics; Indicates the target person The feature extraction result corresponding to the target person image with the smallest timestamp information in the target person image sequence in the real-time video stream corresponding to the mth distributed monitoring device; Indicates the target person The feature extraction result corresponding to the target person image with the largest timestamp information in the target person image sequence in the real-time video stream corresponding to the mth distributed monitoring device; Then, a target person embedding vector set of M distributed monitoring devices is constructed, where the target person embedding vector set of the mth distributed monitoring device is: , Indicates the jth target person in the real-time video stream corresponding to the mth distributed monitoring device The target person embedding vector, represents the number of target personnel embedding vectors of the mth distributed monitoring device, , M represents the total number of distributed monitoring devices.

8. The method for intelligently associating video surveillance risks in a smart park according to claim 7, characterized in that: In the step S4, cross-video stream association matching is performed on target person embedding vectors of different distributed monitoring devices, including: The target persons corresponding to the two groups of target persons whose cross-video stream correlation is higher than the preset correlation threshold are embedded in the vectors as the monitoring and identification results of the same target person in different key areas of the smart park. If the target person has potential risk behaviors, the monitoring and identification results are used to conduct risk screening and post-event tracing of the potential risk behaviors; The target person embedding vector in the mth distributed monitoring device , and the target person embedding vector in the qth distributed monitoring device The cross-video stream association matching process is as follows: S41: Extract target person embedding vector and Appearance features in , calculate the appearance features The similarity of and Appearance relevance , the similarity calculation method of the appearance features is cosine similarity, where , ; S42: Extract target person embedding vector and The feature extraction result corresponding to the target person image with the smallest timestamp information and the feature extraction result corresponding to the target person image with the largest timestamp information are calculated using the timestamp information and the distance between the mth distributed monitoring device and the qth distributed monitoring device to obtain the target person embedding vector and Time constraint relevance ; S43: Embedding vector for target person and Appearance relevance , time constraint association Perform weighting and use the weighted results as target personnel and If the cross-video stream correlation is higher than the preset correlation threshold, it means the target person and For the same target person, monitoring and identification of the target person in different key areas in the smart park is achieved.

9. A smart park video surveillance risk intelligent correlation identification system, characterized in that: The system comprises: The video stream processing module is used to collect real-time video streams from multiple distributed monitoring devices in the smart park, extract and preprocess video frames, obtain preprocessed video frame sequences of distributed monitoring devices, input the preprocessed video frame sequences into the target detection model for target detection, and obtain target person contour image sequences; A feature extraction module is used to extract features from a target person's contour image sequence, and use the feature extraction results to form a target person embedding vector of a distributed monitoring device; An association recognition device is used to perform cross-video stream association matching on target person embedding vectors of different distributed monitoring devices to obtain the cross-video stream association relationship and recognition result of the target person; To realize a smart park video surveillance risk intelligent association method as described in any one of claims 1-8.

Images