Method, system and electronic device for privacy protection of security surveillance video

Abstract

The present invention discloses a method, system, and electronic device for privacy protection of security surveillance videos, relating to the field of video privacy protection. The method comprises: constructing a video directed graph of security surveillance videos; obtaining a user's privacy requirements; applying a minimum set method to determine the privacy requirement core elements of the video directed graph based on the privacy requirements; when the security surveillance is offline, performing invisible processing on a subset of the privacy requirement core elements in the security surveillance video according to the privacy protection time, thereby obtaining a privacy-protected security surveillance video; and when the security surveillance is online, applying a Markov model to predict video privacy events requiring privacy protection in the security surveillance video, and performing invisible processing on a subset of the privacy requirement core elements in the video privacy events, thereby obtaining a privacy-protected security surveillance video. The present invention can improve the usability of a video surveillance system while meeting the user's security and privacy requirements.

Description

Technical Field

[0001] The present invention relates to the field of video privacy protection, and in particular to a method, system and electronic equipment for privacy protection of security monitoring videos. Background Art

[0002] The booming development of video surveillance systems (VSS) has brought convenience to the healthcare industry. VSS monitor public places such as kindergartens, hospitals, and nursing homes, providing immediate feedback to guardians to prevent accidents and losses. However, the continuous wide-angle monitoring of VSS raises the issue of temporal and spatial privacy leakage of user activities. Designing a full-process, cross-device, and predictive video privacy protection solution that meets the privacy needs of individual users while maximizing the usability of all users within the system has become an increasingly popular research issue.

[0003] Existing research often anonymizes users through methods such as blurring, masking, or replacing faces to protect their privacy. However, these solutions can result in significant data availability loss, are targeted only at a single user, and fail to meet diverse needs. Some solutions also fail to prevent attackers from inferring privacy-sensitive information based on the spatiotemporal correlations between videos captured by multiple surveillance devices within the system, posing a certain risk of data leakage. These issues primarily include: 1) Failure to consider the privacy risks associated with the coordinated operation of multiple devices within the system: An intelligent surveillance system typically consists of multiple surveillance devices distributed across different locations. Due to spatial continuity, the videos captured by these devices often have logical correlations. Protecting only a single device cannot effectively defend against cross-device inference attacks. 2) Failure to fully address the issue of sequential privacy: A video is a series of frames, with adjacent frames being highly logically correlated. Therefore, when watching a video, a viewer will learn from the information in the previous frame to predict the content of the next frame. Therefore, even if the system removes a user from a sensitive event in the video, the viewer can still infer the sensitive event. This is referred to as the sequential privacy issue, and existing solutions do not fully address it. 3) Reduced usability: The applied protection techniques significantly reduce the usability of the video. Therefore, there is a trade-off between video privacy and usability. Existing privacy protection schemes sacrifice a significant amount of video usability while maintaining privacy. 4) Targeting only a single user: Existing schemes focus on protecting the privacy of a single user while neglecting to protect the relationships between pairs of users. This characteristic makes these schemes unable to meet the needs of multiple users. 5) Unable to meet differentiated needs: Existing systems provide the same protection to everyone without considering their specific needs, even though differences between these needs may improve the usability of protected videos. Existing solutions do not differentiate to meet customized user needs. Summary of the Invention

[0004] Based on the problems existing in the prior art, the present invention provides a method, system and electronic equipment for privacy protection of security surveillance videos, which can improve the usability of the video surveillance system on the basis of meeting the security and privacy needs of users.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A method for privacy protection of security surveillance videos, the method comprising:

[0007] Construct a video directed graph of security surveillance videos;

[0008] Obtaining users’ privacy needs;

[0009] According to the privacy requirement, applying a greedy algorithm to determine the core elements of the privacy requirement of the video directed graph;

[0010] When the security monitoring is in an offline state, the core elements of the privacy requirements in the security monitoring video are made invisible according to the privacy protection time to obtain the security monitoring video after privacy protection;

[0011] When the security monitoring is online, the Markov model is applied to predict the video privacy events that need to be protected in the security monitoring video, and the core elements of the privacy requirements in the video privacy events are processed invisible to obtain the security monitoring video after privacy protection.

[0012] Optionally, constructing a video directed graph of security surveillance videos specifically includes:

[0013] Obtain security surveillance video;

[0014] Applying a sampling window to divide the security surveillance video into sub-videos with the same number of frames;

[0015] Identifying objects in the sub-video; the objects include users and background objects;

[0016] A video directed graph is constructed according to the objects.

[0017] Optionally, constructing a video directed graph based on the user and the context specifically includes:

[0018] A video directed graph is constructed with each security monitoring camera as a vertex and the distance between the monitoring areas of each camera and the last position of the user in the sub-video as edges; the last position is the relative position of the user to the background object when the user last appears in the sub-video.

[0019] Optionally, the privacy requirement includes user, time, location, background object and viewer; the viewer is a user permission list.

[0020] Optionally, the privacy protection time is a time privacy protection time, a location privacy protection time, a content privacy protection time, or a multi-demand privacy protection time;

[0021] When the user needs to protect time privacy, the start time of the time privacy protection time is a preset time period threshold value ahead of the start time input by the user for privacy protection; the end time of the preset protection time is a preset time period threshold value behind the end time input by the user for privacy protection;

[0022] When the user needs to protect the privacy of the location, the start time of the location privacy protection time is a preset time period threshold before the target user arrives at the privacy location; the end time of the preset protection time is a preset time period threshold after the target user leaves the privacy location;

[0023] When the user needs to protect the privacy of the content, the start time of the content privacy protection time is a preset time period threshold earlier than the time when the target user and the background object meet the set position relationship; the end time of the content privacy protection time is a preset time period threshold later than the time when the target user and the background object no longer meet the set position relationship;

[0024] When a user needs to protect multiple privacy requirements, the start time of the multiple privacy requirements protection time is the earliest time among the start time of the time privacy protection time, the start time of the location privacy protection time, and the start time of the content privacy protection time; the end time of the multiple privacy requirements protection time is the latest time among the end time of the time privacy protection time, the end time of the location privacy protection time, and the end time of the content privacy protection time.

[0025] Optionally, a subset of the core elements of privacy requirements in the video privacy event is made invisible by replacing the screen.

[0026] A security surveillance video privacy protection system is applied to the above-mentioned security surveillance video privacy protection method, and the system includes:

[0027] A construction module for constructing a video directed graph of security surveillance videos;

[0028] Acquisition module, used to obtain the user's privacy needs;

[0029] A core element determination module, configured to determine the core elements of the privacy requirements of the video directed graph by applying a greedy algorithm according to the privacy requirements;

[0030] An offline protection module is used to, when the security monitoring is in an offline state, perform invisible processing on a subset of the core elements of the privacy requirements in the security monitoring video according to the privacy protection time to obtain the security monitoring video after privacy protection;

[0031] The online protection module is used to apply the Markov model to predict video privacy events that require privacy protection in the security monitoring video when the security monitoring is in an online state, and to perform invisible processing on a subset of the core elements of privacy requirements in the video privacy events to obtain the security monitoring video after privacy protection.

[0032] An electronic device includes a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the above-mentioned method for privacy protection of security surveillance videos.

[0033] Optionally, the memory is a readable storage medium.

[0034] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0035] The present invention provides a method, system, and electronic device for privacy protection of security surveillance videos. By constructing a collaborative directed graph between multiple surveillance devices, the system finds the logical and temporal relationships between their video contents, extracts the core elements of user privacy protection needs, finds operations that can meet all privacy needs, and selects the start time of privacy operations based on privacy needs. Sensitive objects that need to be protected are deleted through image processing technology and then delivered to the video surveillance platform. Because the smallest subset of requirements that must be executed is selected, the availability of surveillance videos can be maximized, and the problem of sequential privacy is solved, so that viewers cannot infer the content of sensitive events from the content of the previous frame. While maintaining the functionality and efficiency of the intelligent monitoring system, this solution can resist inference attacks against spatiotemporal logical correlations of multiple devices and temporal logical correlations of single devices, thereby maximizing the protection of user privacy and making the operation more convenient and visualized. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 A flowchart of the method for privacy protection of security surveillance videos provided by the present invention;

[0038] Figure 2A schematic diagram of the implementation process of privacy protection for security surveillance videos provided by the present invention;

[0039] Figure 3 This is a module diagram of the security surveillance video privacy protection system provided by the present invention.

[0040] Description of reference numerals:

[0041] Construction module - 1, acquisition module - 2, core element determination module - 3, offline protection module - 4, online protection module - 5. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0043] The purpose of the present invention is to provide a method, system and electronic device for privacy protection of security surveillance videos. By increasing the defense against inference attacks against spatiotemporal logical correlations of multiple devices and temporal logical correlations of single devices, the availability of the video surveillance system can be maximized while meeting the security and privacy needs of users.

[0044] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] Example 1

[0046] The present invention provides a method for privacy protection of security surveillance videos, which is applied to privacy protection of home video security systems. It is only applicable to monitoring scenes of people, and is not applicable to cars or other objects. It allows users to independently define and describe their privacy needs through a visual interactive interface, and automatically provides corresponding privacy protection for the user. Specifically, Figure 1 and Figure 2 As shown, the method includes:

[0047] Step 101: Construct a video directed graph of security surveillance videos.

[0048] Wherein, step 101 specifically includes:

[0049] Step 1011: Obtain security surveillance video.

[0050] Step 1012: Apply a sampling window to divide the security surveillance video into sub-videos with the same number of frames.

[0051] Step 1013: Identify objects in the sub-video; the objects include users and background objects.

[0052] Step 1014: Construct a video directed graph based on the object.

[0053] Specifically, a video directed graph is constructed with each security monitoring camera as a vertex, and the distance between the monitoring areas of each camera and the last position of the user in the sub-video as edges; the last position is the relative position of the user to the background object when he last appeared in the sub-video.

[0054] Step 102: Obtain the user's privacy requirements; specifically, the privacy requirements include user, time, location, background object and viewer; the viewer is a user permission list.

[0055] Step 103: Based on the privacy requirements, apply a greedy algorithm to determine the core elements of the privacy requirements of the video directed graph. Specifically, according to step 101, determine the set of privacy requirements that need to be protected, and extract a subset of the minimum elements that independently express the privacy requirements as the core element subset of the privacy requirements.

[0056] Step 104: When the security monitoring is in an offline state, the core elements of the privacy requirement in the security monitoring video are made invisible according to the privacy protection time to obtain the security monitoring video after privacy protection.

[0057] Step 105: When the security monitoring is online, a Markov model is applied to predict video privacy events that require privacy protection in the security monitoring video, and the core elements of the privacy requirements in the video privacy events are processed invisible to obtain the security monitoring video after privacy protection.

[0058] As a specific implementation, in step 101, the surveillance video to be processed is divided into a set of sub-videos with equal frame sizes based on the size of the sampling window. Each sub-video is scanned to identify each existing user and background object. During object recognition, existing contour- and gait-based methods are used to match users in the video with objects to be protected, based on predefined outlines provided by the user. Specifically, an open-source neural network-based algorithm, YOLOv4, is employed in this invention to identify protected objects. The predefined outlines are photos uploaded by the user in advance, which help identify and abstract key features of a specific user for continuous tracking. A directed graph consisting of nodes and edges is constructed based on the content of the surveillance video. The directed graph primarily serves to construct the spatiotemporal relationships between different cameras. In step 105, a Markov model providing predictive protection is input to obtain predicted video privacy events. The vertices represent a collection of multiple cameras, and the edges represent the distances between the monitored areas of the cameras and the locations where the identified objects disappear, indicating the locations where specific objects disappear in the video captured by the cameras represented by the vertices. In addition, because the background in the monitoring scene is relatively fixed (the camera is in a fixed position), the background objects are identified first, and then the users are identified for different users, and certain people in the monitoring video are monitored (given by the user, and generally users only care about a few specific people).

[0059] As a specific implementation, in step 102, the user can provide the user's privacy requirements expressed in the form of events through a visual user interaction interface, and the privacy requirements are converted into the form of (user, time, location, background object, viewer) and stored in the corresponding requirement set. Among them, the background object is fixed, such as a tree or door in the background, and the location refers to the relative relationship between the user and these background objects. The viewer can be regarded as a list of authorized users who can view the specific user. The receiving user adds privacy requirements for the node selected by clicking through the visual interaction interface: the requirement can be defined as (user, time, location, background object, viewer). In order to define a rule, the user needs to select at least one of them and give it a non-empty value, which is called the target of the rule.

[0060] As a specific implementation, in step 103, in order to keep most of the information in the video, it is necessary to determine a minimum core subset of the privacy requirement set to be executed, which meets all the requirements set by the user and minimizes the size of the subset. The present invention adopts a greedy algorithm that can reduce computational complexity and processing time. First, find the required maximum requirement subset from the privacy requirement set to be executed. If there are still multiple choices left, then select one randomly. Then delete all the requirements belonging to the maximum requirement subset from the requirement set. Then repeat the last two steps until all the requirements in the set requirements are met. Specifically, for any user and object that needs to be protected, check which subset of the requirement set will be met by protecting this user / object and store the requirement subset. Use the idea of ​​the minimum set to find a minimum subset of the core elements of the requirements that meets all the requirements of the set requirements.

[0061] As a specific implementation, in steps 104 and 105, a subset of the core privacy elements in the video privacy event is made invisible by replacing the screen. In offline scenarios, the start time of the protection operation is selected according to the privacy requirements when performing the privacy protection operation, so that the capture time of the two frames after protection is greater than the inference threshold of the next frame from the previous frame. In online scenarios, a hidden Markov model is used to predict the possible occurrence of events that require privacy protection, and privacy protection operations are performed on them. A visual privacy protection method (such as blind vision or object removal) is used as the privacy protection operation, and the screen of the object to be protected is replaced to achieve the invisible processing effect of deleting the object.

[0062] Furthermore, the privacy protection time in step 104 and step 105 is a time privacy protection time, a location privacy protection time, a content privacy protection time, or a multi-demand privacy protection time. Specifically, the method for determining the privacy protection time includes:

[0063] When the user needs to protect time privacy, the start time of the time privacy protection time is a preset time period threshold ahead of the start time input by the user for privacy protection; the end time of the preset protection time is a preset time period threshold behind the end time input by the user for privacy protection.

[0064] When the user needs to protect the privacy of their location, the start time of the location privacy protection time is a preset time period threshold earlier than the time when the target user arrives at the privacy location; the end time of the preset protection time is a preset time period threshold later than the time when the target user leaves the privacy location.

[0065] When a user needs to protect content privacy, the content privacy protection period begins a preset time threshold before the target user and background objects meet a set positional relationship; the content privacy protection period ends a preset time threshold after the target user and background objects no longer meet the set positional relationship. Specifically, content privacy refers to images of specific objects or people within a video, such as faces or phone screens.

[0066] When a user needs to protect multiple privacy requirements, the start time of the multiple privacy requirements protection time is the earliest time among the start time of the time privacy protection time, the start time of the location privacy protection time, and the start time of the content privacy protection time; the end time of the multiple privacy requirements protection time is the latest time among the end time of the time privacy protection time, the end time of the location privacy protection time, and the end time of the content privacy protection time.

[0067] In practical applications, a time threshold is preset. The time threshold is a preset time period threshold. If the capture time interval of the video frames is greater than the threshold, the mutual dependence between the two frames is small, and the viewer cannot accurately estimate the content of one frame based on the information of the other frame. The start time of the protection operation is selected according to the type of privacy requirements:

[0068]

[0069] When the user-defined privacy requirement is only for time, the user-defined privacy time is the time when the user-defined privacy protection requirement for time begins. For example, if the user defines privacy protection as required from 10 PM to 6 AM, then the privacy time is 10 PM, and the protection operation starts at 10 PM minus the time threshold. When the user-defined privacy requirement is only for location, the target arrival time is the time when the target arrives at the location requiring privacy protection. For example, if the user defines the bedroom as a privacy location, the privacy time is the time when someone enters the bedroom. If the target object remains at the target location from the beginning of the video, then the target arrival time = start time + time threshold, and the protection operation starts at the start time. The time when the privacy requirement content is satisfied is the time when the video content satisfies both the user and scenario policies in the privacy requirement. If the time when the privacy requirement content is satisfied is less than the start time + time threshold, then the time when the privacy requirement content is satisfied is set to = start time + time threshold to ensure that the protection operation starts no earlier than the start time. The non-empty minimum time for the mixed requirement is the non-empty minimum time that combines the privacy requirements of time, location, and background objects (user-defined privacy time, time when the target arrives at the privacy location, and time when the privacy requirement content is satisfied).

[0070] In addition, the method for privacy protection of security surveillance video provided by the present invention also includes:

[0071] The privacy-protected security surveillance video is sent to the video surveillance system platform for the normal operation of the home video surveillance system. Specifically, the privacy-protected surveillance video is sent to the cloud server of the video surveillance system for the normal operation of the video surveillance system.

[0072] The security surveillance video privacy protection method provided by the present invention is used to protect the privacy information of video content. It allows users to specify specific privacy standards and formulate corresponding privacy protection policies based on their needs, thereby maximizing the satisfaction of users' privacy needs without affecting the functionality and efficiency of the security system. The security surveillance video privacy protection method provided by the present invention can be implemented through program instructions to obtain the corresponding security surveillance video privacy protection app.

[0073] The advantages of the privacy protection method for security surveillance videos provided by the present invention are as follows:

[0074] 1) The functionality is complete and can still meet the user's privacy needs even if the attacker has the ability to make reasonable inferences based on part of the video.

[0075] 2) Easy to operate, providing a visual interface for users to describe and meet their security and privacy requirements, and very friendly to users who have not received any relevant training.

[0076] 3) Good compatibility: this method can be used on various existing cloud-based video surveillance systems.

[0077] 4) Good real-time performance, which can accurately protect user privacy without affecting real-time video surveillance.

[0078] Example 2

[0079] In order to execute the method corresponding to the above embodiment 1 and achieve the corresponding functions and technical effects, a system for privacy protection of security surveillance video is provided below. Figure 3 As shown, the system includes:

[0080] Construction module 1 is used to construct a video directed graph of security monitoring videos.

[0081] Acquisition module 2 is used to obtain the user's privacy needs.

[0082] The core element determination module 3 is used to apply a greedy algorithm to determine the core elements of the privacy requirements of the video directed graph based on the privacy requirements.

[0083] The offline protection module 4 is used to make the core elements of privacy requirements in the security monitoring video invisible according to the privacy protection time when the security monitoring is in an offline state, so as to obtain the security monitoring video after privacy protection.

[0084] The online protection module 5 is used to apply the Markov model to predict video privacy events that require privacy protection in the security monitoring video when the security monitoring is in an online state, and to perform invisible processing on the core elements of the privacy requirements in the video privacy events to obtain the security monitoring video after privacy protection.

[0085] Example 3

[0086] An embodiment of the present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the method for privacy protection of security surveillance videos of embodiment 1.

[0087] Optionally, the above-mentioned electronic device may be a server.

[0088] In addition, an embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the method for privacy protection of security surveillance videos of the first embodiment is implemented.

[0089] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0090] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A method for privacy protection of security surveillance videos, characterized in that: The method comprises: Construct a video directed graph of security surveillance videos; Obtaining users’ privacy needs; According to the privacy requirement, applying a greedy algorithm to determine the core elements of the privacy requirement of the video directed graph; When the security monitoring is in an offline state, the core elements of the privacy requirements in the security monitoring video are made invisible according to the privacy protection time to obtain the security monitoring video after privacy protection; When the security monitoring is online, the Markov model is applied to predict the video privacy events that need to be protected in the security monitoring video, and the core elements of the privacy requirements in the video privacy events are processed invisible to obtain the security monitoring video after privacy protection.

2. The method for privacy protection of security surveillance video according to claim 1, characterized in that: The video directed graph of the security surveillance video is constructed as follows: Obtain security surveillance video; Applying a sampling window to divide the security surveillance video into sub-videos with the same number of frames; Identifying objects in the sub-video; the objects include users and background objects; A video directed graph is constructed according to the objects.

3. The method for privacy protection of security surveillance video according to claim 2, characterized in that: Constructing a video directed graph based on the object, specifically including: A video directed graph is constructed with each security monitoring camera as a vertex and the distance between the monitoring areas of each camera and the last position of the user in the sub-video as edges; the last position is the relative position of the user to the background object when the user last appears in the sub-video.

4. The method for privacy protection of security surveillance video according to claim 1, characterized in that: The privacy requirements include user, time, location, background object and viewer; the viewer is a user permission list.

5. The method for privacy protection of security surveillance videos according to claim 4, characterized in that: The privacy protection time is a time privacy protection time, a location privacy protection time, a content privacy protection time, or a multi-demand privacy protection time; When the user needs to protect time privacy, the start time of the time privacy protection time is a preset time period threshold ahead of the start time input by the user for privacy protection; the end time of the time privacy protection time is a preset time period threshold behind the end time input by the user for privacy protection; When the user needs to protect the privacy of the location, the start time of the location privacy protection time is a preset time period threshold before the target user arrives at the privacy location; the end time of the location privacy protection time is a preset time period threshold after the target user leaves the privacy location; When the user needs to protect the privacy of the content, the start time of the content privacy protection time is a preset time period threshold earlier than the time when the target user and the background object meet the set position relationship; the end time of the content privacy protection time is a preset time period threshold later than the time when the target user and the background object no longer meet the set position relationship; When a user needs to protect multiple privacy requirements, the start time of the multiple privacy requirements protection time is the earliest time among the start time of the time privacy protection time, the start time of the location privacy protection time, and the start time of the content privacy protection time; the end time of the multiple privacy requirements protection time is the latest time among the end time of the time privacy protection time, the end time of the location privacy protection time, and the end time of the content privacy protection time.

6. The method for privacy protection of security surveillance video according to claim 1, characterized in that: A subset of the core elements of privacy requirements in the video privacy event is made invisible by replacing the screen.

7. A security surveillance video privacy protection system, characterized in that: The system comprises: A construction module for constructing a video directed graph of security surveillance videos; Acquisition module, used to obtain the user's privacy needs; A core element determination module, configured to determine the core elements of the privacy requirements of the video directed graph by applying a greedy algorithm according to the privacy requirements; An offline protection module is used to make the core elements of privacy requirements in the security monitoring video invisible according to the privacy protection time when the security monitoring is in an offline state, so as to obtain the security monitoring video after privacy protection; The online protection module is used to apply the Markov model to predict video privacy events that require privacy protection in the security monitoring video when the security monitoring is in an online state, and to perform invisible processing on the core elements of the privacy requirements in the video privacy events to obtain the security monitoring video after privacy protection.

8. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the method for privacy protection of security surveillance videos according to any one of claims 1 to 6.

9. The electronic device according to claim 8, characterized in that: The memory is a readable storage medium.

Images