Intelligent monitoring video privacy processing method and device and electronic equipment

Abstract

The invention discloses an intelligent monitoring video privacy processing method and device and electronic equipment, and relates to the technical field of smart home, and the method comprises the steps: collecting an original video stream, and carrying out the real-time semantic recognition of the original video stream, and obtaining a semantic recognition result; the semantic recognition result comprises object areas of at least one privacy related category; determining a processing decision for the object region based on a preset rule set and the semantic recognition result; according to the processing decision, generating differential shielding metadata corresponding to the original video stream; and the original video stream and the differential shielding metadata are encrypted and stored in an associated manner. According to the method, the problems that in the prior art, a static masking scheme cannot dynamically adapt to scene changes and lacks refined privacy management capability are solved, and reversible privacy protection of self-adaptive privacy area recognition based on real-time semantic understanding, strategic masking decision and separated storage of an original video and a masking instruction is realized.

Description

Technical Field

[0001] This invention relates to the field of smart home technology, and in particular to a method, apparatus and electronic device for processing privacy in intelligent surveillance videos. Background Technology

[0002] With the deep integration of artificial intelligence and the Internet of Things (IoT) technologies, intelligent surveillance cameras have been widely used in many fields such as public security, smart homes, and business management. While providing security and behavioral analysis capabilities, surveillance videos inevitably contain personal privacy information such as faces, license plates, and ID numbers. How to effectively protect the privacy information in the videos and avoid infringing on personal privacy rights while fulfilling security responsibilities has become a key issue that video surveillance technology development must address and solve.

[0003] Currently, the mainstream privacy protection solution in the industry is to pre-set fixed masking areas in the video footage. While this type of static masking method is simple to implement, its inherent defects are becoming increasingly apparent: First, it cannot adapt to changes in camera angle or movement of objects within the scene, leading to misalignment of the masking area, which may result in privacy leaks or excessive obstruction of useful monitoring information; second, it lacks the ability to differentiate monitored objects, making it impossible to implement differentiated privacy management strategies; third, the permanent masking of video data destroys the integrity of the original data, rendering it worthless when it needs to be used as legal evidence.

[0004] Therefore, designing a solution that can dynamically and intelligently identify and mask privacy information in video streams while taking into account both monitoring efficiency and the need for original data forensics is a critical technical challenge that urgently needs to be overcome in the field of intelligent surveillance. Summary of the Invention

[0005] This invention provides an intelligent surveillance video privacy processing method to solve the problems of existing static masking schemes being unable to dynamically adapt to scene changes and lacking refined privacy management capabilities. It achieves adaptive privacy region identification based on real-time semantic understanding, strategic masking decision-making, and reversible privacy protection by storing the original video and masking instructions separately.

[0006] This invention provides a method for intelligent surveillance video privacy processing, comprising: The original video stream is acquired, and real-time semantic recognition is performed on the original video stream to obtain semantic recognition results; the semantic recognition results include object regions of at least one privacy-related category; Based on a preset set of rules and the semantic recognition results, a processing decision for the object region is determined; the set of rules includes at least action instructions triggered based on object category, identity, and / or behavior. Based on the processing decision, differential occlusion metadata corresponding to the original video stream is generated, and the differential occlusion metadata records the location information and occlusion method of the object region; The original video stream and the differential masking metadata are encrypted and stored together.

[0007] According to the present invention, a method for intelligent surveillance video privacy processing is provided, wherein the step of performing real-time semantic recognition on the original video stream to obtain a semantic recognition result includes: performing image semantic segmentation on each frame of the original video stream using a first deep learning model to generate a semantic segmentation result; performing object detection on each frame of the original video stream using a second deep learning model, locating and selecting object instances belonging to any privacy-related category in each frame as object detection results; fusing the semantic segmentation result and the object detection result to generate a semantic mask; the semantic mask includes object category, precise boundary contour, and confidence information; extracting feature descriptors of object regions of each identified privacy-related category in each frame and comparing them with a pre-established authorized feature library to obtain the object's identity attributes; and integrating the semantic mask information, identity attribute inference results, and corresponding timestamps of all identified privacy-related category object regions in each frame into a semantic recognition result for the corresponding frame image.

[0008] According to the present invention, an intelligent surveillance video privacy processing method is provided. The step of determining a processing decision for the object region based on a preset rule set and the semantic recognition result includes: matching basic rules from the preset rule set based on the category label of each object region in the semantic recognition result; the basic rules define default processing actions for each category of object; generating a corresponding independent processing decision based on the matching result of each object region, the processing decision including at least one of masking, displaying, or masking and alarming; wherein, when the semantic recognition result contains identity attributes, the identity attributes are associated with the basic rules for judgment; if there are exception rules for a specific identity, the exception rules are used as the processing decision; when the semantic recognition result contains behavioral information, if an abnormal behavioral pattern is detected based on the behavioral information, the basic rules are forcibly overridden, and a masking and alarming decision instruction is generated.

[0009] According to the present invention, an intelligent surveillance video privacy processing method is provided, wherein generating differential masking metadata corresponding to the original video stream based on the processing decision includes: creating corresponding differential recording blocks on a frame-by-frame basis in the original video stream; encoding the contour information of each object region targeted by the processing decision in the form of a compressed polygon sequence in each differential recording block and writing it into a location field; selecting corresponding masking parameters from a predefined masking method set according to the masking method indicated in the processing decision and writing the masking parameters into a method field; serializing the differential recording blocks containing the location field and the method field in frame order and attaching the corresponding frame number and data verification information to form differential masking metadata corresponding to the original video stream.

[0010] According to the present invention, an intelligent surveillance video privacy processing method is provided, the method further includes: for the same object region whose position has not changed in multiple consecutive frames, recording its complete contour information only in the differential recording block of the first frame, and recording it in subsequent frames using a motion vector differential encoding method based on the first frame.

[0011] According to the present invention, an intelligent surveillance video privacy processing method is provided, wherein encrypting and storing the original video stream and the differential masking metadata separately and in association includes: performing real-time frame-level encryption on the original video stream using a first encryption algorithm to generate a first encrypted data stream; performing block-level encryption on the differential masking metadata using a second encryption algorithm to generate a second encrypted data block; establishing a bidirectional mapping relationship between the first encrypted data stream and the second encrypted data block through a timestamp index; writing the encrypted data after establishing the mapping relationship into different logical partitions of the same storage medium, and recording the partition mapping table and key index information in the file header.

[0012] According to the present invention, an intelligent surveillance video privacy processing method further includes, after encrypting and storing the original video stream and the differential masking metadata respectively, the method further includes: responding to a video playback request from a user terminal, obtaining the encrypted original video stream and differential masking metadata for the corresponding time period from the associated storage; performing decryption processing according to the playback request type; if it is a secure playback request, decrypting the original video stream and differential masking metadata respectively; if it is an original playback request, authenticating the user, and decrypting only the original video stream after successful authentication; for secure playback requests, rendering the original video of the corresponding frame in real time according to the decrypted differential masking metadata, performing pixel-level processing on a specified area in the decoded video frame according to the masking method; and streaming the processed video frame to the user terminal for playback.

[0013] The present invention also provides an intelligent surveillance video privacy processing device, comprising: A semantic recognition module is used to acquire raw video streams, perform real-time semantic recognition on the raw video streams, and obtain semantic recognition results; the semantic recognition results include at least one type of object region related to privacy. The processing decision generation module is used to determine the processing decision for the object region based on a preset set of rules and the semantic recognition result; the set of rules includes at least action instructions triggered based on object category, identity and / or behavior. The occlusion metadata generation module is used to generate differential occlusion metadata corresponding to the original video stream based on the processing decision. The differential occlusion metadata records the location information and occlusion method of the object region. The storage module is used to encrypt and store the original video stream and the differential masking metadata respectively.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the above-described intelligent surveillance video privacy processing methods.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the intelligent surveillance video privacy processing method described above.

[0016] The intelligent surveillance video privacy processing method, device, and electronic device provided by this invention accurately locate privacy-related object areas in the video stream through a real-time semantic recognition mechanism. This eliminates the dependence of traditional static masking on fixed screen areas, allowing the masking to dynamically adjust as the target moves. This effectively solves the problems of masking failure or loss of monitoring field of view caused by scene changes. By introducing a preset set of rules and making intelligent decisions based on object category, identity, or behavior, differentiated and refined management of different objects in the same frame is achieved, satisfying the privacy compliance principle of "data minimization" in complex scenarios. By generating and storing differential masking metadata separated from the original video stream, reversible privacy masking is achieved without compromising the integrity of the original video, ensuring the evidentiary value of video data as legal evidence. This achieves an innovative balance between monitoring efficiency and privacy protection. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the hardware environment for an intelligent surveillance video privacy processing method according to an embodiment of this application.

[0019] Figure 2 This is a flowchart illustrating the intelligent surveillance video privacy processing method provided by the present invention.

[0020] Figure 3 This is a schematic diagram of the core hardware architecture and core data flow of the intelligent surveillance camera system provided by the present invention.

[0021] Figure 4 This is a schematic diagram of the workflow and interaction between the core modules of the system provided by the present invention.

[0022] Figure 5 This is a schematic diagram of the intelligent surveillance video privacy processing device provided by the present invention.

[0023] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] According to one aspect of the embodiments of this application, a method for processing privacy in intelligent surveillance videos is provided. This method is widely applicable to whole-house intelligent digital control application scenarios such as smart homes, smart home ecosystems, smart home device ecosystems, and intelligence house ecosystems. Optionally, in this embodiment, the above-mentioned method for processing privacy in intelligent surveillance videos can be applied to, for example... Figure 1 The hardware environment shown consists of terminal device 102 and server 104. For example... Figure 1 As shown, server 104 is connected to terminal device 102 via a network and can be used to provide services (such as application services) to the terminal or clients installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data processing services for server 104.

[0027] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The terminal device 102 may not be limited to PC, mobile phone, tablet computer, smart air conditioner, smart range hood, smart refrigerator, smart oven, smart stove, smart washing machine, smart water heater, smart washing equipment, smart dishwasher, smart projector, smart TV, smart clothes rack, smart curtains, smart audio-visual equipment, smart socket, smart speaker, smart speaker box, smart fresh air equipment, smart kitchen and bathroom equipment, smart bathroom equipment, smart robot vacuum cleaner, smart window cleaning robot, smart mopping robot, smart air purifier, smart steam oven, smart microwave oven, smart water heater, smart air purifier, smart water dispenser, smart door lock, etc.

[0028] The English abbreviations and terms involved in this invention will be explained below.

[0029] Advanced Encryption Standard (AES) is a widely used symmetric encryption algorithm for protecting electronic data.

[0030] Cyclic Redundancy Check (CRC) is a checksum used to detect unexpected changes during digital data transmission or storage.

[0031] A graphics processing unit (GPU) is a microprocessor specifically designed for fast image and graphics computation.

[0032] An initialization vector (IV) is a random or pseudo-random value used in an encryption algorithm to ensure that even if the same plaintext is encrypted multiple times, different ciphertexts will be produced, thus enhancing security.

[0033] Intersection over Union (IoU) is a metric used in object detection to measure the degree of overlap between the predicted bounding box and the ground truth bounding box.

[0034] The following is combined Figures 2 to 6 Embodiments of the present invention are described.

[0035] Figure 2 This invention provides a flowchart illustrating a method for handling privacy in intelligent surveillance videos, comprising the following steps: S210: Acquire the raw video stream, perform real-time semantic recognition on the raw video stream, and obtain the semantic recognition result. The semantic recognition result includes object regions of at least one privacy-related category.

[0036] According to the present invention, an intelligent surveillance video privacy processing method performs real-time semantic recognition on the original video stream to obtain semantic recognition results. The method includes: performing image semantic segmentation on each frame of the original video stream using a first deep learning model to generate a semantic segmentation result; performing object detection on each frame of the original video stream using a second deep learning model, locating and selecting object instances belonging to any privacy-related category in each frame as object detection results; fusing the semantic segmentation results with the object detection results to generate a semantic mask; the semantic mask includes object category, precise boundary contour, and confidence information; extracting feature descriptors of object regions of each identified privacy-related category in each frame and comparing them with a pre-established authorized feature library to obtain the object's identity attributes; and integrating the semantic mask information, identity attribute inference results, and corresponding timestamps of all identified privacy-related category object regions in each frame into the semantic recognition result of the corresponding frame image.

[0037] Specifically, in a real-time semantic recognition unit deployed locally on a smart camera, the first deep learning model is a lightweight DeepLabV3+ semantic segmentation model based on the MobileNetV3 backbone, with its pre-trained weights deployed on the camera's dedicated AI acceleration chip. This model performs pixel-level classification on each frame of the original video image, outputting a semantic segmentation map containing preset privacy-related categories such as "face," "license plate," and "window." This segmentation map can accurately depict detailed boundaries such as facial features and license plate character regions. Simultaneously, the second deep learning model is a YOLO-V5n object detection model, running on the same AI acceleration chip, used to quickly locate and generate bounding boxes and category confidence scores for objects such as "face" and "vehicle" (which can be further associated with license plates) in the frame image.

[0038] The system performs fusion processing. Utilizing the fine contour information provided by semantic segmentation, it performs internal pixel-level verification and edge refinement on the bounding boxes provided by object detection. For example, it calculates the Intersection over Union (IoU) and calibrates the contours of the face regions defined by object detection and the semantically segmented "face" pixel regions, ultimately generating a "semantic mask" that combines the advantages of both models. This semantic mask not only includes object category labels but also a sequence of polygon vertex coordinates describing its precise contours, along with a fusion confidence score. Based on this, the system uses a lightweight face feature extraction network to extract a 128-dimensional feature descriptor for each "face" region identified by the semantic mask. This descriptor is then compared with the cosine similarity of family feature vectors stored in a "family member database" uploaded by the user's app. If the similarity is higher than a preset threshold, the object is identified as a "family member"; otherwise, it is identified as a "stranger." For regions categorized as "license plates," a similar identity comparison can be performed with a vehicle recognition system.

[0039] The system packages and encapsulates the semantic mask (including category, outline, and confidence) of all identified privacy objects (such as a "stranger face", a "family face", and a "license plate") in the current frame, their corresponding identity attribute inference results ("stranger", "family" or "unknown vehicle"), and the precise timestamp of the frame into a complete semantic recognition result for the image of that frame.

[0040] By integrating lightweight semantic segmentation and object detection models, pixel-level precision in identifying and locating privacy-related objects is achieved even with limited computing power on edge devices. This avoids the problem of a single detection box potentially including too much irrelevant background or missing fine details, laying a solid foundation for subsequent accurate occlusion. Identity attribute recognition achieved through feature extraction and authorization library comparison enables the system to go beyond simple category recognition, achieving differentiated understanding of different individuals in monitoring scenarios. This provides crucial input for the application of refined privacy rules based on identity. Integrating all the above information into a timestamped semantic recognition result forms a unified, structured data representation, greatly facilitating subsequent rule engine decision-making and efficient generation of differential metadata, improving the coherence and efficiency of the entire processing flow.

[0041] S220: Based on a preset set of rules and semantic recognition results, determine the processing decision for the object region. The set of rules includes at least action instructions triggered based on object category, identity, and / or behavior.

[0042] According to the present invention, an intelligent surveillance video privacy processing method is provided, which determines the processing decision for an object region based on a preset rule set and semantic recognition results. The method includes: matching basic rules from the preset rule set based on the category label of each object region in the semantic recognition results; defining default processing actions for each category of object; generating a corresponding independent processing decision based on the matching result of each object region, wherein the processing decision includes at least one of masking, displaying, or masking and alarming; wherein, when the semantic recognition results contain identity attributes, the identity attributes are associated with the basic rules for judgment, and if there are exception rules for a specific identity, the exception rules are used as the processing decision; when the semantic recognition results contain behavioral information, if an abnormal behavioral pattern is detected based on the behavioral information, the basic rules are forcibly overridden, and a masking and alarming decision instruction is generated.

[0043] Specifically, the adaptive rule engine maintains a configurable set of rules in the camera's local memory. This set contains basic rules and exception rules. The basic rules are: {Category: "Face", Default Action: "Display"}, {Category: "License Plate", Default Action: "Occlude"}, and {Category: "Window", Default Action: "Occlude"}. When the semantic recognition result indicates that there is a "face" region in a frame (category label: "Face") and the identity attribute is inferred to be "family member", the engine first matches the basic rule {Category: "Face", Default Action: "Display"}. Since the identity attribute exists, the engine further queries the exception rule library and finds a rule {Category: "Face", Identity: "Stranger", Action: "Occlude"}. Because the current identity is "family member", this exception rule is not triggered, so the final processing decision is "Display".

[0044] If the identity of another "face" region in the same frame is inferred to be a "stranger," the aforementioned exception rule is triggered, generating a "masking" decision. Furthermore, the rule base includes behavior trigger rules, such as {Behavior: "Lingering", Duration > 30 seconds, Action: "Mask and Alarm"}. The rule engine receives information from the behavior analysis submodule. When it detects a "face" target lingering in a sensitive area for more than 30 seconds, it is determined to be an "abnormal intrusion" behavior pattern. At this time, the engine will forcibly override any existing basic or identity-based rules for that target, directly generating a "mask and alarm" decision instruction. This instruction consists of two parts: first, executing a "mask" instruction on the face region in the video frame; and second, sending an intrusion alarm push notification to the user's mobile app via the network. All independent processing decisions for different object regions in the frame (e.g., a "family face," a "stranger face," a lingering "intruder face," a "license plate") are aggregated into the processing decision set for that frame.

[0045] By constructing a hierarchical (basic, exception, behavior-triggered) and configurable rule engine, a high degree of flexibility and intelligence in privacy management strategies is achieved. This enables the system to make fine-grained, differentiated decisions based on precise semantic recognition results (category, identity, behavior), such as implementing completely different privacy treatments for family members and strangers, thus ensuring convenience within the family while strictly protecting the privacy of outsiders. The introduction of behavior-triggered rules upgrades the system from passive, static rule matching to proactive, dynamic risk response. When potential security threats are detected, it can promptly implement stricter privacy protection measures and issue alerts, achieving strategic-level linkage between security and privacy protection. This decision-making mechanism based on multiple conditions greatly enhances the system's adaptability to complex and ever-changing monitoring scenarios, making privacy protection no longer a rigid, one-size-fits-all operation, but a customizable, scenario-aware intelligent service.

[0046] S230: Based on the processing decision, generate differential occlusion metadata corresponding to the original video stream. The differential occlusion metadata records the location information and occlusion method of the object region.

[0047] According to the present invention, an intelligent surveillance video privacy processing method generates differential masking metadata corresponding to the original video stream based on a processing decision. The method includes: creating corresponding differential recording blocks on a frame-by-frame basis in the original video stream; encoding the contour information of each object region targeted by the processing decision in a compressed polygon sequence within each differential recording block and writing it into a location field; selecting corresponding masking parameters from a predefined set of masking methods according to the masking method indicated in the processing decision and writing the masking parameters into a method field; serializing the differential recording blocks containing the location field and method field in frame order and appending the corresponding frame number and data verification information to form differential masking metadata corresponding to the original video stream.

[0048] Specifically, during the differential occlusion metadata generation process, the system creates an independent differential recording block for each frame of the original video, using a custom binary format. For each object region determined to be "occluded," the system first obtains the precise contour provided by its "semantic mask." This contour is converted into a polygon sequence consisting of multiple vertex coordinates (x, y). For efficient storage, the system uses the Douglas-Puk algorithm to simplify the polygons, removing redundant vertices while maintaining the basic shape of the contour, resulting in a compressed polygon sequence. For example, a face contour might be compressed from hundreds of original pixels into a sequence defined by 20-30 keypoints. This compressed sequence is encoded and written to the "position field" of the differential recording block. Then, Based on the "masking method" enumeration values ​​(e.g., 1-Gaussian blur, 2-pixelation, 3-virtual sticker) included in the processing decisions output by the rules engine, the corresponding specific parameters are selected from the predefined parameter table and written into the "method field". For example, if the masking method is "Gaussian blur", the "method field" is written with a preset blur radius value (e.g., 5 pixels); if it is "pixelation", the pixel block size (e.g., 8x8 pixels) is written; if it is "virtual sticker", the identifier of the corresponding sticker resource is written. A differential recording block can contain multiple such "position field-method field" pairs to correspond to multiple different masked objects in the same frame. The system serializes the differential recording blocks of all frames in chronological order to form a continuous metadata stream. During serialization, each recording block is prepended with its corresponding original video frame number (as a time index) and a cyclic redundancy check (CRC) code calculated based on the recording block content for data integrity verification during subsequent reading, ultimately forming a complete differential masking metadata file.

[0049] By generating differential occlusion metadata through a structured and lightweight encoding method, efficient and accurate representation of dynamic occlusion commands is achieved. Using compressed polygon sequences to represent object outlines significantly reduces the amount of data required for storage while maintaining the accuracy of the occlusion area, making the metadata file very compact and saving considerable storage space compared to storing a complete second occluded video. By separating and structurally storing position information and rendering parameters, the rendering engine can quickly and accurately parse and execute occlusion commands, supporting the real-time application of complex and diverse occlusion effects. The addition of frame numbers and verification information ensures a strict time synchronization between the metadata and the original video stream, as well as the reliability of the metadata itself during storage or transmission, providing an accurate and reliable data foundation for subsequent reversible rendering and authorized forensics. The entire generation process optimizes data organization, improving overall system processing efficiency and resource utilization.

[0050] The present invention provides an intelligent surveillance video privacy processing method, which further includes: for the same object region whose position has not changed in multiple consecutive frames, recording its complete contour information only in the differential recording block of the first frame, and recording it in subsequent frames using a motion vector differential encoding method based on the first frame.

[0051] Specifically, to further optimize the storage efficiency of differential occlusion metadata, the system performs intelligent differential encoding when recording metadata. When the rule engine determines that the position of the same privacy object (e.g., a "face" stationary in a window or a "license plate" of a parked vehicle) does not change significantly in consecutive video frames (e.g., its outline centroid offset is less than 5 pixels), the metadata generation module initiates differential recording mode. For this object, the system only records its compressed polygon outline sequence to the "position field" in the differential recording block of the frame in which it first appears (called the reference frame).

[0052] For the same object in subsequent consecutive frames, the system no longer records the complete contour coordinates repeatedly. Instead, it creates a special "differential position field" for the object in the differential recording block of the corresponding frame. This field does not contain the polygon sequence; instead, it records an index identifier pointing to the reference frame, as well as a tiny motion vector field from the reference frame contour to the current frame contour. This motion vector field can be calculated using optical flow, and it describes the tiny displacement of each keypoint on the reference frame contour in the current frame.

[0053] During decoding and rendering, the rendering engine first reads the complete outline of the reference frame. Then, based on the motion vector field in the "differential position field" of the current frame, it deforms and translates the reference outline accordingly to reconstruct the accurate outline position of the object in the current frame, and then performs occlusion. When the object begins to move significantly and its displacement exceeds a threshold, the system will end the current differential recording cycle and use the frame in which the object appears at the new position as the new reference frame to restart the complete outline recording.

[0054] By employing a differential coding mechanism, the storage efficiency of differential occlusion metadata is significantly improved. For numerous stationary or slowly moving privacy targets in monitoring scenarios (such as fixed indoor personnel or parked vehicles), it avoids repeatedly storing almost identical contour information in every frame, thereby drastically reducing the overall size of the metadata file and further lowering the system's storage overhead. Simultaneously, this coding method maintains the accuracy of occlusion because the contour reconstructed from the motion vector field can reflect subtle changes in the target's pose or shifts caused by camera shake. This optimization, without compromising privacy protection, makes the system more suitable for long-term, uninterrupted monitoring data recording, extends the effective usage time of local storage media, and reduces network bandwidth consumption when metadata needs to be transmitted back, thus improving the overall system's resource utilization efficiency and practicality.

[0055] S240: Encrypt the original video stream and differential masking metadata separately and store them together.

[0056] The present invention provides an intelligent surveillance video privacy processing method, which encrypts and stores the original video stream and differential masking metadata separately, specifically including: using a first encryption algorithm to perform real-time frame-level encryption on the original video stream to generate a first encrypted data stream; using a second encryption algorithm to perform block-level encryption on the differential masking metadata to generate a second encrypted data block; establishing a bidirectional mapping relationship between the first encrypted data stream and the second encrypted data block through a timestamp index; writing the encrypted data after establishing the mapping relationship into different logical partitions of the same storage medium, and recording the partition mapping table and key index information in the file header.

[0057] Specifically, in the local secure storage module of the camera, the system employs two independent encryption systems. For the raw video stream (S), after H.265 compression encoding, the system uses the AES-256 algorithm in CBC mode for real-time frame-level encryption. Before each frame of video data is written to storage, it is combined with a dynamically generated initialization vector (IV) and encrypted using a dedicated video encryption key (Kev) to generate the first encrypted data stream. For differential masking metadata (F), the system treats it as a series of data blocks and uses the RSA algorithm (or another independent AES key) for block-level encryption to generate the second encrypted data block.

[0058] To ensure a strict correlation between the two, the system appends the same high-precision timestamp (e.g., a Unix timestamp accurate to milliseconds) to each encrypted video frame and its corresponding encrypted metadata block during storage. Based on this timestamp, the system establishes a bidirectional mapping index table in the metadata area of ​​the storage medium. This table records the physical offset address (or logical block address, LBA) of the corresponding encrypted video frame in the storage medium for each timestamp, as well as the address of the corresponding encrypted metadata block. Finally, the first encrypted data stream and the second encrypted data block are written to different logical partitions (e.g., video partition and metadata partition) of the same SD card or NAS storage. At a fixed location on the storage medium (e.g., the file header), the system writes a master file header containing the aforementioned partition mapping table, key index information for identifying different encrypted content types (e.g., a pointer to the Kev storage location, a private key identifier for decrypting metadata), and the version number and integrity check code of the entire storage structure.

[0059] By implementing differentiated dual-stream encryption and a refined associated storage mechanism, the security of surveillance data is greatly enhanced. Different algorithms or keys are used to encrypt the original video and metadata respectively. Even if an attacker cracks one of them, they cannot obtain complete and understandable information (such as seeing a clear face or knowing where it is obscured), achieving a defense-in-depth. A bidirectional mapping is established through timestamps, ensuring that even when the data streams are stored separately, video and obscuration instructions can be accurately and efficiently retrieved and restored synchronously, guaranteeing the reliability of the "reversible obscuration" function. The mapping relationship and key index are centrally recorded in the protected file header, simplifying the data decryption and access process under authorized conditions, while setting a strong "digital seal" for unauthorized access. The entire solution achieves a high level of security while maintaining system processing efficiency and storage structure clarity, providing a solid and reliable technical foundation for balancing privacy protection and data forensic value.

[0060] According to the present invention, an intelligent surveillance video privacy processing method is provided. After encrypting and storing the original video stream and differential masking metadata separately, the method responds to a video playback request from a user terminal by retrieving the encrypted original video stream and differential masking metadata for the corresponding time period from the associated storage. Decryption is performed according to the playback request type: for a secure playback request, both the original video stream and differential masking metadata are decrypted; for an original playback request, user authentication is performed, and only the original video stream is decrypted upon successful authentication. For secure playback requests, the original video of the corresponding frame is rendered in real time based on the decrypted differential masking metadata, and pixel-level processing is performed on a specified area in the decoded video frame according to the masking method. The processed video frame is then streamed to the user terminal for playback.

[0061] Specifically, when a user initiates a video playback request via a mobile app, the system first parses the request parameters to obtain the requested time period and playback type ("secure view" or "raw view"). For a "secure playback request," the system, based on the time period, uses a bidirectional mapping table in the storage file header to locate and read the corresponding encrypted raw video stream data block and encrypted differential masking metadata block from different logical partitions of the storage medium. The system uses their respective decryption keys (Kev for video, Km for metadata) to decrypt them, obtaining the original H.265 bitstream and a sequence of structured metadata record blocks. The decoder decodes the video bitstream frame by frame to recover the original video frames in YUV or RGB format.

[0062] The rendering engine parses the metadata record block corresponding to the timestamp to obtain the compressed polygon outline sequence and occlusion method parameters for all areas to be occluded in the frame. Then, the rendering engine creates a transparent layer in memory with the same size as the original video frame. For each area to be occluded, based on its outline coordinates, it generates an occlusion effect at the corresponding position on the layer using a specified occlusion method (e.g., calling the GPU shader program to perform a Gaussian blur with a specified radius). Finally, the occlusion layer is alpha-blended with the original video frame to generate the final secure video frame, which is then encoded in real-time by the transmission module and streamed to the user's app for playback. For "original playback requests," the user must authenticate via password or fingerprint on the app. After successful authentication, the system only reads and decrypts the encrypted original video stream for the corresponding time segment, directly decodes it, and transmits it to the user without loading, parsing, or applying any metadata, thus presenting the original, unprocessed image.

[0063] By designing two independent video playback paths, the system achieves a perfect balance between privacy protection and data integrity. For "secure playback requests" used in daily viewing, the system dynamically generates a masking effect before transmission through a real-time rendering mechanism, ensuring that privacy information is not leaked at the consumer end, while avoiding the huge overhead of storing an extra masked video. For "original playback requests" in serious scenarios such as judicial evidence collection, the system ensures that authorized users can access complete, untampered original video data through strict identity verification and direct playback that bypasses metadata processing, preserving its core value as legal evidence. This on-demand rendering and distributed access design allows the same stored data to simultaneously meet the dual needs of daily privacy protection and data forensics in special circumstances, greatly improving the system's practicality and compliance, and providing users with a flexible, secure, and efficient video viewing experience.

[0064] Figure 3This invention demonstrates the core hardware architecture and core data flow of the intelligent surveillance camera system. As shown in the figure, the system starts with an image sensor, responsible for acquiring raw, high-resolution video streams. This video stream is directly input to an AI processing chip integrating dedicated AI processing capabilities. Inside this chip, an object detection module and a semantic segmentation module work in parallel, together forming a "real-time semantic segmentation and object detection module." This module performs pixel-level analysis and understanding of the raw video stream, accurately identifying privacy-related objects such as faces and license plates, as well as their outlines. The recognition results are then fed into a rule engine (i.e., an "adaptive rule engine"). This engine dynamically generates processing decisions (such as occlusion or display) for each identified object region based on a preset, configurable rule base (e.g., distinguishing between family members and strangers).

[0065] Based on the rule engine's decision-making, the system generates and outputs two encrypted data streams. One is an encrypted video stream, which is the original video stream compressed and then encrypted using a separate key. The other is an encrypted data unit, namely "differential masking metadata" generated based on the processing decision. It records the location information and masking method of the object area to be masked, and is also encrypted. These two encrypted data streams can be transmitted to the user terminal or cloud service through the transmission module. Simultaneously, they are also securely written to the storage module, where the encrypted video stream and encrypted metadata are stored together but physically separate, providing a data foundation for subsequent "reversible masking" playback. The entire process embodies the core concept of "front-end perception, edge computing," with all AI recognition, rule-based decision-making, and data encryption completed at the camera end, ensuring real-time processing and data privacy security.

[0066] Figure 4 The workflow and interaction between the core modules of the system of this invention are explained. First, the acquired video frames are buffered by the AI ​​engine module and then sent to the AI ​​inference engine (i.e., the semantic segmentation and object detection module) for real-time analysis. The analysis results (semantic recognition results) are passed to the rule engine decision unit, which generates a processing decision based on a preset rule base. The decision result is used to generate metadata, namely differential masking metadata, which contains specific masking instructions. This metadata is stored in the encrypted differential unit. Simultaneously, the original video stream, after video encoding, is sent to the storage module along with the encrypted differential metadata, and is stored in association with the original video file and the encrypted differential unit, respectively.

[0067] When a user initiates a playback request on the app, the system first determines the request type. If it's a secure video request, the system reads the corresponding original video file and encrypted differential unit from storage, decrypts them, decodes the original video, renders the adaptation effect in memory based on metadata, and finally outputs a secure frame for playback. If it's an original video request, an authentication process is triggered. If authentication is successful, the system directly decrypts and plays the original video file without loading or using metadata; if authentication fails, the request is withdrawn and playback is not performed.

[0068] The intelligent surveillance video privacy processing device provided by the present invention is described below. The intelligent surveillance video privacy processing device described below and the intelligent surveillance video privacy processing method described above can be referred to in correspondence.

[0069] Figure 5 A schematic diagram of an intelligent surveillance video privacy processing device provided by the present invention includes the following modules: The semantic recognition module 510 is used to acquire the original video stream, perform real-time semantic recognition on the original video stream, and obtain semantic recognition results; the semantic recognition results include at least one type of object region related to privacy. The processing decision generation module 520 is used to determine the processing decision for the object region based on a preset rule set and the semantic recognition result; the rule set includes at least action instructions triggered based on object category, identity and / or behavior. The occlusion metadata generation module 530 is used to generate differential occlusion metadata corresponding to the original video stream according to the processing decision. The differential occlusion metadata records the location information and occlusion method of the object region. Storage module 540 is used to encrypt and store the original video stream and the differential masking metadata respectively.

[0070] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute an intelligent surveillance video privacy processing method. This method includes: acquiring an original video stream; performing real-time semantic recognition on the original video stream to obtain a semantic recognition result; the semantic recognition result includes at least one category of privacy-related object regions; determining a processing decision for the object regions based on a preset rule set and the semantic recognition result; the rule set includes at least action instructions triggered based on object category, identity, and / or behavior; generating differential masking metadata corresponding to the original video stream according to the processing decision, the differential masking metadata recording the location information and masking method of the object regions; and encrypting and storing the original video stream and the differential masking metadata separately and in association.

[0071] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0072] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by the computer, the computer can execute the intelligent surveillance video privacy processing method provided by the above methods. The method includes: acquiring an original video stream, performing real-time semantic recognition on the original video stream, and obtaining a semantic recognition result; the semantic recognition result includes at least one type of object region related to privacy; determining a processing decision for the object region based on a preset rule set and the semantic recognition result; the rule set includes at least action instructions triggered based on object category, identity, and / or behavior; generating differential masking metadata corresponding to the original video stream according to the processing decision, the differential masking metadata recording the location information and masking method of the object region; and encrypting and storing the original video stream and the differential masking metadata together.

[0073] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the aforementioned intelligent surveillance video privacy processing methods. The method includes: acquiring an original video stream; performing real-time semantic recognition on the original video stream to obtain a semantic recognition result; the semantic recognition result includes at least one category of object regions related to privacy; determining a processing decision for the object region based on a preset rule set and the semantic recognition result; the rule set includes at least action instructions triggered based on object category, identity, and / or behavior; generating differential masking metadata corresponding to the original video stream according to the processing decision, the differential masking metadata recording the location information and masking method of the object region; and encrypting and storing the original video stream and the differential masking metadata together.

[0074] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0075] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing privacy in intelligent surveillance videos, characterized in that, include: The original video stream is acquired, and real-time semantic recognition is performed on the original video stream to obtain the semantic recognition result; The semantic recognition result includes at least one category of privacy-related object regions; Based on a preset set of rules and the semantic recognition results, a processing decision for the object region is determined; the set of rules includes at least action instructions triggered based on object category, identity, and / or behavior. Based on the processing decision, differential occlusion metadata corresponding to the original video stream is generated, and the differential occlusion metadata records the location information and occlusion method of the object region; The original video stream and the differential masking metadata are encrypted and stored together.

2. The intelligent surveillance video privacy processing method according to claim 1, characterized in that, The real-time semantic recognition of the original video stream to obtain the semantic recognition result includes: The first deep learning model is used to perform image semantic segmentation on each frame of the original video stream to generate semantic segmentation results. The second deep learning model performs target detection on each frame of the original video stream, and locates and selects object instances belonging to any privacy-related category in each frame as the target detection result. The semantic segmentation result and the target detection result are fused to generate a semantic mask; the semantic mask includes object category, precise boundary contour and confidence information; Extract the feature descriptor of each object region in each frame of the image that has been identified as a privacy-related category, and compare it with a pre-established authorization feature library to obtain the object's identity attributes; The semantic masking information, identity attribute inference results, and corresponding timestamps of all identified privacy-related object regions in each frame image are integrated into the semantic recognition result of the corresponding frame image.

3. The intelligent surveillance video privacy processing method according to claim 1, characterized in that, The step of determining the processing decision for the object region based on the preset rule set and the semantic recognition result includes: Based on the category label of each object region in the semantic recognition result, basic rules are matched from a preset rule set; the basic rules define the default processing actions for each category of object; Based on the matching results of each object region, a corresponding independent processing decision is generated, and the processing decision includes at least one of masking, displaying, or masking and alarming. Wherein, when the semantic recognition result contains identity attributes, the identity attributes are associated with the basic rules for judgment. If there are exception rules for a specific identity, the exception rules are used as the processing decision. When the semantic recognition result contains behavioral information, if an abnormal behavioral pattern is detected based on the behavioral information, the basic rules are forcibly overridden, and a decision instruction to mask and alarm is generated.

4. The intelligent surveillance video privacy processing method according to claim 1, characterized in that, The step of generating differential masking metadata corresponding to the original video stream based on the processing decision includes: Create corresponding differential recording blocks based on frames of the original video stream; In each of the differential record blocks, the contour information of each object region targeted by the processing decision is encoded in the form of a compressed polygon sequence and written into the position field; Based on the occlusion method indicated in the processing decision, select the corresponding occlusion parameter from the predefined occlusion method set, and write the occlusion parameter into the method field; The differential recording block containing the location field and the mode field is serialized in frame order, and the corresponding frame number and data verification information are appended to form differential masking metadata corresponding to the original video stream.

5. The intelligent surveillance video privacy processing method according to claim 4, characterized in that, The method further includes: For the same object region whose position does not change in multiple consecutive frames, its complete contour information is recorded only in the differential recording block of the first frame, and then recorded in subsequent frames using a motion vector differential encoding method based on the first frame.

6. The intelligent surveillance video privacy processing method according to claim 1, characterized in that, The step of encrypting and storing the original video stream and the differential masking metadata separately and in association specifically includes: The original video stream is encrypted in real time at the frame level using a first encryption algorithm to generate a first encrypted data stream; The differential masking metadata is encrypted at the block level using a second encryption algorithm to generate a second encrypted data block. Establish a bidirectional mapping relationship between the first encrypted data stream and the second encrypted data block using timestamp indexing; After establishing the mapping relationship, the encrypted data is written to different logical partitions of the same storage medium, and the partition mapping table and key index information are recorded in the file header.

7. The intelligent surveillance video privacy processing method according to any one of claims 1 to 6, characterized in that, After encrypting and storing the original video stream and the differential masking metadata respectively, the method further includes: In response to a video playback request from a user terminal, the encrypted original video stream and differential masking metadata for the corresponding time period are obtained from the associated storage. The decryption process is performed based on the playback request type. If it is a secure playback request, the original video stream and differential masking metadata are decrypted separately. If it is an original playback request, the user is authenticated, and only the original video stream is decrypted after successful authentication. For secure playback requests, the original video of the corresponding frame is rendered in real time based on the decrypted differential masking metadata, and pixel-level processing is performed on the specified area in the decoded video frame according to the masking method. The processed video frames are streamed to the user terminal for playback.

8. A smart surveillance video privacy processing device, characterized in that, include: The semantic recognition module is used to acquire the original video stream, perform real-time semantic recognition on the original video stream, and obtain the semantic recognition result. The semantic recognition result includes at least one category of privacy-related object regions; The processing decision generation module is used to determine the processing decision for the object region based on a preset set of rules and the semantic recognition result; the set of rules includes at least action instructions triggered based on object category, identity and / or behavior. The occlusion metadata generation module is used to generate differential occlusion metadata corresponding to the original video stream based on the processing decision. The differential occlusion metadata records the location information and occlusion method of the object region. The storage module is used to encrypt and store the original video stream and the differential masking metadata respectively.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the intelligent surveillance video privacy processing method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the intelligent surveillance video privacy processing method as described in any one of claims 1 to 7.

Images