Privacy protection method and privacy protection system for image

Abstract

The invention belongs to the technical field of image privacy protection, and discloses an image privacy protection method and system. The image privacy protection method comprises the following steps: S1, carrying out privacy detection on an input original image; s2, on the basis of a privacy area segmentation algorithm of 2D scanning, segmenting the original image according to the identified privacy-related area to form a privacy-related image and a privacy-removed image; s3, encoding, encrypting and outputting the privacy-related image; and encoding and outputting the privacy-removed image. By applying the privacy protection method provided by the invention, the segmentation of the privacy content of the original image and the establishment of the privacy-related image can be efficiently completed only by executing assignment and copy operations through a privacy region segmentation algorithm based on 2D scanning, and the method utilizes AI identification, so that the privacy protection efficiency is improved. The method has the advantages that privacy content detection is more intelligent, the processing efficiency is higher, and the processing accuracy is higher.

Description

Technical Field

[0001] The present invention belongs to the technical field of image privacy protection, and in particular, relates to an image privacy protection method and a privacy protection system. Background Art

[0002] With the rapid development of the information age and the popularization of smart devices, people are accustomed to sharing and communicating on social platforms, and the acquisition and dissemination of image and video data has become easier.

[0003] However, images and videos may contain a large amount of personal information and location information, which also brings greater risks of privacy leakage, including but not limited to images and videos generated in public places, social media, security monitoring and other application scenarios. At present, a large number of traditional image and video acquisition systems do not provide effective privacy protection.

[0004] Therefore, there is an urgent need to provide a flexible, efficient, and highly versatile image privacy protection method and privacy protection system to overcome the above problems.

[0005] In view of this, this application is hereby filed. Summary of the invention

[0006] The technical problem to be solved by the present invention is to overcome the deficiency of the conventional image and video acquisition system in the prior art that the image privacy is not effectively protected, and the purpose is to provide an image privacy protection method.

[0007] Another object of the present invention is to provide a privacy protection system using the above-mentioned image privacy protection method.

[0008] In order to solve the above technical problems, the basic concept of the technical solution adopted by the present invention is: to provide an image privacy protection method, comprising:

[0009] S1. Perform privacy detection on the input original image;

[0010] If it is detected that the input original image contains privacy-related content, the detected privacy-related content is framed by a rectangular boundary and marked as a privacy-related area;

[0011] S2. A privacy region segmentation algorithm based on 2D scanning is used to segment the original image according to the identified privacy-related regions to form a privacy-related image and a privacy-removed image;

[0012] S3, encoding, encrypting and outputting the privacy-related image;

[0013] Encode the deprived image and output it.

[0014] According to an embodiment of the present invention, step S1 further includes:

[0015] If it is detected that the input original image does not contain any privacy-related content, the original image is encoded and output.

[0016] According to an embodiment of the present invention, for the original image, the number of areas marked as privacy-related areas is N;

[0017] Wherein, N≥0, N is an integer.

[0018] According to an embodiment of the present invention, the privacy region segmentation algorithm of the 2D scan comprises the steps of:

[0019] A) Preset processing of data;

[0020] B) scanning the 2D plane with the calibration lines in a corresponding predetermined order, and outputting the effective privacy area scanned;

[0021] C) Clear and copy the privacy-related area, and output the privacy-free image and the privacy-related image.

[0022] According to one embodiment of the present invention, the step A) of pre-processing data includes:

[0023] a1) The default privacy background color is black, and the default privacy foreground color is a fixed gray value color;

[0024] a2) Create a blank privacy-related image; set its color to the privacy background color and its size to be consistent with the original image size;

[0025] a3) sorting the horizontal sides and vertical sides of the N input rectangles identified as privacy-related areas on the 2D plane according to the coordinate values;

[0026] After the sorting is completed, the horizontal edge of the 2D plane is divided into horizontal scale positions X by N rectangles marked as privacy-related areas. 1 , X 2 , …, X m , the vertical edge is divided into vertical scale positions Y 1 , Y 2 , …, Y n ;

[0027] a4) Clear the ASS list;

[0028] a5) Clear the partition list G k , where k is G k Subscript variable of the list, K is G k The length of the list; initialize k and K so that k=K=0.

[0029] According to an embodiment of the present invention, the step B) scanning the 2D plane with the calibration lines according to their corresponding predetermined order comprises:

[0030] b1) From the horizontal line H to the bottom, follow the Y 1 , Y 2 , …, Y n Scan the 2D plane in sequence;

[0031] When scanning to Y i When , clear the LR counter and clear the HSS list; where 1≤i≤n;

[0032] b2) Follow the vertical line V from left to right along X 1 , X 2 , …, X m Scan the 2D plane in sequence;

[0033] When scanning to the left edge of any rectangle, the LR counter is incremented by 1, and the position where the LR counter changes from 0 to 1 is recorded as L;

[0034] When the right edge of any rectangle is scanned, the LR counter is reduced by 1, and the position where the LR counter changes from a positive number to 0 is recorded as R;

[0035] Record the line segment of the horizontal line H between L and R as the scanning line segment SS, and add SS to the HSS list;

[0036] When the vertical line V scan is finished, all the scan line segments SS in the ASS list are scanned vertically to the horizontal line H position, and the effective privacy area scanned is output as G k ;

[0037] If the current output G k Compared with the previously output G k’ If the vertical positions are adjacent and the horizontal sizes are the same, G k Merge to G k’ ; If the above adjacent relationship does not exist, the values ​​of k and K are added by 1 respectively;

[0038] b3) Update the ASS list;

[0039] Delete the scan line segment SS in the ASS list that is above the vertical position of the horizontal line H and is no longer associated with any valid rectangular area; that is, if the ordinate value of the scan line segment SS is less than Y i , and SS is outside all rectangular areas intersecting with the current horizontal line H, then SS is deleted from the ASS list;

[0040] Delete the SS that has completed scanning in step b2) from the ASS list;

[0041] Merge the HSS list into the ASS list;

[0042] The horizontal line H continues to scan downward and repeats b2 to b3 until the 2D scan is completed.

[0043] According to an embodiment of the present invention, the step C) clearing and copying the image, and outputting the privacy-removed image and the privacy-related image includes:

[0044] c1) The subscript variable k is in the order of K-1 to 0, and the partition list G obtained by 2D scanning is executed accordingly. k The K partitions contained in ;

[0045] c2) G k The original image is copied to the newly created privacy-related image, and its relative position remains corresponding to the original image;

[0046] At the same time, the G k The pixels in the left column of the left border and the pixels in the upper row of the upper border are copied to the corresponding positions of the newly created privacy-related image;

[0047] c3) G k Clear from the original image and set the color to the privacy foreground color;

[0048] At the same time, the G k The color of the left column of pixels on the left side of the left border and the upper row of pixels on the upper side of the upper border are set as the privacy foreground color; (the logic of selecting the left and upper pixels here is based on the reference logic (reference to the left and upper sides) of the encoder when making intra-frame predictions;)

[0049] c4) outputting the result image;

[0050] Outputting the original image after the above clearing operation as a privacy-removed image;

[0051] The result of completing the content copy operation on the newly created privacy-related image is output as the privacy-related image.

[0052] The present invention also provides an image privacy protection system, which applies the above-mentioned image privacy protection method, and the image privacy protection system includes:

[0053] A privacy detection module, wherein the privacy detection module is configured to detect whether the input original image contains privacy-related content, and to select the privacy content detected in the original image containing the privacy-related content and mark it as a privacy-related area;

[0054] A privacy segmentation module, the privacy segmentation module is electrically connected to the privacy detection module; the privacy segmentation module segments the input original image according to the identified privacy-related area and the original image to form a privacy-free image and a privacy-related image;

[0055] A first encoder, the first encoder is electrically connected to the privacy segmentation module; the first encoder is configured to encode and output the privacy-removed image;

[0056] A second encoder, the second encoder is electrically connected to the privacy segmentation module; the second encoder is configured to encode the privacy-related image;

[0057] An encryption module is electrically connected to the second encoder; the encryption module encrypts and outputs the privacy-related image encoded by the second encoder.

[0058] According to one embodiment of the present invention, the privacy detection module adopts the YOLO target detection algorithm based on the CNN convolutional neural network.

[0059] According to an embodiment of the present invention, the first encoder encodes the privacy-removing image in parallel with the second encoder encodes the privacy-related image.

[0060] Alternatively, the first encoder encodes the privacy-removing image and the second encoder encodes the privacy-related image in series.

[0061] By applying the privacy protection method of an image provided by the present invention, the privacy protection method identifies the privacy-related area through the YOLO target detection algorithm based on the CNN convolutional neural network; through the privacy area segmentation algorithm based on 2D scanning, the segmentation of the privacy content of the original image and the establishment of the privacy-related image can be efficiently completed by only executing the assignment and copy operations; the encoding efficiency of the privacy-related image and the privacy-delisted image is high; at the same time, AI recognition is used, which has the characteristics of more intelligent detection of privacy-related content, higher processing efficiency and higher processing accuracy.

[0062] The specific implementation modes of the present invention are further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings are part of the present invention and are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but do not constitute an improper limitation of the present invention. Obviously, the drawings described below are only some embodiments. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the accompanying drawings:

[0064] Figure 1 is a logic flow chart of a method for protecting the privacy of an image in an embodiment of the present invention;

[0065] Figure 2 is an information transmission relationship diagram between components of an image privacy protection system according to an embodiment of the present invention;

[0066] Figure 3 Schematic diagram of a privacy area segmentation algorithm based on 2D scanning in an embodiment of the present invention;

[0067] Figure 4 Schematic diagram of the operation of the privacy area segmentation algorithm based on 2D scanning in an embodiment of the present invention.

[0068] Description of the main components in the figure:

[0069] 1. Privacy detection module; 2. Privacy segmentation module; 3. First encoder; 4. Second encoder; 5. Encryption module.

[0070] It should be noted that these drawings and textual descriptions are not intended to limit the conceptual scope of the present invention in any way, but are intended to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The following embodiments are used to illustrate the present invention but are not used to limit the scope of the present invention.

[0072] In the description of the present invention, it should be noted that the directions or positional relationships indicated by terms such as “upper”, “lower”, “front”, “back”, “left”, “right”, “vertical”, “inside” and “outside” are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as a limitation on the present invention.

[0073] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0074] like Figures 1 to 4As shown, the privacy protection method of an image described in the present invention comprises the steps of:

[0075] S1. Performing privacy detection on the input original image (or a certain frame or each frame of the video) to detect whether the input original image contains any privacy-related content;

[0076] If it is detected that the input original image contains privacy-related content, the detected privacy-related content is framed by a rectangular boundary and marked as a privacy-related area;

[0077] S2. A privacy region segmentation algorithm based on 2D scanning is used to segment the original image according to the identified privacy-related regions to form a privacy-related image and a privacy-removed image;

[0078] S3, encoding, encrypting and outputting the privacy-related image;

[0079] Encode the deprived image and output it.

[0080] By applying the image privacy protection method provided in the present invention, the privacy protection method uses a privacy area segmentation algorithm based on 2D scanning to segment the privacy content of the original image and establish a privacy-related image. It can be efficiently completed by only executing assignment and copy operations. At the same time, AI recognition is utilized, which has the characteristics of more intelligent detection of privacy-related content, higher processing efficiency and higher processing accuracy.

[0081] It should be noted that, since a video is actually composed of continuous images, the present invention is not only applicable to privacy detection and protection of images, but also to privacy detection and protection of videos, that is, the privacy detection module can perform privacy detection on images extracted frame by frame (or specified frames) from the original video to detect whether they contain privacy-related content.

[0082] In another specific implementation of this embodiment, a standard video decoding module is provided, which can decode the original video and decompose it into frames of images; the video decoding module is electrically connected to the privacy detection module, and the video decoding module transmits the image of a specified frame or each frame in the video to the privacy detection module.

[0083] In a specific implementation of this embodiment, when the image includes a single image, the privacy protection method directly acts on the image itself;

[0084] When the image involves a video, the privacy protection method acts on the image contained in the selected frame rate (or each frame rate) of the video. That is, when the input image is contained in a video, the privacy protection method performs privacy protection processing on each frame of the video; for example, for a video with a frame rate of 30fps and a duration of 10s, the privacy protection is performed on the 300 frames of images contained therein in sequence.

[0085] In a specific implementation of this embodiment, the step S1 further includes:

[0086] If it is detected that the input original image does not contain any privacy-related content, the original image is used as a privacy-removing image, encoded and output.

[0087] In a specific implementation of this embodiment, in the original image, the number of areas marked as privacy-related areas is N;

[0088] Wherein, N≥0, N is an integer.

[0089] It can be understood that if it is detected that the input original image does not contain any privacy-related content, the number of privacy-related areas N is marked as 0; if it is detected that the input original image contains privacy-related content, the number of privacy-related areas N is marked as ≥1.

[0090] In a specific implementation of this embodiment, the privacy area segmentation algorithm of the 2D scan includes:

[0091] A) Preset processing of data;

[0092] B) scanning the 2D plane with the calibration lines in a corresponding predetermined order, and outputting the effective privacy area scanned;

[0093] C) Clear and copy the privacy-related area, and output the privacy-free image and the privacy-related image.

[0094] In a specific implementation of this embodiment, the step A) of pre-setting the data includes:

[0095] a1) Preferably, the system presets the privacy background color pbg (Privacy Background Color) as black, and the privacy foreground color pfg (Privacy Foreground Color) as a fixed gray value color; that is, the privacy foreground color may have the same gray value as the privacy background color (for example, both are black), or may have different gray values ​​(for example, pbg is black, and pfg is gray);

[0096] a2) Create a blank privacy-related image; set its color to the privacy background color and its size to be consistent with the original image size;

[0097] a3) sorting the horizontal sides and vertical sides of the N input rectangles identified as privacy-related areas on the 2D plane according to the coordinate values ​​(of the area in the original image);

[0098] (Please refer to the attached Figure 3 and attached Figure 4 ,) After the sorting is completed, the horizontal edge of the 2D plane is divided into horizontal scale positions X by N rectangles marked as privacy-related areas. 1 , X 2 , …, X m , the vertical edge is divided into vertical scale positions Y 1 , Y 2 , …, Y n ; Here, m and n may not be equal, because the horizontal sides or vertical sides of the privacy-related areas selected by multiple rectangular boundary boxes are adjacent and / or overlap each other; if the horizontal sides and vertical sides of any rectangular area (the privacy-related area selected by the rectangular boundary box) do not overlap each other, then in terms of value, m and n are equal (equal to 2N).

[0099] a4) Clear the ASS list; the ASS stands for Active Scan Segment;

[0100] a5) Clear the partition list G k ,

[0101] Where k is G k Subscript variable of the list, K is G k The length of the list; initialize k and K so that k = K = 0;

[0102] k is the loop variable, K is the upper limit (maximum value) of the loop variable k, that is, the privacy partition area G k The number of .

[0103] In a specific implementation of this embodiment, the step B) scanning the 2D plane with the calibration lines according to their corresponding predetermined order includes:

[0104] b1) From the horizontal line H to the bottom, follow the Y 1 , Y 2 , …, Y n Scan the 2D plane in sequence;

[0105] When scanning to Y i When , clear the LR counter and clear the HSS list; where 1≤i≤n;

[0106] The HSS stands for Horizontal Scan Segment;

[0107] b2) Follow the vertical line V from left to right along X 1 , X 2 , …, X m Scan the 2D plane of the original image in sequence;

[0108] When scanning to the left edge of any rectangle, the LR counter is incremented by 1, and the position where the LR counter changes from 0 to 1 is recorded as L;

[0109] When the right edge of any rectangle is scanned, the LR counter is reduced by 1, and the position where the LR counter changes from a positive number to 0 is recorded as R;

[0110] Record the line segment of the horizontal line H between L and R as a scan line segment SS, and add SS to the HSS list; the SS stands for scan line segment (Scan Segment);

[0111] When the vertical line V scan is finished, all the scan line segments SS in the ASS list are scanned vertically to the horizontal line H position, and the effective privacy area scanned is output as G k ;

[0112] If the current output G k Compared with the previously output G k’ If the vertical positions are adjacent and the horizontal sizes are the same, G k Merge to G k’ ; If the above adjacent relationship does not exist, the values ​​of k and K are added by 1 respectively;

[0113] b3) Update the ASS list;

[0114] Delete the scan line segment SS in the ASS list that is above the vertical position of the horizontal line H and is no longer associated with any valid rectangular area; that is, if the ordinate value of the scan line segment SS is less than Y i , and SS is outside all rectangular areas intersecting with the current horizontal line H, then SS is deleted from the ASS list;

[0115] Delete the SS that has completed scanning in step b2) from the ASS list;

[0116] Merge the HSS list into the ASS list;

[0117] The horizontal line H continues to scan downward and steps b2) to b3) are repeated until the 2D scan is completed.

[0118] In a specific implementation of this embodiment, the step C) clearing and copying the image, and outputting the privacy-removed image and the privacy-related image includes:

[0119] c1) Execute the partition list G obtained by 2D scanning in the order of subscript k from K-1 to 0 k The K partitions contained in ;

[0120] c2) G k Copying from the original image to the newly created blank privacy-related image (block copy), the relative position of which remains the same as that in the original image;

[0121] At the same time, the G k The pixels in the left column of the left border and the pixels in the upper row of the upper border are copied to the corresponding positions of the newly created privacy-related image (pixel copy);

[0122] If the original image G k If the left column of pixels of the left boundary and / or the upper row of pixels of the upper boundary are beyond the boundary of the original image, the pixel copy operation is not performed;

[0123] c3) G k Clear from the original image and set the color to the privacy foreground color (block clear);

[0124] At the same time, the G k The color of the left column of pixels on the left side of the left border and the upper row of pixels on the upper side of the upper border are set as the privacy foreground color (pixel clearing);

[0125] It can be understood that if G in the original image k If the left column of pixels of the left boundary and / or the upper row of pixels of the upper boundary exceed the boundary of the original image, the pixel clearing operation is not performed;

[0126] It should also be pointed out that the data in the original image that disappears due to the pixel clearing operation has been saved in the privacy-related image (generated by copying the content of the newly created privacy-related image) in the aforementioned pixel copying operation; when the application needs it, the data that disappears due to the aforementioned pixel clearing operation can be restored at any time.

[0127] In another specific implementation of this embodiment, the scanning order is different from the above implementation, that is, the scanning order can also be adopted to first scan from left to right along the vertical line V according to X 1 , X 2 , …, X m The 2D plane of the original image is scanned in the order of Y and then the horizontal line H is scanned from top to bottom in the order of Y 1 , Y 2 , …, Y n The 2D plane of the original image is scanned in sequence.

[0128] In the present invention, block copying and pixel copying are performed in sequence according to the index variable k from K-1 to 0, which can ensure that G does not appear inside the privacy-related image. k Pixels outside the borders;

[0129] c4) outputting the result image;

[0130] Outputting the original image after the above clearing operations (block clearing and pixel clearing) as a privacy-removed image;

[0131] The result of completing the content copy operation (block copy and pixel copy) on the newly created privacy-related image is output as the privacy-related image.

[0132] Please see attached Figure 1 and attached Figure 2 The present invention also provides an image privacy protection system using the above-mentioned image privacy protection method, comprising:

[0133] A privacy detection module 1, wherein the privacy detection module 1 is configured to detect whether the input original image contains privacy-related content, and to select the privacy content detected in the original image containing the privacy-related content and mark it as a privacy-related area;

[0134] A privacy segmentation module 2, the privacy segmentation module 2 is electrically connected to the privacy detection module 1; the privacy segmentation module 2 segments the input original image according to the identified privacy-related area and the original image to form a privacy-free image and a privacy-related image;

[0135] A first encoder 3, the first encoder 3 is electrically connected to the privacy segmentation module 2; the first encoder 3 is configured to encode and output the privacy-removed image;

[0136] A second encoder 4, the second encoder 4 is electrically connected to the privacy segmentation module 2; the second encoder 4 is configured to encode the privacy-related image;

[0137] The encryption module 5 is electrically connected to the second encoder 4; the encryption module 5 encrypts the privacy-related image encoded by the second encoder 4 and outputs it.

[0138] In a specific implementation of this embodiment, the second encoder 4 encodes the privacy-related image and sends ( / transmits) it to the encryption module 5 for encryption; in this implementation, the second encoder 4 may not have an internal encryption function, or it may have an internal encryption function, but the encryption function is not enabled.

[0139] In another specific implementation of this embodiment, the second encoder 4 has an internal encryption function; the second encoder 4 encodes and preliminarily encrypts the privacy-related image by itself, and then sends ( / transmits) it to the encryption module 5 for encryption operation.

[0140] In a specific implementation of this embodiment, the data (code stream) encoded by the first encoder 3 and the second encoder 4 is sent to a subsequent subsystem;

[0141] The subsystem includes: one or more of a data storage device (for storage), a data exchange station (for data transmission interaction), and a data display device (for data display).

[0142] In a specific implementation of this embodiment, the first encoder 3 encodes the privacy-removing image in parallel with the second encoder 4 encodes the privacy-related image (without limiting whether they are strictly synchronized), and controls multi-threaded operation.

[0143] In another specific implementation manner of the present application, the first encoder 3 encodes the privacy-removing image and the second encoder 4 encodes the privacy-related image in series, and controls the single-thread operation;

[0144] This implementation includes: firstly running the first encoder 3 to encode the privacy-removing image, and then running the second encoder 4 to encode the privacy-related image;

[0145] The method further includes: firstly running the second encoder 4 to encode the privacy-related image, and then running the first encoder 3 to encode the privacy-removed image.

[0146] In a specific implementation of this embodiment, the privacy detection module 1 includes a YOLO target detection algorithm based on a CNN convolutional neural network, performs privacy detection on the original image, and outputs a privacy area defined by N rectangles (box selections);

[0147] In a specific implementation of this embodiment, if the original image does not contain privacy information, the N value output by the privacy detection module 1 is 0, and its subsequent processing includes outputting the original image as a privacy-removed image and outputting a blank image as its privacy-related image.

[0148] In a specific implementation of this embodiment, the privacy detection module 1 adopts the YOLO series target detection algorithm;

[0149] Preferably, YOLOv8 (or higher) is used.

[0150] The YOLO series of target detection algorithms use a CNN convolutional neural network model, which consists of multiple convolutional layers and pooling layers that can extract image features. By performing convolution and pooling operations on the image, the YOLO model can gradually extract image features at different levels, from low-level edge and texture features to high-level semantic features. After convolution and pooling operations, the fully connected layer predicts the coordinates, size, and category probability of the bounding box.

[0151] The YOLOv8 model inherits the real-time detection features of the YOLO series and optimizes and improves the model structure. YOLOv8 uses a more efficient backbone network and improves feature extraction capabilities by introducing new convolutional layers and connection methods; YOLOv8 uses a new detection head design, making the model more accurate and reliable when predicting target locations and categories; YOLOv8 improves the loss function and optimizes the training strategy, further improving the performance of the model.

[0152] In a specific implementation of this embodiment, the privacy detection module 1 further includes a CNN convolutional neural network model import function, which supports downloading and importing a pre-trained or third-party provided YOLOv8 (or higher version) CNN convolutional neural network model, which is used to initialize or dynamically upgrade the privacy detection module 1;

[0153] For example, the YOLOv8 (or higher) CNN convolutional neural network model uses the ONNX (Open Neural Network Exchange) format; wherein ONNX is an open machine learning model file format for storing trained models. ONNX allows different deep learning frameworks (such as PyTorch, TensorFlow, PaddlePaddle) to store model data in the same format, making it easy to migrate models between various mainstream deep learning frameworks. ONNX not only stores the weight coefficients of the neural network model, but also stores the structural information of the model as well as the input and output of each layer in the network and other auxiliary information, with the advantages of versatility and portability.

[0154] The privacy detection module 1 adopts the CNN-based YOLOv8 (or higher version) target detection algorithm. Under the current technical level, conventional hardware platforms can achieve real-time privacy detection at 30fps or above. It can provide differentiated (specific adaptation scenarios) pre-trained privacy detection models for different application scenarios to achieve the most effective privacy detection. When the algorithm or pre-trained model is upgraded, the privacy detection function can be upgraded accordingly.

[0155] It is understandable that the above implementation of this embodiment is an implementable solution, but it does not exclude the use of other publicly available privacy detection algorithms to implement the functions of the privacy detection module 1.

[0156] In a specific implementation of this embodiment, based on the N rectangular areas output by the privacy detection module 1, the privacy segmentation module 2 segments the original image and outputs a privacy-free image and a privacy-related image;

[0157] The privacy-removed image refers to the remaining portion of the original image after the privacy area is removed; the privacy-related image is an image composed of N rectangular privacy areas detected in the original image and the boundary pixels copied to the newly created privacy-related image.

[0158] It can be understood that after the privacy-removing image and the privacy-related image are decoded and then superimposed, the original image can be completely restored, and the processing efficiency is high.

[0159] The N rectangular areas (marked as privacy-related areas) output by the privacy detection module may overlap with each other. If the N rectangular privacy areas are simply cleared one by one in the original image (or video), repeated clearing operations will result. Similarly, if the N rectangular privacy areas are simply copied one by one to the privacy-related image, repeated copying operations will result. The privacy area segmentation algorithm based on 2D scanning provided in the present application can complete the clearing of the privacy area in the original image and the reconstruction of the privacy-related image in one 2D scan, which can effectively avoid the above-mentioned repeated operations.

[0160] In a specific implementation of this embodiment, the first encoder 3 and the second encoder 4 may be a JPEG encoder, and / or an H.264 AVC (Advanced Video Coding) video encoder, and / or an HEVC (High Efficiency Video Coding) video encoder, and / or a VVC (Versatile Video Coding) video encoder;

[0161] The first encoder 3 is a standard encoder, which inputs a deprivacy image and outputs deprivacy image data after encoding, such as a JPEG image, or an H.264 bitstream, or an HEVC bitstream, or a VVC bitstream;

[0162] The second encoder 4 is a standard encoder, which inputs a privacy-related image and outputs encoded privacy-related image data, such as a JPEG image, or an H.264 code stream, or an HEVC code stream, or a VVC code stream.

[0163] In a specific implementation of this embodiment, the second encoder 4 supports encryption of DCT transformation data, and / or motion vector data, and / or encoding result data within the encoder.

[0164] According to actual application needs, the second encoder 4 can start or stop its own encryption function to start or stop the encryption process of the privacy-related image data;

[0165] In a specific implementation of this embodiment, no matter whether the second encoder 4 encrypts the privacy-related data, the encoded privacy-related image data is sent ( / transferred) to the encryption module 5 for encryption.

[0166] Image (or video) encoders belong to the prior art, but setting the best working mode and working parameters for the encoder for a specific area belongs to one aspect of the present invention. Since the privacy-removed image includes a large range of privacy foreground color rectangular areas, namely the privacy segmentation area G described in the present invention, k (1≤k≤K); the H.264 macroblocks (MB) or HEVC / VVC coding tree units (CTU) in these areas can be effectively optimized during encoding.

[0167] Complexity and K of privacy region segmentation algorithm based on 2D scanning 2 Proportional (K is the privacy partition area G k The number of pixels is independent of the scale (size and resolution) of the original image (or video); it is understandable that the K value is usually small (for example, less than 10), while the original image / video resolution is usually very high (for example, 1920x1080 or even higher).

[0168] In a specific implementation of this embodiment, when the second encoder 4 does not support internal encryption of the encoder, the encryption module 5 encrypts the data output by the second encoder and outputs the privacy-related image data;

[0169] For example, the encryption module supports and adopts one or more existing encryption algorithms such as the national encryption SM4 algorithm, the national encryption ZUC Zu Chongzhi algorithm, and the AES encryption algorithm.

[0170] Please refer to the attached Figure 4 In a specific implementation of this embodiment, the operation process of the privacy area segmentation algorithm based on 2D scanning is as follows:

[0171] Horizontal line H from Y 1 Scan to Y 2 When the privacy partition area G is output 1 = 1 ,B 1 ,C 1 ​,D 1 >

[0172] Then, the horizontal line H is scanned to Y 3 When the privacy partition area G is output 2 = 2 ,B 2 ,C 2 ,D 2 >, G 2 In Y 1 and Y 3 Among them, A 2 , B 2 With A 1 , B 1 At the same level;

[0173] Then, the horizontal line H is scanned to Y 4 When the privacy partition area G is output 3 = 3 ,B 3 ,C 3 ,D 3 >, G 3 In Y 2 and Y 4 between;

[0174] Then, the horizontal line H is scanned to Y 5 When the privacy partition area G is output 4 = 4 ,B 4 ,C 4 ,D 4 >, where A 4 With D 3 Position overlap;

[0175] Then, the horizontal line H is scanned to Y 6 When the privacy partition area G is output 5 = 5 ,B 5 ,C 5 ,D 5 >, where B 5 With C 4 Position overlap;

[0176] Then, the horizontal line H is scanned to Y 7 When the privacy partition area G is output 6 = 6 ,B 6 ,C 6 ,D 6 >

[0177] This completes the 2D scan.

[0178] ​​​​​The privacy-removed image is obtained by the aforementioned privacy region segmentation algorithm based on 2D scanning. Since the privacy-removed image includes a large range of privacy foreground color rectangular areas, namely the privacy segmentation area G in the present invention, k (1≤k≤K); the pixel values ​​in these areas are privacy foreground colors (0 value or other fixed grayscale values), so the H.264 macroblocks or HEVC / VCC coding tree units in these areas can be optimized during prediction, encoding and filtering. Specifically, this embodiment sets the following working modes and working parameters for the encoder 1:

[0179] Setting 1: For JPEG image encoding, a 16x16 macroblock is preferred; for H.264 video encoding, a 16x16 macroblock is preferred; for HEVC video encoding, a 64x64 coding tree unit is preferred; for VVC video encoding, a 128x128 coding tree unit is preferred.

[0180] The reason for setting 1 is that the coding area is monochrome (preferably 0 value or other fixed grayscale value), the picture is simple, and the largest possible coding unit can improve the coding efficiency;

[0181] Setting 2: For H.264 macroblocks or HEVC / VVC coding tree units within the privacy partition, intra-frame prediction technology is preferably used; for H.264 video encoding, mode 0 (vertical prediction) or mode 1 (horizontal prediction) prediction mode is preferred; for HEVC video encoding, mode 10 (horizontal prediction) or mode 26 (vertical prediction) prediction mode is preferred; for VVC video encoding, mode 18 (horizontal prediction) or mode 50 (vertical prediction) prediction mode is preferred.

[0182] The reason for using setting 2 is that, on the one hand, inter-frame prediction can be avoided for encoding in the privacy partition, and intra-frame prediction is simpler to calculate; on the other hand, horizontal or vertical prediction is simpler to calculate than other prediction modes. The aforementioned pixel clearing operation has prepared the boundaries required for horizontal or vertical prediction for the deprivacy image, that is, the left column of pixels and the upper row of pixels in the encoding area have been set as the privacy foreground color, which can be directly used in the preferred intra-frame prediction mode of setting 2;

[0183] Setting 3: Cancel loop filtering operation: including DBF (deblocking filter) deblocking filter, SAO (Sample Adaptive Offset) sample adaptive offset filter, and ALF (Adaptive Loop Filter) adaptive loop filter.

[0184] The reason for using setting 3 is that the coded area is set to a privacy foreground color (monochrome, fixed grayscale value) and no filtering operation is required.

[0185] Setting 4: The DCT transform only calculates the first value (i.e., the DC coefficient) and sets all other DCT transform results directly to 0.

[0186] The reason for setting 4 is that the coding area is set to a privacy foreground color (monochrome, fixed grayscale value). For JPEG image coding or H.264 video coding, after DCT transformation, theoretically only the DC coefficient is non-zero, and other DCT transformation results are 0. The transformation operation involved in HEVC / VCC video coding is similar to the DCT transformation.

[0187] This embodiment further provides a camera system to which the above-mentioned image privacy protection system is applied, and the data processing steps of the camera system include:

[0188] 1. Obtain YUV or RGB raw images from the camera sensor in real time;

[0189] 2. Input the YUV or RGB original image into the privacy detection module for analysis and processing, and output the privacy-related areas defined by N rectangles (boundary box selection);

[0190] This step usually has a delay of tens of milliseconds;

[0191] 3. Input ( / send / transfer) N rectangular areas (privacy-related areas) to the privacy segmentation module for processing, and output the privacy-free image and the privacy-related image;

[0192] 4. The first encoder encodes the privacy-removing image; and outputs the video code stream generated by encoding the privacy-removing image to other subsystems (such as a display subsystem, a storage subsystem, or a network transmission subsystem);

[0193] 5. When the image (or video) encoder supports multi-process / multi-threading, start the second encoder in parallel with step 4 to encode the privacy-related image; when the image encoder does not support multi-process / multi-threading, after completing the encoding of the privacy-removing image in step 4, start the encoding of the privacy-related image; after the encoding is completed, output the encoding result to the encryption module for encryption, and finally output the encrypted video code stream to other subsystems (such as a display subsystem, a storage subsystem, or a network transmission subsystem).

[0194] This camera system provides a complete privacy protection technology solution; it supports real-time privacy protection. The delay is mainly caused by running the YOLO series target detection algorithm, which is usually tens of milliseconds. It has high real-time performance and high processing efficiency in each link.

[0195] This embodiment also provides a traditional network camera, to which the above-mentioned image privacy protection system is applied, and the data processing steps thereof include:

[0196] 1. Obtain video stream data from a traditional network camera, such as H.264 format stream data;

[0197] 2. Use the standard video decoding module to decode the acquired H.264 code stream data to obtain the original image corresponding to the H.264 video frame;

[0198] 3. Perform privacy processing on the original image obtained in step 2, which is completed by the privacy detection module, privacy segmentation module, video encoder, and encryption module.

[0199] It is understandable that in some application scenarios, if it is not necessary to encrypt the privacy-related image, the encryption step can be omitted.

[0200] This embodiment provides a technical solution for privacy protection transformation of traditional monitoring systems; it supports real-time or quasi-real-time privacy protection but the delay will be slightly larger. In addition to the delay caused by acquiring / buffering / decoding the original video stream, it also includes the delay caused by running the YOLO series algorithm for privacy detection, and the delay caused by the video encoder re-encoding, which is usually tens to hundreds of milliseconds.

[0201] The above description is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this patent can make some changes or modifications to equivalent embodiments of equivalent changes by using the technical content suggested above without departing from the scope of the technical solution of the present invention. The implementation scheme in the above embodiment can also be further combined or replaced. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of the solution of the present invention.

Claims

1. A privacy protection method for an image, characterized in that: include: S1. Perform privacy detection on the input original image; If it is detected that the input original image contains privacy-related content, the detected privacy-related content is framed by a rectangular boundary and marked as a privacy-related area; S2. A privacy region segmentation algorithm based on 2D scanning is used to segment the original image according to the identified privacy-related regions to form a privacy-related image and a privacy-removed image; S3, encoding, encrypting and outputting the privacy-related image; Encode the deprived image and output it.

2. The image privacy protection method according to claim 1, characterized in that: The step S1 further comprises: If it is detected that the input original image does not contain any privacy-related content, the original image is encoded and output.

3. The method for protecting image privacy according to claim 1 or 2, characterized in that: For the original image, the number of areas marked as privacy-related areas is N; Wherein, N≥0, N is an integer.

4. The image privacy protection method according to claim 3, characterized in that: The privacy region segmentation algorithm of the 2D scan comprises the steps of: A) Preset processing of data; B) scanning the 2D plane with the calibration lines in a corresponding predetermined order, and outputting the effective privacy area scanned; C) Clear and copy the privacy-related area, and output the privacy-free image and the privacy-related image.

5. The method for protecting image privacy according to claim 4, characterized in that: The step A) of pre-setting the data includes: a1) The default privacy background color is black, and the default privacy foreground color is a fixed gray value color; a2) Create a blank privacy-related image; set its color to the privacy background color and its size to be consistent with the original image size; a3) sorting the horizontal sides and vertical sides of the N input rectangles identified as privacy-related areas on the 2D plane according to the coordinate values; After the sorting is completed, the horizontal edge of the 2D plane is divided into horizontal scale positions X1, X2, ..., X m , the vertical edge is divided into vertical scale positions Y1, Y2, ..., Y n ; a4) Clear the ASS list; a5) Clear the partition list G k , where k is G k Subscript variable of the list, K is G k The length of the list; initialize k and K so that k=K=0.

6. The image privacy protection method according to claim 5, characterized in that: The step B) scanning the 2D plane with the calibration lines according to their corresponding predetermined order comprises: b1) Take the horizontal line H from top to bottom according to Y1, Y2, ..., Y n Scan the 2D plane in sequence; When scanning to Y i When , clear the LR counter and clear the HSS list; where 1≤i≤n; b2) From left to right along the vertical line V, follow X1, X2, ..., X m Scan the 2D plane in sequence; When scanning to the left edge of any rectangle, the LR counter is incremented by 1, and the position where the LR counter changes from 0 to 1 is recorded as L; When the right edge of any rectangle is scanned, the LR counter is reduced by 1, and the position where the LR counter changes from a positive number to 0 is recorded as R; Record the line segment of the horizontal line H between L and R as the scanning line segment SS, and add SS to the HSS list; When the vertical line V scan is finished, all the scan line segments SS in the ASS list are scanned vertically to the horizontal line H position, and the effective privacy area scanned is output as G k ; If the current output G k Compared with the previously output G k’ , vertical positions are adjacent and horizontal sizes are the same, then G k Merge to G k’ ; If the above adjacent relationship does not exist, the values ​​of k and K are added by 1 respectively; b3) Update the ASS list; Delete the scan line segment SS in the ASS list that is above the vertical position of the horizontal line H and is no longer associated with any valid rectangular area; that is, if the ordinate value of the scan line segment SS is less than Y i , and SS is outside all rectangular areas intersecting with the current horizontal line H, then SS is deleted from the ASS list; Delete the SS that has completed scanning in step b2) from the ASS list; Merge the HSS list into the ASS list; The horizontal line H continues to scan downward and repeats b2 to b3 until the 2D scan is completed.

7. The image privacy protection method according to claim 6, characterized in that: The step C) clearing and copying the image, and outputting the privacy-free image and the privacy-related image, includes: c1) The subscript variable k is in the order of K-1 to 0, and the partition list G obtained by 2D scanning is executed accordingly. k The K partitions contained in ; c2) G k The original image is copied to the newly created privacy-related image, and its relative position remains corresponding to the original image; At the same time, the original image G k The pixels in the left column of the left border and the pixels in the upper row of the upper border are copied to the corresponding positions of the newly created privacy-related image; c3) G k Clear from the original image and set the color to the privacy foreground color; At the same time, the original image G k The color of the left column of pixels on the left side of the left border and the upper row of pixels on the upper side of the upper border are set as the privacy foreground color; c4) outputting the result image; Outputting the original image after the clearing operation as a privacy-removed image; The result of completing the content copy operation on the newly created privacy-related image is output as the privacy-related image.

8. An image privacy protection system, using the image privacy protection method according to any one of claims 1 to 7, characterized in that: The image privacy protection system includes: A privacy detection module (1), wherein the privacy detection module (1) is configured to detect whether the input original image contains privacy-related content, and to select the privacy content detected in the original image containing the privacy-related content and mark it as a privacy-related area; A privacy segmentation module (2), the privacy segmentation module (2) being electrically connected to the privacy detection module (1); the privacy segmentation module (2) segments the input original image according to the identified privacy-related area and the original image to form a privacy-removed image and a privacy-related image; A first encoder (3), the first encoder (3) being electrically connected to the privacy segmentation module (2); the first encoder (3) being configured to encode and output the privacy-removed image; A second encoder (4), the second encoder (4) being electrically connected to the privacy segmentation module (2); the second encoder (4) being configured to encode the privacy-related image; An encryption module (5), the encryption module (5) being electrically connected to the second encoder (4); the encryption module (5) encrypts and outputs the privacy-related image encoded by the second encoder (4).

9. The image privacy protection system according to claim 8, characterized in that: The privacy detection module (1) adopts the YOLO target detection algorithm based on the CNN convolutional neural network.

10. The image privacy protection system according to claim 8, characterized in that: The first encoder (3) encodes the privacy-removing image in parallel with the second encoder (4) encodes the privacy-related image. Alternatively, the first encoder (3) encodes the privacy-removing image and the second encoder (4) encodes the privacy-related image in series.

Images