A vehicle data desensitization method based on image block detection

Abstract

The present application provides a vehicle data desensitization method based on image block detection, which performs super-resolution enhancement on areas in the input image that may contain targets, and can effectively reduce the impact of the low resolution of the vehicle-mounted camera on the detection accuracy and improve the detection rate of small targets. Based on this method, license plates and faces with motion blur can also be reconstructed through super-resolution through a diffusion model to reduce the degree of blur, effectively improving the detection rate of sensitive targets, ensuring that this method is particularly suitable for desensitizing images collected by vehicle-mounted image acquisition equipment while driving.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a vehicle data desensitization method based on image block detection. Background Art

[0002] With the continuous advancement of network communication technology, the degree of networking of automobiles is also increasing. With the help of various network communication technologies, people have achieved more control over automobiles, such as remote monitoring of vehicles, remote parking and other functions. These functions of intelligent networked vehicles have brought new convenience and experience to people's lives, but cars will also collect more information, which may involve sensitive data such as faces and license plates outside the car. In order to protect personal privacy, the state has formulated relevant standards to regulate vehicle data collection. For videos or pictures collected by cars outside the car that contain recognizable faces and license plates, data desensitization processing is required during transmission to protect personal privacy information.

[0003] There are some technical solutions for desensitizing private data such as license plates in the prior art. For example, the patent document with publication number CN112347853A discloses a video-based license plate data desensitization method, which extracts the vehicle image of the corresponding frame from the video according to the vehicle position information, and obtains the license plate position information of the corresponding vehicle; determines whether the license plate is a falsely detected license plate by constructing a decision tree binary classifier, and if so, removes the position information of the falsely detected license plate, and records the vehicle as a license plate missed detection vehicle; if not, merges the vehicle position information and the license plate position information and stores them accordingly; uses Kalman filtering to estimate the license plate coordinates of the license plate missed detection vehicle and saves them; mosaics the license plate of the vehicle in the video according to the stored license plate position information and license plate coordinates to achieve license plate data desensitization.

[0004] However, in actual use, it was found that due to the low resolution of images collected by some vehicle-mounted cameras and the fact that most vehicles are driving at high speeds, the collected images will have a certain degree of motion blur, making the faces or license plates in the images smaller and more blurred, resulting in a low detection rate and poor desensitization effect. Summary of the invention

[0005] In order to solve the problem that the detection rate of smaller and blurrier targets in images in existing vehicle data desensitization methods is low, resulting in poor desensitization effect, the present application provides a vehicle data desensitization method based on image block detection, which can effectively reduce the impact of low resolution of on-board cameras on detection accuracy, improve the detection rate of small targets, and improve the impact of motion blur on detection results, thereby improving the desensitization effect of vehicle data.

[0006] The technical solution of the present application is as follows: a vehicle data desensitization method based on image block detection, characterized in that it comprises the following steps:

[0007] S1: Based on the image acquisition device on the vehicle, real-time images are collected, which are recorded as: images to be processed;

[0008] S2: Train and generate image feature extraction network model based on neural network;

[0009] The output feature size of the image feature extraction network model is consistent with the input image size;

[0010] S3: extracting features of the image to be processed based on the image feature extraction network model to obtain a feature map f to be processed;

[0011] The feature map to be processed is denoted as: f∈R H×W×C , where H is the height of f, W is the width of f, and C is the number of feature channels of f;

[0012] S4: Train and generate pixel depth estimation model based on deep learning algorithm;

[0013] The pixel depth estimation model outputs the relative distance between the original target object corresponding to each pixel in the feature map to be processed f and the image acquisition device, which is recorded as: pixel depth; the larger the pixel depth, the farther the original target object corresponding to the pixel is from the image acquisition device;

[0014] S5: Perform feature segmentation on the feature map f to be processed to obtain segmented patch blocks and form a patch set;

[0015] S6: Build a vehicle-person target classification model;

[0016] According to a preset classification target, the vehicle-person target classification model finds an image including the classification target in all input images and outputs the image;

[0017] The classification targets include: one or a combination of vehicles, human bodies, and human portraits;

[0018] S7: input all the patch blocks in the patch block set into the vehicle-person target classification model, record the patch blocks including the classification targets output by the vehicle-person target classification model as positive samples, and record the other patch blocks as negative samples;

[0019] S8: performing super-resolution enhancement on the positive sample, reconstructing the image resolution of the positive sample area, and obtaining a reconstructed positive sample;

[0020] Amplifying the negative sample based on an interpolation technique to obtain an amplified negative sample;

[0021] Rejoining the reconstructed positive sample and the amplified negative sample to obtain a reconstructed feature map to be processed;

[0022] S9: Build a sensitive target detection model based on deep learning algorithm;

[0023] The sensitive targets include: vehicle license plates or human faces;

[0024] The sensitive target detection model detects the input image and outputs the area and confidence level of the detected sensitive target;

[0025] S10: using the sensitive target detection model to detect the reconstructed positive sample, and obtaining: a positive sample detection result;

[0026] S11: using the sensitive target detection model to detect the reconstructed feature map to be processed, and obtaining: a detection result of the feature map to be processed;

[0027] S12: merging the positive sample detection result and the feature map detection result to be processed to obtain a sensitive target detection result corresponding to the image to be processed as an object for subsequent desensitization operation;

[0028] When two test results are combined, the following conditions apply:

[0029] Condition 1: All detection results with confidence levels lower than the threshold are deleted;

[0030] Condition 2: The detection results with high confidence are retained in the areas where the sensitive targets overlap.

[0031] It is further characterized by:

[0032] In step S2, the image feature extraction network model is constructed based on the Encoder network model;

[0033] In step S5, the specific patch block segmentation method includes the following steps:

[0034] a1: According to the preset first-segmentation ratio, the feature map f to be processed is evenly divided into n1*n1 patch blocks, which are recorded as: first-segmentation blocks;

[0035] a2: obtaining the corresponding pixel depth for each pixel on the feature map f to be processed based on the pixel depth estimation model;

[0036] a3: Calculate the average value of the pixel depths corresponding to all the pixels included in each of the first-segmented blocks, denoted as: average pixel depth ave_d;

[0037] a4: Compare each of the average pixel depth ave_d with a preset depth threshold d;

[0038] If there is any ave_d greater than d, the corresponding first-cut block is recorded as the long-distance target block, and step a5 is executed;

[0039] Otherwise, execute step a6;

[0040] a5: divide the long-distance target block evenly again according to the preset secondary division ratio to obtain n2*n2 patch blocks, which are recorded as secondary division blocks; execute step a6;

[0041] a6: All the first-cut blocks and the second-cut blocks obtained after segmentation constitute the patch block set, based on the regional feature r i Represents the i-th patch block;

[0042] ;

[0043] In the formula, i is the number of the patch block, n i Indicates the downsampling multiple of the i-th patch block compared to the feature map to be processed, C i is the number of channels of the feature of the i-th patch block; n i ∈(n1, n1*n2);

[0044] In step S5, n1 is 3 and n2 is 2;

[0045] In step S6, the loss function of the vehicle-person target classification model is implemented based on the cross entropy function;

[0046] In step S8, super-resolution enhancement is performed on the positive sample, which specifically includes the following steps:

[0047] A super-resolution enhancement model is constructed using a wavelet diffusion model WaveDM; the positive sample is processed based on the super-resolution enhancement model to reconstruct the image resolution of the positive sample area;

[0048] During the iteration process of the super-resolution enhancement model, the number of iterations of the super-resolution enhancement model is controlled based on the ResShift model;

[0049] In step S12, the nms algorithm is used to filter repeated and low-confidence detection results, and the final detection result is output.

[0050] The present application provides a vehicle data desensitization method based on image block detection, which performs super-resolution enhancement on the area in the input image that may contain the target, which can effectively reduce the impact of the low resolution of the vehicle camera on the detection accuracy and improve the detection rate of small targets; based on this method, the license plate and face with motion blur can also be reconstructed through the diffusion model to reduce the degree of blur, effectively improve the detection rate of sensitive targets, and ensure that this method is particularly suitable for desensitizing images collected by vehicle-mounted image acquisition equipment while driving. By performing target detection on the area after resolution enhancement and on the entire resolution-enhanced image, based on the mutual reference of the two detection results, the impact of the target size on desensitization is reduced, ensuring that this method is applicable to the image desensitization operation scenarios collected by vehicle-mounted image acquisition equipment of various resolutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flow chart of the vehicle data desensitization method based on image block detection;

[0052] Figure 2 An example of how to split a patch block. DETAILED DESCRIPTION

[0053] like Figure 1 As shown, the present application includes a vehicle data desensitization method based on image block detection, which includes the following steps.

[0054] S1: Based on the image acquisition device on the vehicle, real-time images are collected, which are recorded as: images to be processed; common vehicle-mounted image acquisition devices include: fisheye cameras.

[0055] S2: Train and generate image feature extraction network model based on neural network;

[0056] The feature size of the output image of the image feature extraction network model is consistent with the input image;

[0057] The image feature extraction network is implemented based on a neural network in the prior art. In this embodiment, an image feature extraction network model is constructed based on an Encoder network model.

[0058] S3: Based on the image feature extraction network, feature extraction is performed on the image to be processed to obtain the feature map f to be processed; the feature map to be processed is denoted as: f∈R H×W×C , where H is the height of f, W is the width of f, and C is the number of feature channels of f.

[0059] S4: Train and generate pixel depth estimation model based on deep learning algorithm;

[0060] The pixel depth estimation model outputs the relative distance between the original target object and the image acquisition device corresponding to each pixel in the feature map to be processed f, which is denoted as: pixel depth; the larger the pixel depth, the farther the original target object corresponding to the pixel is from the image acquisition device.

[0061] This application regards the small target in the image to be processed as the target in the image after being captured because the prototype target object is far away from the vehicle-mounted image acquisition device. Therefore, by judging the size of the target in the image to be processed, the distance between the prototype target object and the image acquisition device corresponding to the pixel point is obtained.

[0062] In the specific implementation, a pixel depth estimation model is constructed based on the deep learning algorithm models in the existing technology, such as RNN, DNN, GRU, CNN, etc., and then based on the historical image data collected by the vehicle-mounted image acquisition device, distance labels are added to the targets in the image data to form a training set and a validation set, and the pixel depth estimation model is trained to obtain a trained pixel depth estimation model.

[0063] S5: Perform feature segmentation on the feature map f to be processed, obtain segmented patch blocks, and form a patch set.

[0064] In step S5, the specific patch block segmentation method includes the following steps:

[0065] a1: According to the preset first-segmentation ratio, the feature map f to be processed is evenly divided into n1*n1 patch blocks, which are recorded as: first-segmentation blocks;

[0066] a2: Based on the pixel depth estimation model, each pixel on the feature map f to be processed is processed to obtain the corresponding pixel depth;

[0067] a3: Calculate the average pixel depth of all pixels in each first-cut block, denoted as: average pixel depth ave_d;

[0068] a4: Compare each average pixel depth ave_d with the preset depth threshold d;

[0069] If there is any ave_d greater than d, the corresponding first-cut block is recorded as the long-distance target block, and step a5 is executed;

[0070] Otherwise, execute step a6;

[0071] a5: Divide the distant target block evenly again according to the preset secondary division ratio to obtain n2*n2 patch blocks, which are recorded as secondary division blocks; execute step a6;

[0072] a6: All the first-cut blocks and second-cut blocks obtained after segmentation form a patch block set. Different areas are used to detect targets of different sizes. i The regional features of different sizes are represented as r i ;

[0073] ;

[0074] In the formula, i is the number of the patch block, n i Indicates the downsampling multiple of the i-th patch block compared to the feature map to be processed, C i is the number of channels of the feature of the i-th patch block; n i ∈(n1, n1*n2).

[0075] In this embodiment, n1 is 3 and n2 is 2; for the specific process, refer to Figure 2 After the first split, we get 3*3=9 first split blocks, marked as 1 to 9. i =3. Assume that the average pixel depth ave_d of the first-cut blocks numbered 5 and 8 is greater than d, and the two first-cut blocks are cut again to obtain 2*2=4 second-cut blocks respectively. The second-cut blocks in this embodiment correspond to n i is 6.

[0076] S6: Build a vehicle-person target classification model;

[0077] According to the preset classification target, the vehicle-person target classification model finds the image including the classification target in all input images and outputs it; the classification target includes: one or a combination of vehicles, human bodies, and portraits.

[0078] The vehicle-person target classification model can be constructed based on models such as neural networks, random forests, support vector machines, or k-nearest neighbor algorithms in the existing technologies. The loss function of the vehicle-person target classification model is implemented based on the cross entropy function, which is expressed as follows:

[0079] ;

[0080] In the formula, LP represents the classification loss function, n i is the number of samples, y j represents the true label of the jth sample, p j Represents the classification result predicted by the model for the jth sample.

[0081] The cross entropy function can handle multi-category classification problems well, calculate the loss for each category separately, and converge quickly, which can effectively avoid the problem of gradient disappearance or explosion.

[0082] S7: Input all the patch blocks in the patch block set into the vehicle-person target classification model, and record the patch blocks including the classification targets output by the vehicle-person target classification model as positive samples, and record the other patch blocks as negative samples.

[0083] The vehicle-human target classification model identifies whether the patch area includes sensitive targets such as vehicles and humans, rather than identifying sensitive targets, so the classification work can be completed well based on the patch block. Then, based on the classification results of the vehicle-human target classification model, all the patch blocks are divided into positive and negative samples.

[0084] S8: Perform super-resolution enhancement on the positive sample, reconstruct the image resolution of the positive sample area, and obtain the reconstructed positive sample;

[0085] The negative samples are enlarged based on the interpolation technology to obtain the enlarged negative samples;

[0086] The reconstructed positive samples and the enlarged negative samples are reconnected to obtain the reconstructed feature map to be processed.

[0087] The blurred part of the positive sample is used to restore the image details based on the super-resolution enhancement method. Then the negative sample is enlarged by a simple interpolation technique, and the enlarged negative sample area image is spliced ​​with the super-resolution reconstructed image to obtain the reconstructed whole image. In this way, the resolution of the corresponding sensitive target area in the reconstructed whole image can improve the recognition probability. Super-resolution reconstruction of the divided image based on the diffusion model can effectively reduce the impact of motion blur on desensitization and further improve the detection rate.

[0088] Among them, super-resolution refers to the process of using insufficient information in an existing image to restore image details or other information, usually referred to as super-resolution (SR). The specific super-resolution enhancement technology can be implemented based on any existing super-resolution enhancement technology solution. In this embodiment, super-resolution enhancement is performed on the positive sample, specifically including the following steps:

[0089] Super-resolution enhancement is performed on positive samples, specifically including the following steps:

[0090] A super-resolution enhancement model is constructed using the wavelet diffusion model WaveDM. The positive samples are processed based on the super-resolution enhancement model to reconstruct the image resolution of the positive sample area.

[0091] In the wavelet diffusion model WaveDM, the correspondence between high-resolution and low-resolution input images is directly learned, sampling is performed from Gaussian noise to obtain the low-frequency part of the high-quality image, and HFRM (High-Frequency Refinement Module) is used to predict the high-frequency part of the high-quality image. These two parts are then combined and transformed into a spatial domain RGB image through an inverse wavelet transform.

[0092] During the iteration process of the super-resolution enhancement model built based on WaveDM, the number of iterations of the super-resolution enhancement model is controlled based on the ResShift model. The iteration part is as follows:

[0093] ;

[0094] In the formula, t represents the tth sampling in the process of sampling step from n to 0, where n is the total number of samples in prediction; x t represents the prediction obtained by the current t-th sampling; f φ is the conditional diffusion model, α t β t is a hyperparameter, z is the split patch block; ε t The sampling noise follows a normal distribution;

[0095] , .

[0096] The ResShift model can effectively reduce the number of iterations in the diffusion process, so that the diffusion model only requires a few sampling steps n, greatly shortening the generation time.

[0097] S9: Build a sensitive target detection model based on deep learning algorithm;

[0098] Sensitive targets include: vehicle license plates or human faces;

[0099] The sensitive target detection model detects the input image and outputs the area and confidence level of the detected sensitive target.

[0100] S10: Use the sensitive target detection model to detect the reconstructed positive sample and obtain the positive sample detection result.

[0101] S11: Use a sensitive target detection model to detect the reconstructed feature map to be processed, and obtain: detection result of the feature map to be processed.

[0102] In this application, a sensitive target detection is performed through the positive sample, and another target detection is performed through the reconstructed whole image, and the two detection effects are combined to ensure that the influence of the target size on the desensitized target detection is reduced.

[0103] S12: Merge the positive sample detection result and the feature map detection result to be processed to obtain the sensitive target detection result corresponding to the image to be processed, which is used as the object of subsequent desensitization operation; the specific desensitization technology can be implemented based on the face and license plate number desensitization technology in the existing technology.

[0104] When two test results are combined, the following conditions apply:

[0105] Condition 1: All detection results with confidence levels lower than the threshold are deleted;

[0106] Condition 2: The detection results with high confidence are retained in the areas where the sensitive targets overlap.

[0107] After using the technical solution of the present application, the detection rate of small targets can be greatly improved through the design of image block detection; super-resolution enhancement of the area containing the target in the input image can effectively reduce the impact of the low resolution of the vehicle camera on the detection accuracy and improve the detection rate of small targets. For license plates and faces with motion blur, super-resolution reconstruction can also be performed through the diffusion model to reduce the degree of blur and improve the detection rate. At the same time, image block detection is combined with whole-image detection to reduce the impact of target size on desensitization.

Claims

1. A vehicle data desensitization method based on image block detection, characterized in that: It includes the following steps: S1: Based on the image acquisition device on the vehicle, real-time images are collected, which are recorded as: images to be processed; S2: Train and generate image feature extraction network model based on neural network; The output feature size of the image feature extraction network model is consistent with the input image size; S3: extracting features of the image to be processed based on the image feature extraction network model to obtain a feature map f to be processed; The feature map to be processed is denoted as: f∈R H×W×C , where H is the height of f, W is the width of f, and C is the number of feature channels of f; S4: Train and generate pixel depth estimation model based on deep learning algorithm; The pixel depth estimation model outputs the relative distance between the original target object corresponding to each pixel in the feature map to be processed f and the image acquisition device, which is recorded as: pixel depth; the larger the pixel depth, the farther the original target object corresponding to the pixel is from the image acquisition device; S5: Perform feature segmentation on the feature map f to be processed to obtain segmented patch blocks and form a patch set; S6: Build a vehicle-person target classification model; According to a preset classification target, the vehicle-person target classification model finds an image including the classification target in all input images and outputs the image; The classification targets include: one or a combination of vehicles, human bodies, and human portraits; The vehicle-person target classification model identifies whether the patch area includes the above-mentioned classified targets, rather than identifying sensitive targets; S7: input all the patch blocks in the patch block set into the vehicle-person target classification model, record the patch blocks including the classification targets output by the vehicle-person target classification model as positive samples, and record the other patch blocks as negative samples; S8: performing super-resolution enhancement on the positive sample, reconstructing the image resolution of the positive sample area, and obtaining a reconstructed positive sample; Amplifying the negative sample based on an interpolation technique to obtain an amplified negative sample; Rejoining the reconstructed positive sample and the amplified negative sample to obtain a reconstructed feature map to be processed; S9: Build a sensitive target detection model based on deep learning algorithm; The sensitive targets include: vehicle license plates or human faces; The sensitive target detection model detects the input image and outputs the area and confidence level of the detected sensitive target; S10: using the sensitive target detection model to detect the reconstructed positive sample, and obtaining: a positive sample detection result; S11: using the sensitive target detection model to detect the reconstructed feature map to be processed, and obtaining: a detection result of the feature map to be processed; S12: merging the positive sample detection result and the feature map detection result to be processed to obtain a sensitive target detection result corresponding to the image to be processed as an object for subsequent desensitization operation; When two test results are combined, the following conditions apply: Condition 1: All detection results with confidence levels lower than the threshold are deleted; Condition 2: The detection results with high confidence are retained in the areas where the sensitive targets overlap; In step S5, the specific patch block segmentation method includes the following steps: a1: According to the preset first-segmentation ratio, the feature map f to be processed is evenly divided into n1*n1 patch blocks, which are recorded as: first-segmentation blocks; a2: processing each pixel on the feature map f to be processed based on the pixel depth estimation model to obtain the corresponding pixel depth; a3: Calculate the average value of the pixel depths corresponding to all the pixels included in each of the first-segmented blocks, denoted as: average pixel depth ave_d; a4: Compare each of the average pixel depth ave_d with a preset depth threshold d; If there is any ave_d greater than d, the corresponding first-cut block is recorded as the long-distance target block, and step a5 is executed; Otherwise, execute step a6; a5: divide the long-distance target block evenly again according to the preset secondary division ratio to obtain n2*n2 patch blocks, which are recorded as secondary division blocks; execute step a6; a6: All the first-cut blocks and the second-cut blocks obtained after segmentation constitute the patch block set, based on the regional feature r i Represents the i-th patch block; ； In the formula, i is the number of the patch block, n i Indicates the downsampling multiple of the i-th patch block compared to the feature map to be processed, C i is the number of channels of the feature of the i-th patch block; n i ∈(n1, n1*n2).

2. The vehicle data desensitization method based on image block detection according to claim 1 is characterized in that: In step S2, the image feature extraction network model is constructed based on the Encoder network model.

3. The vehicle data desensitization method based on image block detection according to claim 1 is characterized in that: In step S5, n1 is 3 and n2 is 2.

4. The vehicle data desensitization method based on image block detection according to claim 1 is characterized in that: In step S6, the loss function of the vehicle-person target classification model is implemented based on the cross entropy function.

5. The vehicle data desensitization method based on image block detection according to claim 1 is characterized in that: In step S8, super-resolution enhancement is performed on the positive sample, which specifically includes the following steps: A super-resolution enhancement model is constructed using a wavelet diffusion model WaveDM; the positive sample is processed based on the super-resolution enhancement model to reconstruct the image resolution of the positive sample area.

6. The vehicle data desensitization method based on image block detection according to claim 5 is characterized by: During the iterative process of the super-resolution enhancement model, the number of iterations of the super-resolution enhancement model is controlled based on the ResShift model.

7. The vehicle data desensitization method based on image block detection according to claim 1 is characterized in that: In step S12, the nms algorithm is used to filter repeated and low-confidence detection results, and the final detection result is output.

Images