Privacy protection method, device and system for in-vehicle camera, computer equipment and medium

Abstract

The invention discloses a privacy protection method, device and system for an in-vehicle camera, computer equipment and a medium, and relates to the technical field of automobile intellectualization, and the method comprises the steps: obtaining an image of a person in a vehicle, and carrying out the style conversion processing of the image of the person in the vehicle, so as to reflect the identity information of each person in the image in a fuzzification manner; and extracting features reflecting driver behaviors from the image after style conversion, and detecting the driving state of the driver from the features reflecting the driver behaviors. According to the invention, the privacy problem of data acquisition of a vehicle-mounted camera in a driver monitoring system can be solved.

Description

Technical Field

[0001] The present application relates to the field of automobile intelligent technology, and specifically to a privacy protection method, device, system, computer equipment and medium for an in-vehicle camera in an intelligent networked car. Background Art

[0002] With the widespread use of in-car cameras, the Driver Monitoring System (DMS) has played an important role in improving driving safety. The main functions of in-car cameras include: driver fatigue detection, driver attention monitoring, identity recognition, behavior analysis and gesture control. These functions mainly extract eye closure status, blinking frequency, head posture, line of sight direction, gestures and behaviors from driver images or videos for intelligent analysis. The collected image or video information can be analyzed locally or transmitted to the cloud for computing, depending on the application requirements involved in the system.

[0003] However, the continuous monitoring of the camera also brings the risk of privacy leakage, mainly including: the in-car camera will continuously collect image and video data, which may contain sensitive personal information. The in-car camera may continuously monitor the user, including the behavior and activities of the driver and passengers. Therefore, how to protect the privacy of the people in the car while ensuring the performance of the driver monitoring system has become an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present application provide a privacy protection method, apparatus, system, computer equipment and medium for an in-vehicle camera, which are intended to solve the privacy problem of data collected by the in-vehicle camera in a driver monitoring system.

[0005] In a first aspect, the present application provides a privacy protection method for an in-vehicle camera, comprising:

[0006] Acquire an image of a person in a vehicle, and perform style conversion processing on the image of the person in the vehicle to blur information reflecting the identity of each person in the image;

[0007] Features reflecting the driver's behavior are extracted from the style-converted image, and the driver's driving state is detected from the features reflecting the driver's behavior.

[0008] In some examples, performing style conversion processing on the image of the person in the vehicle includes:

[0009] The image of the occupants is subjected to style conversion processing to blur the facial information of each person in the image and the behavior of the passengers, while retaining the facial features reflecting the driver's driving behavior.

[0010] In some examples, detecting the driving state of the driver from the features reflecting the driver's behavior includes:

[0011] Detecting the driver's driving state from the features reflecting the driver's behavior using a neural network model;

[0012] The method further comprises:

[0013] The style-converted images are used to train the neural network model.

[0014] In some examples, the using the style-converted image to train the neural network model includes:

[0015] Select driver facial images of different genders, ages, races, and driving scenarios as the training dataset;

[0016] Perform style conversion processing on each image in the training data set to blur the facial information of the person in the image and retain the facial features that reflect the driver's driving behavior

[0017] Annotating each image in the training data set to annotate attribute information of the person in the image and facial features reflecting the driver's behavior;

[0018] The neural network model is trained using the labeled training data set.

[0019] In some examples, the method further comprises:

[0020] Set the user's processing rights for the collected images, which include whether to agree to the collected images and the terms of use, as well as the processing of the collected images.

[0021] In some examples, the method further comprises:

[0022] The style-converted images and detected driving status are stored and transmitted, and the data is set not to be tampered with or leaked during the transmission process.

[0023] In a second aspect, the present application provides a privacy protection device for an in-vehicle camera, comprising:

[0024] An image acquisition module, used to acquire images of people in the vehicle;

[0025] A style conversion module, used to perform style conversion processing on the image of the occupants of the vehicle to blur the information reflecting the identity of each person in the image;

[0026] The analysis module is used to extract features reflecting the driver's behavior from the style-converted image, and detect the driver's driving state from the features reflecting the driver's behavior.

[0027] In a third aspect, the present application provides a driver monitoring system including the above-mentioned privacy protection device for an in-vehicle camera.

[0028] In a fourth aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the computer program.

[0029] In a fifth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of any of the above methods when executed by a processor.

[0030] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0031] (1) Through style transfer technology, the privacy of people in the car can be effectively protected and personal identity information can be prevented from being leaked.

[0032] (2) Ensure that the driver monitoring system can accurately extract key features and perform fatigue detection without affecting system performance.

[0033] (3) Real-time image processing and analysis are achieved to meet the high efficiency requirements of the in-vehicle camera system.

[0034] (4) Ensure the integrity and confidentiality of data during transmission through secure storage and transmission mechanisms. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 is a schematic diagram of a privacy protection method for an in-vehicle camera provided in an embodiment of the present application;

[0037] Figure 2 is a schematic diagram of a privacy protection device for an in-vehicle camera provided in an embodiment of the present application;

[0038] Figure 3 It is a schematic diagram of the structure of an embodiment of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0040] In the following description, the specific embodiments of the present application will be described with reference to the steps and symbols performed by one or more computers, unless otherwise specified. Therefore, these steps and operations will be mentioned several times as being performed by a computer, and the computer execution referred to herein includes the operation of a computer processing unit by an electronic signal representing data in a structured form. This operation converts the data or maintains it at a location in the memory system of the computer, which can be reconfigured or otherwise change the operation of the computer in a manner familiar to testers in the field. The data structure maintained by the data is a physical location in the memory, which has specific characteristics defined by the data format. However, the principles of the present application are described in the above text, which does not represent a limitation, and testers in the field will understand that the following various steps and operations can also be implemented in hardware.

[0041] The term "module" or "unit" used in this article can be regarded as a software object executed on the computing system. The different components, modules, engines and services in this article can be regarded as implementation objects on the computing system. The apparatus and method in this article are preferably implemented in software, but can also be implemented in hardware, all within the scope of protection of this application.

[0042] Those skilled in the art will appreciate that, unless expressly stated otherwise, the singular forms "a", "an", "" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other elements, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" as used herein includes all or any unit and all combinations of one or more associated listed items.

[0043] In a first aspect, the present application provides a privacy protection method for an in-vehicle camera, such as Figure 1 As shown, the following steps are included:

[0044] S101: Acquire an image of a person in the vehicle;

[0045] S102: performing style conversion processing on the image of the people in the car to blur the information reflecting the identity of each person in the image;

[0046] S103: extracting features reflecting the driver's behavior from the image after style conversion, and detecting the driver's driving state from the features of the driver's behavior.

[0047] In another specific example, in step S101, images of the driver and passengers may be collected in real time by an in-vehicle camera.

[0048] In another specific example, style conversion processing is performed on an image of a person in a car, including:

[0049] The images of people in the car are processed by style conversion to blur the facial information of each person in the image and the behavior of the passengers, while retaining the facial features that reflect the driver's driving behavior.

[0050] In the embodiment of the present application, the style conversion processing includes but is not limited to image blurring, artistic style conversion (such as oil painting style, cartoon style), etc., so as to make personal identity information in the image difficult to identify.

[0051] In the embodiment of the present application, a style conversion module can be installed in the camera so that the collected images or videos of faces, bodies, etc. can undergo style conversion in real time.

[0052] In the embodiment of the present application, non-critical areas in the image, such as the driver's face shape, hairstyle, hair color, nose, background and details, can be blurred by Gaussian blur or other blurring techniques. However, it is necessary to ensure that key facial features (such as eyes, mouth, etc.) are still clearly visible so that the neural network model can work effectively.

[0053] In another specific example, detecting the driving state of the driver from features reflecting the driver's behavior includes:

[0054] Using a neural network model to detect the driver's driving state from the features reflecting the driver's behavior;

[0055] The above method further includes:

[0056] The style-transferred images are used to train the neural network model.

[0057] In the embodiment of the present application, the features reflecting the driver's behavior include eyes, mouth, head posture, etc. The driving state includes whether the driver is driving fatigued.

[0058] In the embodiment of the present application, the neural network model usually relies on facial features (such as eyes, mouth, head posture, etc.) to detect fatigue status. These key features can still be retained after appropriate style conversion.

[0059] In another specific example, using the style-converted image to train the neural network model includes:

[0060] Select driver facial images of different genders, ages, races, and driving scenarios as the training dataset;

[0061] Perform style conversion on each image in the training dataset to blur the facial information of the person in the image and retain the facial features that reflect the driver's driving behavior

[0062] Annotate each image in the training data set to annotate the attribute information of the person in the image and the facial features reflecting the driver's behavior;

[0063] The neural network model is trained using the labeled training data set.

[0064] In an embodiment of the present application, a style conversion based on StyleGAN can be used. The style conversion of StyleGAN can adjust macro attributes such as gender and age of the generated face, and can also adjust more specific attributes such as skin color and hair color.

[0065] In the embodiment of the present application, the labeled training data set should contain detailed attribute labels (such as gender, age, skin color, hair color, etc.) to facilitate supervised training and attribute editing.

[0066] In the embodiment of the present application, the image data set selected for training should be as close as possible to the actual situation in the driving scene, including different lighting conditions, different postures, etc., to ensure the effectiveness of the generated images in practical applications.

[0067] In the embodiment of this application, the selected dataset needs to be a high-quality face dataset for fine-tuning and attribute editing, such as CelebA: contains rich facial attribute labels. FFHQ: high-quality face images, suitable for high-resolution generation tasks.

[0068] In the embodiment of the present application, the latent space of StyleGAN contains rich facial feature information. By finding the direction of specific attributes in the latent space, these attributes can be controlled. The specific steps are as follows: a. Use existing facial attribute classifiers (such as gender classifiers, age predictors, etc.) to mark samples in the latent space. b. Use linear regression methods to find direction vectors related to these attributes in the latent space. C. By moving along the discovered direction vectors, these attributes can be controlled in the latent space. For example, increase or decrease gender features, age features, etc.

[0069] In an embodiment of the present application, a prepared dataset is used to fine-tune the pre-trained StyleGAN to better adapt to changes in target attributes. Fine-tuning the model: a. The collected driver facial image dataset is used to fine-tune StyleGAN to make the generated images more targeted. b. Use specific attribute labels (gender, age, etc.) for supervised training to enhance the model's control over these attributes. Training: a. Use the Adam optimizer for training. b. Set an appropriate learning rate and batch size. c. Save model checkpoints regularly to monitor the generation effect during training.

[0070] In another specific example, the method further includes:

[0071] Set the user's processing rights for the collected images. The processing rights include whether to agree to the collected images and the terms of use, as well as the processing of the collected images.

[0072] Furthermore, set up transparency and user consent before image collection: Manufacturers should clearly inform users of the functions of in-car cameras, the types and uses of data collection, and obtain users' explicit consent. Provide a detailed privacy policy and set up a user consent interface to ensure that users can clearly understand and choose whether to agree to data collection and usage terms when purchasing and using the vehicle.

[0073] Furthermore, after image acquisition: First, the acquired data should be processed locally as much as possible to reduce the amount of data uploaded to the cloud, and the principle of data minimization should be used to collect only necessary data. Secondly, when doing intelligent analysis, a privacy-preserving neural network model should be trained so that the model can perform driver monitoring analysis on images after style transformation, and realize a series of functions such as fatigue detection and attention monitoring. Finally, users should be given control over their data, including the power to view, download, and delete data, and a user-friendly interface or application should be provided so that users can easily manage their data, set data retention periods, and automatic deletion options.

[0074] In another specific example, the method further includes:

[0075] The style-converted images and detected driving status are stored and transmitted, and the data is set not to be tampered with or leaked during the transmission process.

[0076] In the second aspect, in order to facilitate better implementation of the method provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the above method. The meanings of the terms are the same as those in the above method, and the specific implementation details can refer to the description in the method embodiment.

[0077] See also Figure 2 , Figure 2 A schematic diagram of the structure of a device provided in an embodiment of the present application, wherein the device 200 may include an image acquisition module 201, a style conversion module 202 and an analysis module 203, wherein:

[0078] An image acquisition module 201 is used to acquire images of people in the vehicle;

[0079] A style conversion module 202 is used to perform style conversion processing on the image of the people in the car to blur the information reflecting the identity of each person in the image;

[0080] The analysis module 203 is used to extract features reflecting the driver's behavior from the image after style conversion, and detect the driver's driving state from the features reflecting the driver's behavior.

[0081] In another specific example, the style conversion module 202 is specifically used to perform style conversion processing on the images of people in the car, blur the facial information of each person in the image and the behavior of the passengers, and retain the facial features that reflect the driver's driving behavior.

[0082] In another specific example, the above-mentioned analysis module 203 is specifically used to: use a neural network model to detect the driver's driving state from the features reflecting the driver's behavior; and use the style-converted images to train the neural network model.

[0083] In another specific example, using the style-converted image to train the neural network model includes:

[0084] Select driver facial images of different genders, ages, races, and driving scenarios as the training dataset;

[0085] Perform style conversion on each image in the training dataset to blur the facial information of the person in the image and retain the facial features that reflect the driver's driving behavior

[0086] Annotate each image in the training data set to annotate the attribute information of the person in the image and the facial features reflecting the driver's behavior;

[0087] The neural network model is trained using the labeled training data set.

[0088] In another specific example, the above-mentioned device also includes: a permission processing module, which is used to set the user's processing permission for the collected image, and the processing permission includes whether to agree to the collected image and the terms of use, and the processing of the collected image.

[0089] In another specific example, the above-mentioned device also includes: a security module, which is used to store and transmit the style-converted images and the detected driving status, and to set the data to be prevented from being tampered with and leaked during the transmission process.

[0090] In a third aspect, the present application also provides a driver monitoring system including the above-mentioned privacy protection device for an in-vehicle camera.

[0091] In a fourth aspect, the present application also provides a computer device, such as Figure 3 As shown, it shows a schematic diagram of the structure of the computer device involved in the embodiment of the present application, specifically:

[0092] The computer device may include one or more processing core processors 301, one or more computer readable storage media memory 302, a power supply 303, an input unit 304 and other components. Those skilled in the art will understand that Figure 3 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0093] The processor 301 is the control center of the computer device. It uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the memory 302 and calling data stored in the memory 302, it executes various functions of the computer device and processes data, thereby monitoring the computer device as a whole. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operation storage medium, user interface and application program, etc., and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 301.

[0094] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and data processing by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating storage medium, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 302 may also include a controller to provide the processor 301 with access to the memory 302.

[0095] The computer device also includes a power supply 303 for supplying power to various components. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management storage medium, so as to manage charging, discharging, and power consumption management through the power management storage medium. The power supply 303 can also include any components such as one or more DC or AC power supplies, recharge storage media, power failure detection circuits, power converters or inverters, and power status indicators.

[0096] The computer device may further include an input unit 304, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0097] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 301 in the computer device will load the executable files corresponding to the processes of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application programs stored in the memory 302, thereby implementing the steps in the above method embodiment.

[0098] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0099] To this end, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. The computer program is loaded by a processor to execute the steps in any method provided in the embodiment of the present application.

[0100] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0101] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0102] Since the computer program stored in the computer-readable storage medium can execute the steps in any method provided in the embodiments of the present application, the beneficial effects that can be achieved by any method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0103] The above is a detailed introduction to the privacy protection method, device, system, computer equipment and medium for the in-vehicle camera provided by the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A privacy protection method for an in-vehicle camera, characterized in that: include: Acquire an image of a person in a vehicle, and perform style conversion processing on the image of the person in the vehicle to blur information reflecting the identity of each person in the image; Features reflecting the driver's behavior are extracted from the style-converted image, and the driver's driving state is detected from the features reflecting the driver's behavior.

2. The method according to claim 1, characterized in that: Performing style conversion processing on the image of the person in the vehicle, including: The image of the occupants is subjected to style conversion processing to blur the facial information of each person in the image and the behavior of the passengers, while retaining the facial features reflecting the driver's driving behavior.

3. The method according to claim 1 or 2, characterized in that: Detecting the driving state of the driver from the features reflecting the driver's behavior includes: Detecting the driver's driving state from the features reflecting the driver's behavior using a neural network model; The method further comprises: The style-converted images are used to train the neural network model.

4. The method according to claim 3, characterized in that The method of using the style-converted image to train the neural network model includes: Select driver facial images of different genders, ages, races, and driving scenarios as the training dataset; Perform style conversion processing on each image in the training data set to blur the facial information of the person in the image and retain the facial features that reflect the driver's driving behavior Annotating each image in the training data set to annotate attribute information of the person in the image and facial features reflecting the driver's behavior; The neural network model is trained using the labeled training data set.

5. The method according to claim 1, characterized in that The method further comprises: Set the user's processing rights for the collected images, which include whether to agree to the collected images and the terms of use, as well as the processing of the collected images.

6. The method according to claim 1, characterized in that The method further comprises: The style-converted images and detected driving status are stored and transmitted, and the data is set not to be tampered with or leaked during the transmission process.

7. A privacy protection device for an in-vehicle camera, characterized in that: include: An image acquisition module, used to acquire images of people in the vehicle; A style conversion module, used to perform style conversion processing on the image of the occupants of the vehicle to blur the information reflecting the identity of each person in the image; The analysis module is used to extract features reflecting the driver's behavior from the style-converted image, and detect the driver's driving state from the features reflecting the driver's behavior.

8. A driver monitoring system comprising the privacy protection device for an in-vehicle camera according to claim 7.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.

Images