Privacy protection method, electronic equipment and computer readable storage medium

Abstract

The invention discloses a privacy protection method, electronic equipment and a computer readable storage medium, and the privacy protection method comprises the steps: carrying out the real-time detection of multi-mode perception data collected by a terminal, and obtaining a detection result; wherein the multi-modal perception data comprises at least one of facial expression data, eye movement data and environment image data of the target user; in response to the detection result that at least one privacy risk event is detected, dynamically executing privacy protection operation of a corresponding level according to the type of an application currently operated by the terminal; wherein the privacy protection operation comprises a plurality of levels so as to correspondingly protect various types of applications. According to the method and the device, the multi-modal sensing data is detected in real time, and the privacy protection operation is automatically executed when the privacy risk event is detected, so that active protection can be realized under the condition that a user does not need to carry out additional manual operation, the privacy safety and the user experience are effectively considered, and the user experience is improved. And thus, the privacy protection requirement of the user on the intelligent terminal can be well met.

Description

Technical Field

[0001] This application relates to the field of smart terminal security technology, and in particular to a privacy protection method, electronic device and computer-readable storage medium. Background Technology

[0002] With the widespread adoption of mobile internet, smart mobile terminals (such as smartphones and tablets) have become core devices for handling personal communications, financial transactions, and business secrets, making their privacy protection capabilities crucial.

[0003] In related technologies, terminal privacy is generally protected through static identity authentication mechanisms such as password authentication, fingerprint recognition, or facial recognition.

[0004] However, the aforementioned methods rely on manual user operation, offering only passive defense. This results in cumbersome operation, delayed response, and a negative impact on user experience. Furthermore, they only verify identity and cannot detect environmental threats. Therefore, these methods fail to balance privacy and user experience, and cannot adequately meet users' privacy protection needs for smart terminals. Summary of the Invention

[0005] The main technical problem addressed by this application is to provide a privacy protection method, electronic device, and computer-readable storage medium that can solve the problem that related technologies cannot simultaneously address privacy security and user experience.

[0006] To address the aforementioned technical problems, the first technical solution adopted in this application is to provide a privacy protection method, comprising: real-time detection of multimodal perception data collected by the terminal to obtain a detection result; wherein the multimodal perception data includes at least one of the target user's facial expression data, eye movement data, and environmental image data; in response to the detection result indicating that at least one privacy risk event has been detected, dynamically executing a corresponding level of privacy protection operation based on the application type currently running on the terminal; wherein the privacy protection operation includes multiple levels to protect various types of applications.

[0007] The step of real-time detection of multimodal perception data collected by the terminal to obtain detection results includes: acquiring multimodal perception data collected by the terminal in real time; in response to the multimodal perception data being facial expression data of the target user, matching the facial expression data with micro-expressions in a preset privacy-protected expression library, and determining that a privacy risk event has been detected when the facial expression data is detected to match the micro-expressions in the preset privacy-protected expression library; in response to the multimodal perception data being eye movement data of the target user, determining that a privacy risk event has been detected when the target user's gaze is detected to deviate from the terminal screen based on the eye movement data; and in response to the multimodal perception data being environmental image data, determining that a privacy risk event has been detected when a face image of a non-target user is detected.

[0008] The step of determining that a privacy risk event has been detected when the multimodal perception data is the target user's eye movement data and the target user's gaze is detected to deviate from the terminal screen based on the eye movement data includes: determining that a privacy risk event has been detected when the duration of the target user's gaze deviating from the terminal screen exceeds a preset duration and a change in ambient light intensity data is detected.

[0009] The steps for acquiring multimodal perception data collected by the terminal in real time include: acquiring image data captured within the field of view of the terminal's front-facing camera in real time; performing face detection on the image data to extract the target user's face image data and environmental image data; and acquiring facial expression data and eye movement data based on the face image data.

[0010] The steps include: responding to the detection result of at least one privacy risk event, and dynamically executing the corresponding level of privacy protection operation based on the type of application currently running on the terminal, including: responding to the detection result of at least one privacy risk event, and the application currently running on the terminal is a first type of application, executing a first-level privacy protection operation to blur the terminal screen; responding to the detection result of at least one privacy risk event, and the application currently running on the terminal is a second type of application, executing a second-level privacy protection operation to switch to the system security interface; and responding to the detection result of at least one privacy risk event, and the application currently running on the terminal is a third type of application, executing a third-level privacy protection operation to trigger a fake system and display a preset fake interface.

[0011] The step of performing a first-level privacy protection operation to blur the terminal screen in response to the detection result being that at least one privacy risk event has been detected and the application currently running on the terminal is a first-type application includes: dynamically determining the blur intensity based on the target user's facial expression data and / or eye movement data, and blurring the area of ​​the terminal screen displaying privacy information to a corresponding degree based on the blur intensity.

[0012] The process includes, after responding to the detection result indicating that at least one privacy risk event has been detected and the application currently running on the terminal is a third-type application, performing a third-level privacy protection operation to trigger the disguise system and display a preset fake interface, the process further includes: generating an expression interaction instruction; wherein the expression interaction instruction includes a specified micro-expression; collecting consecutive frame facial images of the target user within a preset time period, and analyzing the consecutive frame facial images to obtain a micro-expression analysis result; and responding to the micro-expression analysis result indicating that the micro-expression made by the target user matches the specified micro-expression, performing subsequent interaction operations.

[0013] The first type of application includes photo album applications or instant messaging software; the second type of application includes audio and video playback software or web browsers; and the third type of application includes financial service software.

[0014] To solve the above-mentioned technical problems, the second technical solution adopted in this application is to provide an electronic device, including: a memory for storing program data, wherein the program data, when executed, implements the steps in the privacy protection method described above; and a processor for executing the program data stored in the memory to implement the steps in the privacy protection method described above.

[0015] To solve the above-mentioned technical problems, the third technical solution adopted in this application is to provide a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps in the privacy protection method described in any of the above claims.

[0016] The beneficial effects of this application are as follows: This application provides a privacy protection method, electronic device, and computer-readable storage medium. By real-time detection of multimodal perception data collected by the terminal, it can intelligently determine whether a privacy risk event has occurred based on at least one dimension of indicator data, such as the target user's facial expression data, eye movement data, and environmental image data, thereby improving the accuracy of the detection results and thus enhancing the precision and reliability of privacy protection triggering. Furthermore, in response to the detection of at least one privacy risk event, it dynamically executes corresponding levels of privacy protection operations according to the type of application currently running on the terminal. This automated response mechanism effectively reduces the risk of privacy leakage. Since privacy protection is automatically triggered, no additional manual operation is required from the user, thus achieving seamless intelligent protection, thereby improving convenience and security, and optimizing the human-computer interaction experience. Furthermore, by implementing multi-level privacy protection operations to provide different levels of protection for different types of applications, it is possible to achieve differentiated and refined privacy protection, thereby further improving the effectiveness of privacy protection. This application detects multimodal sensing data in real time and automatically performs privacy protection operations when a privacy risk event is detected. It can achieve proactive protection without requiring additional manual operation by the user, thus effectively balancing privacy security and user experience, and better meeting the user's privacy protection needs for smart terminals. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1This is a flowchart illustrating the first embodiment of the privacy protection method of this application; Figure 2 This is a flowchart illustrating the second embodiment of the privacy protection method of this application; Figure 3 This is a flowchart illustrating the third embodiment of the privacy protection method of this application; Figure 4 This is a flowchart illustrating the application scenario of the privacy protection method of this application; Figure 5 This is a schematic diagram of one embodiment of the privacy protection device of this application; Figure 6 This is a schematic diagram of the structure of one embodiment of the electronic device of this application; Figure 7 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “said,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms, unless otherwise clearly indicated above. “Multiple” generally includes at least two, but does not exclude the inclusion of at least one.

[0021] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0022] It should be understood that the terms "comprising," "including," or any other variations used herein are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0023] In related technologies, terminal privacy is generally protected through static identity authentication mechanisms such as password authentication, fingerprint recognition, or facial recognition.

[0024] However, the aforementioned methods rely on manual user operation, offering only passive defense. This results in cumbersome operation, delayed response, and a negative impact on user experience. Furthermore, they only verify identity and cannot detect environmental threats. Therefore, these methods fail to balance privacy and user experience, and cannot adequately meet users' privacy protection needs for smart terminals.

[0025] Based on the above, this application provides a privacy protection method, an electronic device, and a computer-readable storage medium that can solve the problem that related technologies cannot simultaneously address privacy security and user experience.

[0026] Specifically, please refer to Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the privacy protection method of this application. In this embodiment, the privacy protection method includes: S11: Perform real-time detection on the multimodal perception data collected by the terminal to obtain the detection results; wherein, the multimodal perception data includes at least one of the target user's facial expression data, eye movement data, and environmental image data.

[0027] In this embodiment, the target user is the owner of the terminal.

[0028] The terminal is equipped with a TEE (Trusted Execution Environment), which is a separate "secure area" created within the terminal's main processor through physical isolation. It is specifically used to process sensitive data (such as biometrics and payment information).

[0029] Specifically, the TEE stores the biometric information of the target user (such as fingerprint features, voiceprint features, facial features, or iris features). The terminal can determine whether the person currently using the terminal is the target user by extracting facial features from the image and matching the extracted facial features with the facial features stored in the TEE.

[0030] In this embodiment, the terminal can be any smart terminal. In some embodiments, the terminal includes any one of a mobile phone, tablet, laptop, or VR (Virtual Reality) device.

[0031] In this embodiment, the multimodal perception data is extracted from real-time continuous video frames acquired by the front-facing camera of the terminal.

[0032] In this embodiment, facial expression data includes micro-expressions.

[0033] Micro-expressions are subtle changes in facial expressions that occur unconsciously and briefly when humans try to hide their true emotions. They typically last for 1 / 25 to 1 / 2 second and are difficult to capture directly with the naked eye, but they contain genuine emotional information.

[0034] In some implementations, if the target user's facial expression data is used to detect a micro-expression that indicates the user is entering a privacy protection mode, the detection result is considered a privacy risk event.

[0035] Understandably, by detecting facial expression data in real time, it is possible to promptly understand the intentions or emotional changes of the target user, and thus determine whether to enable the privacy protection mode.

[0036] In this embodiment, eye-tracking data is eye movement information recorded by algorithms (such as camera-based visual tracking) and is used to analyze a person's visual attention, cognitive state, and behavioral patterns.

[0037] In some implementations, eye-tracking data include eyelid opening / closing ratio (EAR), blink frequency, fixation duration, pupil diameter change rate, and eye movement trajectory.

[0038] Specifically, the eye movement trajectory is obtained by calculating the user's gaze focus in real time using the front-facing camera. When the target user is looking at the terminal screen, their eye movement trajectory will be concentrated on the screen area; when the target user's gaze leaves the terminal screen (turns their head or looks sideways), their eye movement trajectory will deviate from the screen area. The algorithm records the start time, duration, and direction of each deviation.

[0039] In some implementations, if the target user's gaze is detected to be deviating from the terminal screen based on the target user's eye-tracking data, the detection result is considered a privacy risk event.

[0040] Understandably, if a gaze shift is detected, it indicates that the target user is no longer paying attention to the screen content and their attention has shifted. If the target user's gaze suddenly and quickly turns to one side, it usually means that something in the environment (such as someone approaching) has attracted the target user's attention.

[0041] In this embodiment, environmental image data refers to the background image in the image other than the target user's face image.

[0042] In some implementations, if the presence of other facial images of non-target users in the field of view is detected based on environmental image data, the detection result is considered a privacy risk event.

[0043] Understandably, when the background image includes other facial images that are not of the target user, it indicates that there are other people around the target user, who may be spying on the target user's terminal screen, posing a risk of privacy leakage.

[0044] Understandably, by performing real-time detection on the multimodal perception data collected by the terminal, it is possible to make real-time intelligent judgments on whether a privacy risk event has occurred based on at least one dimension of indicator data such as the target user's facial expression data, eye movement data, and environmental image data, so as to improve the accuracy of the detection results and thus improve the precision and reliability of privacy protection triggers.

[0045] S12: In response to the detection result indicating that at least one privacy risk event has been detected, the corresponding level of privacy protection operation is dynamically executed according to the type of application currently running on the terminal; wherein, the privacy protection operation includes multiple levels to protect various types of applications.

[0046] In this embodiment, privacy risk events include detecting specific micro-expression combinations made by the target user based on the target user's facial expression data, detecting the target user's gaze deviating from the terminal screen based on the target user's eye movement data, and detecting faces of non-target users based on environmental image data.

[0047] Among them, the specific micro-expression made by the target user indicates that the user actively issued (or was triggered by tension) a micro-expression instruction that needs to be recognized. Based on this micro-expression instruction, it can be determined that the user has given an active and emotion-driven intention signal, that is, the target user needs to trigger the privacy protection mode at this time to protect the privacy of the terminal.

[0048] When the target user's gaze deviates from the terminal screen, it indicates that the user's attention has shifted and they have not actively given an intention signal to trigger the privacy protection mode. However, since the target user is not looking at the screen, they may not notice whether someone is watching the screen. At this time, there is a risk of privacy leakage. It is necessary to automatically identify the risk and respond, that is, to automatically trigger the privacy protection mode to protect the privacy of the terminal.

[0049] Among them, the detection of faces of non-target users based on environmental image data indicates that there are other people around the target user, which may pose a possibility of others spying on the screen (environmental threat). At this time, there is a risk of privacy leakage, and it is necessary to automatically identify the risk and respond, that is, to automatically trigger the privacy protection mode to protect the privacy of the terminal.

[0050] Understandably, since privacy protection is automatically detected and triggered without requiring any additional manual operation from the user, it can also achieve seamless intelligent protection, thereby improving convenience and security, and thus optimizing the human-computer interaction experience.

[0051] In this embodiment, multiple applications on the terminal are divided into different types, and when at least one privacy risk event is detected, the corresponding level of privacy protection operation is dynamically executed according to the type of application currently running on the terminal.

[0052] In some implementations, the different types include low-risk applications, medium-risk applications, and high-risk applications. Low-risk applications correspond to low-level privacy protection operations, medium-risk applications correspond to intermediate-level privacy protection operations, and high-risk applications correspond to high-level privacy protection operations. The higher the level of privacy protection operation, the stronger the protection performance.

[0053] In this embodiment, in response to the detection result indicating that no privacy risk event was detected, the current display interface of the terminal screen is maintained.

[0054] Unlike related technologies, this implementation method performs real-time detection of multimodal perception data collected by the terminal. Based on at least one dimension of indicator data, such as the target user's facial expression data, eye movement data, and environmental image data, it can intelligently determine whether a privacy risk event has occurred, thereby improving the accuracy of the detection results and enhancing the precision and reliability of privacy protection triggering. Furthermore, in response to the detection of at least one privacy risk event, it dynamically executes corresponding levels of privacy protection operations according to the type of application currently running on the terminal. This automated response mechanism effectively reduces the risk of privacy leakage. Since privacy protection is automatically triggered without requiring additional manual operation from the user, it also achieves seamless intelligent protection, improving convenience and security, and thus optimizing the human-computer interaction experience. Furthermore, by implementing multi-level privacy protection operations to provide different levels of protection for different types of applications, it achieves differentiated and refined privacy protection, thereby further enhancing the effectiveness of privacy protection. This application detects multimodal sensing data in real time and automatically performs privacy protection operations when a privacy risk event is detected. It can achieve proactive protection without requiring additional manual operation by the user, thus effectively balancing privacy security and user experience, and better meeting the user's privacy protection needs for smart terminals.

[0055] Please see Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the privacy protection method of this application. In this embodiment, the privacy protection method includes: S21: Real-time acquisition of multimodal sensing data collected by the terminal.

[0056] In this embodiment, the multimodal perception data includes at least one of the target user's facial expression data, eye movement data, and environmental image data.

[0057] In some implementations, image data captured within the field of view by the terminal's front-facing camera is acquired in real time, and face detection is performed on the image data to extract the target user's facial image data and environmental image data. Then, facial expression data and eye-tracking data are obtained based on the facial image data.

[0058] Specifically, the algorithm locates facial images in real time within continuous video frames captured by the front-facing camera and extracts facial features. These detected facial features are then compared with the facial features of the target user (owner) stored in the TEE (Transparent Environment). If the comparison confirms that the detected facial features match those of the target user, the detected facial image is identified as the target user's facial image data. After extracting the target user's facial image data, the remaining areas of the image data are identified as environmental image data. Furthermore, facial features corresponding to the target user's facial image data are analyzed to obtain facial expression data and eye-tracking data.

[0059] Understandably, by applying different processing procedures and using different target detection algorithms to the raw images captured by the front-facing camera, it is possible to extract the target user's facial image data and environmental image data from the same image stream without integrating data from multiple sensors, thereby reducing the complexity of image processing and lowering the hardware cost of the system.

[0060] S22: In response to the multimodal perception data being the facial expression data of the target user, the facial expression data is matched with micro-expressions in a preset privacy-protected expression library, and when a match is detected between the facial expression data and the micro-expressions in the preset privacy-protected expression library, a privacy risk event is determined to have been detected.

[0061] In this embodiment, the micro-expressions in the preset privacy-protected emoji library are the micro-expressions that the target user has pre-stored when entering the privacy-protected mode.

[0062] In some implementations, the micro-expressions in the preset privacy-preserving expression library can be at least one of eye movements, brief contractions of facial muscles, or slight mouth movements.

[0063] In some specific implementations, eye movements include frowning, raising eyebrows, or blinking rapidly and repeatedly. Mouth micro-movements include raising the corners of the mouth and opening the mouth.

[0064] In this embodiment, facial landmark detection is performed on facial expression data to obtain the coordinates of multiple facial landmarks. The relative distance changes between these coordinates are then analyzed to obtain micro-expression analysis results. The micro-expressions corresponding to the micro-expression analysis results are then matched with micro-expressions in a preset privacy-protected expression library. Upon successful matching, a privacy risk event is detected.

[0065] S23: In response to the multimodal perception data being the eye movement data of the target user, when the target user's gaze is detected to deviate from the terminal screen based on the eye movement data, a privacy risk event is determined to have been detected.

[0066] In this embodiment, the detection of privacy risk events can be based solely on eye-tracking data, or it can be combined with eye-tracking data and ambient light intensity data to detect privacy risk events.

[0067] Ambient light intensity data can be collected through an ambient light sensor integrated into the terminal.

[0068] In some implementations, the eye movement trajectory is obtained by calculating the target user's gaze focus in real time using the front-facing camera. When the eye movement trajectory deviates from the screen area for more than a preset time, it is determined that the target user's gaze has deviated from the terminal screen for more than a preset time, and thus a privacy risk event is detected.

[0069] In other implementations, a privacy risk event is determined to have been detected in response to the detection that the target user's gaze has been deviated from the terminal screen for more than a preset duration and a change in ambient light intensity data is detected.

[0070] Changes in ambient light intensity data could indicate the detection of reflections, which could be caused by someone suddenly approaching.

[0071] The preset duration is greater than or equal to 2 seconds.

[0072] S24: In response to the multimodal perception data being environmental image data, when a face image of a non-target user is detected, it is determined that a privacy risk event has been detected.

[0073] In this embodiment, face recognition is performed from the background image corresponding to the environmental image data. If other face images that are not of the target user are detected, it is determined that a privacy risk event has been detected.

[0074] S25: In response to the detection result indicating that at least one privacy risk event has been detected, the corresponding level of privacy protection operation is dynamically executed according to the type of application currently running on the terminal; wherein, the privacy protection operation includes multiple levels to protect various types of applications.

[0075] Please refer to the description in S12 for the specific process, which will not be repeated here.

[0076] Unlike related technologies, this implementation acquires multimodal perception data collected by the terminal in real time and continuously analyzes this data. It can intelligently determine whether a privacy risk event has occurred based on at least one dimension of indicator data, such as the target user's facial expression data, eye movement data, and environmental image data. This improves the accuracy of the detection results, allowing for automatic risk identification and response even before or simultaneously with privacy leaks, thereby enhancing the accuracy and reliability of privacy protection triggers. By detecting multimodal perception data in real time and automatically executing privacy protection operations upon detecting a privacy risk event, this application achieves proactive protection without requiring additional manual intervention from the user. This effectively balances privacy security and user experience, thus better meeting users' privacy protection needs for smart terminals.

[0077] Please see Figure 3 , Figure 3 This is a flowchart illustrating the third embodiment of the privacy protection method of this application. In this embodiment, the privacy protection method includes: S31: Perform real-time detection on the multimodal perception data collected by the terminal to obtain the detection result; wherein, the multimodal perception data includes at least one of the target user's facial expression data, eye movement data, and environmental image data.

[0078] For details, please refer to the descriptions in S11 and S21~S24, which will not be repeated here.

[0079] S32: In response to the detection result indicating that at least one privacy risk event has been detected, and the application currently running on the terminal is a first-type application, perform a first-level privacy protection operation to blur the terminal screen.

[0080] In this embodiment, the first type of application includes photo album applications or instant messaging software.

[0081] Among them, the first type of application is a medium-risk application, and the first level of privacy protection operation is an intermediate level of privacy protection operation.

[0082] In some implementations, instant messaging software includes SMS, WeChat, email, etc.

[0083] In some implementations, the blur intensity is dynamically determined based on the target user's facial expression data and / or eye movement data, and the area on the terminal screen displaying privacy information is blurred to a corresponding degree based on the blur intensity.

[0084] In some specific implementations, if the micro-expression corresponding to the facial expression data is a frown, then the blur intensity is determined to be a high-level blur intensity, and the corresponding mask image is dark, in order to perform blurring processing with strong occlusion. If the micro-expression corresponding to the facial expression data is an raised eyebrow, then the blur intensity is determined to be a low-level blur intensity, and the corresponding mask image is light, in order to perform blurring processing with weak occlusion.

[0085] In other specific implementations, if eye-tracking data shows that the target user's pupil diameter increases (eyes widening), then the blur intensity is determined to be a high-level blur intensity, and the corresponding mask image is dark, for stronger blurring. If eye-tracking data shows that the target user's pupil diameter decreases (eyes squinting), then the blur intensity is determined to be a low-level blur intensity, and the corresponding mask image is light, for weaker blurring.

[0086] In a specific implementation scenario, when a target user receives a bank SMS message in a public place, if the image data captured by the front-facing camera detects a frowning micro-expression corresponding to the target user's facial expression data, and this frown matches a frowning micro-expression in a preset privacy-protected expression library, the SMS content is immediately blurred. Furthermore, if the SMS content is displayed at the top of the screen, to enhance the deceptiveness of the blurred information, a disguised notification (such as a weather forecast update) can be generated at the top of the screen. Further, if a nodding micro-expression (not a micro-expression indicating entry into privacy protection mode) is detected after the blurred display, the SMS content is automatically restored.

[0087] Understandably, the first type of application includes users' personal privacy information, which is sensitive content. If others spy on it, it may lead to privacy leaks. Therefore, when a privacy risk event is detected, performing a medium-level privacy protection operation to blur the terminal screen can effectively reduce the risk of privacy leaks caused by others spying on it.

[0088] S33: In response to the detection result indicating that at least one privacy risk event has been detected, and the application currently running on the terminal is a second type of application, perform a second-level privacy protection operation to switch to the system security interface.

[0089] In this embodiment, the second type of application includes audio and video playback software or web browsers.

[0090] The second type of application is a low-risk application, and the second level of privacy protection operation is a low-level privacy protection operation.

[0091] In a specific implementation scenario, if a target user is watching a video through video software in a public place, and the image data collected by the front-facing camera detects that the target user's gaze deviates from the screen by more than 15° for more than 2 seconds, and the ambient light sensor detects additional reflections (possibly from someone approaching), then the system switches to the security interface to hide the video software, thereby achieving security locking.

[0092] Understandably, the second type of application is mainly an entertainment application, which usually does not include the user's personal privacy information, but may reveal the target user's entertainment preferences. Therefore, when a privacy risk event is detected, a low-level privacy protection operation is performed, switching to the system security interface to hide the relevant application, which can effectively protect personal preferences from being known by others.

[0093] S34: In response to the detection result indicating that at least one privacy risk event has been detected, and the application currently running on the terminal is a third-type application, execute a third-level privacy protection operation to trigger the disguise system and display a preset fake interface.

[0094] In this embodiment, the third type of application includes financial service software.

[0095] Among them, the third type of application is a high-risk application, and the third level of privacy protection operation is a high-level privacy protection operation.

[0096] In some implementations, financial service software includes banking applications, financial payment software, etc.

[0097] In this scenario, a fake interface can overlay the real interface of the current application and mislead the spy into thinking that they are seeing real content, thus making them unaware that privacy protection has been triggered, or even allowing them to obtain false information.

[0098] Understandably, third-type applications typically involve payments, transfers, or financial services. When using third-type applications, simply interrupting the operation itself may arouse suspicion from eavesdroppers (for example, an eavesdropper might be watching the target user enter their password). In this case, the fake interface presented by the fake system has a strong deceptive effect and can more smoothly and securely reduce the risk of privacy leaks.

[0099] In this embodiment, after a fake interface is displayed, the normal display can be restored through liveness detection technology.

[0100] The system can require the target user to make a randomly specified expression (such as raising an eyebrow) to verify that it is not a photo / video attack, and identify fake faces by the frequency of facial muscle movements (5-12Hz for normal humans).

[0101] In some implementations, if the target user needs to complete the payment, an emoji interaction command is generated, which includes a specified micro-expression. A series of facial images of the target user within a preset time period are captured and analyzed to obtain micro-expression analysis results. In response to the micro-expression analysis result indicating that the target user's micro-expression matches the specified micro-expression, subsequent interactive operations are executed to restore normal display.

[0102] Understandably, by adding liveness detection, the effectiveness of privacy protection can be further improved.

[0103] Unlike related technologies, this implementation method performs real-time detection of multimodal sensing data collected by the terminal. When at least one privacy risk event is detected, it implements multi-level privacy protection operations to provide different levels of protection for different types of applications currently running on the terminal. This enables differentiated and refined privacy protection, thereby further improving the effectiveness of privacy protection. By detecting multimodal sensing data in real time and automatically executing privacy protection operations when a privacy risk event is detected, this application can achieve proactive protection without requiring additional manual operation from the user. This effectively balances privacy security and user experience, thus better meeting users' privacy protection needs for smart terminals.

[0104] Please see Figure 4 , Figure 4 This is a flowchart of the application scenario of the privacy protection method one in this application.

[0105] In this embodiment, multimodal perception data collected by the terminal is first acquired in real time. This multimodal perception data includes at least one of the target user's facial expression data, eye movement data, and environmental image data. The multimodal perception data is then detected in real time. If the multimodal perception data is the target user's facial expression data, it is matched against micro-expressions in a preset privacy-protected expression library. When a match is detected, a privacy risk event is determined. If the multimodal perception data is the target user's eye movement data, a privacy risk event is determined when the target user's gaze is detected to be deviating from the terminal screen. If the multimodal perception data is environmental image data, a privacy risk event is determined when a face image other than the target user's is detected. Further, if the detection result indicates at least one privacy risk event has been detected, and the application currently running on the terminal is a first-type application, a first-level privacy protection operation is performed to blur the terminal screen. If the detection result indicates at least one privacy risk event has been detected, and the application currently running on the terminal is a second-type application, a second-level privacy protection operation is performed to switch to the system security interface. In response to the detection result indicating that at least one privacy risk event has been detected, and the application currently running on the terminal is a third-type application, a third-level privacy protection operation is performed to trigger the disguise system and display a preset fake interface.

[0106] Unlike related technologies, this implementation method performs real-time detection of multimodal perception data collected by the terminal. Based on at least one dimension of indicator data, such as the target user's facial expression data, eye movement data, and environmental image data, it can intelligently determine whether a privacy risk event has occurred, thereby improving the accuracy of the detection results and enhancing the precision and reliability of privacy protection triggering. Furthermore, in response to the detection of at least one privacy risk event, it dynamically executes corresponding levels of privacy protection operations according to the type of application currently running on the terminal. This automated response mechanism effectively reduces the risk of privacy leakage. Since privacy protection is automatically triggered without requiring additional manual operation from the user, it also achieves seamless intelligent protection, improving convenience and security, and thus optimizing the human-computer interaction experience. Furthermore, by implementing multi-level privacy protection operations to provide different levels of protection for different types of applications, it achieves differentiated and refined privacy protection, thereby further enhancing the effectiveness of privacy protection. This application detects multimodal sensing data in real time and automatically performs privacy protection operations when a privacy risk event is detected. It can achieve proactive protection without requiring additional manual operation by the user, thus effectively balancing privacy security and user experience, and better meeting the user's privacy protection needs for smart terminals.

[0107] Correspondingly, this application provides a privacy protection device.

[0108] Please see Figure 5 , Figure 5 This is a schematic diagram of one embodiment of the privacy protection device of this application. In this embodiment, the privacy protection device 50 includes a detection module 51 and a privacy protection operation execution module 52.

[0109] The detection module 51 is used to perform real-time detection on the multimodal perception data collected by the terminal and obtain the detection result; wherein, the multimodal perception data includes at least one of the target user's facial expression data, eye movement data and environmental image data.

[0110] The privacy protection operation execution module 52 is used to respond to the detection result of at least one privacy risk event being detected, and dynamically execute the corresponding level of privacy protection operation according to the type of application currently running on the terminal; wherein, the privacy protection operation includes multiple levels to protect various types of applications.

[0111] For details of the process, please refer to the relevant textual descriptions in S11~S12, S21~S25 and S31~S34, which will not be repeated here.

[0112] Unlike related technologies, this implementation uses a detection module 51 to perform real-time detection of multimodal perception data collected by the terminal. It can intelligently determine whether a privacy risk event has occurred based on at least one dimension of indicator data, such as the target user's facial expression data, eye movement data, and environmental image data, thereby improving the accuracy of the detection results and enhancing the precision and reliability of privacy protection triggering. Furthermore, in response to the detection of at least one privacy risk event, the privacy protection operation execution module 52 dynamically executes the corresponding level of privacy protection operation according to the type of application currently running on the terminal. This automated response mechanism effectively reduces the risk of privacy leakage. Since privacy protection is automatically triggered, no additional manual operation is required from the user, thus achieving seamless intelligent protection, improving convenience and security, and optimizing the human-computer interaction experience. Furthermore, by implementing multi-level privacy protection operations to provide different levels of protection for different types of applications, differentiated and refined privacy protection can be achieved, further enhancing the effectiveness of privacy protection. This application detects multimodal sensing data in real time and automatically performs privacy protection operations when a privacy risk event is detected. It can achieve proactive protection without requiring additional manual operation by the user, thus effectively balancing privacy security and user experience, and better meeting the user's privacy protection needs for smart terminals.

[0113] Correspondingly, this application provides an electronic device.

[0114] Please see Figure 6 , Figure 6This is a schematic diagram of one embodiment of the electronic device of this application. For example... Figure 6 As shown, in this embodiment, the electronic device 60 includes a memory 61 and a processor 62.

[0115] In this embodiment, memory 61 is used to store program data, which, when executed, implements the steps in the privacy protection method described above. Processor 62 is used to execute the program instructions stored in memory 61 to implement the steps in the privacy protection method described above.

[0116] Specifically, processor 62 controls itself and memory 61 to implement the steps in the privacy protection method described above. Processor 62 can also be referred to as a CPU (Central Processing Unit). Processor 62 may be an integrated circuit chip with signal processing capabilities. Processor 62 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 62 can be implemented using multiple integrated circuit chips.

[0117] Unlike related technologies, this implementation uses processor 62 to perform real-time detection of multimodal perception data collected by the terminal. It can intelligently determine whether a privacy risk event has occurred based on at least one dimension of indicator data, such as the target user's facial expression data, eye movement data, and environmental image data, thereby improving the accuracy of the detection results and enhancing the precision and reliability of privacy protection triggering. Furthermore, in response to the detection of at least one privacy risk event, it dynamically executes corresponding levels of privacy protection operations based on the type of application currently running on the terminal. This automated response mechanism effectively reduces the risk of privacy leakage. Since privacy protection is automatically triggered without requiring additional manual operation from the user, it also achieves seamless intelligent protection, improving convenience and security, and thus optimizing the human-computer interaction experience. Furthermore, by implementing multi-level privacy protection operations to provide different levels of protection for different types of applications, it achieves differentiated and refined privacy protection, thereby further enhancing the effectiveness of privacy protection. This application detects multimodal sensing data in real time and automatically performs privacy protection operations when a privacy risk event is detected. It can achieve proactive protection without requiring additional manual operation by the user, thus effectively balancing privacy security and user experience, and better meeting the user's privacy protection needs for smart terminals.

[0118] Correspondingly, this application provides a computer-readable storage medium.

[0119] Please see Figure 7 , Figure 7 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention.

[0120] The computer-readable storage medium 70 includes a computer program 701 stored on it. When executed by the processor, the computer program 701 implements the steps of the privacy protection method described above. Specifically, if the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium 70. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a computer-readable storage medium 70 and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned computer-readable storage medium 70 includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0121] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0122] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0123] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0124] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0125] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A privacy protection method, characterized in that, include: The multimodal perception data collected by the terminal is detected in real time to obtain the detection results; wherein, the multimodal perception data includes at least one of the target user's facial expression data, eye movement data, and environmental image data; In response to the detection result indicating that at least one privacy risk event has been detected, the corresponding level of privacy protection operation is dynamically executed according to the type of application currently running on the terminal; wherein, the privacy protection operation includes multiple levels to protect various types of applications.

2. The privacy protection method according to claim 1, characterized in that, The step of performing real-time detection on the multimodal sensing data collected by the terminal to obtain the detection result includes: The multimodal sensing data collected by the terminal is acquired in real time. In response to the multimodal perception data being the facial expression data of the target user, the facial expression data is matched with micro-expressions in a preset privacy-protected expression library, and when the facial expression data is detected to match the micro-expressions in the preset privacy-protected expression library, the privacy risk event is determined to have been detected; In response to the multimodal perception data being the eye movement data of the target user, when the target user's gaze is detected to deviate from the terminal screen based on the eye movement data, it is determined that the privacy risk event has been detected; In response to the multimodal perception data being the environmental image data, when a face image of a non-target user is detected, it is determined that the privacy risk event has been detected.

3. The privacy protection method according to claim 2, characterized in that, The step of determining that the privacy risk event has been detected when the multimodal perception data is the eye movement data of the target user and the target user's gaze deviates from the terminal screen based on the eye movement data includes: In response to detecting that the target user's gaze has been deviated from the terminal screen for more than a preset duration, and detecting a change in ambient light intensity data, the privacy risk event is determined to have been detected.

4. The privacy protection method according to claim 2, characterized in that, The step of acquiring the multimodal sensing data collected by the terminal in real time includes: Real-time acquisition of image data captured within the field of view of the terminal's front-facing camera; Face detection is performed on the image data to extract the face image data of the target user and the environmental image data; The facial expression data and eye movement data are obtained based on the facial image data.

5. The privacy protection method according to claim 1, characterized in that, The step of dynamically executing a corresponding level of privacy protection operation based on the type of application currently running on the terminal in response to the detection result indicating that at least one privacy risk event has been detected includes: In response to the detection result indicating that at least one of the privacy risk events has been detected, and the application currently running on the terminal is a first type of application, a first-level privacy protection operation is performed to blur the terminal screen. In response to the detection result indicating that at least one of the privacy risk events has been detected, and the application currently running on the terminal is a second type of application, a second-level privacy protection operation is performed to switch to the system security interface; In response to the detection result indicating that at least one of the privacy risk events has been detected, and the application currently running on the terminal is a third-type application, a third-level privacy protection operation is performed to trigger the disguise system and display a preset fake interface.

6. The privacy protection method according to claim 5, characterized in that, The step of performing a first-level privacy protection operation to blur the terminal screen in response to the detection result indicating that at least one privacy risk event has been detected, and the application currently running on the terminal is a first-type application, includes: The blur intensity is dynamically determined based on the facial expression data and / or eye movement data of the target user, and the area of ​​the terminal screen displaying privacy information is blurred to a corresponding degree based on the blur intensity.

7. The privacy protection method according to claim 5, characterized in that, The step of responding to the detection result being that at least one of the privacy risk events has been detected, and the application currently running on the terminal is a third-type application, and performing a third-level privacy protection operation to trigger a fake system and display a preset fake interface includes: Generate facial expression interaction instructions; wherein, the facial expression interaction instructions include a specified micro-expression; Collect consecutive frames of facial images of the target user within a preset time period, and analyze the consecutive frames of facial images to obtain micro-expression analysis results; In response to the micro-expression analysis result indicating that the micro-expression made by the target user matches the specified micro-expression, subsequent interactive operations are performed.

8. The privacy protection method according to any one of claims 5 to 7, characterized in that, The first type of application includes photo album applications or instant messaging software; The second type of application includes audio and video playback software or web browsers; The third type of application includes financial services software.

9. An electronic device, characterized in that, include: A memory for storing program data, which, when executed, implements the steps of the privacy protection method as described in any one of claims 1 to 8; A processor for executing the program data stored in the memory to implement the steps in the privacy protection method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the privacy protection method as described in any one of claims 1 to 8.

Images