Display content processing method and device, chip and electronic equipment
By using neural network models to predict and match video frames in mobile phone anti-peeping screen technology, the problems of inaccurate identification and limited applicable scenarios in the existing technology are solved, and smarter privacy protection and wider applicability.
Patent Information
- Application Number
- CN202310473412.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-27
- Publication Date
- 2024-11-12
AI Technical Summary
The existing mobile phone anti-peeping screen technology is not smart enough in many scenarios, cannot meet the user's real privacy protection needs, and cannot recognize specific facial information, resulting in the inability to hide screen content when needed, and the applicable scenarios are limited.
By inputting the video frame sequence into the trained neural network model for frame prediction, extracting face features, and based on the matching results of the predicted frame's face features and the preset face features, it is decided whether to hide the display content, and adopt The time-cascaded diffusion model and neural network face recognition technology improve recognition accuracy.
It improves the privacy protection of displayed content intelligence, can more accurately identify users and acquaintances, is suitable for a wider range of scenarios, and enhances the user experience.
Smart Images

Figure CN118940330A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to computer processing technology, and more particularly to a method, device, chip and electronic device for processing display content. Background Art
[0002] With the rapid development of mobile phone system-on-chip (SoC) chips and mobile Internet, personal privacy protection, such as anti-spy screen, is increasingly regarded as one of the core needs of users. With the increasing development of artificial intelligence (AI) technology, more and more traditional face recognition algorithms are being replaced by AI algorithms (such as face detection, etc.). As deep learning (DL) technology becomes more and more mature, mobile phone anti-spy screen technology is implemented by deploying a lightweight multi-face recognition neural network model and corresponding control programs. The always-on (AON) function of the mobile phone will continuously read the video frames captured by the front camera of the mobile phone and input them into the trained model. When multiple faces appear in the video frame, the model will output a corresponding signal to notify the anti-spy screen system of the mobile phone to block sensitive information on the screen, such as text messages and pictures.
[0003] Figure 1 It is a processing flow chart of the mobile phone anti-peep screen in the related technology, such as Figure 1 As shown, the video frame is input into the trained model to generate bounding boxes and perform non-maximum suppression to output the number of faces. In the post-processing stage, the anti-peeping screen system controls whether to hide the displayed content on the mobile phone screen through the number of faces recognized by the trained model, for example, if the displayed content is sensitive information.
[0004] However, although the above-mentioned anti-spy screen technology can prevent others from peeping at sensitive information on the screen of mobile phone users to a certain extent, it does not conform to the real thoughts of users in many scenarios; it can be seen that the existing privacy protection of display content has technical problems of not being intelligent enough. Summary of the invention
[0005] The embodiments of the present application provide a method, device, chip and electronic device for processing display content, which can improve the intelligence of privacy protection of display content.
[0006] The technical solution of this application is implemented as follows:
[0007] In a first aspect, an embodiment of the present application provides a method for processing display content, the method being applied in an electronic device, comprising:
[0008] Input the video frame sequence into the trained neural network model for frame prediction to obtain the predicted frame;
[0009] Extracting facial features from the predicted frame to obtain facial features of the predicted frame;
[0010] Based on the matching result between the facial features of the predicted frame and the preset facial features, it is determined whether to hide the display content on the current display interface on the screen of the electronic device.
[0011] In a second aspect, an embodiment of the present application provides a display content processing device, the device is applied in an electronic device, including:
[0012] A prediction module is used to input a video frame sequence into a trained neural network model for frame prediction to obtain a predicted frame;
[0013] A processing module, used for extracting facial features from the prediction frame to obtain facial features of the prediction frame;
[0014] The determination module is used to determine whether to hide the display content on the current display interface on the screen of the electronic device based on the matching result between the facial features of the predicted frame and the preset facial features.
[0015] In a third aspect, an embodiment of the present application provides a chip, including: a processor, used to call and run a computer program from a memory to execute a method for processing display content as described in one or more of the above embodiments.
[0016] In a fourth aspect, an embodiment of the present application provides an electronic device, including:
[0017] A processor and a storage medium storing instructions executable by the processor, wherein the storage medium relies on the processor to perform operations through a communication bus, and when the instructions are executed by the processor, the method for processing display content described in one or more of the above embodiments is executed.
[0018] In a fifth aspect, an embodiment of the present application provides a computer storage medium storing executable instructions. When the executable instructions are executed by one or more processors, the processor executes the method for processing display content described in one or more of the above embodiments.
[0019] The embodiments of the present application provide a method, device, chip and electronic device for processing display content, including: inputting a video frame sequence into a trained neural network model for frame prediction to obtain a predicted frame, extracting facial features from the predicted frame to obtain facial features from the predicted frame, and determining whether to hide the display content on the current display interface on the screen of the electronic device based on the matching results of the facial features of the predicted frame and preset facial features; that is, in the embodiments of the present application, a video frame sequence is predicted by using a trained neural network model to obtain a predicted frame, and determining whether to hide the display content based on the matching results of the facial features of the predicted frame and preset facial features. In this way, the use of an AI algorithm in advance can achieve a prediction of the video frame sequence that is more consistent with the actual situation, thereby improving the accuracy of the predicted frame. Then, determining whether to hide the display content is based on the predicted frame with higher accuracy, thereby improving the intelligence of privacy protection of the display content. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 It is a processing flow chart of the mobile phone anti-peep screen in the related technology;
[0021] Figure 2 A flowchart of an optional method for processing display content provided in an embodiment of the present application;
[0022] Figure 3a Schematic diagram of the current display interface for displaying content in the related art Figure 1 ;
[0023] Figure 3b Schematic diagram of the current display interface for displaying content in the related art Figure 2 ;
[0024] Figure 4 A flowchart of an optional method for processing display content provided in an embodiment of the present application;
[0025] Figure 5a A schematic diagram of an optional current display interface for sensitive information provided in an embodiment of the present application Figure 1 ;
[0026] Figure 5b A schematic diagram of an optional current display interface for sensitive information provided in an embodiment of the present application Figure 2 ;
[0027] Figure 6 A schematic diagram of the structure of an optional display content processing device provided in an embodiment of the present application;
[0028] Figure 7 A schematic diagram of the structure of an optional chip provided in an embodiment of the present application;
[0029] Figure 8 A schematic diagram of the structure of an optional electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0030] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.
[0031] The present application embodiment provides a method for processing display content. Figure 2 A flowchart of an optional method for processing display content provided in an embodiment of the present application is shown as follows: Figure 2 As shown, the method for processing the displayed content may include:
[0032] S201: Inputting the video frame sequence into the trained neural network model to perform frame prediction to obtain a predicted frame;
[0033] Although the current anti-peeping screen technology can prevent others from peeping at the screens of electronic device users to a certain extent, it is not smart enough in many scenarios and cannot reflect the users’ true thoughts, which may cause trouble to users. Figure 3a Schematic diagram of the current display interface for displaying content in the related art Figure 1 ,like Figure 3a As shown in the figure, in daily use scenarios, assuming that the user does not want strangers to see the content displayed on the screen of their mobile phone, for example, sensitive information, at this time, when the mobile phone automatically detects that there are multiple faces in the viewfinder of the front camera, it will automatically hide the sensitive information on the screen to protect the user's privacy. However, when the user wants to watch the content on the mobile phone screen with others, the mobile phone can only recognize the number of faces but cannot recognize the specific face information. Figure 3b Schematic diagram of the current display interface for displaying content in the related art Figure 2 ,like Figure 3b As shown, when a user and an acquaintance are detected, the displayed content on the mobile phone screen will still be hidden, causing trouble to the user and failing to meet the user's real needs; that is, the existing anti-peeping screen technology is only applicable to limited scenarios and has low intelligence.
[0034] In order to improve the intelligence of privacy protection of displayed content, in an embodiment of the present application, a video frame sequence is first obtained, wherein, for an electronic device, when a user is using the electronic device, the electronic device turns on the front camera and captures a video frame sequence to achieve privacy protection of the displayed content on the current display interface; further, if the displayed content on the current display interface of the screen contains sensitive information, the electronic device turns on the front camera and captures the video frame sequence, thereby obtaining a video frame sequence to achieve privacy protection of sensitive information in the displayed content on the current display interface. Here, the embodiment of the present application does not make specific limitations on this.
[0035] After obtaining the video frame sequence, in order to obtain video frames that are more in line with the actual situation for processing the display content, here, the video frame sequence is input into the trained neural network model for frame prediction, so that the predicted frames can be obtained. In this way, the trained neural network model is used to perform frame prediction on the video frame sequence, so that the predicted frames are closer to the actual situation, which is conducive to improving the intelligence of display content processing.
[0036] Among them, the number of frames of the predicted frames obtained above can be one or more, and this embodiment of the present application does not make any specific limitation on this.
[0037] It should be pointed out here that the above-mentioned trained neural network model can be obtained by training one discrete model or two discrete models. Here, the embodiments of the present application do not make specific limitations on this.
[0038] S202: extracting facial features from the predicted frame to obtain facial features of the predicted frame;
[0039] After the predicted frame is obtained through S201, face detection is first performed on the predicted frame to obtain the face area of the predicted frame, and then face features are extracted from the face area in the predicted frame to obtain the face features of the predicted frame. For multiple predicted frames, face detection and face tracking are first performed on the predicted frame, and then face features are extracted to obtain the face features of the predicted frame.
[0040] Here, it should be noted that the above-mentioned method for performing face detection on the predicted frame can adopt AI algorithm for face detection, or can adopt traditional face detection method, and the embodiment of the present application does not make specific limitations on this; the above-mentioned method for performing face tracking on the predicted frame can adopt AI algorithm for face detection, or can adopt traditional face tracking method, and the embodiment of the present application does not make specific limitations on this; the above-mentioned method for performing face feature extraction on the face area of the predicted frame can adopt AI algorithm for extracting face features, or can adopt traditional face feature extraction method, and the embodiment of the present application does not make specific limitations on this.
[0041] S203: Based on the matching result between the facial features of the predicted frame and the preset facial features, determine whether to hide the display content on the current display interface on the screen of the electronic device.
[0042] After obtaining the facial features of the predicted frame through S202, the facial features of the predicted frame are matched with the preset facial features to obtain a matching result, and based on the matching result, it is determined whether to hide the display content on the current display interface on the screen of the electronic device.
[0043] Here, it should be noted that, for the case of multiple prediction frames, after obtaining the facial features of each prediction frame, the facial features of each prediction frame are matched with the preset facial features in turn, and based on the matching results, it is determined whether to hide the display content on the current display interface on the screen of the electronic device.
[0044] Among them, the matching results may include successful matching and failed matching. For the case where multiple facial features will be extracted in the predicted frame, a successful match may be that a part of the facial features in the predicted frame successfully matches the preset facial features, or that all the facial features in the predicted frame successfully matches the preset facial features. Here, the embodiments of the present application do not make specific limitations on this.
[0045] In addition, for the matching failure, it may be that at least one facial feature in the predicted frame fails to match the preset facial feature, or it may be that all the facial features in the predicted frame fail to match the preset facial features. Here, the embodiments of the present application do not make specific limitations on this.
[0046] The above-mentioned display content may be all the display content on the current display interface, or may be part of the display content on the current display interface, for example, the display content involving user privacy in the display content; here, the embodiment of the present application does not make any specific limitation on this.
[0047] In order to obtain a trained neural network model, you can use any of the following two methods to train the trained neural network model:
[0048] In an optional embodiment, the trained neural network model is trained in the following manner:
[0049] Acquire a first training data set from a first sample data set;
[0050] Inputting the first training data set into a preset neural network model for training to obtain a trained neural network model;
[0051] Based on the trained neural network model, a trained neural network model is determined.
[0052] It can be understood that a first sample data set is first obtained by acquisition, wherein the first sample data set includes: a captured video frame sequence and N frames following the next frame of the captured video frame sequence, where N is a positive integer greater than or equal to 2; then a first training data set is obtained from the first sample data set, wherein the first training data set is a plurality of groups of data in the first sample data set, the first training data set is input into a preset neural network model for training, and a loss function between a predicted value of the neural network model and a true value in the first training data set is calculated, for example, the root mean square error between the predicted frame and the true frame is calculated, and the neural network model is optimized based on the loss function, thereby obtaining a trained neural network model, and finally the trained neural network model is determined based on the trained neural network model.
[0053] In this way, through the above training, a trained neural network model can be obtained, and the trained neural network model can be stored and used to perform frame prediction on video frames, so as to obtain predicted frames, making the predicted frames closer to the actual situation, which is conducive to improving the intelligence of privacy protection of displayed content.
[0054] In order to improve the accuracy of the trained neural network model, a first verification data set can be obtained from the collected first sample data set, and the video frame sequence collected in the first verification data set is input into the above-mentioned trained neural network model, so that the predicted frame corresponding to the collected video frame sequence can be obtained.
[0055] In order to verify whether the trained neural network model meets the requirements, here, the predicted frames corresponding to the obtained collected video frame sequence are compared with the predicted frames corresponding to the video frame sequence collected in the verification data set to determine whether the two are similar enough (for example, the root mean square error between the two is small enough). If they are similar enough, it means that the accuracy of the trained neural network model meets the requirements, so the verification passes, and the trained neural network model is directly determined as the trained neural network model. If they are different, it means that the accuracy of the trained neural network model does not meet the requirements, so the verification fails, and it is necessary to re-select the first training data set from the first sample data set, and use the re-selected first training data set to continue training the trained neural network model, so as to obtain the re-trained neural network model until the verification passes, so that the trained neural network model can be obtained.
[0056] Among them, the preset neural network model includes: a diffusion model; here, the preset neural network model adopts a diffusion model. The working principle of the diffusion model is to learn the information attenuation caused by noise, and then use the learned pattern to generate an image. In this way, the trained neural network model trained by the diffusion model is used to predict the video frame sequence to obtain a predicted frame that is closer to the actual situation, which is conducive to improving the intelligence of privacy protection of displayed content.
[0057] In another optional embodiment, the trained neural network model is trained in the following manner:
[0058] Acquire a second training data set from a second sample data set; wherein the second sample data set includes: a captured video frame sequence and a next frame of the captured video frame sequence;
[0059] Inputting the second training data set into a preset neural network model for training to obtain a trained first neural network model;
[0060] Based on the trained first neural network model, determining a trained first neural network model;
[0061] Acquire a third training data set from a third sample data set; wherein the third sample data set includes: the next frame of the acquired video frame sequence and N frames after the next frame of the acquired video frame sequence; wherein N is a positive integer greater than or equal to 2;
[0062] Inputting the third training data set into a preset neural network model for training to obtain a trained second neural network model;
[0063] Based on the trained second neural network model, determining a trained second neural network model;
[0064] The trained first neural network model and the trained second neural network model are determined as trained neural network models.
[0065] It can be understood that a second sample data set is first obtained by acquisition, wherein the second sample data set includes: a captured video frame sequence and the next frame of the captured video frame sequence; then a second training data set is obtained from the second sample data set, wherein the second training data set is a plurality of groups of data in the second sample data set, and the second training data set is input into a preset first neural network model for training, and a loss function between a predicted value of the first neural network model and a true value in the second training data set is calculated, for example, the root mean square error between the predicted frame and the true frame is calculated, and the first neural network model is optimized based on the loss function, so as to obtain a trained first neural network model, and finally the trained first neural network model is determined based on the trained first neural network model.
[0066] A third sample data set is then obtained through collection, wherein the third sample data set includes: the next frame of the collected video frame sequence and the N frames after the next frame of the collected video frame sequence, where N is a positive integer greater than or equal to 2; a third training data set is then obtained from the third sample data set, where the third training data set is multiple groups of data in the third sample data set, the third training data set is input into a preset second neural network model for training, and a loss function between a predicted value of the second neural network model and a true value in the third training data set is calculated, for example, a root mean square error between a predicted frame and a true frame is calculated, and the second neural network model is optimized based on the loss function, so that a trained second neural network model can be obtained, and finally a trained second neural network model is determined based on the trained second neural network model.
[0067] In this way, through the above training, a trained first neural network model and a trained second neural network model can be obtained, and the trained first neural network model and the trained second neural network model are determined as trained neural network models and stored, and the trained first neural network model and the trained second neural network model are used to perform frame prediction on the video frames to obtain predicted frames, so that the predicted frames are closer to the actual situation, which is conducive to improving the intelligence of privacy protection of displayed content.
[0068] In order to improve the accuracy of the trained first neural network model, a second verification data set can be obtained from the collected second sample data set, and the video frame sequence collected in the second verification data set is input into the above-mentioned trained first neural network model, so that the next frame corresponding to the collected video frame sequence can be obtained.
[0069] In order to verify whether the trained first neural network model meets the requirements, here, the next frame corresponding to the obtained collected video frame sequence is compared with the next frame corresponding to the video frame sequence collected in the verification data set to determine whether the two are similar enough (for example, the root mean square error between the two is small enough). If they are similar enough, it means that the accuracy of the trained first neural network model meets the requirements, so the verification passes, and the trained first neural network model is directly determined as the trained first neural network model. If they are different, it means that the accuracy of the trained first neural network model does not meet the requirements, so the verification fails, and it is necessary to re-select the second training data set from the second sample data set, and use the re-selected second training data set to continue training the trained first neural network model, so as to obtain the re-trained first neural network model until the verification passes, so that the trained first neural network model can be obtained.
[0070] In order to improve the accuracy of the trained second neural network model, a third verification data set can be obtained from the collected third sample data set, and the next frame of the video frame sequence collected in the third verification data set is input into the above-mentioned trained second neural network model, so that the next N frames corresponding to the next frame of the collected video frame sequence can be obtained.
[0071] In order to verify whether the trained second neural network model meets the requirements, here, the next N frames corresponding to the next frame of the acquired video frame sequence are compared with the next N frames corresponding to the next frame of the video frame sequence acquired in the verification data set to determine whether the two are similar enough (for example, the root mean square error between the two is small enough). If they are similar enough, it means that the accuracy of the trained second neural network model meets the requirements, so the verification passes, and the trained second neural network model is directly determined as the trained second neural network model. If they are different, it means that the accuracy of the trained second neural network model does not meet the requirements, so the verification fails, and it is necessary to re-select the third training data set from the third sample data set, and use the re-selected third training data set to continue training the trained second neural network model, so as to obtain the re-trained second neural network model until the verification passes, so that the trained second neural network model can be obtained.
[0072] Among them, the first neural network model includes: a diffusion model; the second neural network model includes: a diffusion model; the trained first neural network model and the trained second neural network model obtained by training with the diffusion model are used to predict the video frame sequence to obtain a predicted frame that is closer to the actual situation, which is conducive to improving the intelligence of privacy protection of displayed content.
[0073] In addition, in order to further improve the accuracy of the trained neural network model, online training can be performed. In an optional embodiment, the method further includes:
[0074] When a new video frame sequence is generated in the album in the electronic device, data of the first sample data set is collected from the new video frame sequence;
[0075] Alternatively, when a new video frame sequence is generated in the photo album in the electronic device, data of the second sample data set and data of the third sample data set are collected from the new video frame sequence.
[0076] It can be understood that after the trained neural network model is deployed in an electronic device, in order to further improve the accuracy of the trained neural network model, here, when a new video frame sequence is generated in the album of the electronic device, data of the first sample data set can be collected from the new video frame sequence, and the trained neural network model can be trained online using the first sample data set to update the trained neural network model.
[0077] Alternatively, when a new video frame sequence is generated in the photo album of the electronic device, data of the second sample data set and data of the third sample data set can be collected from the new video frame sequence, and the trained neural network model can be trained online using the second sample data set and the third sample data set to update the trained first neural network model and the trained second neural network, thereby obtaining a trained neural network model.
[0078] After obtaining the above-mentioned trained neural network model, for the trained neural network model including the trained first neural network model and the trained second neural network model, in order to achieve prediction, in an optional embodiment, S201 may include:
[0079] Input the video frame sequence into the trained first neural network model to predict the next frame to obtain the next frame of the video frame sequence;
[0080] The next frame of the video frame sequence is input into the trained second neural network model to predict the N frames following the next frame to obtain a predicted frame.
[0081] It can be understood that after obtaining the video frame sequence, the video frame sequence is input into the trained first neural network model to predict the next frame, and the next frame of the video frame sequence is obtained. Then, the next frame of the video frame sequence is input into the trained second neural network model to predict the next N frames after the next frame, and the next N frames after the next frame of the video frame sequence, i.e., the predicted frames, are obtained. It can be seen that the number of predicted frames is N; in this way, the predicted frame can be obtained through the above prediction method.
[0082] In addition, in order to improve the accuracy of the predicted frame, in an optional embodiment, S201 may include:
[0083] Inputting the video frame sequence into the trained first neural network model to perform at least two predictions to obtain at least two next frames of the video frame sequence;
[0084] Performing average processing on at least two next frames of the video frame sequence to obtain a processed next frame;
[0085] The processed next frame is input into the trained second neural network model to predict the N frames following the next frame, and the predicted frame is output.
[0086] It can be understood that after obtaining the video frame sequence, the video frame sequence is input into the trained first neural network model to predict the next frame to obtain the next frame of the video frame sequence. Here, the video frame sequence is predicted at least twice, so that at least two next frames of the video frame sequence can be obtained.
[0087] At least two next frames of the video frame sequence are averaged, that is, the pixel values at corresponding positions on each next frame are averaged to obtain the processed next frame, and then the processed next frame is input into the trained second neural network model to predict the next N frames after the next frame, to obtain the next N frames after the next frame of the video frame sequence, that is, the predicted frames. It can be seen that the number of predicted frames is N; in this way, the accuracy of the predicted frames can be further improved by averaging at least two next frames.
[0088] In order to process sensitive information based on the matching result between the facial features of the predicted frame and the preset facial features, in an optional embodiment, S203 may include:
[0089] When each facial feature in the predicted frame exists in the preset facial features, it is determined that the match is successful, and the display content on the current display interface is displayed on the screen of the electronic device;
[0090] When at least one of the facial features of the predicted frame does not exist in the preset facial features, it is determined that the match fails, and the display content on the current display interface is hidden on the screen.
[0091] It can be understood that after obtaining the facial features of the predicted frame, it is determined whether the facial features of the predicted frame exist in the preset facial features. Usually, when there are multiple facial features in the predicted frame, it is necessary to determine whether each facial feature exists in the preset facial features. When each facial feature in the facial features of the predicted frame exists in the preset facial features, it means that the faces in the predicted frame are all acquaintances to the user of the electronic device. Therefore, it is determined that the match is successful, and the display content on the current display interface is displayed on the screen of the electronic device.
[0092] In addition, when at least one of the facial features in the predicted frame does not exist in the preset facial features, it means that at least one of the faces in the predicted frame is a stranger to the user of the electronic device. Therefore, it is determined that the match has failed, and the display content on the current display interface is hidden on the screen of the electronic device.
[0093] Among them, the preset facial features include: facial features extracted from pictures in the photo album of the electronic device, that is, the facial features extracted from the pictures in the photo album of the electronic device are used as preset facial features, that is, the faces extracted from the pictures in the photo album of the electronic device are determined to be acquaintances, and the others are strangers.
[0094] Furthermore, the preset facial features include: facial features whose number of facial features extracted from pictures in the photo album of the electronic device is greater than a preset threshold, that is, facial features that appear more frequently in the pictures among the facial features extracted from the pictures in the photo album of the electronic device are used as preset facial features, that is, faces whose number of facial features extracted from pictures in the photo album of the electronic device is greater than a preset threshold are determined to be acquaintances, and the others are strangers; wherein, the preset threshold can be set according to needs.
[0095] In this way, acquaintances and strangers are divided through faces extracted from pictures in the album of the electronic device, so that the function of hiding the displayed content in front of strangers can be realized.
[0096] The following example is used to illustrate the method for processing the display content described in one or more of the above embodiments.
[0097] Based on the anti-peeping screen technology in the related art, which uses electronic devices to detect the number of faces appearing in the video frame sequence collected by the front camera to decide whether to hide sensitive information on the screen, there are problems that cannot meet the real expectations of users and have limited applicable scenarios. This example proposes an anti-peeping screen technology that is closer to the real ideas of users, has a wider range of applicable scenarios, and can improve user experience. In this example, the position prediction technology based on time cascade diffusion and the face recognition technology based on neural networks are integrated into the anti-peeping screen technology. In response to the problem of inaccurate recognition of anti-peeping screen technology in related technologies, this example proposes a two-stage method based on diffusion model to improve the recognition accuracy of anti-peeping screen.
[0098] Figure 4 A flowchart of an optional method for processing display content provided in an embodiment of the present application is shown as follows: Figure 4 As shown, the method for processing the displayed content may include:
[0099] S401: Inputting the captured video frame sequence into the short-term face position prediction difference block for prediction to obtain the next frame of the video sequence frame;
[0100] S402: inputting the next frame of the video frame sequence into the long-term face position prediction difference block for prediction, and obtaining N frames after the next frame of the video frame sequence;
[0101] In the related art, for example, electronic devices, such as mobile phones, usually determine whether the screen of the mobile phone is being peeped at based on the number of faces appearing in the video frame sequence captured by the front camera. Simply using face recognition methods to identify whether the screen of the mobile phone is being peeped at lacks certain robustness. For example, in scenes where the relative positions of people and mobile phones change greatly in subways and buses, there will be inaccurate recognition. Therefore, in this example, the time cascade diffusion of short-term face position prediction difference blocks and long-term face position prediction difference blocks is included, and it is added to the preprocessing module before face detection. The method of combining short-term face position prediction difference blocks and long-term face position prediction difference blocks is mainly based on the following considerations:
[0102] The short-term face position prediction difference block is used for short-term posture prediction. This module reads the input video frame sequence X (wherein the video frame sequence includes k-1 frames), predicts and outputs the next frame, which is the kth frame x k The long-term face position prediction difference block inputs the output of the short-term face position prediction difference block, predicts and outputs the N frames after the predicted k-th frame, which is x k+1 …x k+N .
[0103] Furthermore, in the process of inference using the above-mentioned short-term face position prediction differential block and long-term face position prediction differential block, the short-term face position prediction differential block will be used to predict the video frame sequence 5 times, and 5 k-th frames will be predicted. The pixels at corresponding positions in the 5 k-th frames are averaged to obtain the average value of the 5 k-th frames. The average value is input into the long-term face position prediction differential block, and the N frames after the k-th frame are output as predicted frames.
[0104] Thus, due to the effective division of long-term and short-term tasks, this method enables the short-term face position prediction differential block to focus on predicting a small number of samples and improving its accuracy in short-term predictions, and due to its accurate short-term predictions, the long-term face position prediction differential block can focus on more received samples, giving it a longer-term prediction capability. Therefore, the cascade result of the two diffusion models is a better overall and specific long-term prediction, making the trained neural network model more robust in scenarios where the phone is shaking violently.
[0105] Among them, the short-term face position prediction difference block and the long-term face position prediction difference block are obtained by training based on the diffusion model, wherein the short-term face position prediction difference block is obtained by training based on the second sample data set mentioned above, and the long-term face position prediction difference block is obtained by training based on the third sample data set mentioned above. In this way, the two-stage position prediction block has the ability to deal with the problem of partial face occlusion. Specifically, for the diffusion model, Gaussian noise is added to the occluded face in the real video frame sequence X input to the model, and according to the Markov chain, it is repeated t times, as shown in the following formula (1):
[0106]
[0107] Among them, β t ∈(0, 1) is the variance hyperparameter of the Gaussian distribution and increases with t. Then we get the video frame sequence X with added noise m , where the specific video frame x mt ∈X m , including the noise area and the non-occluded area. In the inference stage, the noise added in the forward stage is removed, and the distribution q(x t-1 |x t ), as shown in the following formula (2):
[0108] p θ (x t-1 |x t )=N(x t-1 ;μ θ (x t , t), ∑ θ (x t , t)) (2)
[0109] Among them, θ is the model parameter. After k rounds of iterations, the model finally learns the fitting distribution and outputs the video frame sequence x that fills the occluded face part k+1 …x k+N .
[0110] S403: inputting the predicted frame into a face detection and face tracking module to obtain a predicted frame with a face boundary frame;
[0111] S404: inputting the predicted frame with the face bounding box into the face recognition module to obtain the predicted frame with the face bounding box and the face label;
[0112] S405: Input the predicted frame with the face bounding box and the face label to the video frame quality judgment module to judge whether the video frame quality meets the requirements. If yes, execute S406; if no, execute S402;
[0113] S406: Compare the face tag with the face tag extracted from the picture in the mobile phone album to determine whether each face tag in the face tag exists in the face tag extracted from the picture in the mobile phone album. If yes, execute S407; if not, execute S408;
[0114] S407: Display sensitive information on the screen of the mobile phone.
[0115] S408: Hide sensitive information on the screen of the mobile phone.
[0116] In S403-S408, after the temporal cascade difference block completion outputs the face position in the preprocessing step, the predicted video frame x with the face position k+1 …x k+N ∈X p It will be used as the input of the face detection, face tracking and face recognition modules. The face detection, face tracking and face recognition modules are implemented using a neural network method. The face recognition module will continuously track the faces in the input video frame and output the corresponding feature map, namely the face label. Thereafter, the face recognition module will analyze the face feature map segmented by the face tracking module to obtain the analysis result, which is a plurality of face labels. The face label must be compared with the existing face labels in the mobile phone security zone, and then the anti-peep screen system will decide whether to hide the sensitive information on the screen based on the comparison result. The entire anti-peep screen processing process will continuously iterate and select video frames containing more valid information for analysis, ignoring blurred video frames, so as to improve the accuracy and robustness of the final recognition result. The basic formula of the neural network is given by the following formula (3):
[0117] a l =σ(z l )=σ(a l-1 *W l +b l ) (3)
[0118] Where l represents the number of layers, * represents the convolution operation, b represents the bias term, and σ represents the activation function. During the training process, the gradient is updated by the back propagation algorithm based on the chain rule of formula (4). In the inference stage, in order to meet the low power consumption requirements of embedded devices, model optimization algorithms such as quantization and pruning are also introduced; where formula (4) is as follows:
[0119]
[0120] Among them, in this example, the sensitive information in the display content on the current display interface is mainly used to determine whether to hide the information. Therefore, in S407, when it is determined that there is a stranger, the sensitive information is displayed on the screen of the mobile phone. In S408, when it is determined that there is an acquaintance, the sensitive information is hidden on the screen of the mobile phone.
[0121] Among them, the above-mentioned sensitive information refers to all information that may be detrimental to the implementation of national interest plans or to the personal privacy rights enjoyed by individuals according to law if used improperly or accessed or modified without authorization, such as ID number, bank card number, account balance, deposit amount, expenditure amount and account income.
[0122] It should be pointed out that the entire training and reasoning process is divided into two stages: pre-training and online learning. During the pre-training process, the short-term and long-term face position prediction difference blocks, face detection, face tracking and recognition neural network models will be trained on the server with a large amount of face data, and the resulting pre-trained models will be deployed in advance to the mobile phone. When the mobile phone begins to be used, over time, the mobile phone album may store a variety of pictures captured by the camera, including face pictures. At this time, the pictures containing faces are screened out from the pictures generated by the mobile phone camera and input into the short-term and long-term face position prediction difference blocks and neural network modules as data for subsequent online learning. When the photos of certain specific people in the mobile phone album exceed the preset threshold, the anti-peep screen system will add them to the category of acquaintances (equivalent to the above-mentioned preset facial features). When a specific person is judged as an acquaintance, the anti-peep screen system will choose to partially hide or not hide the sensitive information on the screen based on its face label when it is detected on the screen next time.
[0123] Figure 5a A schematic diagram of an optional current display interface for sensitive information provided in an embodiment of the present application Figure 1 ,like Figure 5a As shown, corresponding to S408, in the case where the facial features include the mobile phone user and the stranger, the sensitive information is hidden; Figure 5b A schematic diagram of an optional current display interface for sensitive information provided in an embodiment of the present application Figure 2 ,like Figure 5b As shown, corresponding to S407, when the facial features include the mobile phone user and an acquaintance, sensitive information is displayed; in this way, due to continuous online learning, the mobile phone anti-peeping screen system can more intelligently display / hide sensitive information on the screen and improve the anti-peeping screen effect.
[0124] Among them, the target tracking algorithm used in the above-mentioned face tracking can be a yolo target tracking algorithm, and the face recognition algorithm can be replaced from a neural network method to other face recognition algorithms such as feature faces, which is not specifically limited in the embodiments of the present application.
[0125] It can be seen that this example solves the problem that the current mobile phone anti-peeping screen recognition standard is too single. The anti-peeping screen system can choose to display or hide sensitive information on the screen according to the specific face. The preset threshold can be adjusted to make the anti-peeping screen system flexible and adaptable to different scenarios. After the deployment on the mobile phone is completed, the relevant parameters can be automatically optimized to improve the accuracy. The addition of a long-term and short-term face position prediction module makes face recognition more robust to different scenarios. The entire module is scalable and can replace a lighter neural network, etc., which meets the needs of multiple scenarios.
[0126] An embodiment of the present application provides a method for processing display content, including: inputting a video frame sequence into a trained neural network model for frame prediction to obtain a predicted frame, performing face detection and face feature extraction on the predicted frame to obtain the face features of the predicted frame, and determining whether to hide the display content on the current display interface on the screen of the electronic device based on the matching results of the face features of the predicted frame and preset face features; that is, in an embodiment of the present application, a video frame sequence is predicted by using a trained neural network model to obtain a predicted frame, and determining whether to hide the display content based on the matching results of the face features of the predicted frame and preset face features. In this way, the use of an AI algorithm in advance can achieve a prediction of the video frame sequence that is more consistent with the actual situation, thereby improving the accuracy of the predicted frame. Then, whether to hide the display content is determined based on the predicted frame with higher accuracy, thereby improving the intelligence of privacy protection of the display content.
[0127] Based on the same inventive concept, the embodiment of the present application provides a display content processing device, Figure 6 A schematic diagram of the structure of an optional display content processing device provided in an embodiment of the present application, such as Figure 6 As shown, the display content processing device may include:
[0128] Prediction module 61, used for inputting the video frame sequence into the trained neural network model to perform frame prediction and obtain predicted frames;
[0129] A processing module 62 is used to extract facial features from the prediction frame to obtain facial features of the prediction frame;
[0130] The determination module 63 is used to determine whether to hide the display content on the current display interface on the screen of the electronic device based on the matching result between the facial features of the predicted frame and the preset facial features.
[0131] In an optional embodiment, the device further includes a first training module, which is used to:
[0132] Acquire a first training data set from a first sample data set; wherein the first sample data set includes: a captured video frame sequence and N frames following the next frame of the captured video frame sequence; wherein N is a positive integer greater than or equal to 2;
[0133] Inputting the first training data set into a preset neural network model for training to obtain a trained neural network model;
[0134] Based on the trained neural network model, a trained neural network model is determined.
[0135] In an optional embodiment, the preset neural network model includes: a diffusion model.
[0136] In an optional embodiment, the device further includes a first training module, which is used to:
[0137] Acquire a second training data set from a second sample data set; wherein the second sample data set includes: a captured video frame sequence and a next frame of the captured video frame sequence;
[0138] Inputting the second training data set into a preset neural network model for training to obtain a trained first neural network model;
[0139] Based on the trained first neural network model, determining a trained first neural network model;
[0140] Acquire a third training data set from a third sample data set; wherein the third sample data set includes: the next frame of the acquired video frame sequence and N frames after the next frame of the acquired video frame sequence; wherein N is a positive integer greater than or equal to 2;
[0141] Inputting the third training data set into a preset neural network model for training to obtain a trained second neural network model;
[0142] Based on the trained second neural network model, determining a trained second neural network model;
[0143] The trained first neural network model and the trained second neural network model are determined as trained neural network models.
[0144] In an optional embodiment, the first neural network model includes: a diffusion model; the second neural network model includes: a diffusion model.
[0145] In an optional embodiment, the device is also used for:
[0146] When a new video frame sequence is generated in the album in the electronic device, data of the first sample data set is collected from the new video frame sequence;
[0147] Alternatively, when a new video frame sequence is generated in the photo album in the electronic device, data of the second sample data set and data of the third sample data set are collected from the new video frame sequence.
[0148] In an optional embodiment, the prediction module 61 is specifically configured to:
[0149] Input the video frame sequence into the trained first neural network model to predict the next frame to obtain the next frame of the video frame sequence;
[0150] Input the next frame of the video frame sequence into the trained second neural network model to predict the N frames after the next frame to obtain a predicted frame;
[0151] The trained neural network model includes a trained first neural network model and a trained second neural network model.
[0152] In an optional embodiment, the prediction module 61 is specifically configured to:
[0153] Inputting the video frame sequence into the trained first neural network model to perform at least two predictions to obtain at least two next frames of the video frame sequence;
[0154] Performing average processing on at least two next frames of the video frame sequence to obtain a processed next frame;
[0155] The processed next frame is input into the trained second neural network model to predict the N frames after the next frame, and the predicted frame is output;
[0156] The trained neural network model includes a trained first neural network model and a trained second neural network model.
[0157] In an optional embodiment, the determination module 63 is specifically configured to:
[0158] When each facial feature in the predicted frame exists in the preset facial features, it is determined that the match is successful, and the display content on the current display interface is displayed on the screen of the electronic device;
[0159] When at least one of the facial features of the predicted frame does not exist in the preset facial features, it is determined that the match fails, and the display content on the current display interface is hidden on the screen.
[0160] In an optional embodiment, the preset facial features include: facial features whose number of facial features extracted from pictures in an album of the electronic device is greater than a preset threshold.
[0161] In practical applications, the prediction module 61, processing module 62 and determination module 63 can be implemented by a processor located on a processing device for displaying content, specifically a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA).
[0162] The embodiment of the present application provides a chip, Figure 7 A schematic diagram of the structure of an optional chip provided in an embodiment of the present application, such as Figure 7 As shown, the chip 700 includes: a processor 71, which is used to call and run a computer program from a memory 72 to execute the display content processing method as described in one or more of the above embodiments.
[0163] Alternatively, if Figure 7 As shown, the chip 700 may also include a memory 72. The processor 71 may call and run a computer program from the memory 72 to implement the method in the embodiment of the present application. The memory 72 may be a separate device independent of the processor 71, or may be integrated in the processor 71. Optionally, the chip 700 may also include an input interface 73. The processor 81 may control the input interface 73 to communicate with other devices or chips, specifically, to obtain information or data sent by other devices or chips. Optionally, the chip 700 may also include an output interface 73. The processor 71 may control the output interface 74 to communicate with other devices or chips, specifically, to output information or data to other devices or chips.
[0164] An embodiment of the present application provides an electronic device, Figure 8 A schematic diagram of the structure of an optional electronic device provided in an embodiment of the present application, such as Figure 8 As shown, the electronic device 800 includes: a processor 81 and a storage medium 82 storing instructions executable by the processor; the storage medium 82 relies on the processor 81 to perform operations through a communication bus 83, and when the instructions are executed by the processor, the sensitive information processing method executed on the processor side in one or more of the above-mentioned embodiments is executed.
[0165] It should be noted that in actual application, the various components in the terminal are coupled together through the communication bus 83. It is understandable that the communication bus 83 is used to realize the connection and communication between these components. In addition to the data bus, the communication bus 83 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 8 Various buses are labeled as communication buses 83.
[0166] An embodiment of the present application provides a computer storage medium storing executable instructions. When the executable instructions are executed by one or more processors, the processors execute the sensitive information processing method described in one or more of the above embodiments.
[0167] Among them, the computer-readable storage medium can be a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM) and other memories.
[0168] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.
[0169] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0170] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0171] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0172] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.
Claims
1. A method for processing display content, characterized in that: The method is applied to an electronic device, comprising: Input the video frame sequence into the trained neural network model for frame prediction to obtain the predicted frame; Extracting facial features from the predicted frame to obtain facial features of the predicted frame; Based on the matching result between the facial features of the predicted frame and the preset facial features, it is determined whether to hide the display content on the current display interface on the screen of the electronic device.
2. The method according to claim 1, characterized in that The trained neural network model is trained in the following way: Acquire a first training data set from a first sample data set; wherein the first sample data set includes: a captured video frame sequence and N frames following the next frame of the captured video frame sequence; wherein N is a positive integer greater than or equal to 2; Inputting the first training data set into a preset neural network model for training to obtain a trained neural network model; Based on the trained neural network model, the trained neural network model is determined.
3. The method according to claim 2, characterized in that The preset neural network model includes: a diffusion model.
4. The method according to claim 1, characterized in that: The trained neural network model is trained in the following way: Acquire a second training data set from a second sample data set; wherein the second sample data set includes: a captured video frame sequence and a next frame of the captured video frame sequence; Inputting the second training data set into a preset neural network model for training to obtain a trained first neural network model; Based on the trained first neural network model, determining a trained first neural network model; Acquire a third training data set from a third sample data set; wherein the third sample data set includes: the next frame of the acquired video frame sequence and N frames after the next frame of the acquired video frame sequence; wherein N is a positive integer greater than or equal to 2; Inputting the third training data set into a preset neural network model for training to obtain a trained second neural network model; Determining a trained second neural network model based on the trained second neural network model; The trained first neural network model and the trained second neural network model are determined as the trained neural network models.
5. The method according to claim 4, characterized in that The first neural network model includes: a diffusion model; the second neural network model includes: a diffusion model.
6. The method according to claim 2 or 4, characterized in that: The method further comprises: When a new video frame sequence is generated in the photo album in the electronic device, collecting data of the first sample data set from the new video frame sequence; Alternatively, when a new video frame sequence is generated in the photo album in the electronic device, data of the second sample data set and data of the third sample data set are collected from the new video frame sequence.
7. The method according to claim 1, characterized in that The step of inputting the video frame sequence into the trained neural network model to perform frame prediction to obtain a predicted frame includes: Inputting the video frame sequence into the trained first neural network model to predict the next frame, thereby obtaining the next frame of the video frame sequence; Inputting the next frame of the video frame sequence into the trained second neural network model to predict the N frames following the next frame, to obtain the predicted frame; The trained neural network model includes the trained first neural network model and the trained second neural network model.
8. The method according to claim 1, characterized in that: The step of inputting the video frame sequence into the trained neural network model to perform frame prediction to obtain a predicted frame includes: Inputting the video frame sequence into a trained first neural network model to perform at least two predictions to obtain at least two next frames of the video frame sequence; Performing average processing on at least two next frames of the video frame sequence to obtain a processed next frame; Inputting the processed next frame into the trained second neural network model to predict the N frames after the next frame, and outputting the predicted frame; The trained neural network model includes the trained first neural network model and the trained second neural network model.
9. The method according to claim 1, characterized in that: Determining whether to hide the content on the current display interface on the screen of the electronic device according to the matching result between the facial feature of the predicted frame and the preset facial feature includes: When each facial feature in the predicted frame exists in the preset facial features, it is determined that the match is successful, and the display content on the current display interface is displayed on the screen of the electronic device; When at least one of the facial features of the predicted frame does not exist in the preset facial features, it is determined that the match fails, and the display content on the current display interface is hidden on the screen.
10. The method according to claim 1, characterized in that The preset facial features include: facial features whose number of facial features extracted from pictures in the album of the electronic device is greater than a preset threshold.
11. A display content processing device, characterized in that: The device is arranged in an electronic device and comprises: A prediction module is used to input a video frame sequence into a trained neural network model for frame prediction to obtain a predicted frame; A processing module, used for extracting facial features from the prediction frame to obtain facial features of the prediction frame; The determination module is used to determine whether to hide the display content on the current display interface on the screen of the electronic device based on the matching result between the facial features of the predicted frame and the preset facial features.
12. A chip, characterized in that: It is characterized in that The invention comprises: a processor, configured to call and run a computer program from a memory, and execute the method for processing display content as claimed in any one of claims 1 to 10.
13. An electronic device, characterized in that: include: A processor and a storage medium storing instructions executable by the processor, wherein the storage medium relies on the processor to perform operations via a communication bus, and when the instructions are executed by the processor, the method for processing display content described in any one of claims 1 to 10 is executed.
14. A computer storage medium, characterized in that: Executable instructions are stored, and when the executable instructions are executed by one or more processors, the processors execute the method for processing display content as described in any one of claims 1 to 10.
