Electronic device anti-leakage display method, system and electronic device

Abstract

The present application discloses an anti-leakage display method, system and electronic device for electronic devices, which divide the screen image into regions and adopt a convolutional neural network model and a converter-based context coding model to fully utilize the correlation information between the high-dimensional image features of each region, thereby achieving global classification. Further, by introducing a meta-weighted vector based on a comparison consistency factor to weight the semantic feature vector, the consistency between the obtained semantic feature vector and the original feature map is ensured, thereby improving the fidelity of the original content information as much as possible. In this way, the accuracy of identifying the theme of the displayed content can be improved, thereby avoiding leakage caused by the related content not being hidden.

Description

Technical Field

[0001] The present invention relates to the field of anti-peeping of electronic devices, and more specifically, to an anti-leakage display method and system for electronic devices and electronic devices. Background Art

[0002] At present, with the development of display technology and network technology, more and more people are shopping or conducting account transactions on the Internet. During the above operations, operators often need to enter personal information on display devices such as computers, mobile phones, ATMs, and automatic ticket machines, which can easily cause personal information leakage. Therefore, the anti-voyeurism property of display devices has received more and more widespread attention, in an effort to ensure the security of operators' personal information.

[0003] However, not all information on the display interface of an electronic device needs to be hidden or displayed in a way that is invisible to the human eye. For example, in some special industries, when transmitting confidential information, it is expected that the display of confidential information will be anti-peeping, while the display of other information is normal. For example, information showing the progress of information transmission does not need to be anti-peeping.

[0004] Therefore, in order to improve the accuracy of identifying the subject of the displayed content and to better realize the anti-peeping function of the display device, an anti-leakage display solution for an electronic device is desired. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiments of the present application provide an anti-leakage display method, system and electronic device for an electronic device, which divides the screen image into regions, and adopts a convolutional neural network model and a converter-based context coding model to fully utilize the correlation information between the high-dimensional image features of each region, thereby realizing global classification. Further, by introducing a meta-weighted vector based on a comparison consistency factor to weight the semantic feature vector, the consistency between the obtained semantic feature vector and the original feature map is ensured, thereby improving the fidelity of the original content information as much as possible. In this way, the accuracy of identifying the subject of the displayed content can be improved, thereby avoiding leakage caused by the related content not being hidden.

[0006] According to one aspect of the present application, a method for preventing leakage display of an electronic device is provided, which includes:

[0007] Inputting the acquired screen image of the electronic device into a convolutional neural network to obtain a display feature map;

[0008] Dividing the screen image into regions based on screen display content and mapping the positions of the respective regions in the screen image into the display feature map to obtain respective sub-feature maps corresponding to the screen content of the respective regions;

[0009] Vectorizing each of the sub-feature graphs to obtain a plurality of sub-feature vectors;

[0010] Inputting the plurality of sub-feature vectors into a transformer-based context encoding model to obtain a plurality of semantic feature vectors;

[0011] Performing global mean pooling on each of the sub-feature maps along the channel dimension direction to obtain a feature matrix corresponding to each of the sub-feature maps;

[0012] Performing matrix multiplication on each of the feature matrices and the semantic feature vector corresponding thereto to map the semantic feature vector into the feature space of the feature matrix to obtain a plurality of comparison matrices;

[0013] Calculating consistency factors of each of the comparison matrices to obtain a plurality of consistency factors, wherein the consistency factors are related to the maximum eigenvalue in the comparison matrix and the size of the comparison matrix;

[0014] constructing a plurality of meta-weight vectors using the plurality of consistency factors;

[0015] Performing positional point multiplication of the meta-weighted vector and the semantic feature vector corresponding thereto to obtain a plurality of classification feature vectors;

[0016] Inputting the plurality of classification feature vectors into a classifier respectively to obtain a plurality of classification results, wherein each of the classification results is used to indicate whether the screen content in the corresponding area is private content; and

[0017] Based on the multiple classification results, the display mode of each area of ​​the screen of the electronic device is controlled.

[0018] In the above-mentioned electronic device anti-leakage display method, the acquired screen image of the electronic device is input into a convolutional neural network to obtain a display feature map, including: the convolutional neural network spatially encodes the screen image according to the following formula to obtain the display feature map;

[0019] Wherein, the formula is:

[0020] f i =active(N i ×f i-1 +B i )

[0021] Among them, f i-1 is the input of the i-th layer of convolutional neural network, f i is the output of the i-th layer of convolutional neural network, N i is the filter of the i-th layer of convolutional neural network, and B iis the bias matrix of the i-th layer convolutional neural network, and Active represents the nonlinear activation function.

[0022] In the above-mentioned anti-leakage display method of the electronic device, vectorization is performed on each of the sub-feature graphs to obtain a plurality of sub-feature vectors, including: performing vector projection along a specific dimension on each of the sub-feature graphs to obtain the sub-feature vectors.

[0023] In the above-mentioned electronic device anti-leakage display method, calculating the consistency factor of each comparison matrix to obtain multiple consistency factors includes: calculating the consistency factor of each comparison matrix to obtain multiple consistency factors using the following formula, wherein the formula is:

[0024]

[0025] where λ max is the largest eigenvalue of the comparison matrix, and m is the size of the comparison matrix.

[0026] In the above-mentioned electronic device anti-leakage display method, using the multiple consistency factors to construct multiple element weighted vectors includes: using the multiple consistency factors to construct the multiple element weighted vectors according to the following formula; wherein the formula is:

[0027]

[0028] where v e is the eigenvalue λ max The corresponding feature vector.

[0029] In the above-mentioned anti-leakage display method of the electronic device, the multiple classification feature vectors are respectively input into the classifier to obtain multiple classification results, including: the classification feature vectors are respectively input into the Softmax classification function of the classifier to obtain a first probability that the screen content of the corresponding area belonging to the classification feature vector is private content and a second probability that the screen content of the corresponding area belonging to the classification feature vector is non-privacy content; and, based on a comparison between the first probability and the second probability, determining the classification result.

[0030] In the above-mentioned anti-leakage display method of the electronic device, based on the multiple classification results, the display mode of each area of ​​the screen of the electronic device is controlled, including: in response to the classification result that the screen content of the corresponding area is private content, the pixels of the corresponding area of ​​the screen of the electronic device are controlled to operate in the second mode; and, in response to the classification result that the screen content of the corresponding area is non-privacy content, the pixels of the corresponding area of ​​the screen of the electronic device are controlled to operate in the first mode.

[0031] According to another aspect of the present application, there is provided an anti-leakage display system for an electronic device, comprising:

[0032] A display feature map generating unit, used for inputting the acquired screen image of the electronic device into a convolutional neural network to obtain a display feature map;

[0033] a mapping unit, configured to divide the screen image obtained by the display feature map generating unit into regions based on screen display content and map the positions of the respective regions in the screen image to the display feature map obtained by the display feature map generating unit to obtain respective sub-feature maps corresponding to the screen content of the respective regions;

[0034] A vectorization unit, used for vectorizing the sub-feature maps obtained by each of the mapping units to obtain a plurality of sub-feature vectors;

[0035] A converter processing unit, configured to input the plurality of sub-feature vectors obtained by the vectorization unit into a converter-based context coding model to obtain a plurality of semantic feature vectors;

[0036] A global pooling unit, used for performing global mean pooling along the channel dimension direction on the sub-feature maps obtained by each of the mapping units to obtain a feature matrix corresponding to each of the sub-feature maps;

[0037] a matrix multiplication unit, configured to perform matrix multiplication on the feature matrix obtained by each of the global pooling units and the semantic feature vector obtained by the corresponding converter processing unit to map the semantic feature vector into a feature space of the feature matrix to obtain a plurality of comparison matrices;

[0038] A consistency factor calculation unit, used for calculating the consistency factors of the comparison matrices obtained by the matrix multiplication units to obtain a plurality of consistency factors, wherein the consistency factors are related to the maximum eigenvalue in the comparison matrix and the size of the comparison matrix;

[0039] a meta-weighted vector construction unit, configured to construct a plurality of meta-weighted vectors using the plurality of consistency factors obtained by the consistency factor calculation unit;

[0040] a classification feature vector generating unit, configured to perform positional point multiplication on the meta-weighted vector obtained by the meta-weighted vector constructing unit and the semantic feature vector obtained by the corresponding converter processing unit to obtain a plurality of classification feature vectors;

[0041] a classification unit, configured to input the plurality of classification feature vectors obtained by the classification feature vector generation unit into a classifier to obtain a plurality of classification results, wherein each of the classification results is used to indicate whether the screen content in a corresponding area is private content; and

[0042] A control unit is used to control the display mode of each area of ​​the screen of the electronic device based on the multiple classification results obtained by the classification unit.

[0043] In the above-mentioned anti-leakage display system of the electronic device, the display feature map generating unit is further used for: the convolutional neural network spatially encodes the screen image according to the following formula to obtain the display feature map; wherein the formula is:

[0044] f i =active(N i ×f i-1 +B i )

[0045] Among them, f i-1 is the input of the i-th layer of convolutional neural network, f i is the output of the i-th layer of convolutional neural network, N i is the filter of the i-th layer of convolutional neural network, and B i is the bias matrix of the i-th layer convolutional neural network, and Active represents the nonlinear activation function.

[0046] In the anti-leakage display system of the electronic device, the vectorization unit is further used to: perform vector projection along a specific dimension on each of the sub-feature graphs to obtain the sub-feature vector.

[0047] In the above-mentioned electronic device anti-leakage display system, the consistency factor calculation unit is further used to: calculate the consistency factor of each comparison matrix according to the following formula to obtain multiple consistency factors, wherein the formula is:

[0048]

[0049] where λ max is the largest eigenvalue of the comparison matrix, and m is the size of the comparison matrix.

[0050] In the above electronic device anti-leakage display system, the element weighted vector construction unit is further used to: use the multiple consistency factors to construct the multiple element weighted vectors according to the following formula; wherein the formula is:

[0051]

[0052] where v e is the eigenvalue λ max The corresponding feature vector.

[0053] In the above-mentioned anti-leakage display system of the electronic device, the classification unit is further used to: input the classification feature vectors into the Softmax classification function of the classifier respectively to obtain a first probability that the screen content of the corresponding area to which the classification feature vector belongs is private content and a second probability that the screen content of the corresponding area to which the classification feature vector belongs is non-private content; and determine the classification result based on a comparison between the first probability and the second probability.

[0054] In the above-mentioned anti-leakage display system of the electronic device, the control unit is further used to: in response to the classification result that the screen content of the corresponding area is private content, control the pixels of the corresponding area of ​​the screen of the electronic device to operate in the second mode; and in response to the classification result that the screen content of the corresponding area is non-privacy content, control the pixels of the corresponding area of ​​the screen of the electronic device to operate in the first mode.

[0055] According to another aspect of the present application, an electronic device is provided, comprising: a processor; and a memory, in which computer program instructions are stored, and when the computer program instructions are executed by the processor, the processor executes the anti-leakage display method of the electronic device as described above.

[0056] According to yet another aspect of the present application, a computer-readable medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the anti-leakage display method of the electronic device as described above.

[0057] Compared with the prior art, the anti-leakage display method, system and electronic device provided by the present application divide the screen image into regions, and adopt a convolutional neural network model and a converter-based context coding model to fully utilize the correlation information between the high-dimensional image features of each region, thereby achieving global classification. Further, by introducing a meta-weighted vector based on a comparison consistency factor to weight the semantic feature vector, the consistency between the obtained semantic feature vector and the original feature map is ensured, thereby improving the fidelity of the original content information as much as possible. In this way, the accuracy of identifying the subject of the displayed content can be improved, thereby avoiding leakage caused by the related content not being hidden. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0059] Figure 1 This is a diagram of an application scenario of the anti-leakage display method of an electronic device according to an embodiment of the present application;

[0060] Figure 2 is a flow chart of a method for preventing leakage display of an electronic device according to an embodiment of the present application;

[0061] Figure 3 A schematic diagram of the system architecture of the anti-leakage display method of the electronic device according to an embodiment of the present application;

[0062] Figure 4 is a block diagram of an anti-leakage display system for an electronic device according to an embodiment of the present application;

[0063] Figure 5 is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0064] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0065] Scenario Overview

[0066] As mentioned above, at present, with the development of display technology and network technology, more and more people are shopping or conducting account transactions on the Internet. During the above operations, operators often need to enter personal information on display devices such as computers, mobile phones, ATMs, and automatic ticket machines, which can easily cause personal information leakage. Therefore, the anti-voyeurism property of display devices has received more and more widespread attention, in an effort to ensure the security of the operator's personal information.

[0067] However, not all information on the display interface of an electronic device needs to be hidden or displayed in a way that cannot be seen by human eyes. For example, in some special industries, when confidential information is transmitted, it is expected that the display of confidential information is anti-peeping, while the display of other information is normal, for example, the information showing the progress of information transmission does not need to be anti-peeping. Therefore, in order to improve the accuracy of identifying the subject of the displayed content and better realize the anti-peeping function of the display device, an anti-leakage display solution for electronic devices is desired.

[0068] Accordingly, in the embodiment of the present application, a special display screen is firstly used, including: a substrate, a plurality of pixel units arranged in an array on the substrate; each pixel unit includes: a first self-luminous device and a second self-luminous device stacked in each sub-pixel region, and a quantum dot light-emitting device arranged in at least two sub-pixel regions; and the light-emitting colors of the quantum dot light-emitting devices in different sub-pixel regions are different. In one working mode of the display screen, when the first self-luminous device is turned on and the second self-luminous device is turned off, a normal picture is displayed. When the first self-luminous device and the second self-luminous device are turned on at the same time, an interference picture is displayed.

[0069] In the technical solution of the present application, if part of the content in the screen is private content, the pixel units in the corresponding area operate in the second mode; if part of the content in the screen is non-private content, the pixel units in the corresponding area operate in the first mode, thereby achieving intelligent display control at the pixel level.

[0070] Accordingly, in the technical solution of the present application, the content displayed on the screen can be divided into regions, and the content of each region can be extracted. And considering that when the subject of the content displayed on the screen is identified, it is necessary to make full use of the association information between the content subject of the region and the content subject of other regions, it is obvious that the accuracy of subject identification can be improved through such association, and leakage caused by the related content not being hidden can be avoided. That is, when the content of the screen area is identified as sensitive information, the second display mode is used, and when the content of the screen area is identified as non-sensitive information, the first display mode is used.

[0071] Based on this, in the technical solution of the present application, first, the screen image is input into a convolutional neural network to obtain a feature map, and then the screen image is divided into regions to obtain each sub-feature map corresponding to the content of each region. Then, each sub-feature map is vectorized to obtain a plurality of sub-feature vectors. For example, the feature map can be converted into a feature vector by vector projection, or further input into a convolutional neural network as a filter to obtain a feature vector. Furthermore, the plurality of sub-feature vectors are input into a context encoding model based on a converter to obtain a plurality of semantic feature vectors.

[0072] At the same time, in order to maintain the consistency between each sub-feature map and its corresponding semantic feature vector, a meta-weight vector is constructed using a comparison consistency factor to weight the semantic feature vector. Specifically, the sub-feature map is first averaged in the channel direction to obtain a feature matrix, and then the feature matrix is ​​multiplied by the semantic feature vector to obtain a comparison matrix, and the consistency factor of the comparison matrix is ​​calculated:

[0073]

[0074] where λ max is the largest eigenvalue of the comparison matrix, and m is the size of the comparison matrix, then the element weight vector is:

[0075]

[0076] where v e is the eigenvalue λ max The corresponding feature vector.

[0077] Then, the meta-weighted vector is point-multiplied with the semantic feature vector to obtain a classification feature vector, which is then input into the classifier to obtain the classification result.

[0078] It should be understood that by converting the sub-feature maps of each area of ​​the feature map into feature vectors and then performing context-based encoding, the semantic feature vector can contain the context information between the feature vectors, thereby making full use of the correlation information between the high-dimensional image features of each area and realizing global classification.

[0079] Moreover, by introducing a meta-weighted vector based on a comparative consistency factor to weight the semantic feature vector, it is ensured that the semantic feature vector obtained after vector projection and context encoding is still consistent with the original feature map. That is, while extracting related information, the fidelity of the original content information is improved as much as possible.

[0080] Based on this, the present application proposes an anti-leakage display method for an electronic device, which includes: inputting the acquired screen image of the electronic device into a convolutional neural network to obtain a display feature map; dividing the screen image into regions based on the screen display content and mapping the positions of each region in the screen image to the display feature map to obtain each sub-feature map corresponding to the screen content of each region; vectorizing each of the sub-feature maps to obtain a plurality of sub-feature vectors; inputting the plurality of sub-feature vectors into a converter-based context coding model to obtain a plurality of semantic feature vectors; performing global mean pooling along the channel dimension direction on each of the sub-feature maps to obtain a feature matrix corresponding to each of the sub-feature maps; and combining each of the feature matrices and the semantic feature vectors corresponding thereto. The invention relates to a method for controlling the display mode of each area of ​​the screen of the electronic device, comprising: performing matrix multiplication on the semantic feature vector to map the semantic feature vector into the feature space of the feature matrix to obtain a plurality of comparison matrices; calculating the consistency factor of each comparison matrix to obtain a plurality of consistency factors, wherein the consistency factor is related to the maximum eigenvalue in the comparison matrix and the size of the comparison matrix; using the plurality of consistency factors to construct a plurality of meta-weighted vectors; performing positional point multiplication on the meta-weighted vectors and the corresponding semantic feature vectors to obtain a plurality of classification feature vectors; inputting the plurality of classification feature vectors into a classifier to obtain a plurality of classification results, wherein each classification result is used to indicate whether the screen content in the corresponding area is private content; and controlling the display mode of each area of ​​the screen of the electronic device based on the plurality of classification results.

[0081] Figure 1 The figure shows an application scenario diagram of the anti-leakage display method of the electronic device according to the embodiment of the present application. Figure 1 As shown, in this application scenario, first, an electronic device (for example, Figure 1 A display screen (e.g., as shown in FIG. Figure 1 The image P) shown in the figure, wherein the electronic device includes but is not limited to a computer, a mobile phone, an ATM, an automatic ticket machine, etc. Then, the screen image of the electronic device is input into a server (for example, Figure 1 S), wherein the server can process the screen image of the electronic device with the anti-leakage display algorithm of the electronic device to generate a classification result indicating whether the screen content in the corresponding area is private content. Further, based on the multiple classification results, the display mode of each area of ​​the screen of the electronic device is controlled.

[0082] Specifically, in response to the classification result that the screen content of the corresponding area is private content, the pixels of the corresponding area of ​​the screen of the electronic device are controlled to work in the second mode; in response to the classification result that the screen content of the corresponding area is non-private content, the pixels of the corresponding area of ​​the screen of the electronic device are controlled to work in the first mode. In this way, the accuracy of identifying the subject of the displayed content can be improved, thereby avoiding leakage caused by the related content not being hidden, so that intelligent display control at the pixel level can be achieved.

[0083] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0084] Exemplary Methods

[0085] Figure 2 The figure shows a flow chart of a method for preventing leakage of electronic equipment. Figure 2 As shown, according to the embodiment of the present application, the anti-leakage display method of the electronic device includes: S110, inputting the acquired screen image of the electronic device into a convolutional neural network to obtain a display feature map; S120, dividing the screen image into regions based on the screen display content and mapping the positions of each region in the screen image to the display feature map to obtain each sub-feature map corresponding to the screen content of each region; S130, vectorizing each of the sub-feature maps to obtain a plurality of sub-feature vectors; S140, inputting the plurality of sub-feature vectors into a converter-based context coding model to obtain a plurality of semantic feature vectors; S150, performing global mean pooling along the channel dimension direction on each of the sub-feature maps to obtain a feature matrix corresponding to each of the sub-feature maps; S160, combining each of the feature matrices with the semantic feature vectors corresponding thereto. S170, calculating the consistency factor of each comparison matrix to obtain a plurality of consistency factors, wherein the consistency factor is related to the maximum eigenvalue in the comparison matrix and the size of the comparison matrix; S180, using the plurality of consistency factors to construct a plurality of meta-weighted vectors; S190, performing positional point multiplication on the meta-weighted vector and the corresponding semantic feature vector to obtain a plurality of classification feature vectors; S200, inputting the plurality of classification feature vectors into a classifier respectively to obtain a plurality of classification results, wherein each of the classification results is used to indicate whether the screen content in the corresponding area is private content; and, S210, controlling the display mode of each area of ​​the screen of the electronic device based on the plurality of classification results.

[0086] Figure 3FIG. 1 is a schematic diagram of the structure of the anti-leakage display method of the electronic device according to the embodiment of the present application. Figure 3 As shown, in the network architecture of the electronic device anti-leakage display method, first, the screen image of the electronic device (for example, Figure 3 P) is input into a convolutional neural network (e.g., Figure 3 ) to obtain the display feature map (e.g., Figure 3 F); then, the screen image is divided into regions based on the screen display content and the positions of the respective regions in the screen image are mapped into the display feature map to obtain the screen contents of the respective regions (for example, Figure 3 Each sub-feature graph (for example, Figure 3 Then, each of the sub-feature graphs is vectorized to obtain a plurality of sub-feature vectors (for example, Figure 3 V1 as shown in FIG); then, the plurality of sub-feature vectors are input into a context coding model based on a converter (e.g., Figure 3 C as shown in FIG) to obtain multiple semantic feature vectors (for example, Figure 3 Then, each of the sub-feature maps is globally averaged pooled along the channel dimension to obtain a feature matrix corresponding to each of the sub-feature maps (for example, Figure 3 MF1 shown in the figure); then, each of the feature matrices and the semantic feature vector corresponding thereto are matrix-multiplied to map the semantic feature vector to the feature space of the feature matrix to obtain a plurality of comparison matrices (for example, Figure 3 Then, the consistency factor of each comparison matrix is ​​calculated to obtain a plurality of consistency factors (for example, Figure 3 CF shown in FIG); then, using the plurality of consistency factors to construct a plurality of element weight vectors (eg, Figure 3 Then, the meta-weight vector and the semantic feature vector corresponding thereto are multiplied by position to obtain a plurality of classification feature vectors (for example, Figure 3 Then, the plurality of classification feature vectors are respectively input into a classifier (for example, Figure 3 Circle S shown in the figure) to obtain multiple classification results, wherein each classification result is used to indicate whether the screen content in the corresponding area is private content; and finally, based on the multiple classification results, the display mode of each area of ​​the screen of the electronic device is controlled.

[0087] In step S110 and step S120, the acquired screen image of the electronic device is input into a convolutional neural network to obtain a display feature map, and the screen image is divided into regions based on the screen display content and the positions of the respective regions in the screen image are mapped to the display feature map to obtain respective sub-feature maps corresponding to the screen content of the respective regions. As described above, considering that when the subject is identified for the content displayed on the screen, there is a correlation between the content subject of the region and the content subject of other regions. Therefore, in the technical solution of the present application, it is necessary to make full use of the correlation information between the content subject of the region and the content subject of other regions, so as to improve the accuracy of the subject identification through the correlation information, thereby avoiding the leakage caused by the associated content not being hidden.

[0088] That is, in a specific example, first, the acquired screen image of the electronic device needs to be input into the convolutional neural network for processing to extract the feature distribution representation of the local features of the screen image in the high-dimensional space, thereby obtaining a display feature map. Here, the electronic devices include but are not limited to computers, mobile phones, ATMs, automatic ticket machines, etc. Next, the screen feature map is divided into regions based on the screen display content to facilitate the subsequent extraction of the association information between the content themes of the various regions. Then, the positions of the various regions in the screen image are mapped to the display feature map to obtain the various sub-feature maps corresponding to the screen content of each region. That is, the screen display content of each region is mapped to the high-dimensional feature space corresponding to it, so as to facilitate the subsequent extraction of the association information between the high-dimensional image features.

[0089] Specifically, in an embodiment of the present application, the process of inputting the acquired screen image of the electronic device into a convolutional neural network to obtain a display feature map includes: the convolutional neural network spatially encodes the screen image according to the following formula to obtain the display feature map;

[0090] Wherein, the formula is:

[0091] f i =active(N i ×f i-1 +B i )

[0092] Among them, f i-1 is the input of the i-th layer of convolutional neural network, f i is the output of the i-th layer of convolutional neural network, N i is the filter of the i-th layer of convolutional neural network, and B i is the bias matrix of the i-th layer convolutional neural network, and Active represents the nonlinear activation function.

[0093] In step S130 and step S140, each of the sub-feature graphs is vectorized to obtain a plurality of sub-feature vectors, and the plurality of sub-feature vectors are input into a context coding model based on a converter to obtain a plurality of semantic feature vectors. That is, in the technical solution of the present application, after obtaining the respective sub-feature graphs, each of the sub-feature graphs is then vectorized to obtain a plurality of sub-feature vectors. Accordingly, in a specific example, each of the sub-feature graphs can be vector-projected along a specific dimension to obtain the sub-feature vector. In particular, in another specific example, it can also be further input into a convolutional neural network as a filter for processing to obtain the sub-feature vector. Then, the plurality of sub-feature vectors are input into a context coding model based on a converter to obtain a plurality of semantic feature vectors. It should be understood that by converting the sub-feature graphs of each region of the feature graph into the feature vector and then performing context-based encoding, the semantic feature vector can contain the context information between the respective feature vectors, thereby making full use of the association information between the high-dimensional image features of the respective regions to achieve global-based classification.

[0094] In step S150 and step S160, global mean pooling is performed on each of the sub-feature maps along the channel dimension direction to obtain a feature matrix corresponding to each of the sub-feature maps, and each of the feature matrices and the corresponding semantic feature vectors are matrix multiplied to map the semantic feature vectors to the feature space of the feature matrix to obtain multiple comparison matrices. It should be understood that in order to maintain the consistency between each of the sub-feature maps and the corresponding semantic feature vectors, in the technical solution of the present application, a comparison consistency factor is used to construct a meta-weight vector to weight the semantic feature vectors. That is, first, it is necessary to perform global mean pooling processing along the channel dimension direction on each of the sub-feature maps to reduce the number of parameters and reduce overfitting, so as to obtain a feature matrix corresponding to each of the sub-feature maps. Then, each of the feature matrices and the corresponding semantic feature vectors are matrix multiplied to map the semantic feature vectors to the feature space of the feature matrix, so as to obtain multiple comparison matrices, so that it is convenient to calculate the comparison consistency factor later.

[0095] In step S170, the consistency factors of each comparison matrix are calculated to obtain multiple consistency factors, and the consistency factors are related to the maximum eigenvalue in the comparison matrix and the size of the comparison matrix. That is, in the technical solution of the present application, after obtaining the comparison matrix, the consistency factors of each comparison matrix are calculated to obtain multiple consistency factors, so as to facilitate the subsequent weighting of the semantic feature vector, so as to maintain the consistency between each sub-feature graph and the corresponding semantic feature vector, where the consistency factor is related to the maximum eigenvalue in the comparison matrix and the size of the comparison matrix.

[0096] Specifically, in an embodiment of the present application, calculating the consistency factor of each comparison matrix to obtain multiple consistency factors includes: calculating the consistency factor of each comparison matrix to obtain multiple consistency factors using the following formula, wherein the formula is:

[0097]

[0098] where λ max is the largest eigenvalue of the comparison matrix, and m is the size of the comparison matrix.

[0099] In step S180 and step S190, the multiple consistency factors are used to construct multiple meta-weighted vectors, and the meta-weighted vectors and the corresponding semantic feature vectors are point-multiplied by position to obtain multiple classification feature vectors. That is, in the technical solution of the present application, after obtaining the consistency factor of the comparison matrix, multiple meta-weighted vectors are then constructed based on the multiple consistency factors to weight the corresponding semantic feature vectors, thereby obtaining multiple classification feature vectors for classification. It should be understood that by introducing the meta-weighted vector based on the comparison consistency factor to weight the semantic feature vector, it is ensured that after the original feature map is subjected to vector projection and context encoding, the obtained semantic feature vector is still consistent with the original feature map, that is, while extracting the associated information, the fidelity of the original content information is improved as much as possible to improve the accuracy of subsequent classification.

[0100] Specifically, in the embodiment of the present application, using the multiple consistency factors to construct multiple element weighted vectors includes: using the multiple consistency factors to construct the multiple element weighted vectors using the following formula; wherein the formula is:

[0101]

[0102] where v e is the eigenvalue λ max The corresponding feature vector.

[0103] In step S200 and step S210, the multiple classification feature vectors are respectively input into the classifier to obtain multiple classification results, wherein each of the classification results is used to indicate whether the screen content in the corresponding area is private content, and based on the multiple classification results, the display mode of each area of ​​the screen of the electronic device is controlled. Accordingly, in a specific example, first, the classification feature vectors are respectively input into the Softmax classification function of the classifier to obtain a first probability that the screen content of the corresponding area attributed to the classification feature vector is private content and a second probability that the screen content of the corresponding area attributed to the classification feature vector is non-private content. Then, based on the comparison between the first probability and the second probability, the classification result is determined. Specifically, when the first probability is greater than the second probability, the classification result is that the screen content of the corresponding area is private content; when the first probability is less than the second probability, the classification result is that the screen content of the corresponding area is non-private content. Finally, based on the multiple classification results, the display mode of each area of ​​the screen of the electronic device is controlled. Specifically, in response to the classification result that the screen content of the corresponding area is private content, the pixels of the corresponding area of ​​the screen of the electronic device are controlled to operate in the second mode; in response to the classification result that the screen content of the corresponding area is non-privacy content, the pixels of the corresponding area of ​​the screen of the electronic device are controlled to operate in the first mode.

[0104] In summary, the anti-leakage display method of the electronic device of the embodiment of the present application is explained, which divides the screen image into regions and adopts a convolutional neural network model and a converter-based context coding model to fully utilize the correlation information between the high-dimensional image features of each region, thereby realizing global classification. Further, by introducing a meta-weighted vector based on a comparison consistency factor to weight the semantic feature vector, the consistency between the obtained semantic feature vector and the original feature map is ensured, thereby improving the fidelity of the original content information as much as possible. In this way, the accuracy of identifying the theme of the displayed content can be improved, thereby avoiding leakage caused by the related content not being hidden, so that intelligent display control at the pixel level can be achieved.

[0105] Exemplary Systems

[0106] Figure 4 FIG. 1 is a block diagram of an anti-leakage display system for an electronic device according to an embodiment of the present application. Figure 4As shown, according to an embodiment of the present application, the anti-leakage display system 400 of an electronic device includes: a display feature map generating unit 410, which is used to input the acquired screen image of the electronic device into a convolutional neural network to obtain a display feature map; a mapping unit 420, which is used to divide the screen image obtained by the display feature map generating unit 410 into regions based on the screen display content and map the positions of each region in the screen image to the display feature map obtained by the display feature map generating unit 410 to obtain each sub-feature map corresponding to the screen content of each region; a vectorization unit 430, which is used to map each of the mapping units 420 0 to obtain a plurality of sub-feature vectors; a converter processing unit 440, used to input the plurality of sub-feature vectors obtained by the vectorization unit 430 into a context coding model based on a converter to obtain a plurality of semantic feature vectors; a global pooling unit 450, used to perform global mean pooling along the channel dimension direction on the sub-feature maps obtained by each of the mapping units 420 to obtain a feature matrix corresponding to each of the sub-feature maps; a matrix multiplication unit 460, used to multiply the feature matrix obtained by each of the global pooling units 450 and the feature matrix obtained by the corresponding converter processing unit 440 The semantic feature vector is matrix-multiplied to map the semantic feature vector into the feature space of the feature matrix to obtain a plurality of comparison matrices; a consistency factor calculation unit 470 is used to calculate the consistency factors of the comparison matrices obtained by each of the matrix multiplication units 460 to obtain a plurality of consistency factors, wherein the consistency factors are related to the maximum eigenvalue in the comparison matrix and the size of the comparison matrix; a meta-weighted vector construction unit 480 is used to construct a plurality of meta-weighted vectors using the plurality of consistency factors obtained by the consistency factor calculation unit 470; a classification feature vector generation unit 490 is used to map the plurality of meta-weighted vectors into the classification feature space; a consistency factor calculation unit 470 is used to calculate the consistency factors of the comparison matrices obtained by the matrix multiplication units 460 to obtain a plurality of consistency factors, wherein the consistency factors are related to the maximum eigenvalue in the comparison matrix and the size of the comparison matrix; a meta-weighted vector construction unit 480 is used to construct a plurality of meta-weighted vectors using the plurality of consistency factors obtained by the consistency factor calculation unit 470; a classification feature vector generation unit 490 is used to map the meta-weighted vectors into the classification feature space; The meta-weight vector obtained by the weight vector construction unit 480 and the corresponding semantic feature vector obtained by the converter processing unit 440 are point-multiplied by position to obtain multiple classification feature vectors; a classification unit 500 is used to input the multiple classification feature vectors obtained by the classification feature vector generation unit 490 into a classifier respectively to obtain multiple classification results, wherein each of the classification results is used to indicate whether the screen content in the corresponding area is private content; and a control unit 510 is used to control the display mode of each area of ​​the screen of the electronic device based on the multiple classification results obtained by the classification unit 500.

[0107] In one example, in the anti-leakage display system 400 of the electronic device, the display feature map generating unit 410 is further used to: the convolutional neural network spatially encodes the screen image according to the following formula to obtain the display feature map; wherein the formula is:

[0108] f i =active(N i ×f i-1 +B i )

[0109] Among them, f i-1 is the input of the i-th layer of convolutional neural network, f i is the output of the i-th layer of convolutional neural network, N i is the filter of the i-th layer of convolutional neural network, and B i is the bias matrix of the i-th layer convolutional neural network, and Active represents the nonlinear activation function.

[0110] In one example, in the anti-leakage display system 400 for the electronic device, the vectorization unit 430 is further used to: perform vector projection along a specific dimension on each of the sub-feature graphs to obtain the sub-feature vector.

[0111] In one example, in the above-mentioned electronic device anti-leakage display system 400, the consistency factor calculation unit 470 is further used to: calculate the consistency factor of each comparison matrix according to the following formula to obtain multiple consistency factors, wherein the formula is:

[0112]

[0113] where λ max is the largest eigenvalue of the comparison matrix, and m is the size of the comparison matrix.

[0114] In one example, in the electronic device anti-leakage display system 400, the element weighted vector construction unit 480 is further used to: use the multiple consistency factors to construct the multiple element weighted vectors using the following formula; wherein the formula is:

[0115]

[0116] where v e is the eigenvalue λ max The corresponding feature vector.

[0117] In one example, in the anti-leakage display system 400 of the above-mentioned electronic device, the classification unit 500 is further used to: input the classification feature vectors into the Softmax classification function of the classifier respectively to obtain a first probability that the screen content of the corresponding area to which the classification feature vector belongs is private content and a second probability that the screen content of the corresponding area to which the classification feature vector belongs is non-privacy content; and, based on a comparison between the first probability and the second probability, determine the classification result.

[0118] In one example, in the anti-leakage display system 400 of the above-mentioned electronic device, the control unit 510 is further used to: in response to the classification result that the screen content of the corresponding area is private content, control the pixels of the corresponding area of ​​the screen of the electronic device to operate in the second mode; and, in response to the classification result that the screen content of the corresponding area is non-privacy content, control the pixels of the corresponding area of ​​the screen of the electronic device to operate in the first mode.

[0119] Here, those skilled in the art will appreciate that the specific functions and operations of the various units and modules in the electronic device anti-leakage display system 400 have been described in detail above. Figures 1 to 3 The invention has been described in detail in the description of the anti-leakage display method of the electronic device, and therefore, its repeated description will be omitted.

[0120] As described above, the anti-leakage display system 400 of the electronic device according to the embodiment of the present application can be implemented in various terminal devices, such as a server of the anti-leakage display algorithm of the electronic device. In one example, the anti-leakage display system 400 of the electronic device according to the embodiment of the present application can be integrated into the terminal device as a software module and / or a hardware module. For example, the anti-leakage display system 400 of the electronic device can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the anti-leakage display system 400 of the electronic device can also be one of the many hardware modules of the terminal device.

[0121] Alternatively, in another example, the electronic device's anti-leakage display system 400 and the terminal device may also be separate devices, and the electronic device's anti-leakage display system 400 may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in an agreed data format.

[0122] Exemplary Electronic Devices

[0123] Below, reference Figure 5 To describe the electronic device according to the embodiment of the present application. Figure 5 As shown, the electronic device 10 includes one or more processors 11 and a memory 12. The processor 11 may be a central processing unit (CPU) or other processing units having data processing capability and / or instruction execution capability, and may control other components in the electronic device 10 to perform desired functions.

[0124] The memory 12 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may run the program instructions to implement the functions of the anti-leakage display method of the electronic device of each embodiment of the present application described above and / or other desired functions. Various contents such as comparison matrices, element weighted vectors, etc. may also be stored in the computer-readable storage medium.

[0125] In one example, the electronic device 10 may further include: an input system 13 and an output system 14, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0126] The input system 13 may include, for example, a keyboard, a mouse, and the like.

[0127] The output system 14 can output various information to the outside, including classification results, etc. The output system 14 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0128] Of course, to simplify, Figure 5 Only some of the components related to the present application in the electronic device 10 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 10 may also include any other appropriate components.

[0129] Exemplary computer program products and computer-readable storage media

[0130] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to perform the steps in the functions of the anti-leakage display method of an electronic device according to various embodiments of the present application described in the above-mentioned "Exemplary Method" section of this specification.

[0131] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0132] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the anti-leakage display method of an electronic device described in the above "Exemplary Method" section of this specification.

[0133] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, system or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0134] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0135] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0136] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0137] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0138] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for preventing leakage of information in an electronic device, characterized in that: include: Inputting the acquired screen image of the electronic device into a convolutional neural network to obtain a display feature map; Dividing the screen image into regions based on screen display content and mapping the positions of the respective regions in the screen image into the display feature map to obtain respective sub-feature maps corresponding to the screen content of the respective regions; Vectorizing each of the sub-feature graphs to obtain a plurality of sub-feature vectors; Inputting the plurality of sub-feature vectors into a transformer-based context encoding model to obtain a plurality of semantic feature vectors; Performing global mean pooling on each of the sub-feature maps along the channel dimension direction to obtain a feature matrix corresponding to each of the sub-feature maps; Performing matrix multiplication on each of the feature matrices and the semantic feature vector corresponding thereto to map the semantic feature vector into the feature space of the feature matrix to obtain a plurality of comparison matrices; Calculating consistency factors of each of the comparison matrices to obtain a plurality of consistency factors, wherein the consistency factors are related to the maximum eigenvalue in the comparison matrix and the size of the comparison matrix; constructing a plurality of meta-weight vectors using the plurality of consistency factors; Performing positional point multiplication of the meta-weighted vector and the semantic feature vector corresponding thereto to obtain a plurality of classification feature vectors; Inputting the plurality of classification feature vectors into a classifier respectively to obtain a plurality of classification results, wherein each of the classification results is used to indicate whether the screen content in the corresponding area is private content; and Based on the multiple classification results, controlling the display mode of each area of ​​the screen of the electronic device; The step of calculating the consistency factors of each comparison matrix to obtain a plurality of consistency factors comprises: The consistency factor of each comparison matrix is ​​calculated by the following formula to obtain multiple consistency factors, wherein the formula is: ， in is the largest eigenvalue of the comparison matrix, and is the size of the comparison matrix; Wherein, using the plurality of consistency factors to construct a plurality of element weighted vectors comprises: The plurality of consistency factors are used to construct the plurality of element weight vectors using the following formula; wherein the formula is: ， in is the eigenvalue The corresponding feature vector.

2. The electronic device anti-leakage display method according to claim 1, wherein: The acquired screen image of the electronic device is input into the convolutional neural network to obtain a display feature map, including: The convolutional neural network spatially encodes the screen image using the following formula to obtain the display feature map; Wherein, the formula is: ， in, is the input of the i-th layer of convolutional neural network, is the output of the i-th layer of convolutional neural network, is the filter of the i-th layer of the convolutional neural network, and is the bias matrix of the i-th layer convolutional neural network, represents a non-linear activation function.

3. The electronic device anti-leakage display method according to claim 2, wherein: Vectorizing each of the sub-feature graphs to obtain a plurality of sub-feature vectors includes: Perform vector projection along the dimension on each of the sub-feature maps to obtain the sub-feature vector.

4. The electronic device anti-leakage display method according to claim 3, wherein: Inputting the plurality of classification feature vectors into a classifier respectively to obtain a plurality of classification results, including: Inputting the classification feature vectors into the Softmax classification function of the classifier respectively to obtain a first probability that the screen content of the corresponding area to which the classification feature vector belongs is private content and a second probability that the screen content of the corresponding area to which the classification feature vector belongs is non-private content; and The classification result is determined based on a comparison between the first probability and the second probability.

5. The electronic device anti-leakage display method according to claim 4, wherein: Based on the multiple classification results, controlling the display mode of each area of ​​the screen of the electronic device includes: In response to the classification result that the screen content of the corresponding area is private content, controlling the pixels of the corresponding area of ​​the screen of the electronic device to operate in a second mode; and In response to the classification result that the screen content of the corresponding area is non-privacy content, the pixels of the corresponding area of ​​the screen of the electronic device are controlled to operate in a first mode.

6. An anti-leakage display system for electronic equipment, characterized in that: include: A display feature map generating unit, used for inputting the acquired screen image of the electronic device into a convolutional neural network to obtain a display feature map; a mapping unit, configured to divide the screen image obtained by the display feature map generating unit into regions based on screen display content and map the positions of the respective regions in the screen image to the display feature map obtained by the display feature map generating unit to obtain respective sub-feature maps corresponding to the screen content of the respective regions; A vectorization unit, used for vectorizing the sub-feature maps obtained by each of the mapping units to obtain a plurality of sub-feature vectors; A converter processing unit, configured to input the plurality of sub-feature vectors obtained by the vectorization unit into a converter-based context coding model to obtain a plurality of semantic feature vectors; A global pooling unit, used for performing global mean pooling along the channel dimension direction on the sub-feature maps obtained by each of the mapping units to obtain a feature matrix corresponding to each of the sub-feature maps; a matrix multiplication unit, configured to perform matrix multiplication on the feature matrix obtained by each of the global pooling units and the semantic feature vector obtained by the corresponding converter processing unit to map the semantic feature vector into a feature space of the feature matrix to obtain a plurality of comparison matrices; A consistency factor calculation unit, used for calculating the consistency factors of the comparison matrices obtained by the matrix multiplication units to obtain a plurality of consistency factors, wherein the consistency factors are related to the maximum eigenvalue in the comparison matrix and the size of the comparison matrix; a meta-weighted vector construction unit, configured to construct a plurality of meta-weighted vectors using the plurality of consistency factors obtained by the consistency factor calculation unit; a classification feature vector generating unit, configured to perform positional point multiplication on the meta-weighted vector obtained by the meta-weighted vector constructing unit and the semantic feature vector obtained by the corresponding converter processing unit to obtain a plurality of classification feature vectors; a classification unit, configured to input the plurality of classification feature vectors obtained by the classification feature vector generation unit into a classifier to obtain a plurality of classification results, wherein each of the classification results is used to indicate whether the screen content in a corresponding area is private content; and a control unit, configured to control a display mode of each area of ​​a screen of the electronic device based on the plurality of classification results obtained by the classification unit; The consistency factor calculation unit is used to calculate the consistency factor of each comparison matrix using the following formula to obtain multiple consistency factors, wherein the formula is: ， in is the largest eigenvalue of the comparison matrix, and is the size of the comparison matrix; The meta-weighted vector construction unit is used to use the multiple consistency factors to construct the multiple meta-weighted vectors using the following formula; wherein the formula is: ， in is the eigenvalue The corresponding feature vector.

7. The electronic device anti-leakage display system according to claim 6, wherein: The display feature map generating unit is further used for: The convolutional neural network spatially encodes the screen image using the following formula to obtain the display feature map; Wherein, the formula is: ， in, is the input of the i-th layer of convolutional neural network, is the output of the i-th layer of convolutional neural network, is the filter of the i-th layer of the convolutional neural network, and is the bias matrix of the i-th layer convolutional neural network, represents a non-linear activation function.

8. An electronic device comprising: processor; as well as A memory, in which computer program instructions are stored, and when the computer program instructions are executed by the processor, the processor executes the anti-leakage display method for an electronic device according to any one of claims 1 to 5.

Images