Active perception interface display method, system and electronic device

Abstract

The present application discloses an active-aware interface display method, system and electronic device, which use a converter model based on a word embedding layer and a conditional random field model to annotate a sequence of feature vectors generated based on content in a screen image based on permission information to determine the subject tags of the content in the screen, and further use a dense prediction model architecture to generate a screen image from the text feature vector with a tag, so as to realize interface display based on text-based tagging. In this way, only the part of the displayable content with the subject tag is displayed, so that end-to-end active-aware interface display can be realized, thereby improving the security of screen content information.

Description

Technical Field

[0001] The present invention relates to the field of intelligent anti-peeping, and more specifically, to an active awareness interface display method, system and electronic device. Background Art

[0002] With the development of information science and technology, the Internet and various electronic products of network terminals have become inseparable from people's daily life. Although there have been great changes in the human-computer interaction mode, such as the early keyboard input to the current touch screen input, wearable devices supported by sensor technology and new display devices represented by Google Glass, the current electronic product information display mode is still mostly conveyed to users through the screen, which can easily lead to the user's information being peeped and leaked, especially when certain confidentiality is very strong and personal private information is to be processed.

[0003] There are two main methods to prevent peeping on computer screens: active peeping and passive peeping. Passive peeping is to use a special screen film to prevent the screen content from being seen at a certain angle. Active peeping detects whether someone has reached the visible range, and if someone has, the computer window will be actively hidden or the screen saver mode will be directly entered to achieve the purpose of preventing peeping. Comparing the above two methods, active peeping is obviously more secure and suitable for some occasions with relatively high confidentiality requirements.

[0004] However, most active anti-peeping methods are to put the screen into protection mode once it is detected that someone other than the user is peeping at the contents of the screen, but when the screen is in protection mode, the user cannot see the contents on the screen.

[0005] In actual application scenarios, some people have relevant permissions to view screen content, and accordingly, when these people view the screen, there is no need to start the screen saver mode. In addition, although some people have permissions to view screen content, permissions for different people may be different, that is, some people may only be allowed to view content of a part of the subject, while other people may only be allowed to view content of another part of the subject.

[0006] Therefore, we look forward to an active awareness interface display solution that coordinates identity with screen display content. Summary of the invention

[0007] In order to solve the above technical problems, the present application is proposed. The embodiments of the present application provide an active perceptual interface display method, system and electronic device, which annotates a sequence of feature vectors generated based on the content in the screen image based on permission information through a converter model and a conditional random field model based on a word embedding layer to determine the subject label of the content in the screen, and further generates a screen image from the text feature vector with a label through a dense prediction model architecture to realize interface display based on text-based labeling. In this way, only the part with the subject label as the displayable content is displayed, so that end-to-end active perceptual interface display can be realized, thereby making the security of the screen content information better.

[0008] According to one aspect of the present application, there is provided an active awareness interface display method, which includes:

[0009] Acquire a screen image of an electronic device to be displayed;

[0010] extracting text data from the screen image;

[0011] After word segmentation, the text data is passed through a converter model including a word embedding layer to obtain a sequence of word feature vectors;

[0012] The sequence of word feature vectors is passed through a conditional random field to label each word feature vector in the sequence of word feature vectors based on permission information to obtain a category label of each word feature vector;

[0013] Based on the category label of each of the word feature vectors, extracting multiple groups of one or more word feature vectors whose label categories belong to displayable content from the sequence of the word feature vectors;

[0014] Two-dimensionally concatenate the feature vectors of one or more words in each group whose label category belongs to displayable content to obtain a plurality of feature matrices with different scales;

[0015] The plurality of feature matrices with different scales are fused into a projection matrix using a projection recombination function of a dense prediction model architecture, wherein the projection recombination function of the dense prediction model architecture includes three-dimensionally splicing all the plurality of feature matrices to obtain a three-dimensional feature map and then projecting the three-dimensional feature map onto a two-dimensional plane;

[0016] Inputting the projection matrix into a generator model of the dense prediction model architecture to generate a reconstructed screen image of the electronic device, wherein the convolution layer of the generator model is suitable for concatenating and upsampling convolution feature maps obtained by performing convolution operations using different convolution kernels; and

[0017] The reconstructed screen image of the electronic device is displayed on the screen of the electronic device.

[0018] In the above active perception interface display method, extracting text data from the screen image includes: performing text recognition processing on the screen image to obtain the text data.

[0019] In the above-mentioned active perception interface display method, the text data is segmented and then passed through a converter model including a word embedding layer to obtain a sequence of word feature vectors, including: the text data is segmented and then obtained to obtain a word sequence corresponding to the text data; each word in the word sequence is input into the word embedding layer of the converter model to map each word in the word sequence to a word vector to obtain a sequence of word vectors; and, the sequence of word vectors is input into the converter of the converter model to obtain the sequence of word feature vectors.

[0020] In the above-mentioned active perception interface display method, the projection recombination function of the dense prediction model architecture is used to merge the multiple feature matrices with different scales into a projection matrix, including: using the dense prediction model architecture to perform three-dimensional stitching on the multiple feature matrices with different scales to obtain the three-dimensional feature map; and, using the dense prediction model architecture to project the three-dimensional feature map onto a two-dimensional plane to obtain the two-dimensional projection matrix.

[0021] In the above-mentioned active perception interface display method, the dense prediction model architecture is used to perform three-dimensional stitching on the multiple feature matrices with different scales to obtain the three-dimensional feature map, including: using the dense prediction model architecture to perform scale adjustment based on linear transformation on each feature matrix of the multiple feature matrices with different scales to unify the scales of the multiple feature matrices.

[0022] In the above-mentioned active perceptual interface display method, the generator model is a convolutional neural network consisting of alternating convolutional layers, upsampling layers and activation layers; wherein, the projection matrix is ​​input into the generator model of the dense prediction model architecture to generate a reconstructed screen image of the electronic device, including: using the convolutional layer of the generator model to convolve the projection matrix to obtain a convolutional feature map; using the upsampling layer of the generator model connected to the convolutional layer to upsample the convolutional feature map to obtain an upsampled feature map; and, using the activation layer of the generator model connected to the upsampling layer to activate the upsampled feature map to generate an activated feature map and output it to the convolutional layer of the next layer of the generator model.

[0023] In the above-mentioned active perception interface display method, in the generator model, the convolution layers of different layers use different convolution kernels to perform convolution operations.

[0024] According to another aspect of the present application, there is provided an active awareness interface display system, comprising:

[0025] A screen image acquisition unit, used to acquire a screen image of the electronic device to be displayed;

[0026] a text data extraction unit, configured to extract text data from the screen image obtained by the screen image acquisition unit;

[0027] A converter processing unit, configured to perform word segmentation processing on the text data obtained by the text data extraction unit and then obtain a sequence of word feature vectors through a converter model including a word embedding layer;

[0028] A conditional random field unit, configured to pass the sequence of word feature vectors obtained by the converter processing unit through a conditional random field to label each word feature vector in the sequence of word feature vectors based on permission information to obtain a category label of each word feature vector;

[0029] an extraction unit, configured to extract, based on the category labels of the word feature vectors obtained by each of the conditional random field units, a plurality of groups of one or more word feature vectors whose label categories belong to displayable content from the sequence of the word feature vectors obtained by the converter processing unit;

[0030] A two-dimensional splicing unit, used for two-dimensionally splicing the groups of one or more word feature vectors of the label category belonging to the displayable content obtained by the text data extraction unit to obtain a plurality of feature matrices with different scales;

[0031] A fusion unit, used to fuse the multiple feature matrices obtained by the two-dimensional splicing units with different scales into a projection matrix using a projection recombination function of a dense prediction model architecture, wherein the projection recombination function of the dense prediction model architecture includes three-dimensionally splicing all the multiple feature matrices to obtain a three-dimensional feature map and then projecting the three-dimensional feature map onto a two-dimensional plane;

[0032] a generator model processing unit, configured to input the projection matrix obtained by the fusion unit into the generator model of the dense prediction model architecture to generate a reconstructed screen image of the electronic device, wherein the convolution layer of the generator model is suitable for concatenating and upsampling convolution feature maps obtained by performing convolution operations using different convolution kernels; and

[0033] A display unit is used to display the reconstructed screen image of the electronic device obtained by the generator model processing unit on the screen of the electronic device.

[0034] In the above active perception interface display system, the text data extraction unit is further used to: perform text recognition processing on the screen image to obtain the text data.

[0035] In the above-mentioned active perception interface display system, the converter processing unit includes: a word segmentation subunit, which is used to perform word segmentation processing on the text data to obtain a word sequence corresponding to the text data; a word embedding subunit, which is used to input each word in the word sequence obtained by the word segmentation subunit into the word embedding layer of the converter model to map each word in the word sequence to a word vector to obtain a sequence of word vectors; and a converter subunit, which is used to input the sequence of word vectors obtained by the word embedding subunit into the converter of the converter model to obtain a sequence of word feature vectors.

[0036] In the above-mentioned active perception interface display system, the fusion unit is further used to: use the dense prediction model architecture to perform three-dimensional stitching on the multiple feature matrices with different scales to obtain the three-dimensional feature map; and use the dense prediction model architecture to project the three-dimensional feature map onto a two-dimensional plane to obtain the two-dimensional projection matrix.

[0037] In the above-mentioned active perception interface display system, the fusion unit is further used to: use the dense prediction model architecture to perform scale adjustment based on linear transformation on each feature matrix of the multiple feature matrices with different scales to unify the scales of the multiple feature matrices.

[0038] In the above-mentioned active perceptual interface display system, the generator model is a convolutional neural network consisting of alternating convolutional layers, upsampling layers and activation layers; and the generator model processing unit is further used to: use the convolutional layer of the generator model to convolve the projection matrix to obtain a convolutional feature map; use the upsampling layer of the generator model connected to the convolutional layer to upsample the convolutional feature map to obtain an upsampled feature map; and use the activation layer of the generator model connected to the upsampling layer to activate the upsampled feature map to generate an activated feature map and output it to the convolutional layer of the next layer of the generator model.

[0039] In the above-mentioned active perception interface display system, in the generator model, the convolution layers of different layers use different convolution kernels to perform convolution operations.

[0040] According to another aspect of the present application, an electronic device is provided, comprising: a processor; and a memory, wherein computer program instructions are stored in the memory, and when the computer program instructions are executed by the processor, the processor executes the active awareness interface display method as described above.

[0041] According to yet another aspect of the present application, a computer-readable medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the active awareness interface display method as described above.

[0042] Compared with the prior art, the active perceptual interface display method, system and electronic device provided by the present application use a converter model based on a word embedding layer and a conditional random field model to label a sequence of feature vectors generated based on the content in the screen image based on permission information to determine the subject tags of the content in the screen, and further use a dense prediction model architecture to generate a screen image from the text feature vector with a label to achieve interface display based on text-based labeling. In this way, only the part of the displayable content with the subject tag is displayed. In this way, end-to-end active perceptual interface display can be achieved, thereby improving the security of the screen content information. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0044] Figure 1 This is an application scenario diagram of the active perception interface display method according to an embodiment of the present application;

[0045] Figure 2 is a flow chart of an active perception interface display method according to an embodiment of the present application;

[0046] Figure 3 A schematic diagram of the system architecture of the active perception interface display method according to an embodiment of the present application;

[0047] Figure 4 A flowchart of obtaining a sequence of word feature vectors through a converter model including a word embedding layer after word segmentation processing is performed on the text data in the active perceptual interface display method according to an embodiment of the present application;

[0048] Figure 5 is a block diagram of an active perception interface display system according to an embodiment of the present application;

[0049] Figure 6 A block diagram of a converter processing unit in an active perceptual interface display system according to an embodiment of the present application;

[0050] Figure 7 is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0051] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0052] Scenario Overview

[0053] As mentioned above, there are two main methods to prevent peeping on computer screens: active peeping and passive peeping. Passive peeping is to use a special screen film to prevent the screen content from being seen at a certain angle. Active peeping detects whether someone has reached the visible range, and if someone has, the computer window is actively hidden or the screen saver mode is directly entered to achieve the purpose of preventing peeping. Comparing the above two methods, it is obvious that active peeping is more secure and is suitable for some occasions with relatively high confidentiality requirements.

[0054] However, most active anti-peeping methods are to put the screen into protection mode once it is detected that someone other than the user is peeping at the contents of the screen, but when the screen is in protection mode, the user cannot see the contents on the screen.

[0055] In actual application scenarios, some people have relevant permissions to view screen content. Accordingly, when these people view the screen, there is no need to start the screen saver mode. Moreover, although some people have the permission to view screen content, the permissions of different people may be different, that is, some people may only be allowed to view the content of a part of the subject, while other people may only be allowed to view the content of another part of the subject. Therefore, an active awareness interface display solution that coordinates identity and screen content is expected.

[0056] Accordingly, considering the advantages of text data in semantic understanding and labeling, in the technical solution of the present application, it is expected to mark the content displayed on the screen based on the text data and then generate a screen image, thereby realizing an end-to-end active awareness interface display method.

[0057] Based on this, in the technical solution of the present application, after the text data corresponding to the content displayed on the screen is segmented, it is input into the converter model containing the word embedding layer to obtain a sequence of feature vectors corresponding to the text data, and then the sequence of feature vectors is annotated based on the permission information through the conditional random field to obtain the label category of each feature vector.

[0058] Then, a screen image is generated based on a feature vector of displayable content with a label category through a dense prediction model architecture. Specifically, one or more first feature vectors corresponding to the label of the first displayable content are first obtained, and they are two-dimensionally spliced ​​to obtain a first feature matrix, and the second to nth feature matrices are obtained similarly.

[0059] Afterwards, a screen image is generated through a generator model, such as a generative adversarial model. First, the feature matrices of different scales are directly fused using the projection recombination function of the dense prediction model architecture, that is, all feature matrices are spliced ​​in three dimensions to obtain feature maps, and then projected onto a two-dimensional plane to obtain a projection matrix. Then, the projection matrix is ​​input into the generator model for fusion of the dense prediction model architecture to obtain the desired user interface. The convolution layer in the generator model splices and upsamples the feature maps obtained by convolution operations using different convolution kernels.

[0060] It should be understood that by generating screen images from labeled text feature vectors through a dense prediction model architecture, interface display can be achieved based on text-based labeling, fully ensuring the accuracy of content classification.

[0061] Moreover, when the dense prediction model architecture is splicing features, the feature matrix can not only express the content features of each part of the image by the eigenvalues ​​in the matrix, but also express the scale of the part in the final displayed image by the scale of the feature matrix. For example, the larger the feature matrix, the more important the content of the part is, and therefore the higher the weight it should have in the displayed image.

[0062] In addition, the dense prediction model architecture has a global receptive field of view by abandoning downsampling when recombining and fusing features, and maintains a constant dimensionality of feature representation, so that the images generated by the generator model can better reconstruct details, that is, improve global consistency in areas that are challenging for convolutional structures, such as large uniform areas.

[0063] Based on this, the present application proposes an active awareness interface display method, which includes: acquiring a screen image of an electronic device to be displayed; extracting text data from the screen image; performing word segmentation on the text data and then passing it through a converter model including a word embedding layer to obtain a sequence of word feature vectors; passing the sequence of word feature vectors through a conditional random field to annotate each word feature vector in the sequence of word feature vectors based on permission information to obtain a category label for each word feature vector; based on the category label of each word feature vector, extracting multiple groups of one or more word feature vectors whose label categories belong to displayable content from the sequence of word feature vectors; and labeling each group of one or more word feature vectors whose label categories belong to displayable content. The method comprises the steps of: performing two-dimensional stitching on all the multiple feature matrices to obtain a plurality of feature matrices with different scales; using the projection and recombination function of the dense prediction model architecture to fuse the plurality of feature matrices with different scales into a projection matrix, wherein the projection and recombination function of the dense prediction model architecture comprises the following steps of: performing three-dimensional stitching on all the multiple feature matrices to obtain a three-dimensional feature map and then projecting the three-dimensional feature map onto a two-dimensional plane; inputting the projection matrix into the generator model of the dense prediction model architecture to generate a reconstructed screen image of the electronic device, wherein the convolution layer of the generator model is suitable for stitching and up-sampling the convolution feature maps obtained by convolution operations using different convolution kernels; and displaying the reconstructed screen image of the electronic device on the screen of the electronic device.

[0064] Figure 1 The figure shows an application scenario diagram of the active perception interface display method according to an embodiment of the present application. Figure 1 As shown, in this application scenario, first, from an electronic device (for example, Figure 1 The screen image of the electronic device to be displayed is obtained from the T) and the cloud storage end (for example, Figure 1 B) obtaining permission information, wherein the permission information indicates a list of content topics that the identity can view, and the electronic device includes but is not limited to a computer. Then, the obtained screen image and the permission information are input into a server that is deployed with an active perceptual interface display algorithm (for example, Figure 1 The server S shown in FIG. 1 is capable of processing the screen image and the permission information using an active perceptual interface display algorithm to generate a reconstructed screen image of the electronic device. Then, the reconstructed screen image of the electronic device is displayed on the screen of the electronic device (for example, Figure 1 R) shown in the figure can realize end-to-end active awareness interface display, thereby improving the security of screen content information.

[0065] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0066] Exemplary Methods

[0067] Figure 2 The figure shows a flow chart of the active perceptual interface display method. Figure 2 As shown, according to the active perception interface display method of the embodiment of the present application, it includes: S110, obtaining a screen image of an electronic device to be displayed; S120, extracting text data from the screen image; S130, performing word segmentation on the text data and then obtaining a sequence of word feature vectors through a converter model including a word embedding layer; S140, passing the sequence of word feature vectors through a conditional random field to annotate each word feature vector in the sequence of word feature vectors based on permission information to obtain a category label of each word feature vector; S150, based on the category label of each word feature vector, extracting multiple groups of one or more word feature vectors whose label category belongs to displayable content from the sequence of word feature vectors; S160, labeling each group of one or more word feature vectors whose label category belongs to displayable content. or multiple word feature vectors are two-dimensionally spliced ​​to obtain multiple feature matrices with different scales; S170, using the projection and recombination function of the dense prediction model architecture to merge the multiple feature matrices with different scales into a projection matrix, wherein the projection and recombination function of the dense prediction model architecture includes three-dimensionally splicing all the multiple feature matrices to obtain a three-dimensional feature map and then projecting the three-dimensional feature map onto a two-dimensional plane; S180, inputting the projection matrix into the generator model of the dense prediction model architecture to generate a reconstructed screen image of the electronic device, wherein the convolution layer of the generator model is suitable for splicing and up-sampling the convolution feature maps obtained by convolution operations using different convolution kernels; and, S190, displaying the reconstructed screen image of the electronic device on the screen of the electronic device.

[0068] Figure 3 FIG. 1 is a schematic diagram of the architecture of the active perception interface display method according to an embodiment of the present application. Figure 3 As shown, in the network architecture of the active perception interface display method, first, the screen image (for example, Figure 3 IN) to extract text data (for example, Figure 3 Then, the text data is segmented and then passed through a converter model including a word embedding layer (for example, Figure 3 ) to obtain a sequence of word feature vectors (e.g., Figure 3 Then, the sequence of word feature vectors is passed through a conditional random field (e.g., Figure 3 The CRF shown in FIG. 10 is based on the obtained permission information (e.g., Figure 3 The Q shown in FIG. 1 is used to label each word feature vector in the sequence of word feature vectors to obtain a category label for each word feature vector (for example, Figure 3 CL as shown in the figure); then, based on the category label of each of the word feature vectors, extract multiple groups of one or more word feature vectors (for example, Figure 3 Then, the feature vectors of one or more words in each group whose label category belongs to the displayable content are concatenated in two dimensions to obtain multiple feature matrices with different scales (for example, Figure 3 MF as shown in ); then, using a dense prediction model architecture (e.g., Figure 3 The projection recombination function of the DP (shown in FIG. 1 ) merges the multiple feature matrices with different scales into a projection matrix (e.g., Figure 3 Then, the projection matrix is ​​input into the generator model of the dense prediction model architecture (e.g., as Figure 3 GM) to generate a reconstructed screen image of the electronic device (e.g., Figure 3 OUT as shown in FIG); and, finally, displaying the reconstructed screen image of the electronic device on the screen of the electronic device.

[0069] In step S110 and step S120, a screen image of the electronic device to be displayed is obtained; and text data is extracted from the screen image. As mentioned above, considering the advantages of text data in semantic understanding and labeling, in the technical solution of the present application, it is expected to mark the content displayed on the screen based on the text data and then generate a screen image, so as to realize an end-to-end active awareness interface display method. That is, first, it is necessary to obtain a screen image of the electronic device to be displayed from the electronic device, and the electronic device includes but is not limited to a computer, etc. Then, the text data is extracted from the obtained screen image. In a specific example, the screen image can be subjected to text recognition processing to obtain the text data.

[0070] In step S130 and step S140, the text data is segmented and then passed through a converter model including a word embedding layer to obtain a sequence of word feature vectors; and the sequence of word feature vectors is passed through a conditional random field to annotate each word feature vector in the sequence of word feature vectors based on permission information to obtain a category label for each word feature vector. That is, in the technical solution of the present application, after the text data corresponding to the content displayed on the screen is segmented, it is input into a converter model including a word embedding layer to obtain a sequence of feature vectors corresponding to the text data. Further, it should be understood that the conditional random field model can be applied to annotating a single type of label. Therefore, in the technical solution of the present application, then, the conditional random field can be used to annotate each word feature vector in the sequence of word feature vectors based on permission information to obtain a category label for each word feature vector.

[0071] Specifically, in an embodiment of the present application, the process of obtaining a sequence of word feature vectors after word segmentation of the text data through a converter model including a word embedding layer includes: first, the text data is segmented to obtain a word sequence corresponding to the text data. It should be understood that word segmentation can avoid word order confusion, which is conducive to the subsequent semantic understanding of the text data. Then, each word in the word sequence is input into the word embedding layer of the converter model to map each word in the word sequence to a word vector to obtain a sequence of word vectors; finally, the sequence of word vectors is input into the converter of the converter model to obtain the sequence of word feature vectors. It should be understood that in this way, the semantic information of the text data corresponding to the content displayed on the screen can be extracted to facilitate subsequent labeling.

[0072] Figure 4 The figure shows a flow chart of obtaining a sequence of word feature vectors after word segmentation of the text data in the active perceptual interface display method according to an embodiment of the present application through a converter model including a word embedding layer. Figure 4 As shown, after word segmentation, the text data is processed by a converter model including a word embedding layer to obtain a sequence of word feature vectors, including: S210, after word segmentation, the text data is processed to obtain a word sequence corresponding to the text data; S220, each word in the word sequence is input into the word embedding layer of the converter model to map each word in the word sequence to a word vector to obtain a sequence of word vectors; and, S230, the sequence of word vectors is input into the converter of the converter model to obtain the sequence of word feature vectors.

[0073] In step S150 and step S160, based on the category label of each of the word feature vectors, multiple groups of one or more word feature vectors whose label category belongs to the displayable content are extracted from the sequence of the word feature vectors, and each group of one or more word feature vectors whose label category belongs to the displayable content are two-dimensionally spliced ​​to obtain multiple feature matrices with different scales. That is, in the technical solution of the present application, after obtaining the category label of each of the word feature vectors, a screen image is generated based on the feature vector whose label category is the displayable content through a dense prediction model architecture. Specifically, first, based on the category label of each of the word feature vectors, one or more first feature vectors corresponding to the label of the first displayable content are obtained from the sequence of the word feature vectors, and then two-dimensionally spliced ​​to obtain a first feature matrix, and similarly obtain the second to nth feature matrices. It is worth mentioning that here, the first nth feature matrix has different scales. It should be understood that when the dense prediction model architecture is splicing features, the feature matrix can not only express the content features of each part of the image by the eigenvalues ​​in the matrix, but also express the scale of the part in the final displayed image by the scale of the feature matrix. For example, the larger the feature matrix, the higher the importance of the content of the part, and therefore the higher the weight it should have in the displayed image.

[0074] In step S170, the projection and recombination function of the dense prediction model architecture is used to fuse the multiple feature matrices of different scales into a projection matrix, wherein the projection and recombination function of the dense prediction model architecture includes three-dimensional splicing of all the multiple feature matrices to obtain a three-dimensional feature map and then projecting the three-dimensional feature map onto a two-dimensional plane. That is, further, a screen image is generated by a generator model, such as an adversarial generation model, and before that, it is first necessary to directly use the projection and recombination function of the dense prediction model architecture to fuse the feature matrices of different scales, that is, to splice all the feature matrices in three dimensions to obtain a feature map, and then project them onto a two-dimensional plane to obtain a projection matrix.

[0075] Specifically, in an embodiment of the present application, the process of fusing the multiple feature matrices with different scales into a projection matrix using the projection reorganization function of the dense prediction model architecture includes: first, using the dense prediction model architecture to perform three-dimensional splicing on the multiple feature matrices with different scales to obtain the three-dimensional feature map. In a specific example, the dense prediction model architecture is used to perform linear transformation-based scale adjustment on each feature matrix of the multiple feature matrices with different scales to unify the scales of the multiple feature matrices. Then, the dense prediction model architecture is used to project the three-dimensional feature map onto a two-dimensional plane to obtain the two-dimensional projection matrix.

[0076] In step S180 and step S190, the projection matrix is ​​input into the generator model of the dense prediction model architecture to generate a reconstructed screen image of the electronic device, wherein the convolution layer of the generator model is suitable for splicing and upsampling the convolution feature map obtained by convolution operation using different convolution kernels, and the reconstructed screen image of the electronic device is displayed on the screen of the electronic device. That is, after obtaining the projection matrix, the projection matrix is ​​then input into the generator model for fusion of the dense prediction model architecture to obtain the desired user interface. It is worth mentioning that here, the convolution layer in the generator model splices and upsamples the convolution feature map obtained by convolution operation using different convolution kernels. It should be understood that the dense prediction model architecture has a global perception field of view by abandoning downsampling when recombining and fusion of features, and maintains a constant dimension of feature representation, so that the image generated by the generator model can better reconstruct details, that is, improve global consistency in areas that are challenging to the convolution structure, such as larger uniform areas. Finally, the reconstructed screen image of the electronic device is displayed on the screen of the electronic device.

[0077] In particular, it should be understood that by generating a screen image from the text feature vector with a label through the dense prediction model architecture, an interface display can be implemented based on labeling based on the text, which fully ensures the accuracy of content classification.

[0078] Specifically, in the embodiment itself, the generator model is a convolutional neural network composed of alternating convolutional layers, upsampling layers and activation layers; wherein, the process of inputting the projection matrix into the generator model of the dense prediction model architecture to generate a reconstructed screen image of the electronic device includes: first, using the convolutional layer of the generator model to convolve the projection matrix to obtain a convolutional feature map; then, using the upsampling layer of the generator model connected to the convolutional layer to upsample the convolutional feature map to obtain an upsampled feature map; finally, using the activation layer of the generator model connected to the upsampling layer to activate the upsampled feature map to generate an activated feature map and output it to the convolutional layer of the next layer of the generator model. It is worth mentioning that in the generator model, the convolutional layers of different layers use different convolutional kernels for convolution operations.

[0079] In summary, the active perceptual interface display method of the embodiment of the present application is explained, which uses a converter model based on a word embedding layer and a conditional random field model to label a sequence of feature vectors generated based on the content in the screen image based on permission information to determine the subject tags of the content in the screen, and further uses a dense prediction model architecture to generate a screen image from the text feature vector with a label to achieve interface display based on text-based labeling. In this way, only the part of the displayable content with the subject tag is displayed. In this way, end-to-end active perceptual interface display can be achieved, thereby improving the security of the screen content information.

[0080] Exemplary Systems

[0081] Figure 5 FIG. 1 is a block diagram of an active perceptual interface display system according to an embodiment of the present application. Figure 5As shown, according to an embodiment of the present application, the active perceptual interface display system 500 includes: a screen image acquisition unit 510, which is used to acquire a screen image of an electronic device to be displayed; a text data extraction unit 520, which is used to extract text data from the screen image obtained by the screen image acquisition unit 510; a converter processing unit 530, which is used to perform word segmentation processing on the text data obtained by the text data extraction unit 520 and then obtain a sequence of word feature vectors through a converter model including a word embedding layer; a conditional random field unit 540, which is used to pass the sequence of word feature vectors obtained by the converter processing unit 530 through a conditional random field to annotate each word feature vector in the sequence of word feature vectors based on permission information to obtain a category label of each word feature vector; an extraction unit 550, which is used to extract multiple groups of one or more word feature vectors whose label categories belong to displayable content from the sequence of word feature vectors obtained by the converter processing unit 530 based on the category label of each word feature vector obtained by the conditional random field unit 540; a two-dimensional splicing unit 5 60, used for two-dimensionally splicing the one or more word feature vectors of each group of displayable content of the label category obtained by the text data extraction unit 550 to obtain multiple feature matrices with different scales; a fusion unit 570, used to use the projection recombination function of the dense prediction model architecture to fuse the multiple feature matrices obtained by the two-dimensional splicing unit 560 with different scales into a projection matrix, wherein the projection recombination function of the dense prediction model architecture includes three-dimensionally splicing all the multiple feature matrices to obtain a three-dimensional feature map and then projecting the three-dimensional feature map onto a two-dimensional plane; a generator model processing unit 580, used to input the projection matrix obtained by the fusion unit 570 into the generator model of the dense prediction model architecture to generate a reconstructed screen image of the electronic device, wherein the convolution layer of the generator model is suitable for splicing and up-sampling the convolution feature map obtained by convolution operations using different convolution kernels; and a display unit 590, used to display the reconstructed screen image of the electronic device obtained by the generator model processing unit 580 on the screen of the electronic device.

[0082] In one example, in the active awareness interface display system 500, the text data extraction unit 520 is further used to: perform text recognition processing on the screen image to obtain the text data.

[0083] In one example, in the active perception interface display system 500, as Figure 6As shown, the converter processing unit 530 includes: a word segmentation subunit 531, which is used to perform word segmentation processing on the text data to obtain a word sequence corresponding to the text data; a word embedding subunit 532, which is used to input each word in the word sequence obtained by the word segmentation subunit 531 into the word embedding layer of the converter model to map each word in the word sequence to a word vector to obtain a sequence of word vectors; and a converter subunit 533, which is used to input the sequence of word vectors obtained by the word embedding subunit 532 into the converter of the converter model to obtain a sequence of word feature vectors.

[0084] In one example, in the above-mentioned active perception interface display system 500, the fusion unit 570 is further used to: use the dense prediction model architecture to perform three-dimensional stitching on the multiple feature matrices with different scales to obtain the three-dimensional feature map; and use the dense prediction model architecture to project the three-dimensional feature map onto a two-dimensional plane to obtain the two-dimensional projection matrix.

[0085] In one example, in the above-mentioned active perception interface display system 500, the fusion unit 570 is further used to: use the dense prediction model architecture to perform scale adjustment based on linear transformation on each feature matrix of the multiple feature matrices with different scales to unify the scales of the multiple feature matrices.

[0086] In one example, in the above-mentioned active perceptual interface display system 500, the generator model is a convolutional neural network consisting of alternating convolutional layers, upsampling layers and activation layers; and the generator model processing unit 580 is further used to: use the convolutional layer of the generator model to convolve the projection matrix to obtain a convolutional feature map; use the upsampling layer of the generator model connected to the convolutional layer to upsample the convolutional feature map to obtain an upsampled feature map; and use the activation layer of the generator model connected to the upsampling layer to activate the upsampled feature map to generate an activated feature map and output it to the convolutional layer of the next layer of the generator model.

[0087] In one example, in the active perceptual interface display system 500, in the generator model, the convolution layers of different layers use different convolution kernels to perform convolution operations.

[0088] Here, those skilled in the art will appreciate that the specific functions and operations of the various units and modules in the active perception interface display system 500 have been described in detail above. Figures 1 to 4 The active perception interface display method has been introduced in detail in the description of the active perception interface display method, and therefore, its repeated description will be omitted.

[0089] As described above, the active perceptual interface display system 500 according to the embodiment of the present application can be implemented in various terminal devices, such as a server of an active perceptual interface display algorithm. In one example, the active perceptual interface display system 500 according to the embodiment of the present application can be integrated into the terminal device as a software module and / or a hardware module. For example, the active perceptual interface display system 500 can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the active perceptual interface display system 500 can also be one of the many hardware modules of the terminal device.

[0090] Alternatively, in another example, the active perception interface display system 500 and the terminal device may be separate devices, and the active perception interface display system 500 may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0091] Exemplary Electronic Devices

[0092] Below, reference Figure 7 To describe the electronic device according to the embodiment of the present application. Figure 7 As shown, the electronic device 10 includes one or more processors 11 and a memory 12. The processor 11 may be a central processing unit (CPU) or other processing units having data processing capability and / or instruction execution capability, and may control other components in the electronic device 10 to perform desired functions.

[0093] The memory 12 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may run the program instructions to implement the functions of the active perceptual interface display method of each embodiment of the present application described above and / or other desired functions. Various contents such as text data, projection matrix, etc. may also be stored in the computer-readable storage medium.

[0094] In one example, the electronic device 10 may further include: an input system 13 and an output system 14, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0095] The input system 13 may include, for example, a keyboard, a mouse, and the like.

[0096] The output system 14 can output various information to the outside, including the reconstructed screen image of the electronic device, etc. The output system 14 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0097] Of course, to simplify, Figure 7 Only some of the components related to the present application in the electronic device 10 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 10 may also include any other appropriate components.

[0098] Exemplary computer program products and computer-readable storage media

[0099] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the functions of the active perception interface display method according to various embodiments of the present application described in the above-mentioned "Exemplary Method" section of this specification.

[0100] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0101] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the active perception interface display method described in the above "Exemplary Method" section of this specification.

[0102] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, system or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0103] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0104] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0105] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0106] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0107] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. An active perception interface display method, characterized in that: include: Acquire a screen image of an electronic device to be displayed; extracting text data from the screen image; After word segmentation, the text data is passed through a converter model including a word embedding layer to obtain a sequence of word feature vectors; The sequence of word feature vectors is passed through a conditional random field to label each word feature vector in the sequence of word feature vectors based on permission information to obtain a category label for each word feature vector; Based on the category label of each of the word feature vectors, extracting multiple groups of one or more word feature vectors whose label categories belong to displayable content from the sequence of the word feature vectors; Two-dimensionally concatenate the feature vectors of one or more words in each group whose label category belongs to displayable content to obtain a plurality of feature matrices with different scales; Using a projection recombination function of a dense prediction model architecture to fuse the multiple feature matrices with different scales into a projection matrix, wherein the projection recombination function of the dense prediction model architecture includes three-dimensionally splicing all the multiple feature matrices to obtain a three-dimensional feature map and then projecting the three-dimensional feature map onto a two-dimensional plane; Inputting the projection matrix into a generator model of the dense prediction model architecture to generate a reconstructed screen image of the electronic device, wherein the convolution layer of the generator model is suitable for concatenating and upsampling convolution feature maps obtained by convolution operations using different convolution kernels; and Displaying the reconstructed screen image of the electronic device on a screen of the electronic device; The projection recombination function of the dense prediction model architecture is used to fuse the multiple feature matrices with different scales into a projection matrix, including: Using the dense prediction model architecture, three-dimensionally concatenate the multiple feature matrices with different scales to obtain the three-dimensional feature map; and Projecting the three-dimensional feature map onto a two-dimensional plane using the dense prediction model architecture to obtain a two-dimensional projection matrix; The method of using the dense prediction model architecture to perform three-dimensional splicing on the multiple feature matrices with different scales to obtain the three-dimensional feature map includes: Using the dense prediction model architecture, performing linear transformation-based scale adjustment on each of the plurality of feature matrices with different scales to unify the scales of the plurality of feature matrices; Wherein, the generator model is a convolutional neural network consisting of alternating convolutional layers, upsampling layers and activation layers; The step of inputting the projection matrix into the generator model of the dense prediction model architecture to generate a reconstructed screen image of the electronic device includes: Using the convolution layer of the generator model to convolve the projection matrix to obtain a convolution feature map; Upsampling the convolutional feature map using an upsampling layer of the generator model connected to the convolutional layer to obtain an upsampled feature map; and The upsampled feature map is activated using the activation layer of the generator model connected to the upsampled layer to generate an activated feature map and output to the convolution layer of the next layer of the generator model.

2. The active perception interface display method according to claim 1, wherein: Extracting text data from the screen image includes: performing text recognition processing on the screen image to obtain the text data.

3. The active perception interface display method according to claim 2, wherein: After word segmentation, the text data is processed through a converter model including a word embedding layer to obtain a sequence of word feature vectors, including: Performing word segmentation processing on the text data to obtain a word sequence corresponding to the text data; Inputting each word in the word sequence into the word embedding layer of the transformer model to map each word in the word sequence to a word vector to obtain a sequence of word vectors; and The sequence of word vectors is input into the converter of the converter model to obtain the sequence of word feature vectors.

4. The active perception interface display method according to claim 3, wherein: In the generator model, the convolution layers at different layers use different convolution kernels to perform convolution operations.

5. An active perception interface display system, characterized in that: include: A screen image acquisition unit, used to acquire a screen image of the electronic device to be displayed; a text data extraction unit, configured to extract text data from the screen image obtained by the screen image acquisition unit; A converter processing unit, configured to perform word segmentation processing on the text data obtained by the text data extraction unit and then obtain a sequence of word feature vectors through a converter model including a word embedding layer; A conditional random field unit, configured to pass the sequence of word feature vectors obtained by the converter processing unit through a conditional random field to label each word feature vector in the sequence of word feature vectors based on permission information to obtain a category label of each word feature vector; an extraction unit, configured to extract, based on the category labels of the word feature vectors obtained by each of the conditional random field units, a plurality of groups of one or more word feature vectors whose label categories belong to displayable content from the sequence of the word feature vectors obtained by the converter processing unit; A two-dimensional splicing unit, used for two-dimensionally splicing the groups of one or more word feature vectors of the label category belonging to the displayable content obtained by the text data extraction unit to obtain a plurality of feature matrices with different scales; A fusion unit, used to fuse the multiple feature matrices obtained by the two-dimensional splicing units with different scales into a projection matrix using a projection recombination function of a dense prediction model architecture, wherein the projection recombination function of the dense prediction model architecture includes three-dimensionally splicing all the multiple feature matrices to obtain a three-dimensional feature map and then projecting the three-dimensional feature map onto a two-dimensional plane; a generator model processing unit, configured to input the projection matrix obtained by the fusion unit into the generator model of the dense prediction model architecture to generate a reconstructed screen image of the electronic device, wherein the convolution layer of the generator model is suitable for concatenating and upsampling convolution feature maps obtained by performing convolution operations using different convolution kernels; and A display unit, configured to display the reconstructed screen image of the electronic device obtained by the generator model processing unit on a screen of the electronic device; The fusion unit is used to: use the dense prediction model architecture to perform three-dimensional splicing on the multiple feature matrices with different scales to obtain the three-dimensional feature map; and use the dense prediction model architecture to project the three-dimensional feature map onto a two-dimensional plane to obtain the two-dimensional projection matrix; The fusion unit is used to: use the dense prediction model architecture to perform scale adjustment based on linear transformation on each feature matrix of the multiple feature matrices with different scales to unify the scales of the multiple feature matrices; Wherein, the generator model is a convolutional neural network consisting of alternating convolutional layers, upsampling layers and activation layers; and the generator model processing unit is used to: use the convolutional layer of the generator model to convolve the projection matrix to obtain a convolutional feature map; use the upsampling layer of the generator model connected to the convolutional layer to upsample the convolutional feature map to obtain an upsampled feature map; and use the activation layer of the generator model connected to the upsampling layer to activate the upsampled feature map to generate an activated feature map and output it to the convolutional layer of the next layer of the generator model.

6. The active perception interface display system according to claim 5, wherein: The converter processing unit comprises: A word segmentation subunit, used for performing word segmentation processing on the text data to obtain a word sequence corresponding to the text data; A word embedding subunit, configured to input each word in the word sequence obtained by the word segmentation subunit into the word embedding layer of the converter model to map each word in the word sequence to a word vector to obtain a sequence of word vectors; and A converter subunit is used to input the sequence of word vectors obtained by the word embedding subunit into the converter of the converter model to obtain the sequence of word feature vectors.

7. An electronic device comprising: processor; as well as A memory, in which computer program instructions are stored, and when the computer program instructions are executed by the processor, the processor executes the active perception interface display method as described in any one of claims 1-4.

Images