Display device and screen projection processing method thereof

Abstract

The invention belongs to the technical field of screen projection, and provides display equipment and a screen projection processing method thereof. The decoder is configured to respond to the mirror image screen projection request of the source end equipment, and decode the coded data stream carried in the mirror image screen projection request to obtain decoded video data; the processor is respectively connected with the display screen and the decoder and is configured to analyze effective display areas of the plurality of video pictures in the decoded video data to obtain feature information of the plurality of video pictures; if it is determined that the effective display area contains the sensitive information based on the feature information of any video image, performing fuzzy processing on the sensitive information according to pixel point information corresponding to the sensitive information in the effective display area to obtain processed video data; and displaying the processed video data based on the display screen. The technical effects that the sensitive information in the mirror image projection content of the source end equipment is effectively processed by the display equipment, and the user privacy is protected are achieved.

Description

Technical Field

[0001] The present application belongs to the field of screen projection technology, and more specifically, to a display device and a screen projection processing method thereof. Background Art

[0002] With the popularization of devices such as smartphones, tablets and smart TVs, screen projection technology has gradually become an important means to connect different devices and achieve large-screen sharing.

[0003] However, during the screen projection process, sensitive information such as personal privacy or business secrets may be involved. If this information is leaked, it may cause losses to individuals or companies. At the same time, as people's awareness of privacy protection increases, how to protect sensitive information during the screen projection process has become an urgent problem to be solved.

[0004] However, there is currently no better solution in the industry. Generally, when users operate sensitive information, they must first disconnect the screen projection and then reconnect it after the user has completed the operation, which is time-consuming and laborious. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a display device and a screen projection processing method thereof, aiming to solve the technical problem that the prior art cannot effectively protect sensitive information in the projection video data during the projection process.

[0006] To achieve the above object, according to a first aspect of the present application, a display device is provided, the method comprising:

[0007] Display screen;

[0008] A decoder is configured to decode the encoded data stream carried in the mirroring screen projection request in response to the mirroring screen projection request of the source device to obtain decoded video data;

[0009] The processors connected to the display screen and the decoder are configured as follows:

[0010] Analyzing effective display areas of multiple video pictures in the decoded video data to obtain feature information of the multiple video pictures;

[0011] If it is determined based on the feature information of any one of the video images that sensitive information is contained in the effective display area, blurring the sensitive information according to the pixel point information corresponding to the sensitive information in the effective display area to obtain processed video data;

[0012] The processed video data is displayed based on the display screen.

[0013] Optionally, in a possible implementation manner of the first aspect, before the processor performs the analyzing of the effective display areas of the multiple video pictures in the decoded video data to obtain the feature information of the multiple video pictures, the processor is further configured to:

[0014] According to the preset interval duration, video frames are captured from the decoded video data to obtain the multiple video frames.

[0015] Optionally, in a possible implementation manner of the first aspect, before the processor performs the analyzing of the effective display areas of the multiple video pictures in the decoded video data to obtain the feature information of the multiple video pictures, the processor is further configured to:

[0016] Obtaining pixel points at different coordinate positions in each of the video images;

[0017] Determine an invalid display area of ​​each video screen according to the pixel points in each video screen;

[0018] The invalid display area of ​​each video screen is removed to obtain the valid display area of ​​each video screen.

[0019] Optionally, in a possible implementation manner of the first aspect, the processor is further configured to:

[0020] Inputting the feature information of the plurality of video images into a pre-trained video image analysis model, wherein the video image analysis model is obtained by training a convolutional neural network model based on a plurality of training samples, each of the training samples comprising: feature information of a sample video image and a sensitive information label;

[0021] By using the video picture analysis model, characteristic information of each video picture is analyzed respectively to obtain a comparative analysis result corresponding to each video picture;

[0022] According to the comparison and analysis results corresponding to each of the video images, it is determined whether each of the effective display areas contains the sensitive information.

[0023] Optionally, in a possible implementation manner of the first aspect, the processor performs blurring processing on the sensitive information according to pixel point information corresponding to the sensitive information in the effective display area to obtain processed video data, and is further configured to:

[0024] Determine the starting coordinates and area size parameters corresponding to the sensitive information according to the pixel point information corresponding to the sensitive information in the effective display area, wherein the starting coordinates include: a starting horizontal coordinate and a starting vertical coordinate, and the area size parameters include: width and height;

[0025] Determine the area to be blurred within the effective display area according to the starting coordinates and the area size parameters;

[0026] The pixel points in the area to be blurred are blurred to obtain the processed video data.

[0027] Optionally, in a possible implementation manner of the first aspect, before the processor performs blur processing on the pixels in the to-be-blurred area, the processor is further configured to:

[0028] Identify the boundary area between other areas in the effective display area and the area to be blurred;

[0029] The color values ​​of the pixels on the boundary area are changed to achieve a smooth and uniform transition between the area to be blurred and the other areas.

[0030] Optionally, in a possible implementation manner of the first aspect, the processor, during the process of displaying the processed video data based on the display screen, or after displaying the processed video data based on the display screen, is further configured to:

[0031] Continuously analyzing the feature information of each video picture through the video picture analysis model to obtain a comparative analysis result corresponding to each video picture;

[0032] If it is determined based on the comparative analysis results corresponding to each of the video images that none of the effective display areas contains the sensitive information, the blurring of the sensitive information is canceled to obtain restored video data;

[0033] The restored video data is displayed based on the display screen.

[0034] According to a second aspect of the present application, a screen projection processing method of a display device is provided, comprising:

[0035] In response to a mirroring screen projection request from a source device, decoding the encoded data stream carried in the mirroring screen projection request to obtain decoded video data;

[0036] Analyzing effective display areas of multiple video pictures in the decoded video data to obtain feature information of the multiple video pictures;

[0037] If it is determined based on the feature information of any one of the video images that sensitive information is contained in the effective display area, blurring the sensitive information according to the pixel point information corresponding to the sensitive information in the effective display area to obtain processed video data;

[0038] The processed video data is displayed based on a display screen of the display device.

[0039] Optionally, in a possible implementation manner of the second aspect, the method further includes:

[0040] Inputting the feature information of the plurality of video images into a pre-trained video image analysis model, wherein the video image analysis model is obtained by training a convolutional neural network model based on a plurality of training samples, each of the training samples comprising: feature information of a sample video image and a sensitive information label;

[0041] By using the video picture analysis model, characteristic information of each video picture is analyzed respectively to obtain a comparative analysis result corresponding to each video picture;

[0042] According to the comparison and analysis results corresponding to each of the video images, it is determined whether each of the effective display areas contains the sensitive information.

[0043] Optionally, in a possible implementation manner of the second aspect, the blurring the sensitive information according to the pixel point information corresponding to the sensitive information in the effective display area to obtain processed video data includes:

[0044] Determine the starting coordinates and area size parameters corresponding to the sensitive information according to the pixel point information corresponding to the sensitive information in the effective display area, wherein the starting coordinates include: a starting horizontal coordinate and a starting vertical coordinate, and the area size parameters include: width and height;

[0045] Determine the area to be blurred within the effective display area according to the starting coordinates and the area size parameters;

[0046] The pixel points in the area to be blurred are blurred to obtain the processed video data.

[0047] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0048] According to a third aspect of the present application, a display device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the display device implements any of the methods described in one embodiment.

[0049] According to a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the above is implemented.

[0050] According to a fifth aspect of the present application, a computer program product is provided. When the computer program product is run on a display device, the display device executes any one of the methods described in the first aspect.

[0051] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0052] The embodiment of the present application provides a display device and a screen projection processing method thereof. Through the coordinated work of components such as a decoder, a processor, and a display screen in the display device, the encoded data stream carried in the mirror projection request of the source device is first decoded and processed, and then the effective display areas of multiple video pictures in the decoded video data are analyzed to obtain feature information of the multiple video pictures, so as to identify sensitive information in the effective display area of ​​the video picture based on the feature information of the video picture, and blur the sensitive information according to the pixel point information corresponding to the sensitive information in the effective display area to obtain processed video data, and finally display the processed video data on the display screen, thereby achieving the technical effect of effective processing of sensitive information in the mirror projection content of the source device by the display device and protection of user privacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 is a structural schematic diagram of a display device provided in an embodiment of the present application;

[0055] Figure 2a It is a flowchart of an optional screen projection processing method provided in an embodiment of the present application;

[0056] Figure 2b It is a flowchart of another optional screen projection processing method provided in an embodiment of the present application;

[0057] Figure 3 is a structural diagram of an optional video picture analysis model provided in an embodiment of the present application;

[0058] Figure 4It is a schematic diagram of the structure of an optional convolution operation module provided in an embodiment of the present application;

[0059] Figure 5 is a flowchart of an optional screen projection processing method provided in another embodiment of the present application;

[0060] Figure 6 It is a structural schematic diagram of a screen projection processing device provided in an embodiment of the present application;

[0061] Figure 7 It is a structural schematic diagram of a display device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0063] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0064] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0065] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0066] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0067] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0068] First, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0069] Mirroring screen: refers to the real-time and original copying and projection of the entire screen content of the source device (such as mobile phones, tablets, computers, etc.), including interface layout, operation process, display effects, etc., to the display device (such as smart TVs, projectors, display screens and other receiving devices).

[0070] Adaptive Bayer Filter: In the field of image processing, it is a filter used for single-frame noise reduction in the Raw domain. It first uses a bilateral filter to remove noise and edges, and then performs soft threshold processing.

[0071] The above is a brief introduction to the nouns involved in the embodiments of the present application, which will not be repeated below.

[0072] This application example provides an example of a display device, please refer to Figure 1 As shown, Figure 1 The structure diagram of a display device provided by the present application is shown, and the display device 100 includes:

[0073] Display screen 101;

[0074] The decoder 102 is configured to decode the encoded data stream carried in the mirroring screen projection request in response to the mirroring screen projection request of the source device to obtain decoded video data;

[0075] The processors connected to the display screen 101 and the decoder 102 are configured as follows:

[0076] Analyze the effective display areas of multiple video pictures in the decoded video data to obtain feature information of the multiple video pictures;

[0077] If it is determined based on the feature information of any video screen that the effective display area contains sensitive information, the sensitive information is blurred according to the pixel point information corresponding to the sensitive information in the effective display area to obtain processed video data;

[0078] The processed video data is displayed based on a display screen.

[0079] Optionally, the display device provided in the example of the present application can be applied to, but is not limited to, the following technical scenarios: the display device is communicatively connected to a source device, and when the source device uses wireless technology to mirror the display device, sensitive information in the video data of the mirrored screen is protected.

[0080] Optionally, in the example of the present application, the display screen is a visual window used by the display device to present image content to the user. The type of the display screen may be, for example, a liquid crystal display (LCD), an organic light emitting diode display (OLED), etc.

[0081] Optionally, the decoder is used to decode the coded data stream of a specific coding format (for example, H.264 coding format, H.265 coding format, etc.) received by the display device to obtain decoded video data. When the source device (such as a mobile phone, computer, or other device that initiates screen projection) initiates a mirror screen projection request to the display device through any screen projection application (such as miracast and airplay, etc.), after the screen projection application in the display device receives the mirror screen projection request, it forwards the coded data stream carried in the mirror screen projection request to the decoder, and the decoder decodes the coded data stream according to the corresponding video coding rules (for example, H.264 coding standard, H.265 coding standard, etc.), thereby restoring the coded data stream into decoded video data that can be further processed and displayed by the display device.

[0082] In this application example, Figure 2a As shown, the processor in the display device is connected to the display screen and the decoder respectively. The processor is used to receive the decoded video data output by the decoder and analyze the multiple video pictures contained in the decoded video data to obtain the feature information corresponding to each of the multiple video pictures.

[0083] It should be understood that the above-mentioned multiple video screens refer to multiple frames of static images at different time points in the decoded video data, and the effective display area of ​​each video screen refers to the part of each video screen that truly carries meaningful image content and / or image content that needs to be displayed to the user for viewing. The effective display area is distinguished from invalid display areas such as black screen areas in the video screen.

[0084] In the example of the present application, the processor can determine the effective display area in the video screen by analyzing the color, brightness, texture and other characteristics of the pixels in each video screen through an image analysis algorithm, and extract the characteristic information of each video screen according to the effective display area of ​​each video screen. The above-mentioned characteristic information may include but is not limited to: software content characteristics (such as whether the application currently displayed on the video screen is communication software, shopping software, video software or online banking, and the specific text information and control information involved in the screen), color distribution characteristics (such as whether the overall color tone of the video screen involves sensitive information, etc.), screen layout (such as the positional relationship of each element in the video screen, etc.), etc.

[0085] The processor analyzes the feature information of each video screen obtained based on the image feature recognition algorithm to determine whether sensitive information is contained in the effective display area. In the example of this application, the above sensitive information may be, but is not limited to, content involving user privacy (such as communication software, unobstructed account password information, private chat records, etc.), or content that does not comply with relevant regulations, is prohibited or inappropriate for display (such as violence, pornography and other bad screen elements). Figure 2a As shown, if it is determined that any type of sensitive information exists in the effective display area based on the characteristic information of a certain video screen, the processor performs a fuzzy processing operation on the sensitive information according to the pixel point information corresponding to the sensitive information in the effective display area.

[0086] It should be understood that the above pixel information includes the position coordinates of each pixel in the video image and the color value it represents (for example, in the RGB color mode, each pixel corresponds to the specific values ​​of the three channels of red, green and blue). Specifically, a specific blur processing algorithm can be used to process the pixels in the area where the sensitive information is located in the effective display area, change the original color value and other information of the pixels in the area where the sensitive information is located, and then make the sensitive information blurred and difficult to recognize. After the above blur processing, the processed video data is obtained.

[0087] Still Figure 2a As shown, after the processor completes the blurring of sensitive information and obtains the processed video data, it transmits the processed video data to the display screen in the display device. After receiving the processed video data, the display screen displays the image content corresponding to the processed video data to the user according to the preset display principle and driving mechanism, so that the user finally sees the video picture after the processing and blurring of sensitive information, ensuring that the display content of the display device meets the user's privacy protection and related display requirements.

[0088] As an optional specific embodiment, Figure 2bAs shown, the screen projection application of the source device generates a mirror screen projection request in response to the user operation, and sends the mirror screen projection request to the screen projection application of the display device. After receiving the mirror screen projection request, the screen projection application wakes up the image abstraction layer of the display device and wakes up the display screen of the display device, and sends the mirror screen projection request to the decoder through the screen projection application of the display device. The decoder decodes the encoded data stream carried in the mirror screen projection request to obtain decoded video data, and the decoder sends the decoded video data to the display screen, and the display screen displays the decoded video data through the image display abstraction layer.

[0089] Afterwards, the screen recording module of the display device is used to capture multiple video screens in the decoded video data, and the invalid display area of ​​each video screen is removed. Then, the effective display area of ​​the multiple video screens in the decoded video data is analyzed by the recognition algorithm module in the processor to obtain the characteristic information of the multiple video screens; based on the characteristic information of each video screen, it is determined that the effective display area contains sensitive information. The recognition algorithm module of the processor sends a notification message to the screen projection application of the display device to notify the screen projection application that the effective display area contains sensitive information. Moreover, the fuzzy processing module in the processor blurs the sensitive information according to the pixel point information corresponding to the sensitive information in the effective display area to obtain the processed video data. Afterwards, the processed video data is sent to the display screen through the fuzzy processing module in the processor, and then the processed video data is displayed on the display screen.

[0090] Through the collaborative work of components such as the decoder, processor and display screen in the display device, the encoded data stream carried in the mirror projection request of the source device is first decoded and processed, and then the effective display areas of multiple video pictures in the decoded video data are analyzed to obtain feature information of the multiple video pictures, so as to identify sensitive information in the effective display area of ​​the video picture based on the feature information of the video picture, and blur the sensitive information according to the pixel information corresponding to the sensitive information in the effective display area to obtain the processed video data, and finally display the processed video data on the display screen, thereby achieving the technical effect of effective processing of sensitive information in the mirror projection content of the source device on the display device and protection of user privacy.

[0091] It should be noted that this application example has no specific requirements for the network environment. Since the resolution of the display screen of the source device is generally 1080P, the minimum network bandwidth needs to reach the transmission rate that can support the video code stream. In addition, this application example has no specific requirements for the device model of the source device, and can support the screen projection protocol between the source device and the display device.

[0092] In a possible implementation, in the process of the display device processing the mirror projection content of the source device, before the processor performs analysis on the effective display area of ​​multiple video pictures in the decoded video data and obtains the feature information of the multiple video pictures, it is also configured as follows:

[0093] According to the preset interval duration, video frames are captured from the decoded video data to obtain multiple video frames.

[0094] Specifically, the above-mentioned preset interval duration is a time interval parameter pre-set by the display device. For example, it can be pre-set to every 500 milliseconds (the specific duration can be determined based on actual needs and factors such as display device performance, and this application example is not specifically limited to this) as a time interval duration to ensure that the captured video images can cover a certain range of video content without being too frequent (avoiding excessive data processing that causes excessive performance pressure on the display device) or too sparse (ensuring that important image information is not missed).

[0095] In one example, the processor can perform a video screen capture operation on the decoded video data that has been processed by the decoder based on the capture interface in the display device, by transmitting the level of the current screen (which node of the decoded video data it belongs to), and the starting coordinates (x, y), width, height and other information of the screen to be captured, according to the set preset interval duration. It should be understood that the above-mentioned decoded video data is a collection of a series of video screen information arranged in chronological order. The quality of the captured screen is consistent with the resolution of the current video stream, and the capture speed is within 100ms / frame, which does not affect the performance of the display device. The captured multiple video screen information exists in a digital form, including pixel point data, color information and other content of each video screen.

[0096] For example, assuming that the preset interval length is 1 second, and the decoded video data corresponds to a 10-second video content, the processor can capture the corresponding video frames from the entire decoded video data at the 1st second, 2nd second, 3rd second... and so on, until the 10th second and other time nodes, and obtain a total of 10 video frames.

[0097] The processor captures the decoded video data at preset intervals, providing the necessary data basis for the processor to conduct in-depth analysis of the video and ensure that the content displayed on the screen complies with sensitive information protection requirements.

[0098] In a possible implementation, before the processor analyzes the effective display areas of multiple video pictures in the decoded video data to obtain feature information of the multiple video pictures, the processor is further configured to:

[0099] Get the pixel points at different coordinate positions in each video screen;

[0100] Determine the invalid display area of ​​each video screen according to the pixel points in each video screen;

[0101] The invalid display area of ​​each video picture is removed to obtain the valid display area of ​​each video picture.

[0102] In the example of this application, before the processor analyzes the effective display area of ​​the video screen to obtain feature information, it needs to pre-process each video screen, that is, by obtaining the pixel points at different coordinate positions in each video screen, determining and removing the invalid display area in each video screen, so as to obtain the effective display area in each video screen. The purpose of this is to exclude some irrelevant parts in each video screen that may interfere with the analysis and judgment of the processor, and ensure that the feature information of the video screen can be extracted more accurately based on the accurate effective display area, and then effectively determine whether there is sensitive information in the video screen.

[0103] It should be understood that, in essence, each video screen is composed of multiple pixels, and the multiple pixels are distributed in the video screen in a certain arrangement (such as a two-dimensional row and column form), and each pixel has a corresponding coordinate position. For example, in an image coordinate system with the upper left corner as the coordinate origin (the coordinate value is (0,0)), the horizontal direction of the screen is the x-axis, and the vertical direction is the y-axis, and each pixel can be represented by a unique (x,y) coordinate to represent its position in the screen.

[0104] The processor accesses the data of each pixel in each video screen, for example, including the coordinate position of each pixel in the video screen and the color information represented by the pixel, so as to obtain all pixels at different coordinate positions, so as to determine the invalid display area of ​​each video screen according to the pixel in each video screen.

[0105] Optionally, the invalid display area refers to the part of the video screen that does not carry valid visual content and does not need to participate in subsequent analysis, such as a black screen area, a part without an image due to a transmission error or device adaptation problem, etc. As an optional example, the processor can determine the main judgment method of the invalid display area based on the acquired pixel information, but is not limited to the following two methods:

[0106] Judgment based on color value: Under normal circumstances, the color values ​​of the pixels in the invalid display area are relatively simple and meet specific characteristics. For example, in the black screen area, the values ​​of the red, green, and blue channels are all close to 0 (i.e., they appear black) in the RGB color mode. The processor traverses each pixel in each video screen. If it detects whether the color value of each pixel is consistent with the color value range of the pixel in the invalid display area, it is marked as an invalid display area.

[0107] Combined with the judgment of pixel distribution law: In addition to color values, the distribution law of pixel points can also assist in judging invalid display areas. For example, if the color values ​​of pixels in a certain area have no regular changes, or lack a reasonable connection and transition with the pixels in the surrounding normal display area, it can be determined that the area may be the missing part of the image due to data loss or error, that is, the invalid display area. Through the above comprehensive analysis of the color values, distribution laws, etc. of the pixels in the video screen, the processor can determine the specific location and specific range of the invalid display area in each video screen. After determining the invalid display area, the processor takes corresponding operations to remove the invalid display area, thereby obtaining an area containing only valid visual content, that is, the valid display area. In the example of this application, the operation of removing the invalid display area can be achieved by image cropping, data removal, etc. For example, if it is determined that the upper left corner of a video screen is an invalid display area, and the range of coordinate positions is a rectangular area with a horizontal coordinate ranging from 0 to 100 and a vertical coordinate ranging from 0 to 50, the processor removes the pixel point data in the above rectangular area from the pixel data set of the video screen by adjusting the index range of the image data, and only retains the pixel point data corresponding to the remaining valid display area, so as to facilitate further feature analysis and other operations based on the valid display area, thereby ensuring the accuracy and effectiveness of the video screen processing flow.

[0108] In a possible implementation, the processor is further configured to:

[0109] Inputting feature information of multiple video images into a pre-trained video image analysis model, wherein the video image analysis model is obtained by training a convolutional neural network model based on multiple training samples, each training sample including: feature information of a sample video image and a sensitive information label;

[0110] Through the video picture analysis model, the feature information of each video picture is analyzed respectively to obtain the comparative analysis result corresponding to each video picture;

[0111] Based on the comparison and analysis results corresponding to each video screen, determine whether each effective display area contains sensitive information.

[0112] In the screen projection processing flow of the display device, the processor uses a pre-trained video picture analysis model to determine whether sensitive information is contained in the effective display area. The video picture analysis model is obtained by pre-training the convolutional neural network CNN model based on a large number of training samples. Each training sample contains two important pieces of information: feature information of the sample video picture and a sensitive information label. The feature information of the sample video picture is the feature information obtained by feature extraction of the sample video picture, including various features such as texture, color distribution, object shape, contour, etc.; the sensitive information label is used to mark whether the sample video picture contains sensitive information. For example, "1" indicates that sensitive information is contained, and "0" indicates that sensitive information is not contained. Through training, the video picture analysis model can learn the characteristics of sensitive information, and then the video picture analysis model can analyze and judge new video pictures, thereby improving the accuracy and efficiency of detecting sensitive information in video pictures.

[0113] It should be understood that the convolutional neural network (CNN) model is a deep learning network model that is mostly used in image recognition, analysis and other fields. Through structures such as convolutional layers, pooling layers and fully connected layers, it can automatically learn the feature information in the image, and is particularly good at extracting local features in the image and combining local features into more advanced feature representations.

[0114] The processor uses the feature information extracted from multiple video images as the input data of the pre-trained video image analysis model, and uses any calculation and analysis method, including but not limited to: threshold calculation, random forest, multi-layer perceptron (MLP), deep learning, etc., to analyze the feature information of each video image respectively, and obtain the comparative analysis results corresponding to each video image: Figure 3 In the video picture analysis model shown, each block represents an operation module, which is a superposition of convolution calculations, depthwise separable convolution operations, linear integration units and other calculation methods. The connection between adjacent modules is represented by arrows, and the horizontal connection between adjacent modules (horizontal arrows) represents that the result of the operation of the previous operation module is directly input into the next operation module.

[0115] In the video image analysis model, the feature information of the input video image is first convolved through the convolution layer to automatically extract features at different levels. After multiple convolution, pooling and other operations, the extracted features are finally mapped to the final output results through the fully connected layer, that is, the comparative analysis results corresponding to each video image are obtained. Optionally, the comparative analysis result can be a probability value, indicating the probability that the video image contains sensitive information, or a category label, such as "contains sensitive information" or "does not contain sensitive information".

[0116] The processor determines whether the effective display area contains sensitive information based on the comparative analysis result obtained from the video image analysis model. For example, if the comparative analysis result is a probability value, a threshold value, such as 0.5, can be set. When the probability value is greater than the predetermined threshold value, the processor determines that the effective display area of ​​the video image contains sensitive information; for another example, if the comparative analysis result is a category label, the processor directly determines whether sensitive information exists based on the label.

[0117] In one example, the output of the video picture analysis model is an array matrix with I rows and J columns, and the input data is an array matrix with M rows and N columns and a convolution kernel kernel. The output result is:

[0118] output(i,j)=∑ M ∑ N input(m,n)×kernel(im,jn);

[0119] In the video picture analysis model, if the depth separable convolution operation module is a variant of the convolution operation module, the original convolution operation module can be added or replaced in the depth separable convolution operation module according to the empirical effect. For the linear integration unit, all pixels of the intermediate operation matrix undergo a nonlinear calculation. The nonlinear function f(x) of the linear integration unit is:

[0120] f(x)=max(0,x);

[0121] In an optional embodiment, any nonlinear function can replace the linear integration unit, such as sigmoid function, tanh function, etc. However, in the experimental results of this embodiment, the linear integration unit has the smallest calculation amount and the best effect.

[0122] In some embodiments, Figure 4 As shown, the convolution operation module is sequentially executed by convolution calculation, linear integration unit, depth-separable convolution operation, linear integration unit, and convolution operation. The calculation process is the process of encoding or decoding the input image into abstract features.

[0123] Optionally, in the example of the present application, some sample video images with the same resolution as the video stream data to be subsequently projected and marked with sensitive information label values ​​0 and 1 can be obtained in advance to participate in pre-training. The video image analysis model finally trained makes the output of the two numerical values ​​[value1, value2] between 0 and 1 in the cross entropy calculation involving the Sigmoid function, and the maximum value of [value1, value2] is the final output result. Specifically, the output result predict of the video image analysis model is a binary classification result.

[0124] Among them, the training loss Loss can be obtained by cross entropy calculation, that is:

[0125] Loss=label×ln[σ(predict)]+[1-label]×ln[σ(predict)];

[0126] In the above formula, σ is the Sigmoid function, namely:

[0127]

[0128] By leveraging the powerful feature learning and classification capabilities of the deep learning model, the processor can more intelligently and accurately judge the sensitive information in the video screen, improve the display device's ability to detect and process sensitive information, and based on whether the effective display area contains sensitive information, the processor can take corresponding processing measures for the effective display area containing sensitive information in the subsequent screen projection processing flow, such as blurring or other security protection operations, to ensure that the projection content played by the display device meets the user's information security and privacy requirements.

[0129] In a possible implementation, the processor performs blurring processing on the sensitive information according to pixel information corresponding to the sensitive information in the effective display area to obtain processed video data, and is further configured as follows:

[0130] According to the pixel point information corresponding to the sensitive information in the effective display area, the starting coordinates and area size parameters corresponding to the sensitive information are determined.

[0131] According to the starting coordinates and area size parameters, the area to be blurred within the effective display area is determined.

[0132] The pixels in the blurred area are blurred to obtain processed video data.

[0133] Optionally, the starting coordinates include: a starting horizontal coordinate and a starting vertical coordinate, and the area size parameters include: width and height.

[0134] In the example of this application, in the processing flow of the display device, after the processor determines that there is sensitive information in the effective display area, it determines the specific area where the sensitive information is located, and blurs the pixels in the area, and finally obtains the processed video data to protect privacy or comply with relevant regulations.

[0135] The processor first determines the starting horizontal coordinate and starting vertical coordinate of the sensitive information in the video screen according to the pixel point information corresponding to the sensitive information in the effective display area. The starting horizontal coordinate and the starting vertical coordinate jointly determine the starting coordinate of the area where the sensitive information is located. For example, in an image coordinate system with the upper left corner as the coordinate origin (0,0), for a sensitive information located in the video screen, the starting horizontal coordinate x and the starting vertical coordinate y are (300,200), respectively, indicating that the area where the sensitive information is located starts from the position of 300 pixels at the horizontal coordinate and 200 pixels at the vertical coordinate.

[0136] At the same time, the processor also needs to determine the area size parameters of the area where the sensitive information is located based on the pixel information corresponding to the sensitive information. In the example of this application, the area size parameters are represented by the two parameters of width and height. Specifically, the width represents the number of pixels occupied by the sensitive information area in the horizontal direction, and the height represents the number of pixels occupied by the sensitive information area in the vertical direction. According to the starting horizontal coordinate x and the starting vertical coordinate y of the sensitive information area, the width and height of the sensitive information area can accurately describe the scope of the sensitive information area.

[0137] Based on the starting coordinates and area size parameters corresponding to the sensitive information, the processor can delineate an area to be blurred that needs to be blurred within the effective display area to avoid interfering with other normal display areas. It should be noted that the blurred area is formed by using the starting coordinates as the upper left corner vertex of the rectangular area, and the length and width of the area to be blurred are determined by the width and height in the area size parameters. For example, with the starting coordinates (300, 200), a width of 100 pixels, and a height of 50 pixels, the area to be blurred can be understood as a rectangular area with the upper left vertex at (300, 200) and the lower right vertex at (400, 250).

[0138] In one example, the processor can call the Video blur interface to blur the pixels in the blurred area, for example, changing the color value of the pixel to blur the image details in the area and making it difficult to identify sensitive information. An optional blurring method is to use a weighted average algorithm to calculate the weighted average of the color value of each pixel in the blurred area and the color values ​​of the surrounding pixels. For example, in RGB color mode, for any pixel that needs to be blurred, the color values ​​of the red, green, and blue channels of the pixels around the pixel can be calculated according to the predetermined weights to finally obtain a new, smoother color value.

[0139] The processor traverses all the pixels in the area to be blurred and calculates all the pixels in the area to be blurred according to the selected blur algorithm (such as Gaussian blur algorithm). In this way, the pixels that originally clearly displayed sensitive information will become blurred, and the image contours and details of the area to be blurred will gradually disappear, thereby achieving the effect of blurring sensitive information.

[0140] After the processor completes the blurring of the pixels in the blurred area, the original sensitive information area in the video data is blurred. Finally, the sensitive information area in the processed video data has become blurred, while other unprocessed areas remain the same. The processor passes the processed video data containing blurred sensitive information to the display screen, so that when the display screen plays the video data, the user sees the blurred video picture, which can not only watch non-sensitive video data normally, but also avoid the leakage of sensitive information in the video data, thereby ensuring the security and compliance of screen projection in the display device.

[0141] In a possible implementation, before the processor performs blur processing on the pixels in the to-be-blurred area, the processor is further configured to:

[0142] Identify the boundary area between other areas in the effective display area and the area to be blurred;

[0143] Change the color value of the pixel points on the boundary area to make the transition between the blurred area and other areas smooth and uniform.

[0144] When the display device processes the area to be blurred where sensitive information is located in the video screen, it is necessary not only to blur the pixels in the area to be blurred, but also to consider the connection between the area to be blurred and other normal display areas in the video screen. By changing the color value of the pixels in the boundary area between other areas and the area to be blurred, a smooth and uniform transition is achieved between the area to be blurred and other areas, avoiding obvious visual faults, so that the processed video screen displayed on the display screen looks more natural and coordinated, improving the user's viewing experience, and ensuring the visual coherence of the video screen after the sensitive information is blurred.

[0145] Specifically, in the example of this application, the boundary area is located in the effective display area, between the area to be blurred (i.e., the area containing sensitive information that needs to be blurred) and other areas in the picture that are normally displayed and not affected by the blurring process. It can be understood as a "band-shaped" area surrounding the area to be blurred, and the width can be determined by a preset rule or according to the actual situation of the video picture, for example, pre-set to a specific width range of N pixels.

[0146] The processor calls the Video Blur interface and uses the HW module (hardware) to identify areas in the image where color changes form contour lines. These areas are usually parts of the image where color transitions are more obvious, that is, boundary areas or edge areas. It determines whether each pixel is within a predetermined range extending outward from the edge of the area to be blurred (the predetermined range is set according to the width of the boundary area). If a pixel is within the predetermined range, the pixel is considered to belong to the boundary area.

[0147] For example, assuming that the starting coordinates of a region to be blurred are (100, 100), the width in the region size parameters is 50 pixels, the height is 30 pixels, and the width of the boundary region is set to 2 pixels, then the pixels with a horizontal coordinate range of 98 to 102 and 148 to 152, and a vertical coordinate range of 98 to 102 and 128 to 132 will be identified as pixels in the boundary region.

[0148] In order to achieve a smooth and uniform transition effect, the hardware module applies a filter (such as an adaptive Bayer filter ABF) to adjust the color value of the pixel in the boundary area. In one example, the processor can determine the new color value of the pixel to be adjusted based on the color difference between adjacent pixels of the pixel to be adjusted and the relative position relationship with the area to be blurred and other areas.

[0149] For example, in the RGB color mode, for the pixel points in the boundary area close to the side of the area to be blurred, the color value of the pixel points can be adjusted to gradually approach the color value of the pixel points in the area to be blurred after subsequent blurring; and for the pixel points close to other normal areas, the color value of the pixel points can be adjusted to approach the color value of the pixel points in the normal area. Through the above-mentioned color uniformity adjustment method, the boundary area can naturally transition from the color of the area to be blurred to the color of other areas, thereby reducing the sharp edges in the image.

[0150] In an optional example, it is assumed that after the area to be blurred is blurred, the RGB color values ​​of the pixels at the edge of the area to be blurred are (100, 100, 100), while the color values ​​of the pixels in other normal areas adjacent to the area to be blurred are (200, 200, 200), and the width of the boundary area is 3 pixels. For a row of pixels in the boundary area that is closest to the area to be blurred, the processor can adjust the color values ​​of the row of pixels to color values ​​close to (100, 100, 100) but slightly transitioning to (200, 200, 200), such as (120, 120, 120) according to a predetermined algorithm (such as a linear interpolation algorithm, allocating color value weights according to the distance ratio to the areas on both sides); for the pixels in the middle of the boundary area, continue to adjust their color values ​​to be closer to the color values ​​of the normal area, such as (150, 150, 150); and for the row of pixels in the boundary area that is closest to the normal area, adjust their color values ​​to values ​​that are close to the color values ​​of the normal area but slightly have the color characteristics of the area to be blurred, such as (180, 180, 180).

[0151] Through the above example, before blurring the pixels in the area to be blurred, the processor adjusts the color values ​​of the pixels in the boundary area row by row and point by point, so as to achieve a smooth and uniform transition in color between the area to be blurred and other areas, ensuring that when the processed video data is displayed, the transition from the area to be blurred to other normal areas is natural and smooth, and there is no obvious sense of visual separation in the entire video picture.

[0152] In a possible implementation, the processor is further configured to:

[0153] Continuously analyzing the feature information of each video picture through the video picture analysis model to obtain a comparative analysis result corresponding to each video picture;

[0154] If it is determined based on the comparative analysis results corresponding to each video screen that no sensitive information is contained in each valid display area, the blurring of the sensitive information is canceled to obtain restored video data;

[0155] The restored video data is displayed based on the display screen.

[0156] In the process of display devices processing projection video data, it is necessary not only to detect and blur the possible sensitive information in the initial stage and display the processed video data, but also to continuously monitor the subsequent video images. The video image analysis model is used to continuously determine whether there is still sensitive information in the video image. If each valid display area of ​​the subsequent multiple video images does not contain sensitive information, the previous blurring of the pixels in the blurred area is canceled to restore the original clear state of the video image, so as to ensure that the user is provided with clear and complete viewing content to the maximum extent possible while meeting information security requirements, thereby improving the user viewing experience.

[0157] During the process of displaying processed video data based on the display screen, that is, while the video screen is being played or after the processed video data is displayed, the processor continuously and regularly (the specific time interval can be set according to factors such as device performance and actual needs) obtains new video screens.

[0158] The processor takes the feature information extracted from multiple video pictures as input and passes it to a pre-trained video picture analysis model. The feature information of each video picture is analyzed separately through the knowledge learned by the video picture analysis model during the training process. For example, the feature information of the pre-acquired sample video picture is used as a reference standard and compared with the feature information of each newly acquired video picture to determine the degree of similarity between each video picture and the sample video picture, thereby obtaining the comparative analysis results corresponding to each video picture.

[0159] The processor determines whether there is still sensitive information in the effective display area of ​​each video screen according to the comparative analysis results output by the video screen analysis model. If, after analysis, the effective display area of ​​all video screens no longer contains content similar to the sample video screen features that can be determined to be sensitive information, it is determined that the current video screen meets the conditions for canceling the blurring process.

[0160] Since the area to be blurred where the sensitive information is located has been identified during the blurring process, the processor can now use the opposite logic to restore the information such as the color values ​​of the pixels in the area to be blurred that have been changed by the blurring process to their original state before blurring. For example, the blurring effect was previously achieved by changing the color values ​​of the pixels in the area to be blurred by algorithms such as weighted average. Now, the color values ​​of the above pixels can be reversed, and the color values ​​of these pixels can be restored according to the original data records or through the corresponding restoration algorithm to restore them to the state before the sensitive information blurring process was performed, thereby obtaining the restored video data.

[0161] After obtaining the restored video data, the processor passes the restored video data to the display screen. The display screen displays the image content corresponding to the restored video data according to the display principle and driving mechanism, and then the video screen seen by the user changes from the previous blurred state back to the original clear state. When the video screen displayed on the screen no longer contains sensitive information, the display device adjusts the display state in real time to present all video screens completely and clearly, while ensuring the security and compliance of the projection information, optimizing the user's viewing experience as much as possible.

[0162] This application example provides an example of a screen projection processing method for a display device, please refer to Figure 5 As shown, Figure 5 A schematic flow chart of a screen projection processing method for a display device provided by the present application is shown. As an example but not a limitation, the method can be applied to or run in any display device. The method includes:

[0163] S501, in response to a mirroring screen projection request from a source device, decoding the encoded data stream carried in the mirroring screen projection request to obtain decoded video data;

[0164] S502, analyzing effective display areas of multiple video pictures in the decoded video data to obtain feature information of the multiple video pictures;

[0165] S503, if it is determined based on the feature information of any video screen that the effective display area contains sensitive information, the sensitive information is blurred according to the pixel point information corresponding to the sensitive information in the effective display area to obtain processed video data;

[0166] S504, displaying the processed video data based on a display screen of a display device.

[0167] It should be understood that the above-mentioned multiple video screens refer to multiple frames of static images at different time points in the decoded video data, and the effective display area of ​​each video screen refers to the part of each video screen that truly carries meaningful image content and / or image content that needs to be displayed to the user for viewing. The effective display area is different from the invalid black screen area that may appear in the video screen.

[0168] In the example of the present application, an image analysis algorithm can be used to analyze the color, brightness, texture and other features of the pixels in each video screen to determine the effective display area in the video screen, and the feature information of each video screen can be extracted based on the effective display area of ​​each video screen. The above feature information can include but is not limited to: software content features (such as whether the application currently displayed on the video screen is communication software, shopping software, video software or online banking, and the specific text information and control information involved in the screen), color distribution characteristics (such as whether the overall color tone of the video screen involves sensitive information, etc.), screen layout (such as the positional relationship of each element in the video screen, etc.), etc.

[0169] In the example of this application, the characteristic information of each video screen obtained by analysis is analyzed one by one based on the image feature recognition algorithm to determine whether sensitive information is contained in the effective display area. Optionally, the above-mentioned sensitive information may be, but is not limited to, content involving user privacy (such as unobstructed account password information, private chat records, etc.), or content that does not comply with relevant regulations, is prohibited or inappropriate for display (such as violence, pornography and other bad screen elements). If it is determined that any type of sensitive information exists in the effective display area based on the characteristic information of a certain video screen, a fuzzy processing operation is performed on the sensitive information according to the pixel point information corresponding to the sensitive information in the effective display area.

[0170] It should be understood that the above pixel information includes the position coordinates of each pixel in the video image and the color value it represents (for example, in the RGB color mode, each pixel corresponds to the specific values ​​of the three channels of red, green and blue). Specifically, a specific blur processing algorithm can be used to process the pixels in the area where the sensitive information is located in the effective display area, change the original color value and other information of the pixels in the area where the sensitive information is located, and then make the sensitive information blurred and difficult to recognize. After the above blur processing, the processed video data is obtained.

[0171] After the sensitive information is blurred and processed video data is obtained, the processed video data is transmitted to the display screen in the display device. After receiving the processed video data, the display screen displays the image content corresponding to the processed video data to the user according to the preset display principle and driving mechanism, so that the user finally sees the processed video picture with blurred protection of sensitive information, ensuring that the display content of the display device meets the user's privacy protection and related display requirements.

[0172] The method first decodes the encoded data stream carried in the mirror projection request of the source device, and then analyzes the effective display areas of multiple video pictures in the decoded video data to obtain feature information of the multiple video pictures, so as to identify sensitive information in the effective display area of ​​the video picture based on the feature information of the video picture, and blur the sensitive information according to the pixel point information corresponding to the sensitive information in the effective display area to obtain processed video data, and finally displays the processed video data on the display screen, thereby achieving the technical effect of effective processing of sensitive information in the mirror projection content of the source device on the display device and protection of user privacy.

[0173] In a possible implementation, the method further includes:

[0174] Inputting feature information of multiple video images into a pre-trained video image analysis model, wherein the video image analysis model is obtained by training a convolutional neural network model based on multiple training samples, each training sample including: feature information of a sample video image and a sensitive information label;

[0175] Through the video picture analysis model, the feature information of each video picture is analyzed respectively to obtain the comparative analysis result corresponding to each video picture;

[0176] Based on the comparison and analysis results corresponding to each video screen, determine whether each effective display area contains sensitive information.

[0177] In the screen projection processing flow of the display device, a pre-trained video picture analysis model can be used to determine whether sensitive information is contained in the effective display area. The video picture analysis model is obtained by pre-training the convolutional neural network (CNN) model based on a large number of training samples. Each training sample contains two important pieces of information: feature information of the sample video picture and a sensitive information label. The feature information of the sample video picture is the feature information obtained by extracting features from the sample video picture, including various features such as texture, color distribution, object shape, and contour; the sensitive information label is used to mark whether the sample video picture contains sensitive information. For example, "1" indicates that sensitive information is contained, and "0" indicates that sensitive information is not contained. Through training, the video picture analysis model can learn the characteristics of sensitive information, and then the video picture analysis model can analyze and judge new video pictures, thereby improving the accuracy and efficiency of detecting sensitive information in video pictures.

[0178] It should be understood that the convolutional neural network (CNN) model is a deep learning network model that is mostly used in image recognition, analysis and other fields. Through structures such as convolutional layers, pooling layers and fully connected layers, it can automatically learn the feature information in the image, and is particularly good at extracting local features in the image and combining local features into more advanced feature representations.

[0179] In one example, feature information extracted from multiple video images is used as input data for a pre-trained video image analysis model. The feature information of each video image is analyzed separately using the knowledge learned by the video image analysis model during the training process to obtain a comparative analysis result corresponding to each video image: In the video image analysis model, the input feature information is first convolved through a convolution layer to automatically extract features at different levels. After multiple convolution, pooling and other operations, the extracted features are finally mapped to the final output results through a fully connected layer, i.e., a comparative analysis result corresponding to each video image is obtained. Optionally, the comparative analysis result can be a probability value indicating the likelihood that the video image contains sensitive information, or a category label, such as: "contains sensitive information" or "does not contain sensitive information."

[0180] Whether the effective display area contains sensitive information is determined based on the comparative analysis result obtained from the video image analysis model. For example, if the comparative analysis result is a probability value, a threshold value, such as 0.5, can be set. When the probability value is greater than the predetermined threshold value, it is determined that the effective display area of ​​the video image contains sensitive information; for another example, if the comparative analysis result is a category label, it can be determined based on the category label whether the effective display area of ​​the video image contains sensitive information.

[0181] By leveraging the powerful feature learning and classification capabilities of deep learning models, it is possible to more intelligently and accurately judge sensitive information in video images, improve the display device's ability to detect and process sensitive information, and based on whether the effective display area contains sensitive information, in the subsequent screen projection processing flow, take corresponding processing measures for the effective display area containing sensitive information, such as blurring or other security protection operations, to ensure that the projection content played by the display device meets the user's information security and privacy requirements.

[0182] In a possible implementation, the sensitive information is blurred according to the pixel information corresponding to the sensitive information in the effective display area to obtain processed video data, including:

[0183] According to the pixel point information corresponding to the sensitive information in the effective display area, the starting coordinates and area size parameters corresponding to the sensitive information are determined.

[0184] According to the starting coordinates and area size parameters, the area to be blurred within the effective display area is determined.

[0185] The pixels in the blurred area are blurred to obtain processed video data.

[0186] Optionally, the above starting coordinates include: a starting horizontal coordinate and a starting vertical coordinate, and the area size parameters include: width and height.

[0187] In the example of this application, in the processing flow of the display device, after determining that sensitive information exists in the effective display area, the specific area where the sensitive information is located is determined, and the pixels in the area are blurred to finally obtain processed video data to protect privacy or comply with relevant regulations.

[0188] According to the pixel point information corresponding to the sensitive information in the effective display area, the starting horizontal coordinate and the starting vertical coordinate of the sensitive information in the video screen are first determined, and the starting horizontal coordinate and the starting vertical coordinate jointly determine the starting coordinate of the area where the sensitive information is located. For example, in an image coordinate system with the upper left corner as the coordinate origin (0,0), for a sensitive information located in the video screen, the starting horizontal coordinate x and the starting vertical coordinate y are (300,200), respectively, indicating that the area where the sensitive information is located starts from the position of 300 pixels at the horizontal coordinate and 200 pixels at the vertical coordinate.

[0189] At the same time, the area size parameters of the area where the sensitive information is located can also be determined based on the pixel information corresponding to the sensitive information. In the example of this application, the area size parameters are represented by the two parameters of width and height. Specifically, the width represents the number of pixels occupied by the sensitive information area in the horizontal direction, and the height represents the number of pixels occupied by the sensitive information area in the vertical direction. According to the starting horizontal coordinate x and the starting vertical coordinate y of the sensitive information area, the width and height of the sensitive information area can accurately describe the scope of the sensitive information area.

[0190] Based on the starting coordinates and area size parameters corresponding to the sensitive information, a blurred area that needs to be blurred can be delineated within the effective display area to avoid interfering with other normal display areas. It should be noted that the blurred area is formed by using the starting coordinates as the upper left corner vertex of the rectangular area, and the length and width of the blurred area are determined by the width and height in the area size parameters. For example, taking the starting coordinates (300, 200), the width is 100 pixels, and the height is 50 pixels, the area to be blurred can be understood as a rectangular area with the upper left corner vertex at (300, 200) and the lower right corner vertex at (400, 250).

[0191] In one example, the processor can call the Video blur interface to blur the pixels in the blurred area, for example, changing the color value of the pixel to blur the image details in the area and making it difficult to identify sensitive information. An optional blurring method is to use a weighted average algorithm to calculate the weighted average of the color value of each pixel in the blurred area and the color value of the surrounding pixels. For example, in RGB color mode, for any pixel that needs to be blurred, the color values ​​of the red, green, and blue channels of the pixels around the pixel can be calculated according to the predetermined weights to finally obtain a new, smoother color value.

[0192] By traversing all the pixels in the area to be blurred, all the pixels in the area to be blurred are calculated according to the selected blur algorithm (such as Gaussian blur algorithm). Then the pixels that originally clearly displayed sensitive information will become blurred, and the image contours and details of the area to be blurred will gradually disappear, thereby achieving the effect of blurring sensitive information.

[0193] After blurring the pixels in the blurred area, the original sensitive information area in the video data is blurred. Finally, the sensitive information area in the processed video data has become blurred, while other unprocessed areas remain the same. The processed video data containing the blurred sensitive information is passed to the display screen, so that when the display screen plays the video data, the user sees the blurred video picture, which can not only watch the video data of non-sensitive information normally, but also avoid the leakage of sensitive information in the video data, thereby ensuring the security and compliance of screen projection in the display device.

[0194] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0195] Corresponding to the screen projection processing method of the display device in the above embodiment, Figure 6 is a schematic diagram of the structure of a screen projection processing device of a display device provided in an embodiment of the present application. The device can be implemented as part or all of a computer device by software, hardware, or a combination of both. The computer device can be Figure 7 Display device shown.

[0196] Reference Figure 6 , the screen projection processing device of the display device includes:

[0197] The decoding module 601 is used to decode the encoded data stream carried in the mirroring screen projection request in response to the mirroring screen projection request of the source device to obtain decoded video data;

[0198] An analysis module 602 is used to analyze effective display areas of multiple video pictures in the decoded video data to obtain feature information of the multiple video pictures;

[0199] A blur processing module 603 is used for, if it is determined based on the feature information of any video screen that the effective display area contains sensitive information, blurring the sensitive information according to the pixel point information corresponding to the sensitive information in the effective display area to obtain processed video data;

[0200] The display module 604 is used to display the processed video data based on the display screen of the display device.

[0201] Further, based on any of the above embodiments, as an example of the present application, the device further includes:

[0202] An input module is used to input feature information of multiple video images into a pre-trained video image analysis model, wherein the video image analysis model is obtained by training a convolutional neural network model based on multiple training samples, and each training sample includes: feature information of a sample video image and a sensitive information label;

[0203] The analysis module is used to analyze the feature information of each video picture through the video picture analysis model to obtain the comparative analysis result corresponding to each video picture;

[0204] The determination module is used to determine whether each effective display area contains sensitive information based on the comparison analysis results corresponding to each video screen.

[0205] Further, based on any of the above embodiments, as an example of the present application, the fuzzy processing module includes:

[0206] A first determination submodule is used to determine the starting coordinates and area size parameters corresponding to the sensitive information according to the pixel point information corresponding to the sensitive information in the effective display area, wherein the starting coordinates include: a starting horizontal coordinate and a starting vertical coordinate, and the area size parameters include: width and height;

[0207] The second determination submodule is used to determine the area to be blurred within the effective display area according to the starting coordinates and the area size parameters;

[0208] The blur processing submodule is used to perform blur processing on the pixels in the blur area to obtain processed video data.

[0209] It should be noted that the screen projection processing device of the display device provided in the above embodiment is only illustrated by the division of the above-mentioned functional modules. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0210] The functional units and modules in the above embodiments may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit, and the above integrated units may be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present application.

[0211] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0212] An embodiment of the present application further provides a display device, the display device comprising one or more processors and a memory;

[0213] The memory is coupled to one or more processors, and the memory is used to store computer program code, the computer program code includes computer instructions, and one or more processors call the computer instructions to enable the display device to execute the screen projection processing method of the display device shown above.

[0214] Figure 7 The structural diagram of a display device provided in an embodiment of the present application is shown in FIG. 700 . The display device 700 may be a mobile phone, a smart screen, a tablet computer, a wearable display device, a vehicle display device, an augmented reality (AR) device, a virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a projector, or a communication device such as a server, a storage device, a base station, or a smart car, etc. The embodiment of the present application does not impose any restrictions on the specific type of the display device.

[0215] The memory 701 can be used to store computer software programs 702 and modules. The processor 703 executes various functional applications and data processing of the display device by running the software programs and modules stored in the memory 701. The memory 701 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the display device (such as audio data, a phone book, etc.), etc. In addition, the memory 701 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0216] Among them, the processor 703 may include one or more processors such as a central processing unit, an application processor (AP), a baseband processor, etc. The processor may be the nerve center and command center of the wireless router. The processor 703 may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions. The memory 701 may be used to store computer executable program codes, and the executable program codes include instructions. The processor 703 executes various functional applications and data processing of the network device by running the instructions stored in the memory. The memory 701 may include a program storage area and a data storage area, such as storing data of a sound signal to be played. For example, the memory may be a double rate synchronous dynamic random access memory DDR or a flash memory Flash.

[0217] An embodiment of the present application also provides a computer-readable storage medium, in which computer instructions are stored; when the computer-readable storage medium runs on a display device, the display device executes the screen projection processing method of the display device shown above.

[0218] The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium, or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0219] An embodiment of the present application also provides a computer program product including computer instructions. When the computer program product is run on a display device, the display device can execute the screen projection processing method of the display device shown above.

[0220] The computer storage medium and computer program product provided in the above-mentioned embodiments of the present application are used to execute the method provided above. Therefore, the beneficial effects that can be achieved can refer to the corresponding beneficial effects of the method provided above, and will not be repeated here.

[0221] In the above embodiments, it can also be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (such as: coaxial cable, optical fiber, data subscriber line (Digital Subscriber Line, DSL)) or wireless (such as: infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a digital versatile disc (DVD)), or a semiconductor medium (such as a solid state disk (SSD)).

[0222] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0223] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments applied for herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0224] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0225] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0226] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A display device, characterized in that: include: Display screen; A decoder is configured to decode the encoded data stream carried in the mirroring screen projection request in response to the mirroring screen projection request of the source device to obtain decoded video data; The processors connected to the display screen and the decoder are configured as follows: Analyzing effective display areas of multiple video pictures in the decoded video data to obtain feature information of the multiple video pictures; If it is determined based on the feature information of any one of the video images that sensitive information is contained in the effective display area, blurring the sensitive information according to the pixel point information corresponding to the sensitive information in the effective display area to obtain processed video data; The processed video data is displayed based on the display screen.

2. The display device according to claim 1, characterized in that Before the processor performs the analysis of the effective display areas of the multiple video pictures in the decoded video data to obtain the feature information of the multiple video pictures, the processor is further configured to: According to the preset interval duration, video frames are captured from the decoded video data to obtain the multiple video frames.

3. The display device according to claim 1, characterized in that Before the processor performs the analysis of the effective display areas of the multiple video pictures in the decoded video data to obtain the feature information of the multiple video pictures, the processor is further configured to: Obtaining pixel points at different coordinate positions in each of the video images; Determine an invalid display area of ​​each video screen according to the pixel points in each video screen; The invalid display area of ​​each video screen is removed to obtain the valid display area of ​​each video screen.

4. The display device according to claim 1, characterized in that The processor is further configured to: Inputting the feature information of the plurality of video images into a pre-trained video image analysis model, wherein the video image analysis model is obtained by training a convolutional neural network model based on a plurality of training samples, each of the training samples comprising: feature information of a sample video image and a sensitive information label; By using the video picture analysis model, characteristic information of each video picture is analyzed respectively to obtain a comparative analysis result corresponding to each video picture; According to the comparison and analysis results corresponding to each of the video images, it is determined whether each of the effective display areas contains the sensitive information.

5. The display device according to claim 1, characterized in that The processor performs blurring of the sensitive information according to the pixel point information corresponding to the sensitive information in the effective display area to obtain processed video data, and is further configured to: Determine the starting coordinates and area size parameters corresponding to the sensitive information according to the pixel point information corresponding to the sensitive information in the effective display area, wherein the starting coordinates include: a starting horizontal coordinate and a starting vertical coordinate, and the area size parameters include: width and height; Determine the area to be blurred within the effective display area according to the starting coordinates and the area size parameters; The pixel points in the area to be blurred are blurred to obtain the processed video data.

6. The display device according to claim 5, characterized in that Before the processor performs blur processing on the pixels in the area to be blurred, the processor is further configured to: Identify the boundary area between other areas in the effective display area and the area to be blurred; The color values ​​of the pixels on the boundary area are changed to achieve a smooth and uniform transition between the area to be blurred and the other areas.

7. The display device according to any one of claims 1 to 5, characterized in that: The processor is further configured to: during the process of displaying the processed video data based on the display screen, or after the process of displaying the processed video data based on the display screen: Continuously analyzing the feature information of each video picture through the video picture analysis model to obtain a comparative analysis result corresponding to each video picture; If it is determined based on the comparative analysis results corresponding to each of the video images that none of the effective display areas contains the sensitive information, the blurring of the sensitive information is canceled to obtain restored video data; The restored video data is displayed based on the display screen.

8. A screen projection processing method for a display device, characterized in that: include: In response to a mirroring screen projection request from a source device, decoding the encoded data stream carried in the mirroring screen projection request to obtain decoded video data; Analyzing effective display areas of multiple video pictures in the decoded video data to obtain feature information of the multiple video pictures; If it is determined based on the feature information of any one of the video images that sensitive information is contained in the effective display area, blurring the sensitive information according to the pixel point information corresponding to the sensitive information in the effective display area to obtain processed video data; The processed video data is displayed based on a display screen of the display device.

9. The method according to claim 8, characterized in that The method further comprises: Inputting the feature information of the plurality of video images into a pre-trained video image analysis model, wherein the video image analysis model is obtained by training a convolutional neural network model based on a plurality of training samples, each of the training samples comprising: feature information of a sample video image and a sensitive information label; By using the video picture analysis model, characteristic information of each video picture is analyzed respectively to obtain a comparative analysis result corresponding to each video picture; According to the comparison and analysis results corresponding to each of the video images, it is determined whether each of the effective display areas contains the sensitive information.

10. The method according to claim 8, characterized in that The fuzzy processing of the sensitive information according to the pixel point information corresponding to the sensitive information in the effective display area to obtain processed video data includes: Determine the starting coordinates and area size parameters corresponding to the sensitive information according to the pixel point information corresponding to the sensitive information in the effective display area, wherein the starting coordinates include: a starting horizontal coordinate and a starting vertical coordinate, and the area size parameters include: width and height; Determine the area to be blurred within the effective display area according to the starting coordinates and the area size parameters; The pixel points in the area to be blurred are blurred to obtain the processed video data.

Images