Display apparatus and display method

Abstract

The invention provides a display device and a display method, belongs to the technical field of display, and can solve the problem that conference experience of participants is affected by conference interruption caused by an existing anti-shooting system. The display device comprises an acquisition module, a behavior identification module, a sensitive content identification module, a blanking layer generation module and a display module. The acquisition module is configured to acquire an environment image around the display equipment; the behavior recognition module is configured to judge whether a target object exists in the environment image or not and generate a first control instruction when the target object exists; the sensitive content identification module is configured to respond to the first control instruction to determine first area information; the blanking layer generation module is configured to generate a blanking layer at least according to the first area information; and the display module is configured to perform image display according to the to-be-displayed frame image and the blanking layer. The display device disclosed by the invention can prevent sensitive contents from being recorded and shot and does not interfere with the thinking of participants.

Description

Technical Field

[0001] The present disclosure belongs to the field of display technology, and particularly relates to a display device and a display method. Background Art

[0002] With the rapid development of information technology, the performance of video recording equipment is becoming more and more superior and its popularity is becoming more and more widespread. For example, portable mobile phones, tablet computers, laptops, small voice recorders or cameras can all capture high-definition images, videos and record audio. However, some important occasions or confidential meetings often involve sensitive information such as commercial secrets, policy discussions, and scientific research results. Once these important information are illegally filmed and leaked, they may cause huge economic losses and reputation damage to organizations or individuals. At the same time, when participants know that effective anti-filming measures have been taken in the meeting environment, they are more likely to freely exchange views and share information, which is crucial to promoting open discussions and enhancing teamwork.

[0003] In order to prevent the information displayed by the display device from being illegally recorded and leaked, there are currently two commonly used measures. One is manual intervention, such as using detection instruments to detect or confiscate the devices carried by participants at the venue entrance, or arranging supervisors to supervise the entire meeting and manually stop or delete the recorded content when recording is discovered; however, this method is difficult to completely eliminate the effect of recording, and it seriously affects the participants' experience. The other is to use an anti-secret camera alarm system at the venue, that is, using a camera to collect images of the venue in real time and detect whether there are shooting postures in the images. Once an image with a shooting posture is detected, the display device is controlled to interrupt the original meeting content and display a warning message; however, since this method will interrupt the meeting process when shooting is detected, it will affect the continuity of the meeting, increase the time cost of participants, and cause a bad meeting experience. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art. On the one hand, a display device is provided, which includes: an acquisition module, a behavior recognition module, a sensitive content recognition module, a hidden layer generation module and a display module; the acquisition module is configured to acquire an environmental image around the display device; the behavior recognition module is configured to determine whether there is a target object in the environmental image, and generate a first control instruction when the target object exists; the sensitive content recognition module is configured to obtain position information of sensitive content in the frame image to be displayed in response to the first control instruction, and determine first area information according to the position information of the sensitive content; the hidden layer generation module is configured to generate a hidden layer according to at least the first area information; the display module is configured to display an image according to the received frame image to be displayed and the hidden layer.

[0005] In some examples, the behavior recognition module includes an image recognition model, a posture recognition model and a first judgment unit; the image recognition model is configured to receive the environmental image, obtain a first object and a similarity between the first object and a first standard object; the posture recognition model is configured to receive the environmental image, obtain a user posture and a similarity between the user posture and the standard user posture; the first judgment unit is configured to determine that the target object exists in the environmental image and generate the first control instruction when the similarity between the first object and the first standard object, and the similarity between the user posture and the standard user posture meet preset conditions.

[0006] In some examples, the sensitive content identification module is specifically configured to, in response to the first control instruction, obtain location information of sensitive content in the frame image to be displayed, determine first area information based on the location information of the sensitive content, and obtain background image data of the first area in the frame image to be displayed; the hidden layer generation module is specifically configured to generate the hidden layer based on the first area information and the background image data.

[0007] In some examples, a position in the hidden layer corresponding to the first area is a non-transparent monochrome image.

[0008] In some examples, the sensitive content includes sensitive text, and the sensitive content identification module includes a sensitive text recognition model and a second judgment unit; the sensitive text recognition model is configured to receive the frame image to be displayed, obtain a second object and a similarity between the second object and a second standard object; the second judgment unit is configured to determine that the sensitive text exists in the frame image to be displayed and obtain location information of the sensitive text when the similarity between the second object and the second standard object meets a preset condition.

[0009] In some examples, the image recognition model uses a trained YOLOv7 network model or a trained YOLOv8 network model.

[0010] In some examples, the gesture recognition model uses a trained Swift YOLO network model.

[0011] In some examples, the sensitive text recognition model uses a trained CNN model.

[0012] In some examples, the display module includes a main display module and a sub-display module; the main display module is configured to display an image based on the received frame image to be displayed and the hidden layer; the sub-display module is configured to display the environmental image when the target object exists in the environmental image.

[0013] In a second aspect, the present invention provides a display method of a display device described in any of the above examples, wherein the display device includes an acquisition module, a behavior recognition module, a sensitive content recognition module, a hidden layer generation module, and a display module; wherein the display method includes:

[0014] Using the acquisition module to acquire an environmental image around the display device;

[0015] Using the behavior recognition module to determine whether a target object exists in the environment image, and generating a first control instruction if the target object exists;

[0016] The sensitive content identification module obtains the position information of the sensitive content in the frame image to be displayed in response to the first control instruction, and determines the first area information according to the position information of the sensitive content;

[0017] The hidden layer generating module generates a hidden layer at least according to the first region information;

[0018] The display module displays an image according to the received frame image to be displayed and the hidden layer.

[0019] In some examples, the behavior recognition module includes an image recognition model, a posture recognition model, and a first judgment unit; using the behavior recognition module to determine whether there is a target object in the environment image, and generating a first control instruction when the target object exists, includes:

[0020] Inputting the environment image into the image recognition model, and using the image recognition model to obtain a first object and a similarity between the first object and a first standard object;

[0021] Inputting the environment image into the gesture recognition model, and using the gesture recognition model to obtain the user gesture and the similarity between the user gesture and the standard user gesture;

[0022] The first judgment unit determines that the target object exists in the environment image and generates the first control instruction when the similarity between the first object and the first standard object, and the similarity between the user posture and the standard user posture both meet preset conditions.

[0023] In some examples, the sensitive content identification module is specifically configured to, in response to the first control instruction, obtain location information of sensitive content in the frame image to be displayed, determine first area information according to the location information of the sensitive content, and obtain background image data of the first area in the frame image to be displayed; the hidden layer generation module generates a hidden layer according to at least the first area information, specifically including:

[0024] The sensitive content identification module obtains location information of sensitive content in the frame image to be displayed, determines the first area information according to the location information of the sensitive content, and obtains background image data of the first area in the frame image to be displayed;

[0025] The hidden layer generating module generates the hidden layer according to the first region information and the background image data.

[0026] In some examples, the sensitive content includes sensitive text, and the sensitive content recognition module includes a sensitive text recognition model and a second judgment unit; the sensitive content recognition module obtains position information of the sensitive content in the frame image to be displayed in response to the first control instruction, including:

[0027] Inputting the frame image to be displayed into the sensitive word recognition model, and using the sensitive word recognition model to obtain a second object and a similarity between the second object and a second standard object;

[0028] When the similarity between the second object and the second standard object meets a preset condition, the second judgment unit determines that the sensitive text exists in the frame image to be displayed, and obtains the position information of the sensitive text.

[0029] In some examples, the display module includes a main display module and a sub-display module; the display method further includes: when the target object exists in the environmental image, using the sub-display module to display the environmental image. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings, in which:

[0031] Figure 1 A schematic diagram of the structure of a display device provided in an embodiment of the present disclosure;

[0032] Figure 2 A schematic diagram of the structure of a display device including a main display module and a sub-display module provided by the present disclosure;

[0033] Figure 3 It is a structural diagram of the behavior recognition module disclosed in the present invention;

[0034] Figure 4 It is a schematic diagram of the structure of the YOLOv8 network model;

[0035] Figure 5 This is a detailed structural diagram of each part of the YOLOv8 network model;

[0036] Figure 6 A schematic diagram of the structure of the sensitive content identification module provided by the present disclosure;

[0037] Figure 7 This is an effect diagram of blanking the frame image to be displayed in the present disclosure;

[0038] Figure 8 A flow chart of the display method provided by the present disclosure;

[0039] Fig. 9 A specific flow chart showing step SS2 in the method;

[0040] Fig.10 It is a specific flow chart showing step SS3 in the method. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0042] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" do not indicate quantity restrictions, but indicate that there is at least one. Similar words such as "include" or "comprise" mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Similar words such as "connect" or "connected" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0043] As used herein, "parallel" and "perpendicular" include the described situation and the situation similar to the described situation, and the range of the similar situation is within the acceptable deviation range, wherein the acceptable deviation range is determined by a person of ordinary skill in the art taking into account the measurement in question and the error associated with the measurement of a specific quantity (i.e., the limitations of the measurement system). For example, "parallel" includes absolute parallelism and approximate parallelism, wherein the acceptable deviation range of approximate parallelism can be, for example, a deviation within 5°; "perpendicular" includes absolute perpendicularity and approximate perpendicularity, wherein the acceptable deviation range of approximate perpendicularity can also be, for example, a deviation within 5°.

[0044] It will be understood that when a layer or an element is referred to as being on another layer or substrate, it can be directly on the other layer or substrate, or intervening layers may be present between the layer or element and the other layer or substrate.

[0045] Exemplary embodiments are described herein with reference to cross-sectional views and / or plan views that are idealized exemplary drawings. In the drawings, the thickness of layers and regions are exaggerated for clarity. Therefore, variations in shape relative to the drawings due to, for example, manufacturing techniques and / or tolerances are conceivable. Therefore, the exemplary embodiments should not be interpreted as being limited to the shapes of the regions shown herein, but include deviations in shape due to, for example, manufacturing. Therefore, the regions shown in the drawings are schematic in nature, and their shapes are not intended to illustrate the actual shapes of regions of the device, and are not intended to limit the scope of the exemplary embodiments.

[0046] In order to ensure the security of the conference content, the relevant technology has proposed some conference or screen anti-shooting systems or methods. For example, a patent has proposed a conference anti-sneak shooting method and device, which includes an external camera, a conference robot and a photosensitive sensor, wherein the conference robot includes a depth camera, a control computer, a laser projection device and a wireless communication module; the implementation process of the device generally includes the following steps: 1. Use an external camera to collect video data of the venue area; 2. Transmit the video data of the venue area to the control computer, and use the anti-photography detection model loaded on the control computer to detect whether there is an image of a photo pose in the video data; 3. When the control computer detects an image of a photo pose, a warning message is generated and displayed on the laser projection device to prompt the venue staff; 4. After the warning message prompt ends, restore the content displayed in the meeting. For another example, a patent proposes a screen anti-shooting system, which includes a detection module, a monitoring module, and a switching module. The implementation process of the system is roughly as follows: 1. Use the detection module to extract the information displayed on the display screen, and compare the information on the display screen with the confidential information. When the comparison result indicates that the information on the display screen is confidential information, the optical camera of the display device is self-checked, and a first warning message is generated when it is determined that the optical camera is blocked or interfered with; 2. Use the monitoring module to monitor the area around the display screen, and identify whether there is any shooting behavior in the monitoring information; if there is any shooting behavior, a second warning message is generated; 3. When the first warning message or the second warning message is received, use the switching module to switch the meeting content displayed on the display screen so that the display screen displays a warning message.

[0047] Although the above-mentioned related technologies can close the meeting information in time when the shooting behavior is found, and display a warning signal on the display screen to remind the participants, which can stop the shooting behavior and prevent the meeting information from being stolen, this method will interrupt the meeting process. When there are many people shooting or the shooting behavior is frequent, the high-frequency suspension of the meeting will seriously affect the meeting experience of the participants, and to a certain extent prolong the duration of the meeting, increasing the time cost of the participants. Therefore, it is urgent to provide a display device with an anti-recording function, which not only needs to have a good anti-recording effect, but also needs to ensure the continuity of the meeting process as much as possible to avoid frequent suspension of the meeting and affect the meeting experience of the participants.

[0048] To this end, in one aspect, the present disclosure provides a display device. Figure 1 A schematic diagram of the structure of a display device provided in an embodiment of the present disclosure is shown in FIG. Figure 1As shown, the display device provided by the present disclosure includes: an acquisition module, a behavior recognition module, a sensitive content recognition module, a hidden layer generation module and a display module. Specifically, the acquisition module is configured to acquire the environment image around the display device; the behavior recognition module is configured to determine whether there is a target object in the environment image acquired by the acquisition module, and generate a first control instruction when the target object exists in the environment image; the sensitive content recognition module is configured to obtain the position information of the sensitive content in the frame image to be displayed in response to the first control instruction issued by the behavior recognition module, and determine the first area information according to the position information of the sensitive content; the hidden layer generation module is configured to generate a hidden layer according to at least the first area information; the display module is configured to display the image according to the received frame image to be displayed and the hidden layer.

[0049] Those skilled in the art can understand that when the behavior recognition module in the display device is used to identify whether there is a recording behavior in the environmental image, the target object refers to the recording behavior, such as secretly photographing behavior, recording behavior, etc. Sensitive content refers to the more important text information, data, images or videos in the meeting content.

[0050] The display device provided by the present disclosure can, when detecting the presence of a target object (such as candid photography) in the environment around the display device, use the sensitive content recognition module to obtain the position information of the sensitive content (such as important text, data or images, etc.) in the frame image to be displayed, and the hidden layer generation module can generate a hidden layer according to this position information, and superimpose the frame image to be displayed with the hidden layer and transmit it to the display module, so that the display module displays the frame image to be displayed superimposed with the hidden layer. It can be seen that when the display device provided by the present disclosure detects the recording behavior, it can timely hide the sensitive information in the next frame display content, and all parts except the sensitive content are displayed normally. Such a setting can prevent important information from being further stolen on the one hand, and on the other hand, it will not interrupt the meeting process and maintain the continuity of the meeting, so it can avoid affecting the meeting experience of the participants to a large extent.

[0051] In some examples, the display module of the display device provided by the present disclosure includes a main display module and a sub-display module. Figure 2 is a schematic diagram of the structure of a display device including a main display module and a sub-display module, such as Figure 2 As shown, the main display module is configured to display images according to the received frame image to be displayed and the hidden layer; the auxiliary display module is configured to display the environmental image when there is a target object in the environmental image. For example, the auxiliary display module can be a monitoring display, which is supervised by the staff responsible for security. When the auxiliary display module displays an environmental image with recording behavior, the staff can record and stop the recording person in time, so as to avoid interfering with other participants.

[0052] In some examples, the behavior recognition module of the display device provided by the present disclosure mainly includes an image recognition model, a posture recognition model and a first judgment unit. Figure 3 FIG. 1 is a schematic diagram showing the structure of a behavior recognition module of a display device disclosed in the present invention. Figure 3 As shown, the behavior recognition module includes an image recognition model, a posture recognition model and a first judgment unit. The image recognition model is configured to receive an environment image i, obtain a first object and the similarity between the first object and the first standard object; the posture recognition model is configured to receive an environment image i, obtain a user posture and the similarity between the user posture and the standard user posture; the first judgment unit is configured to determine that there is a target object in the environment image i and generate a first control instruction when the similarity between the first object and the first standard object and the similarity between the user posture and the standard user posture meet the preset conditions; when either the similarity between the first object and the first standard object and the similarity between the user posture and the standard user posture do not meet the preset conditions, the next environment image i+1 is input into the behavior recognition model for judgment.

[0053] It should be noted that when the target object is defined as recording behavior, the above-mentioned first standard object can be various types of recording devices, such as mobile phones, voice recorders, miniature cameras, etc.; the above-mentioned standard user posture can be the posture of the user holding the recording device with one hand or both hands, etc.

[0054] As an optional embodiment, the image recognition model of the present disclosure may adopt a trained YOLOv7 network model or a trained YOLOv8 network model.

[0055] The YOLO (You Only Look Once) algorithm is a deep learning algorithm in the field of computer vision, mainly used for detection and recognition of single-frame images. Compared with the target detection method based on manual feature extraction, it has higher detection accuracy and faster detection speed.

[0056] The following uses the YOLOv8 network model as an example to introduce the network structure and training process of the image recognition model disclosed in the present invention. The training process of the YOLOv8 network model generally includes the following steps:

[0057] Step S101, collecting an environmental image around the display device to generate a first original data set; performing data enhancement on the first original data set to generate a first extended data set; dividing the first extended data set into three sub-data sets in a ratio of 7:2:1, the three sub-data sets being a first training sub-data set, a first test sub-data set, and a first validation sub-data set; wherein the first training sub-data set is used to train a YOLOv8 network model, the first test sub-data set is used to evaluate the performance and generalization ability of the trained YOLOv8 network model, and the first validation sub-data set is used to adjust the hyperparameters of the YOLOv8 network model during the training process.

[0058] Since there are many kinds of existing recording and shooting equipment, including common mobile phones, tablet computers, voice recorders, laptop computers, camera pens, etc., in order to improve the accuracy of the image recognition model, the types of environmental images collected in step S101 should be diversified and the number should be as large as possible. For example, the environmental images should at least include: images of no recording equipment at the conference site, images of recording equipment at different distances or angles or under different light, and other types of images; it should be noted that the recording and shooting equipment in the environmental images should at least include several types commonly found on the market. For example, the recording and shooting equipment in the image should not only include the above-mentioned common mobile phones, tablet computers, voice recorders, camera pens, etc., but also include invisible cameras and wearable devices for sneak shots (such as watches, ties, glasses, etc. with implanted cameras). The number of environmental images can be tens of thousands or hundreds of thousands. Those skilled in the art can understand that the data enhancement in step S101 includes a series of transformations and processing such as flipping, rotating, scaling, translating, adjusting brightness, adding noise, and cropping the environment image; through the data enhancement operation, the number of data sets can be effectively expanded, the generalization ability of the image recognition model can be improved, and the risk of overfitting can be reduced, thereby improving the performance and accuracy of the image recognition model. In addition, the division ratio of the first extended data set can be other, for example, it can be divided according to a ratio of 8:1:1 or 6:2:2, and the present disclosure does not limit this.

[0059] Step S102: Create an initial YOLOv8 network model, which includes an input layer, a Backbone network, a Neck network, a Head network, and an output layer.

[0060] Figure 4 This is a schematic diagram of the network structure of the initial YOLOv8 network model; Figure 5This is a schematic diagram of the Backbone network, the Neck network, and the Head network. In the YOLOv8 network model, the Backbone network is responsible for extracting features from the environment image input by the Input layer to obtain feature maps of different scales. Specifically, the Backbone network uses a series of convolutional layers Cov and deconvolutional layers, and uses residual connections and bottleneck structures to reduce the size of the network model and improve the performance of the network model; the size of the convolutional layer can be as follows Figure 5 As shown, the size of the convolution kernel is k=3, the sliding step of the convolution kernel on the input data is s=2, and the number of pixels filled at the edge of the input data is p=1; Figure 5 As shown in the figure, the Backbone network also uses the c2f module as the basic building block. The c2f module has fewer parameters and stronger feature extraction capabilities. The Neck network is responsible for multi-scale feature fusion. It enhances the feature representation capability by fusing feature maps from different stages in the Backbone network. This part includes the SPPF (Spatial Pyramid Pooling Fast) module, the PAA (Probabilistic Anchor Assignment) module, and the PAN (Path Aggregation Network) module. The SPPF module is used to perform pooling operations on feature maps of different scales, splicing feature maps of different scales together to improve the detection capability of targets of different sizes. The PAA module is used to intelligently assign anchor frames to optimize the selection of positive and negative samples and improve the training effect of the network model. The PAN module is used for path aggregation of features at different levels, and enhances the expression capability of feature maps through bottom-up and top-down paths. The head network is used for the final target detection and classification tasks. It is mainly responsible for further processing the multi-scale fusion features output by the neck network, converting the feature map into the specific information required for target detection, including category, location, and confidence, so as to generate the final detection result. Figure 5 As shown, this part includes three detection head Detect modules, among which the first detection head Detect module is used to detect small objects, and its input feature map scale is 80×80, the second detection head Detect module is used to detect medium-sized objects, and its input feature map scale is 40×40, and the third detection head Detect module is used to detect large objects, and its input feature map scale is 20×20.

[0061] Step S103, input the first training sub-data set into the initial YOLOv8 network model created in step S102 for iterative training. During the training process, input the first verification sub-data set into the YOLOv8 network model to adjust the hyperparameters of the model. When the performance index of the YOLOv8 network model for the first test sub-data set reaches a preset accuracy threshold, stop the training process, save the parameter weights of the YOLOv8 network model at this time, and obtain a trained YOLOv8 network model as the image recognition model adopted in the embodiment of the present disclosure.

[0062] Those skilled in the art can understand that the first test sub-dataset is used to evaluate the performance and generalization ability of the trained YOLOv8 network model. Specifically, after the training is completed, the first test sub-dataset is input into the YOLOv8 network model, and the accuracy, precision and other performance indicators of the YOLOv8 network model are evaluated by comparing the prediction results of the YOLOv8 network model with the actual labels of the first test sub-dataset. Therefore, the first test sub-dataset should be independent of the first training sub-dataset, that is, there is no intersection, so as to ensure that the YOLOv8 network model can perform well on unseen data.

[0063] As an optional embodiment, the posture recognition model disclosed in the present invention may adopt a trained Swift YOLO network model.

[0064] The following takes the Swift YOLO network model as an example to introduce the training process of the disclosed posture recognition model. The training process of the SwiftYOLO network model generally includes the following steps:

[0065] Step S201: collecting an environmental image around the display device to generate a second original data set, and performing data enhancement on the second original data set to generate a second extended data set; dividing the second extended data set into three sub-data sets in a ratio of 7:2:1, and the three sub-data sets are a second training sub-data set, a second test sub-data set, and a second verification sub-data set; wherein the second training sub-data set is used to train a gesture recognition model, the second test sub-data set is used to evaluate the performance and generalization ability of the trained gesture recognition model, and the second verification sub-data set is used to adjust the hyperparameters of the gesture recognition model during the training process.

[0066] In order to improve the accuracy of the gesture recognition model, the types of environmental images collected in step S201 should be diversified and the number should be as large as possible. For example, the environmental images should at least include: images of participants not using recording equipment, images of participants holding recording equipment with one hand or both hands at different distances or different angles or different light, and images of recording equipment being clamped and fixed at a certain position for recording, etc.; it should be noted that the recording equipment in the environmental images should at least include several common types on the market. For example, the recording equipment in the image should not only include the above-mentioned common mobile phones, tablet computers, voice recorders, camera pens, etc., but also include invisible cameras, sneak-shooting wearable devices (such as watches, ties, glasses, etc. with implanted cameras). The number of environmental images can be tens of thousands or hundreds of thousands. It can be understood by those skilled in the art that the data enhancement in step S201 includes a series of transformations and processing such as flipping, rotating, scaling, translating, adjusting brightness, adding noise, and cropping the environment image; through the data enhancement operation, the number of data sets can be effectively expanded, the generalization ability of the gesture recognition model can be improved, and the risk of overfitting can be reduced, thereby improving the performance and accuracy of the gesture recognition model. In addition, the division ratio of the second extended data set can be other, for example, it can be divided according to a ratio of 8:1:1 or 6:2:2, and the present disclosure does not limit this.

[0067] Step S202: Create an initial Swift YOLO network model.

[0068] Step S203, input the second training sub-data set into the initial Swift YOLO network model created in step S202 for iterative training. During the training process, input the second verification sub-data set into the Swift YOLO network model to adjust the hyperparameters of the model. When the performance index of the Swift YOLO network model for the second test sub-data set reaches a preset accuracy threshold, stop the training process, save the parameter weights of the Swift YOLO network model at this time, and obtain the trained SwiftYOLO network model as the posture recognition model adopted by the embodiment of the present disclosure.

[0069] Those skilled in the art can understand that the second test sub-dataset is used to evaluate the performance and generalization ability of the trained Swift YOLO network model. Specifically, after the training is completed, the second test sub-dataset is input into the Swift YOLO network model, and the accuracy, precision and other performance indicators of the Swift YOLO network model are evaluated by comparing the prediction results of the Swift YOLO network model with the actual labels of the second test sub-dataset. Therefore, the second test sub-dataset should be independent of the second training sub-dataset, that is, there is no intersection, so as to ensure that the Swift YOLO network model can perform well on unseen data.

[0070] Further, the sensitive content identification module of the display device provided by the present disclosure is specifically configured to obtain the position information of the sensitive content in the frame image to be displayed in response to the first control instruction, determine the first area information according to the position information of the sensitive content, and obtain the background image data of the first area in the frame image to be displayed. The hidden layer generation module is specifically configured to generate the hidden layer according to the first area information and the background image data.

[0071] Sensitive content usually includes sensitive text, sensitive pictures or videos, etc. Therefore, the sensitive content recognition module disclosed in the present invention at least includes a sensitive text recognition model and a second judgment unit. Figure 6 The following is a workflow diagram of the sensitive text recognition model and the second judgment unit of the present disclosure, such as Figure 6 As shown, after receiving the first control instruction, the sensitive text recognition model obtains the second object and the similarity between the second object and the second standard object from the received frame image to be displayed, and inputs the similarity between the second object and the second standard object into the second judgment unit; when the similarity between the second object and the second standard object meets the preset conditions, the second judgment unit determines that there are sensitive texts in the frame image to be displayed, obtains and outputs the location information of the sensitive text, and when the similarity between the second object and the second standard object does not meet the preset conditions, it determines that there are no sensitive texts in the frame image to be displayed.

[0072] Those skilled in the art can understand that the above-mentioned second object refers to the text in the frame image to be displayed, and the second standard object refers to the more important and sensitive text determined manually; it should be noted that the text here can include Arabic numerals. Figure 7 The display device of the present disclosure generates an effect diagram of a hidden layer for a frame image to be displayed, wherein: Figure 7 Sub-image A is an exemplary frame image to be displayed. For example, the sensitive content may be defined as "number" in advance, that is, the sensitive content recognition module is enabled to recognize the "number" in the image and output its position information, determine the first area information according to the position information of the "number", and obtain the background image data at the first area; Figure 7 Sub-image B is a hidden layer generated by the hidden layer generation module according to the first region information and the background image data. It can be seen from sub-image B that the image color of the area corresponding to the first region in the hidden layer is the same as the background color of the first region in the frame image to be displayed, and the remaining areas are transparent; Figure 7Sub-image C is a rendering of the superposition of the frame image to be displayed and the hidden layer. It can be seen from sub-image C that the sensitive content of the superposition of the frame image to be displayed and the hidden layer, relative to the frame image to be displayed in sub-image A, is hidden. At this time, the display module displays the superposition of the frame image to be displayed and the hidden layer, which can not only prevent the sensitive content in the frame image to be displayed from being stolen, but also avoid interrupting the thoughts and meeting progress of the participants. Of course, the area corresponding to the first area in the hidden layer can also be a non-transparent monochrome image, and the present disclosure does not limit the color of the area corresponding to the first area in the hidden layer.

[0073] As an optional embodiment, the sensitive text recognition model provided by the present disclosure may adopt a trained convolutional neural network (CNN) model. Specifically, the training process of the CNN model generally includes the following steps:

[0074] Step S301, generate a third original data set containing a large number of text images, and perform data preprocessing and data enhancement on it to obtain a third extended data set; divide the third extended data set to obtain three sub-data sets, namely a third training sub-data set, a third verification sub-data set and a third test sub-data set; wherein the third training sub-data set is used to train the sensitive text recognition model, the third test sub-data set is used to evaluate the performance and generalization ability of the trained sensitive text recognition model, and the third verification sub-data set is used to adjust the hyperparameters of the sensitive text recognition model during the training process.

[0075] Those skilled in the art will appreciate that labels for different text sensitivities in text images can be determined based on the topic and content of the meeting; it should be noted that the text in the text image includes numbers. Data preprocessing includes data standardization (scaling the data size to a range of 0-1), centering (subtracting the data mean from the original data to make the data centered on 0) and data cleaning (eliminating inconsistent and erroneous data); data enhancement includes rotating, cropping, scaling, adding random noise and color jittering to the text image.

[0076] Step S302: Create an initial CNN model; it includes an input layer, a convolutional layer, an activation function, a pooling layer, a fully connected layer, and an output layer.

[0077] Step S303, input the third training sub-dataset into the initial CNN model created in step S302 for iterative training. During the training process, input the third verification sub-dataset into the CNN model to adjust the hyperparameters of the model. When the performance index of the CNN model for the third test sub-dataset reaches a preset accuracy threshold, stop the training process, save the parameter weights of the CNN model at this time, and obtain the trained CNN model as the sensitive text recognition model adopted in the embodiment of the present disclosure.

[0078] It should be noted that the present disclosure does not restrict the algorithms and network structures of the image recognition model, gesture recognition model, and sensitive text recognition model, and they can be set as needed according to specific scenario requirements when applied.

[0079] Those skilled in the art can understand that the hardware part of the display device disclosed in the present invention includes a processor chip. During the production process of the display device, the image recognition model, the posture recognition model and the sensitive text recognition model can be trained in advance, and the trained image recognition model, the posture recognition model and the sensitive text recognition model can be implanted in the processor chip; the acquisition module can be an external camera, and the processor chip is connected to the external camera, and can receive the environmental image acquired by the external camera; the display module can include a main display screen and a sub-display screen, both of which are connected to the processor chip, and can receive the frame image to be displayed transmitted by the processor chip. Specifically, when the processor chip does not recognize the recording behavior, it processes and transmits the original frame image to be displayed to the main display screen and the sub-display screen for display. When the processor chip recognizes the recording behavior, it performs a blanking process on the frame image to be displayed, and transmits the blanked frame image to be displayed to the main display screen for display, and transmits the image of the recording behavior to the sub-display screen for display.

[0080] In a second aspect, the present disclosure provides a display method of the display device provided in the first aspect, Figure 8 A flow chart of a display method provided by the present disclosure, such as Figure 8 As shown, the display method includes:

[0081] Step SS1: Use the acquisition module to acquire the environment image around the display device.

[0082] Step SS2: Using the behavior recognition module, determine whether there is a target object in the environment image, and generate a first control instruction if there is a target object.

[0083] Specifically, the behavior recognition module includes an image recognition model, a posture recognition model and a first judgment unit. Therefore, in step SS2, the behavior recognition module is used to determine whether there is a target object in the environment image, and if there is a target object, a first control instruction is generated, which specifically includes steps SS21-SS23. Fig. 9The specific process of step SS2 is as follows: Fig. 9 As shown, step SS2 specifically includes:

[0084] Step SS21: Input the environment image into the image recognition model, and use the image recognition model to obtain the first object and the similarity between the first object and the first standard object. The image recognition model can use a trained YOLOv7 network model or a trained YOLOv8 network model; the first standard object refers to a recording device, such as a mobile phone, a voice recorder, a miniature camera, etc.

[0085] Step SS22: Input the environment image into the posture recognition model, and use the posture recognition model to obtain the user posture and the similarity between the user posture and the standard user posture. The posture recognition model can use a trained Swift YOLO network model; the standard user posture refers to the posture of the user when recording.

[0086] Step SS23: When the similarity between the first object and the first standard object, and the similarity between the user posture and the standard user posture meet preset conditions, the first judgment unit determines that there is a target object in the environment image and generates a first control instruction.

[0087] Step SS3: The sensitive content identification module obtains the position information of the sensitive content in the frame image to be displayed in response to the first control instruction, and determines the first area information according to the position information of the sensitive content.

[0088] Specifically, the sensitive content includes sensitive text, and the sensitive content recognition module includes a sensitive text recognition model and a second judgment unit. In step SS3, the sensitive content recognition module responds to the first control instruction to obtain the position information of the sensitive content in the frame image to be displayed, including steps SS31-SS32. Fig.10 is a flow chart of step SS3, as shown in Fig.10 As shown, step SS3 includes:

[0089] Step SS31: Input the frame image to be displayed into the sensitive text recognition model, and use the sensitive text recognition model to obtain the second object and the similarity between the second object and the second standard object. The sensitive text recognition model can use a trained CNN model; the second standard object refers to sensitive text and numbers.

[0090] Step SS32: When the similarity between the second object and the second standard object meets a preset condition, the second judgment unit determines that sensitive text exists in the frame image to be displayed, and obtains the position information of the sensitive text.

[0091] Furthermore, after obtaining the position information of the sensitive content in the frame image to be displayed and determining the first area information according to the position information of the sensitive content, the sensitive content identification module may also obtain background image data of the first area in the frame image to be displayed.

[0092] Step SS4: The hidden layer generation module generates a hidden layer at least according to the first region information.

[0093] Specifically, the hidden layer generation module in step SS4 generates the hidden layer at least according to the first region information, including: generating the hidden layer according to the first region information and the background image data of the first region in the frame image to be displayed.

[0094] Step SS5: The display module displays the image according to the received frame image to be displayed and the hidden layer.

[0095] Furthermore, the display module includes a main display module and a sub-display module, and the display method further includes: when there is a target object in the environment image, using the sub-display module to display the environment image.

[0096] The display method of the display device provided in the present disclosure can identify and hide the sensitive content in the frame image to be displayed when the target object (i.e., recording behavior) is identified in the environmental image, so that the display module displays the frame image to be displayed after the hide is displayed. This can prevent important content in the meeting content from being further recorded, and can also avoid interrupting the meeting to interfere with the participants' thinking and affect the participants' meeting experience.

[0097] Those skilled in the art will appreciate that all or some of the steps, systems, and functional modules / units in the apparatus disclosed above may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0098] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed by several physical components in cooperation.

[0099] Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit (CPU), a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-temporary medium) and a communication medium (or temporary medium). As known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH) or other disk storage; read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical disk storage; magnetic cassettes, magnetic tapes, disk storage or other magnetic storage; any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0100] The present disclosure has disclosed example embodiments, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly stated, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A display device, comprising: Acquisition module, behavior recognition module, sensitive content recognition module, hidden layer generation module and display module; The acquisition module is configured to acquire an environmental image around the display device; The behavior recognition module is configured to determine whether a target object exists in the environment image, and generate a first control instruction if the target object exists; The sensitive content identification module is configured to obtain, in response to the first control instruction, location information of sensitive content in the frame image to be displayed, and determine first area information according to the location information of the sensitive content; The hidden layer generation module is configured to generate a hidden layer at least according to the first region information; The display module is configured to display an image according to the received frame image to be displayed and the hidden layer.

2. The display device according to claim 1, wherein: The behavior recognition module includes an image recognition model, a posture recognition model and a first judgment unit; The image recognition model is configured to receive the environment image, obtain a first object and a similarity between the first object and a first standard object; The gesture recognition model is configured to receive the environment image, obtain the user gesture and the similarity between the user gesture and a standard user gesture; The first judgment unit is configured to determine that the target object exists in the environment image and generate the first control instruction when the similarity between the first object and the first standard object, and the similarity between the user posture and the standard user posture both meet preset conditions.

3. The display device according to claim 1, wherein: The sensitive content identification module is specifically configured to, in response to the first control instruction, obtain location information of sensitive content in the frame image to be displayed, determine first area information according to the location information of the sensitive content, and obtain background image data of the first area in the frame image to be displayed; The hidden layer generation module is specifically configured to generate the hidden layer according to the first area information and the background image data.

4. The display device according to claim 1, wherein: The area in the hidden layer corresponding to the first area is a non-transparent monochrome image.

5. The display device according to claim 1, wherein: The sensitive content includes sensitive words, and the sensitive content identification module includes a sensitive word identification model and a second judgment unit; The sensitive text recognition model is configured to receive the frame image to be displayed, obtain a second object and a similarity between the second object and a second standard object; The second judgment unit is configured to determine that the sensitive text exists in the frame image to be displayed and obtain position information of the sensitive text when the similarity between the second object and the second standard object meets a preset condition.

6. The display device according to claim 2, wherein: The image recognition model uses a trained YOLOv7 network model or a trained YOLOv8 network model.

7. The display device according to claim 2, wherein: The posture recognition model uses a trained Swift YOLO network model.

8. The display device according to claim 5, wherein: The sensitive text recognition model adopts a trained CNN model.

9. The display device according to any one of claims 1 to 8, wherein: The display module includes a main display module and a secondary display module; The main display module is configured to display an image according to the received frame image to be displayed and the hidden layer; The auxiliary display module is configured to display the environment image when the target object exists in the environment image.

10. A display method of a display device according to any one of claims 1 to 9, wherein the display device comprises a collection module, a behavior recognition module, a sensitive content recognition module, a hidden layer generation module and a display module; wherein: The display method comprises: Using the acquisition module to acquire an environmental image around the display device; Using the behavior recognition module to determine whether a target object exists in the environment image, and generating a first control instruction if the target object exists; The sensitive content identification module obtains the position information of the sensitive content in the frame image to be displayed in response to the first control instruction, and determines the first area information according to the position information of the sensitive content; The hidden layer generating module generates a hidden layer at least according to the first region information; The display module displays an image according to the received frame image to be displayed and the hidden layer.

11. The display method according to claim 10, wherein: The behavior recognition module includes an image recognition model, a posture recognition model and a first judgment unit; The step of using the behavior recognition module to determine whether a target object exists in the environment image, and generating a first control instruction when the target object exists, includes: Inputting the environment image into the image recognition model, and using the image recognition model to obtain a first object and a similarity between the first object and a first standard object; Inputting the environment image into the gesture recognition model, and using the gesture recognition model to obtain the user gesture and the similarity between the user gesture and the standard user gesture; The first judgment unit determines that the target object exists in the environment image and generates the first control instruction when the similarity between the first object and the first standard object, and the similarity between the user posture and the standard user posture both meet preset conditions.

12. The display method according to claim 10, wherein: The sensitive content identification module is specifically configured to, in response to the first control instruction, obtain location information of sensitive content in the frame image to be displayed, determine first area information according to the location information of the sensitive content, and obtain background image data of the first area in the frame image to be displayed; The hidden layer generating module generates a hidden layer at least according to the first area information, specifically including: The sensitive content identification module obtains location information of sensitive content in the frame image to be displayed, determines the first area information according to the location information of the sensitive content, and obtains background image data of the first area in the frame image to be displayed; The hidden layer generating module generates the hidden layer according to the first region information and the background image data.

13. The display method according to claim 10, wherein: The sensitive content includes sensitive words, and the sensitive content identification module includes a sensitive word identification model and a second judgment unit; The sensitive content identification module obtains the position information of the sensitive content in the frame image to be displayed in response to the first control instruction, including: Inputting the frame image to be displayed into the sensitive word recognition model, and using the sensitive word recognition model to obtain a second object and a similarity between the second object and a second standard object; When the similarity between the second object and the second standard object meets a preset condition, the second judgment unit determines that the sensitive text exists in the frame image to be displayed, and obtains the position information of the sensitive text.

14. The display method according to claim 10, wherein: The display module includes a main display module and a secondary display module; The display method further includes: When the target object exists in the environment image, the environment image is displayed using the auxiliary display module.

Images