Protecting displayed information using adaptive digital watermarks

Abstract

The claimed technical solution relates to the field of protecting digital data displayed on the screen of an electronic device by using a digital watermarking mechanism. The technical result consists in providing a more effective method for protecting on-screen digital data by enhancing the robustness of digital watermarks during the generation of a protective layer. The claimed result is achieved by means of a computer-implemented method for protecting information displayed on the screen of a computing device, which is executed by a processor and comprises the steps of: generating a digital watermark in the form of a data block containing encoded information at least identifying the user of the computing device, wherein said information is encoded into a quaternary numeral system where each symbol from "0" to "3" is represented by a corresponding graphical element in the form of a line that lies at a given angle to an orthogonal coordinate axis, and the start of the data block is encoded as intersecting lines; generating a protective layer consisting of a set of data blocks and covering at least part of the display area of the screen of the computing device, wherein the colour and transparency of the digital watermark are determined; and applying the protective layer to the display area of the screen of the computing device.

Description

PROTECTING DISPLAYED INFORMATION WITH ADAPTIVE DIGITAL TAGS AREA OF TECHNOLOGY

[0001] The claimed technical solution relates to the field of protecting digital data displayed on the screen of an electronic device using a mechanism for introducing digital tags (DTs). LEVEL OF TECHNOLOGY

[0002] The use of digital images in the field of digital information protection is a common solution, in which encoded information is embedded into an image, allowing the identification of its owner or the person responsible for its leakage and / or unauthorized access.

[0003] Typically, such approaches use a specified graphic element or image region (mask) containing the digital mark. This mark can be either discernible or indiscernible to the human eye. Steganography is one example of an indiscernible mark.

[0004] The vulnerability of information displayed on a monitor screen is primarily due to the human-computer interaction process, which includes displaying information on the screen and visually perceiving it. To ensure information security, software is being developed that displays watermarks (labels) on device screens, similar to the watermarks used to mark documents. This mechanism serves both as a tool for identifying the source of a leak and as a deterrent to attackers. However, it currently appears difficult to develop an approach that simultaneously addresses these two inherently contradictory requirements: - Reliability of tag recognition and, as a result, accurate decoding of the information contained in them; - User comfort when working on devices with installed tags.

[0005] A known principle for forming a protective layer based on digital markings is disclosed in U.S. Patent 9,239,910 (Markany Inc., January 19, 2016). The solution involves creating an invisible protective layer consisting of digital markings, which is used as a background layer displayed on the device's screen and is invisible to the user.

[0006] A drawback of the existing approach is its lack of effectiveness, due to the fact that the protective layer is generated using a digital image (DEM), which is a text or graphic primitive selected from a database and then used to generate the space fill. This makes the DEM generation process sensitive to quality, and when subsequently capturing the screen image with an external device, such as a smartphone or camera, changing the camera angle or capturing a portion of the screen containing the DEM, this reduces the effectiveness of the DEM identification in identifying the source of the data leak. ESSENCE OF THE INVENTION

[0007] The proposed approach allows us to solve the technical problem of low stability (robustness) of the method of protecting digital data when they are recorded by external means from various angles and shooting quality, which leads to the appearance of interference, leading to a decrease in the effectiveness of this principle of protecting digital data.

[0008] The technical result consists in increasing the efficiency of the method for protecting digital data on device screens by increasing the stability of recognizing digital marks when forming a protective layer.

[0009] The stated result is achieved through a computer-implemented method for protecting data displayed on the screen of a computing device (CD), executed by a processor and containing the following stages: - a digital mark (DM) is formed in the form of a data block containing encoded information that at least identifies the user of the device, and the information is encoded in a quaternary number system, where each symbol from “0” to “3” corresponds to a graphic element in the form of a line located at a given angle to the orthogonal coordinate axis, and the beginning of the data block is encoded as intersecting lines; - a protective layer is formed, consisting of a set of the mentioned data blocks, covering at least part of the display area of ​​the VU screen, while the color and transparency of the CM are determined, and - apply a protective layer to the display area of ​​the VU screen.

[0010] In one of the particular embodiments of the method, information about the date and time is additionally encoded in the digital memory. [UN] In another particular embodiment of the method, the transparency of the CM is determined on the basis of calculating the average color as the average value for each of the RGB components of the VU screen.

[0012] In another particular embodiment of the method, the contrast coefficient is calculated between the brightness of the mixed color of the CM with the average color of the screen of the VU and the brightness of the color of the screen of the VU.

[0013] In another particular embodiment of the method, the contrast ratio is compared with a threshold value for adjusting the transparency of the digital matrix.

[0014] In another particular embodiment of the method, display areas on the VU screen are determined that contain alphanumeric information.

[0015] In another particular embodiment of the method, the information display areas are determined based on the calculation of color dispersion.

[0016] In another particular embodiment of the method, for CMs located in areas containing a color dispersion index below a given threshold value, the color mask is reset.

[0017] The claimed solution is also implemented using a computer-implemented method for processing protected data, performed using a processor and containing the steps of: - an image is obtained containing at least a portion of a screen image with information protected using the above method; - determine at least one CM; - perform decoding of information from the received digital memory by converting graphic symbols into a quaternary number system and then converting them into a decimal number system.

[0018] In one particular embodiment of the method, the obtained image is processed, including at least one of: contrast adjustment, brightness adjustment, and color correction.

[0019] In another particular embodiment of the method, at least one area with a CM in the image is marked with an indicator.

[0020] The claimed solution is also implemented using a system for protecting data displayed on the screen of a computing device, comprising at least one processor and at least one memory containing machine-readable instructions that, when executed by the processor, implement the above-mentioned method.

[0021] The claimed solution is also implemented using a system for processing protected data, comprising at least one processor and at least one memory containing machine-readable instructions, which, when executed by the processor, implement the above-mentioned method. BRIEF DESCRIPTION OF DRAWINGS

[0022] Fig. 1 illustrates the general operating principle of the claimed solution.

[0023] Fig. 2A illustrates an example of a CM.

[0024] Fig. 2B illustrates the principle of information coding in the digital memory.

[0025] Fig. 3 illustrates a block diagram of a method for implementing information protection using a digital memory.

[0026] Fig. 4 illustrates an example of placing a mask with a digital image on the device screen.

[0027] Fig. 5 illustrates a block diagram of a method for decoding information from a digital image.

[0028] Fig. 6 illustrates an example of capturing an image of information from the device screen.

[0029] Fig. 7 illustrates the general appearance of the computing device. IMPLEMENTATION OF THE INVENTION

[0030] Fig. 1 shows the general concept of the technical implementation of the claimed solution. As a rule, such solutions are aimed at protecting sensitive, confidential or other alphanumeric information displayed on the screen (111) of the computing device (CD) (CD) of the user, for example, a computer, laptop, tablet, etc. The protection of data on the screen (111) is carried out by introducing a digital memory (10), in which the relevant information is encoded for the subsequent identification of the responsible person who allowed the leakage of such information, or the unauthorized receipt of information outside the protected perimeter of the infrastructure, for example, by photographing, video recording or capturing (screenshot) the image on the screen (111) using external devices (smartphone, camera, etc.), including the subsequent printing of images with the received information.

[0031] Each CM (10) is a block of data that forms a protective layer (101) that covers most or all of the display area on the screen. (I ll) VU (PO). Layer (101) can be made invisible or barely visible, without distorting the image displayed on the device screen.

[0032] Fig. 2A shows the principle of forming the digital memory (10), which is a data block in which information identifying the user of the device (110) is encoded, for example, a personal identifier, personnel number, device number, etc. When encoding information, it is converted from the decimal number system to the quaternary number system, and then from the quaternary to the graphic one, by encoding the values ​​into graphic elements, as shown in Fig. 2B. Each symbol from "0" to "3" corresponds to a graphic element in the form of a line (12-15), located at a given angle to the orthogonal coordinate axis, and the beginning of the data block is encoded as intersecting lines, for example, "X" (11), as shown in Figs. 2A-2B.

[0033] Typically, the data embedded in the digital memory (10) is necessary to identify the device (software) or the user of the device, such as an employee with access to certain information. This data may include: personnel number, name, user image ID, IP address, MAC address, and the unique digital memory identifier. This information may be used individually or in any combination. Time and / or date information may also be encoded, such as the time the digital memory (10) was generated or the current date. Date and time information may be dynamically updated to reflect current information during the digital memory (10) encoding process.

[0034] When encoding information in the digital image sensor (10), the screen is divided into blocks, where each block is a separate encoding object that can be uniquely decoded. When forming the protective layer (101) from the digital image sensor (10), the transparency of the digital image sensor (10) is adapted to the background conditions of the screen (111), ensuring optimal visual perception of the information and recognition quality. Furthermore, functionality can be employed that adapts the filling of the digital image sensor (10) on the screen by analyzing its contents to place it only on areas containing alphanumeric information.

[0035] This principle allows for the most efficient use of screen space and ensures ease of reading and perception of protected information.

[0036] Let's take a closer look at the principle of information encoding in the digital memory (10). When encoding information, including, for example, a user ID and a date accurate to the hour, a transformation into a quaternary number system is used.

[0037] Seven characters are allocated for the date, and 10 characters for the user identifier, allowing for encoding up to 1,048,576 unique users. One character is used as a start character, defining the beginning of the information sequence. The encoded information is represented as lines, organized into a 6x3 block structure (6 characters in 3 rows). In this structure, the first element represents the start identifier.

[0038] In Fig. 2A-2B, the presented example of the digital memory (10) contains the following data: employee ID, time and date of creation of the digital memory. As an example, the employee ID is 777666. The time and date are February 7, 18:00, but this information will be considered in terms of the number of hours since the beginning of the year, which is equal to 906 hours. These numerical values ​​are converted from the decimal system to the quaternary system of calculation, which allows us to obtain the following type of encoded data: 777666 ^ 2331313002 906 - 0032022 Glue: “2331313002” + “0032022” -” 23313130020032022.

[0039] As a result, a block of graphic elements of the CM (10) is formed, shown in Fig. 2A, having its display in the form of lines encoding information as follows: X 2 3 3 1 3 1 3 0 0 2 0 0 3 2 0 2 2

[0040] As shown in Fig. 3, the claimed method (200) for protecting digital information comprises a series of sequential stages. At the first stage (201), the digital memory (10) is formed from a set of graphic elements (11 - 15) that encode the required data. In the presented example of the formation of the digital memory (10), the elements (11 - 15) are represented in the form of oblique lines arranged in increments of 45 degrees to the orthogonal coordinate axis, however, their shape and placement principle may be different (e.g., at an angle of 30, 50, 60, etc. degrees), ensuring compliance with the principle of delimitation of information encoding in the quaternary number system, and forming a pattern of the digital memory (10) block for encoding information within it.

[0041] At step (202), a protective layer (101) is formed in the form of a CM placement (10), which at step (203) is applied to the display area of ​​the screen (111) and fills most or all of the space of the screen (111). The use of this approach allows determine the value of the CM (10) in any area of ​​the screen (111) regardless of the coordinates or scale of the screenshot, photocopy of the screen contents, etc. Fig. 4 shows an example of a formed protective layer (101) based on a set of CM (10).

[0042] The number of digital micrographs (10) is determined based on the resolution of the screen (111) and the digital micrograph dimension (10). Typically, digital micrographs (10) are selected to be of equal dimension, but may also be arranged in an alternating order to form a layer (101), which also does not violate the technical implementation of the claimed solution in terms of subsequent data identification during the decoding process.

[0043] To ensure the adaptive transparency of the digital image (10) during user interaction with the screen (111), the VU (software) solution automatically matches the transparency of the object mask, i.e., the digital image and background, to the content on the screen (111). At each iteration, the program captures a screenshot at specified intervals and calculates the average color on it. The contrast ratio between the luminance of the mixed color of the marks with the average screen color and the luminance of the screen color is then calculated. The average color is defined as the average value for each RGB component of the screen image (111) when generating the digital image (10).

[0044] If the contrast is insufficient (the coefficient is below the specified threshold), the algorithm reduces the object mask's opacity to increase contrast. If the contrast is excessive (the coefficient is above the threshold plus the epsilon), the algorithm increases the object mask's opacity. If the contrast is within the acceptable range, the adaptation is skipped. The mask on the screen is then updated to account for the changed object opacity.

[0045] To solve this problem, the N and EPS parameters are selected empirically and determine the screen capture frequency and the acceptable contrast deviation from the reference value, respectively. The algorithm's input parameters include a screen mask matrix (111), of dimensions h x w x 4, where h is the screen height, w is the screen width, a label color array containing the label color obtained from the configuration, and the desired contrast value. This results in an updated screen color mask (111) used to generate the protective layer (101).

[0046] Method for calculating the contrast ratio of a color relative to another (if there is a color with an alpha channel): 1. The color of the marks and the average color of the screen are mixed (r, g, b, a are the color components of each pixel): b = b x * A г + b2* a2* (1 - a t ) 2. For mixed color and screen color, brightness 1 and 12 are calculated as: Result: C = 0.2126 * r z + 0.7152 * g t + 0.0722 * b L 3. After the brightness of the colors has been calculated and 12 it is possible to calculate the contrast coefficient K l2+ 0.05 - - — — if l2> Zi Zi + 0.05 2 1 K = Zi + 0.05 - - - - , otherwise l2+ 0.05

[0047] Additionally, this solution implements the ability to adjust the parameters of the CM, such as the frequency of access to the server for configuration updates; the thickness of graphic elements (lines); the distance between CM; the distance between CM elements; the offset of blocks along the x / y axis relative to each other; the color of labels in RGBA format; etc.

[0048] In one particular implementation of the method (200), when applying the protective layer (101) to the display area of ​​the screen (111), only areas containing any alphanumeric information can be determined. Such information display areas are typically determined based on a color dispersion calculation. In this case, the color mask is reset for the protective layer (101) and / or digital image (10) placed on areas whose color dispersion is below a predetermined threshold, rendering them invisible.

[0049] When the color variance is close to zero, this means that the given screen area represents either a background with no information or a uniform color image. This algorithm executes by capturing the screen for subsequent calculations. To further optimize the calculations, this algorithm is not called every n seconds, but rather upon a trigger event, such as a user clicking the mouse. When the algorithm runs, a screenshot is captured, which is then converted to grayscale. The screenshot is then divided into blocks, along with a screen mask, and the variance of each block of the mask is checked. If the variance of a block is less than the specified CONST value, the color mask for that block is reset to zero. Otherwise, the algorithm moves on to the next block. The screen mask is then updated to reflect the changes. This algorithm uses the CONST parameter, which is selected empirically and determines the variance threshold for mask blocks. The algorithm's input parameters include the screen mask matrix and the block size array. The output is an updated screen color mask.

[0050] After the protective layer (101) is formed, it is superimposed on the display area of ​​the VU (PO) screen (111) at step (203), such that the entire screen coverage area is covered by the superimposed layer (101). This approach allows the value of the digital mean (10) to be determined in any area of ​​the screen (111), regardless of the coordinates or scale of the screenshot or photocopy.

[0051] Next, we will consider the process of decoding information protected by the digital memory, shown in Fig. 5 - Fig. 6.

[0052] Fig. 5 illustrates a block diagram of the method (300) for decoding information from an image captured from the screen (111) of the CU (110). In the first step (301), an image (410) is received by the computing module (for example, a processor), which was made using an external device (400) and contains part or all of the information presented on the screen (111) of the CU (110), as shown in Fig. 6.

[0053] Next, the obtained image (410) undergoes a processing stage (302), at which the presence of the CM (10) in the image (410) is determined, for example, using neural network approaches to solving the detection problem or in another way.

[0054] The process of decoding information from an image (410) involves several stages. First, the image is processed, which involves one or more of the following algorithms: 1) The image is subject to contrast adjustment to increase the visibility of the CM (10) against the background of the image (410); 2) Adjusting the brightness of the image, which helps to highlight the details of the CM (10) and make them more visible; 3) Image color correction, which uses various filters or color transformations to highlight specific colors of the image (10) for better visibility.

[0055] Additionally, a mechanism for automatic or manual highlighting or selection of the CM (10) in the image (410) can be used, for example, using neural network approaches to solving the detection problem with the formation of indicators (bounding frames, etc.) for highlighting or selecting areas with graphic elements (11-15) that form the CM (10).

[0056] After selecting the CM (10), the user can generate a display of their graphical representations.

[0057] At step (303), the graphic representations of the digital memory (10) are converted into digital data for subsequent decoding, in particular, the graphic elements (11-15) are processed to obtain their representation in the symbolic form of the quaternary number system for further conversion into the decimal number system.

[0058] Data is read from the digital memory (10) from left to right, line by line, top to bottom. After mapping the graphic labels to digital ones, the following data is obtained for the example given earlier: 23313130020032022.

[0059] To decode the user ID, the first 10 characters (2331313002) are converted to decimal, which corresponds to 777666. To convert the date, the remaining characters (0032022) are converted to decimal, which corresponds to 906, which is February 7 at 6:00 PM in the current year.

[0060] The result of decoding the CM (10) is: - Employee ID 777666 - Date / Time February 7, 18:00 (Moscow, standard time).

[0061] The claimed solution allows achieving the following effects: - increasing the reliability of recognition of encoded information by introducing special marks that have a high degree of stability and recognition accuracy, even in low light conditions or in the presence of interference, which ensures more reliable coding and recognition of information, increasing the efficiency of using information marking technology on the screen; - improving the user's visual comfort when working with labels by implementing a number of improvements in terms of adapting the color characteristics of labels and the visibility of labels, taking into account the contents of the screen.

[0062] Fig. 7 shows a general view of a computing device (500) suitable for performing the methods (200, 300). The device (500) may be, for example, a server or another type of computing device that can be used to implement the claimed technical solution. This includes being part of a cloud computing platform.

[0063] In general, the computing device (500) comprises one or more processors (501), memory means such as RAM (502) and ROM (503), input / output interfaces (504), input / output devices (505), and a device for network interaction (506), connected by a common information exchange bus.

[0064] The processor (501) (or several processors, multi-core processor) can be selected from a range of devices that are widely used at the present time, for example, from Intel™, AMD™, Apple™, Samsung Exynos™, MediaTEK™, Qualcomm Snapdragon™, etc. A graphics processor can also be used as the processor (501), for example, from Nvidia, AMD, Graphcore, etc.

[0065] RAM (502) is random access memory (RAM) and is designed to store machine-readable instructions executed by the processor (501) to perform the necessary logical data processing operations. RAM (502) typically contains executable instructions from the operating system and corresponding software components (applications, software modules, etc.).

[0066] ROM (503) represents one or more permanent data storage devices, such as a hard disk drive (HDD), a solid-state drive (SSD), flash memory (EEPROM, NAND, etc.), optical storage media (CD-R / RW, DVD-R / RW, BlueRay Disc, MD), etc.

[0067] To organize the operation of the device components (500) and to organize the operation of external connected devices, various types of I / O interfaces (504) are used. The choice of the appropriate interfaces depends on the specific design of the computing device, which may include, but are not limited to: PCI, AGP, PS / 2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini, type C), TRS / Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc.

[0068] To ensure user interaction with the computing device (500), various I / O information means (505) are used, for example, a keyboard, a display (monitor), a touch display, a touchpad, a joystick, a mouse, a light pen, a stylus, a touch panel, a trackball, speakers, a microphone, augmented reality means, optical sensors, a tablet, light indicators, a projector, a camera, biometric identification means (a retinal scanner, a fingerprint scanner, a voice recognition module), etc.

[0069] The network interaction means (506) ensures the transmission of data by the device (500) via an internal or external computer network, for example, an Intranet, the Internet, a LAN, etc. One or more means (506) may be, but are not limited to: an Ethernet card, a GSM modem, a GPRS modem, an LTE modem, a 5G modem, a satellite communication module, an NFC module, a Bluetooth and / or BLE module, a Wi-Fi module, etc.

[0070] Additionally, satellite navigation tools included in the device (500) can also be used, for example, GPS, GLONASS, BeiDou, Galileo.

[0071] The submitted application materials disclose preferred examples of the implementation of the technical solution and should not be interpreted as limiting other, particular examples of its implementation that do not go beyond the scope of the requested legal protection, which are obvious to specialists in the relevant field of technology.

Claims

FORMULA 1. A computer-implemented method for protecting data displayed on the screen of a computing device (CD), executed by a processor and comprising the steps of: - a digital mark (DM) is formed in the form of a data block containing encoded information that at least identifies the user of the device, and the information is encoded in a quaternary number system, where each symbol from “O” to “3” corresponds to a graphic element in the form of a line located at a given angle to the orthogonal coordinate axis, and the beginning of the data block is encoded as intersecting lines; - a protective layer is formed, consisting of a set of the mentioned data blocks, covering at least part of the display area of ​​the VU screen, while the color and transparency of the CM are determined, and - apply a protective layer to the display area of ​​the VU screen.

2. The method according to paragraph 1, characterized in that the digital memory additionally encodes information about the date and time.

3. The method according to paragraph 1, characterized in that the transparency of the digital material is determined on the basis of calculating the average color as the average value for each of the RGB components of the display device.

4. The method according to paragraph 3, characterized in that the contrast coefficient is calculated between the brightness of the mixed color of the digital camera with the average color of the display unit screen and the brightness of the color of the display unit screen.

5. The method according to paragraph 4, characterized in that the contrast coefficient is compared with a threshold value for adjusting the transparency of the digital image sensor.

6. The method according to paragraph 1, characterized in that display areas on the VU screen containing alphanumeric information are determined.

7. The method according to claim 6, characterized in that the information display areas are determined on the basis of calculating the color dispersion.

8. The method according to paragraph 7, characterized in that for CMs located in areas containing a color dispersion index below a specified threshold value, the color mask is reset.

9. A computer-implemented method for processing protected data, performed by a processor and comprising the steps of: - an image is obtained containing at least a portion of a screen image with information protected using the method according to any of paragraphs 1-9; - determine at least one CM; - perform decoding of information from the received digital memory by converting graphic symbols into a quaternary number system and then converting them into a decimal number system.

10. The method according to claim 9, characterized in that the obtained image is processed, including at least one of: contrast adjustment, brightness adjustment, color correction.

11. The method according to claim 9, characterized in that at least one area with a CM in the image is marked with an indicator.

12. A system for protecting data displayed on the screen of a computing device, comprising at least one processor and at least one memory containing machine-readable instructions that, when executed by the processor, implement the method according to any one of paragraphs 1-8.

13. A system for processing protected data, comprising at least one processor and at least one memory containing machine-readable instructions that, when executed by the processor, implement the method according to any one of paragraphs 9-11.

Images