Invisible QR code embedding, detection and repair method based on neural network

Abstract

The present invention provides an invisible two-dimensional code embedding, detection and repair method based on a neural network, comprising: using nonlinear time psycho-visual modulation technology to embed a two-dimensional code into a video so that the two-dimensional code cannot be observed by human eyes; collecting a set of data containing the two-dimensional code from the video after the two-dimensional code is embedded, and using a first neural network to locate the two-dimensional code in the data; using a second neural network to calibrate the located two-dimensional code to obtain a calibrated two-dimensional code; inputting the calibrated two-dimensional code into a third neural network for repair to obtain a repaired two-dimensional code. The invisible two-dimensional code embedding, detection and repair method based on a neural network provided by the present invention can effectively improve the embedding quality of the invisible two-dimensional code and the efficiency of detection and repair.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for embedding, detecting and repairing invisible two-dimensional codes based on a neural network. Background Art

[0002] Nowadays, people are often forced to accept many additional patterns (such as QR codes, watermarks, and hyperlinks) when watching videos or movies. Even though these patterns provide effective information in a small space, they are meaningless to humans and even distract people from the video itself. Therefore, it is urgent to find ways to embed these patterns on digital screens so that people cannot observe the patterns with the naked eye, but can obtain the information contained in the patterns through shooting devices such as mobile phones.

[0003] Wu proposed temporal psycho-visual modulation (TPVM) in "Temporal psychovisual modulation: A new paradigm of information display" and has been applied to information security, movie anti-piracy, medical image visualization, and visible light communication. TPVM uses the consensus of visual psychophysics that in most cases, the human visual system cannot perceive changes in light signals exceeding 60Hz, while the refresh rate of displays can now be 120Hz or even more than 240Hz. Combining the original image with additional information at a refresh rate exceeding 60Hz, people will not be able to distinguish the difference between the processed video and the original video, but can use a camera to capture the changes.

[0004] Gao proposed the concept of invisible QR code in "The invisible QR code". They proposed to embed QR code in video using linear method, and then detect and repair QR code using traditional binarization method. However, the brightness and gray value of pixels are not linearly related. Using linear embedding makes the brightness of the video change in the time domain, which will reduce the quality of the video; the detection using traditional methods also lacks robustness. Song used U-Net to improve the repair part of invisible QR code in "Deep restoration of invisible QR code from tpvm display". However, before using U-Net to restore the QR code, they need to manually locate the four corners of the invisible QR code, which is not feasible in practical applications.

[0005] After searching, the Chinese invention patent CN201810032597.6 discloses a video playback method and display system, which performs time domain frequency modulation on the original video; performs spatial frequency modulation on the video after time domain frequency modulation, and plays the video after spatial frequency modulation at a matching frame rate; so that the video after spatial frequency modulation contains a first image that is invisible to the human eye. Since the time domain and spatial domain frequencies of the first image are both greater than the maximum frequency that the human eye can distinguish, and are within the frequency range that the shooting device can distinguish, the human eye cannot distinguish the first image when the video is played, and the shooting device will capture the video screen including the first image when shooting, thereby keeping the original picture clarity unchanged when the human eye watches, and effectively preventing the video screen from being captured by the shooting device, ensuring the security of the video content. The above invention patent replaces each frame of the image with N frames of the image, which means that it cannot be realized without a player with a higher frame rate and a display with a higher refresh rate; at the same time, the invention is similar to the above linear method, and experiments show that the linear method affects the video quality.

[0006] These problems existing in the current methods urgently require the development of a more robust and efficient invisible QR code embedding, detection and repair method. Summary of the invention

[0007] In view of the above-mentioned deficiencies in the prior art, the object of the present invention is to provide an invisible two-dimensional code embedding, detection and repair method based on a neural network.

[0008] The first invention of the present invention provides an invisible two-dimensional code embedding, detection and repair method based on a neural network, comprising:

[0009] The two-dimensional code is embedded into the video by using nonlinear temporal psycho-visual modulation technology, so that the two-dimensional code cannot be observed by human eyes;

[0010] Collect a set of data containing the QR code from the video after the QR code is embedded, and locate the QR code in the data using a first neural network;

[0011] Using a second neural network to calibrate the located two-dimensional code to obtain a calibrated two-dimensional code;

[0012] The calibrated two-dimensional code is input into a third neural network for repair to obtain a repaired two-dimensional code.

[0013] Optionally, the temporal psychovisual modulation technology comprises:

[0014] Combining the original video with additional information, wherein the refresh frequency of the additional information is greater than the maximum frequency that can be distinguished by the human visual system and is within the frequency range that can be distinguished by the shooting device;

[0015] The human visual system cannot tell the difference between the combined video and the original video, but using a camera will be able to capture the changes.

[0016] Optionally, embedding the QR code into the video includes: embedding the QR code of opposite colors at the same position in two adjacent frames of the video, and the brightness of the video remains unchanged within a set time period.

[0017] Optionally, the QR code with opposite colors is embedded in the same position in two adjacent frames of the video, wherein: the RGB value representing the QR code is added to the RGB channel of the odd frame in the two adjacent frames, and the corresponding RGB value is subtracted from the even frame.

[0018] Optionally, the nonlinear temporal psychovisual modulation technology is specifically:

[0019] The brightness and grayscale value of a pixel are not in a linear relationship, and there is an error in the RGB values ​​after the adjacent frames are embedded with QR codes of opposite colors.

[0020] Optionally, the using the first neural network to locate the two-dimensional code in the data includes: using the first neural network to take the data containing the two-dimensional code as input data to obtain an external rectangular frame where the two-dimensional code is located.

[0021] Optionally, the calibrating the located two-dimensional code by using the second neural network includes:

[0022] The coordinates of the four vertices of the two-dimensional code are obtained by using the second neural network, and the two-dimensional code is calibrated into a square by using the rotation matrix A; wherein:

[0023]

[0024] In the above formula, θ is the rotation angle, (t x , t y ) are the displacement coordinates.

[0025] Optionally, the calibrated two-dimensional code is input into a third neural network for repair, including: using the third neural network to remove the background in the calibrated two-dimensional code and restore the two-dimensional code to black and white.

[0026] The second aspect of the present invention provides an invisible two-dimensional code embedding, detection and repair device, including a memory, a processor and a computer program stored in the memory and capable of running on the processor, and the processor is used to execute the neural network-based invisible two-dimensional code embedding, detection and repair method when executing the program.

[0027] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, it is used to execute the invisible two-dimensional code embedding, detection and repair method based on neural network.

[0028] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0029] The neural network-based invisible QR code embedding, detection and repair method provided by the present invention utilizes nonlinear temporal psychovisual modulation technology to embed the QR code into a video, and then uses a neural network to locate, calibrate and repair the QR code, which can effectively improve the quality of invisible QR code embedding, the robustness and efficiency of QR code detection and repair, and can achieve efficient and real-time detection and repair.

[0030] The invisible QR code embedding, detection and repair method based on neural network provided by the present invention embeds QR codes of opposite colors at the same position in two adjacent frames to ensure that the brightness of the video remains unchanged within a certain period of time so that it cannot be observed by the human eye. A camera device is used to collect a group of data containing the QR code from the video and a neural network is used to locate the QR code in the data. The method is suitable for different shooting angles and lighting conditions, and the robustness is greatly improved.

[0031] The present invention abandons the linear method and adopts a nonlinear method to embed the two-dimensional code with opposite colors, so that the video quality effect is better.

[0032] The present invention has no additional requirements on the original video, the video player, and the display, and can add a QR code starting from any frame of the video and at any position. Therefore, the application can highlight the human-computer interactivity and can be used to convey more additional information. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0034] Figure 1 This is a flow chart of an invisible two-dimensional code embedding, detection and repair method based on a neural network according to an embodiment of the present invention;

[0035] Figure 2 This is an overall flow chart of an invisible two-dimensional code embedding, detection and repair method based on a neural network according to a preferred embodiment of the present invention;

[0036] Figure 3 FIG. 4 is a relationship between the RGB value and brightness of a pixel according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following is a detailed description of the embodiments of the present invention: This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given. It should be pointed out that for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

[0038] Figure 1 FIG. 1 is a flow chart of an invisible two-dimensional code embedding, detection and repair method based on a neural network according to an embodiment of the present invention. Figure 1 As shown, the invisible two-dimensional code embedding, detection and repair method based on neural network in this embodiment includes the following steps:

[0039] S1: The QR code is embedded into the video using nonlinear temporal psychovisual modulation technology, making the QR code invisible to human eyes;

[0040] S2: collecting a set of data containing the QR code from the video embedded with the QR code, and locating the QR code in the data using the first neural network;

[0041] S3: Using the second neural network to calibrate the located QR code to obtain a calibrated QR code;

[0042] S4: Input the calibrated QR code into the third neural network for repair to obtain a repaired QR code.

[0043] In this embodiment, nonlinear temporal psychovisual modulation technology is used to embed the QR code into the video, and then the neural network is used to locate, calibrate and repair the QR code, which can effectively improve the quality of invisible QR code embedding, the robustness and efficiency of QR code detection and repair.

[0044] In the above S1, the temporal psychovisual modulation technology is: combining the original video with additional information, the refresh frequency of the additional information is greater than the maximum frequency that the human visual system can distinguish, and is within the frequency range that the shooting device can distinguish. Specifically, the human visual system cannot perceive the changes in light signals exceeding 60Hz. By combining the original image with additional information with a refresh rate exceeding 60Hz, people will not be able to distinguish the difference between the processed video and the original video, but can use a camera to capture the changes.

[0045] In some embodiments, in S1, the QR code is embedded in the video by embedding the QR code of opposite colors in the same position in two adjacent frames to ensure that the brightness of the video remains unchanged within a certain period of time. Further, the QR code of opposite colors can be embedded by adding the RGB value representing the QR code to the RGB channel of the odd frame in two adjacent frames of the video, and subtracting the corresponding RGB value in the even frame.

[0046] Figure 3The relationship between the RGB value and brightness of a pixel in an embodiment of the present invention is shown in FIG. Figure 3 As shown, in actual applications, the brightness and grayscale value of pixels are not linearly related. Using a linear method to embed makes the brightness of the video change in the time domain, which will reduce the quality of the video. In some embodiments, in order to solve this problem, in the above S1, a nonlinear time psycho-visual modulation technology is used, that is, the two-dimensional code is nonlinearly embedded in the video using the time psycho-visual modulation technology, so that after the RGB values ​​of the two-dimensional codes with opposite colors are embedded in the above adjacent frames, there is a certain error in the RGB values ​​of the adjacent frames. This is consistent with the brightness and grayscale value of the pixels in actual applications, ensuring the quality of the video.

[0047] Figure 2 This is a general flow chart of an invisible two-dimensional code embedding, detection and repair method based on a neural network according to a preferred embodiment of the present invention. Figure 2 In order to better illustrate the technical solution of the present invention, the following process is described in conjunction with a preferred application example:

[0048] In this preferred embodiment, the invisible two-dimensional code embedding, detection and repair method based on neural network includes the following steps:

[0049] The first step is to use nonlinear temporal psycho-visual modulation technology to embed the QR code into the video so that it cannot be observed by the human eye;

[0050] Specifically, by embedding a QR code with opposite colors in the same position in two adjacent frames, the brightness of the video is guaranteed to remain unchanged within a certain period of time. The RGB value representing the QR code is added to the RGB channel of the odd-numbered frames of the video, where the black part of the QR code is 0 and the white part is 25. At the same time, the corresponding RGB value is subtracted from the even-numbered frames, where the black part is 0 and the white part is 11. -1 (2*l(I)-l(I+25)). Where I represents the RGB value of the original pixel, function l(.) represents the brightness value corresponding to the RGB value, and function l -1 (.) indicates the RGB value corresponding to the brightness value.

[0051] Step 2: Use a camera to collect a set of data containing a QR code from the video and use the first neural network to locate the QR code in the data;

[0052] Specifically, the first neural network is used to obtain the bounding rectangle where the two-dimensional code is located in the input data (data containing the two-dimensional code). Because the embodiment of the present invention is to achieve efficient and real-time detection and repair, the first neural network mentioned above can choose to use YOLOV3. The loss function uses the position coordinates of the bounding rectangle where the two-dimensional code is located and the relative position relationship of the four vertices of the two-dimensional code as constraints.

[0053] Step 3: Use the second neural network to perform some post-processing on the located QR code to obtain a calibrated QR code;

[0054] Specifically, the coordinates of the four vertices of the QR code are obtained by using the second neural network. Because the embodiment of the present invention is to achieve efficient and real-time detection and repair, the second neural network uses a lightweight MixNet. The loss function uses the coordinates of the four vertices of the QR code and the lengths of the four edges as constraints.

[0055] Then the QR code is calibrated into a square by rotating the matrix A, where:

[0056]

[0057] In the above formula, θ is the rotation angle, (t x , t y ) are the displacement coordinates.

[0058] Step 4: Get the repaired QR code through the third neural network;

[0059] Specifically, the third neural network is used to extract the background of the calibrated QR code and restore the QR code to black and white. Because the present invention aims to detect and repair efficiently and in real time, the neural network uses a lightweight Efficient U-Net. The loss function uses the pixel value of each pixel as a constraint.

[0060] In the above preferred embodiments of the present invention, the network output targets are different, and using different networks at different stages can obtain the best and fastest effect, which is conducive to improving the accuracy and speed of repair.

[0061] In the above embodiment of the present invention, the appearance of the repaired two-dimensional code is substantially the same as that of the original two-dimensional code, and the contents obtained by scanning are the same.

[0062] Implementation effect:

[0063] In order to verify the effectiveness of the invisible two-dimensional code embedding, detection and repair method based on neural network provided in the above embodiment of the present invention, a data set named 'IQRCodes' is first established to verify the detection and repair effect of invisible two-dimensional codes. The data is embedded using five types of two-dimensional codes and several videos, and one to four two-dimensional codes are randomly embedded at any position of each frame of the video. Then the camera of the mobile phone is set to slow motion mode and captured from various angles to ensure that the invisible two-dimensional code in the data set contains various types of geometric distortion. Finally, images that do not contain two-dimensional codes or are severely obscured by two-dimensional codes are deleted to generate a data set containing more than 2,500 images and more than 5,000 two-dimensional codes. In the experiment, 75% of the data set is used as a training set, 5% as a verification set, and the rest as a test set. The experimental results show that compared with the existing results, the accuracy of the present invention exceeds 99% with little increase in time consumption, and it is suitable for different shooting angles and lighting conditions, which greatly improves the robustness.

[0064] The invisible QR code embedding, detection and repair method based on neural network in the above preferred embodiment of the present invention utilizes nonlinear time psycho-visual modulation technology to embed the QR code into the video, and embeds the QR code of opposite color at the same position in two adjacent frames to ensure that the brightness of the video remains unchanged within a certain period of time, so that it cannot be observed by the human eye. Then, a camera is used to collect a group of data containing the QR code from the video and the neural network is used to locate the QR code in the data. Finally, a second neural network is used to perform certain post-processing on the located QR code to obtain a calibrated QR code, and then a third neural network is used to obtain the repaired QR code. The steps of the above steps can effectively improve the quality of invisible QR code embedding, the robustness and efficiency of QR code detection and repair.

[0065] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for embedding, detecting and repairing invisible two-dimensional codes based on neural networks, characterized in that: include: The two-dimensional code is embedded into the video by using nonlinear temporal psycho-visual modulation technology, so that the two-dimensional code cannot be observed by human eyes; Collect a set of data containing the QR code from the video after the QR code is embedded, and locate the QR code in the data using a first neural network; Using a second neural network to calibrate the located two-dimensional code to obtain a calibrated two-dimensional code; Inputting the calibrated two-dimensional code into a third neural network for repair to obtain a repaired two-dimensional code; The method of embedding the QR code into the video comprises: Embed a QR code with opposite colors at the same position in two adjacent frames of the video, and keep the brightness of the video unchanged within the set time period; The method of using the first neural network to locate the two-dimensional code in the data includes: Using the first neural network, taking the data containing the two-dimensional code as input data, and obtaining a circumscribed rectangular frame where the two-dimensional code is located; The method of calibrating the located two-dimensional code by using the second neural network includes: The coordinates of the four vertices of the two-dimensional code are obtained by using the second neural network, and the two-dimensional code is calibrated into a square by using the rotation matrix A; wherein: In the above formula, θ is the rotation angle, (t x , t y ) is the displacement coordinate; Inputting the calibrated two-dimensional code into a third neural network for repair, including: The background in the calibrated two-dimensional code is extracted by using a third neural network, and the two-dimensional code is restored to black and white.

2. According to the neural network-based invisible two-dimensional code embedding, detection and repair method of claim 1, it is characterized in that: The temporal psychovisual modulation technology comprises: Combining the original video with additional information, wherein the refresh frequency of the additional information is greater than the maximum frequency that can be distinguished by the human visual system and is within the frequency range that can be distinguished by the shooting device; The human visual system cannot tell the difference between the combined video and the original video, but using a camera will be able to capture the changes.

3. The method for embedding, detecting and repairing invisible two-dimensional codes based on a neural network according to claim 1, characterized in that: The two-dimensional codes with opposite colors are embedded in the same position in two adjacent frames of the video, wherein: The RGB value representing the QR code is added to the RGB channel of the odd-numbered frame in two adjacent frames, and the corresponding RGB value is subtracted from the even-numbered frame.

4. The method for embedding, detecting and repairing invisible two-dimensional codes based on a neural network according to claim 3, characterized in that: The nonlinear temporal psycho-visual modulation technology is specifically: The brightness and grayscale value of a pixel are not in a linear relationship, and there is an error in the RGB values ​​after the adjacent frames are embedded with QR codes of opposite colors.

5. An invisible two-dimensional code embedding, detection and repair device, comprising a memory, a processor and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the program, it is used to execute the invisible two-dimensional code embedding, detection and repair method based on neural network as described in any one of claims 1-4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it is used to execute the invisible two-dimensional code embedding, detection and repair method based on neural network as described in any one of claims 1-4.

Images