A QR code-attached video recognition and analysis method based on object perception

Abstract

This invention provides a QR code attachment video recognition and analysis method based on object perception, including a mobile terminal equipped with a QR code scanning and verification module; it also includes the following steps: S1: real-time video acquisition and preprocessing; S2: feature extraction; S3: feature fusion and classification; S4: result output and anti-cheating judgment. This invention overcomes the shortcomings of existing QR code inspection technologies, such as the inability to identify the physical properties of the carrier, slow response speed, and susceptibility to deception by pre-stored images / videos. By forcing the mobile terminal to capture a short video (usually 2-3 seconds) containing the target QR code in real time, and quickly analyzing the dynamic visual features of the QR code area and its surrounding background in the video frame sequence (total time ≤ 5 seconds), it determines whether the QR code is attached to a three-dimensional physical surface or only exists on a two-dimensional planar medium (image or screen), thereby preventing cheating behavior using pre-stored QR code images / videos from the source.

Description

Technical Field

[0001] This invention relates to the intersection of computer vision, mobile applications, and anti-counterfeiting technology, and in particular to a method for QR code-attached video recognition and analysis based on object perception. Background Technology

[0002] Existing sales auditing systems typically require auditors to complete audits by scanning QR codes on products. However, some auditors, in order to save time, may directly scan pre-prepared QR code images instead of the actual product. This cheating behavior leads to distorted backend data and consequently affects decision-making.

[0003] Currently, there are some anti-cheating methods on the market, such as:

[0004] 1. Require uploading multiple photos from different angles: This method still leaves the possibility that users may use pre-stored images in their albums, and manual review is costly.

[0005] 2. Using a single image for 3D information inference: Due to the limited information in a single image, the accuracy is low and it cannot cope with advanced cheating methods (such as high-quality printed images).

[0006] 3. Video-based anti-counterfeiting methods: Some methods use video, but they usually take a long time to process (more than 10 seconds), which is difficult to meet the requirements of mobile devices for real-time response (within 5 seconds), and allows photo album uploads, which may have the vulnerability of pre-recorded video fraud.

[0007] Therefore, a method is needed that can quickly and accurately distinguish whether the carrier of a QR code is a physical object or an image, and to prevent cheating at the source (i.e., to force the use of real-time video). Summary of the Invention

[0008] The purpose of this invention is to address the shortcomings of existing technologies by proposing a QR code-attached video recognition and analysis method based on object perception.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A QR code-attached video recognition and analysis method based on object perception includes a mobile terminal equipped with a QR code scanning and verification module; it also includes the following steps:

[0011] S1: Real-time video acquisition and preprocessing;

[0012] S11: Startup and constraints of the QR code scanning verification module;

[0013] On the mobile terminal, start the QR code scanning verification module, set the permissions of the QR code scanning verification module, restrict it to only calling the rear camera and block the album access interface, and the user uses the QR code scanning verification module to take a picture of the object with the QR code to be recognized attached.

[0014] S12: Short video recording;

[0015] After the user triggers the recording, a short video of a certain duration is recorded continuously.

[0016] S13: Keyframe extraction and decoding;

[0017] From the recorded video, samples are taken at fixed intervals to extract valid frames that have been successfully recognized by QR codes. Each frame is preprocessed, and the QR code region is located and decoded using a fast QR code recognition library. The bounding box coordinates, center point coordinates, and area of ​​the QR code in each frame are recorded. The frame image is then converted into a grayscale image.

[0018] S2: Feature extraction;

[0019] S21: Spatiotemporal feature extraction of QR code region;

[0020] S211: Position / Attitude Change Analysis;

[0021] S212: Surface Deformation Consistency Analysis;

[0022] S213: Motion mode determination;

[0023] S214: Eigenvalue output;

[0024] S22: Extraction of background features around the QR code;

[0025] S221: Background motion consistency analysis;

[0026] S222: Obtain background depth cues: Use a lightweight depth estimation model to output a depth map and obtain the depth gradient of the QR code boundary;

[0027] S223: Background stability analysis;

[0028] S23: Extracting image quality and anti-spoofing features;

[0029] The QR code region of the middle valid frame is selected for analysis, and the high-frequency noise energy, moiré effect and edge sharpness of the QR code region are analyzed.

[0030] S3: Feature fusion and classification;

[0031] S31: Extract feature vectors;

[0032] S32: Feature vector input classification model;

[0033] S4: Result output and anti-cheating judgment;

[0034] Receive the output of the classification model and determine whether it is a three-dimensional physical carrier or a two-dimensional planar carrier.

[0035] Preferably, step S211 involves calculating motion features based on the bounding box coordinates, center point coordinates, and area of ​​the QR code in adjacent frames. The motion features include: the displacement vector and displacement magnitude of the bounding box center point of the entire frame sequence, the average displacement, the maximum displacement, the displacement standard deviation, the bounding box area change rate, the average area change rate, and the maximum area change rate.

[0036] Preferably, step S212 is to calculate the cosine similarity between all pairs of non-zero motion vectors, and then calculate the average cosine similarity of all inter-frame motion vectors in the entire sequence as a motion consistency index.

[0037] S212a: Obtain a non-zero motion vector;

[0038] Obtain the displacement vectors of the center points of the bounding boxes of all adjacent QR codes to get a set of motion vectors. Remove motion vectors with a magnitude of <10^(-6) to get valid motion vectors. Combine the valid vectors into a set of non-zero motion vectors.

[0039] S212b: Calculate the cosine similarity between any two distinct vectors in a set of non-zero motion vectors;

[0040] Select valid vector pairs and calculate the cosine similarity of all valid vector pairs;

[0041] S212c: Obtain motion consistency index;

[0042] Calculate the average cosine similarity of all vector pairs to obtain the motion consistency index.

[0043] Preferably, it further includes:

[0044] S213a: Determine whether it is a 2D motion marker;

[0045] Determine whether the movement of the QR code is a two-dimensional planar motion: If it meets the following conditions: average displacement ≤ 15 pixels, displacement standard deviation < 10 pixels, average area change rate ≤ 5% and motion consistency > 0.7, it is determined to be a two-dimensional planar motion; if it does not meet these conditions, it is determined not to be a two-dimensional planar motion, and the determination result is converted to 1.0 or 0.0 via Boolean value.

[0046] S213b: Determine whether it is a rigid planar marker;

[0047] Determine if the QR code is attached to a rigid plane: If the following conditions are met: average area change rate ≤ 3%, maximum area change rate ≤ 10%, and displacement standard deviation < 8 pixels, then it is determined to be attached to a rigid plane. If these conditions are not met, then it is determined not to be attached to a rigid plane. The determination result is converted to 1.0 or 0.0 via Boolean value.

[0048] Preferably, in step S221, a background region is selected outside the QR code bounding box, and the optical flow fields of the background region and the QR code region are calculated using the Farneback optical flow method. The difference between the average optical flow vector of the background region and the average optical flow vector of the QR code region is compared.

[0049] S221a: Calculate the optical flow field of the background region and the QR code region;

[0050] First, obtain the QR code region and background region: extract the grayscale image of a set of adjacent frames of the middle valid frame, and crop out the rectangular region as the QR code region according to the coordinates of the QR code's bounding box; select a background region outside the QR code's bounding box.

[0051] Next, feature points are extracted: first, feature points of the QR code region and background region of a valid frame are extracted, and then feature point matching is used to extract feature points of the QR code region and background region of another frame.

[0052] Finally, the optical flow vectors of the feature points in the QR code region and the background region are calculated separately.

[0053] The optical flow vector of the QR code region is obtained by comparing the difference between the matching feature points in the QR code region, and the optical flow vector of the background region is obtained by comparing the difference between the matching feature points in the background region.

[0054] S221b: Calculate the average optical flow vector of all feature points in the QR code region and the background region respectively;

[0055] S221c: Calculate motion difference;

[0056] The motion difference is obtained by calculating the magnitude of the difference between the average optical flow vector of the QR code region and the average optical flow vector of the background region.

[0057] Preferably, step S222 includes:

[0058] Step 1: Preprocessing of valid frame images;

[0059] Obtain grayscale images of valid frames, and generate images that meet the input requirements of a lightweight MiDaS model through resizing and normalization.

[0060] Step 2: Depth map generation;

[0061] The preprocessed valid frame images are input into the lightweight MiDaS model, and the MiDaS model infers and outputs a depth map.

[0062] Step 3: Extract the QR code boundaries;

[0063] Based on the obtained bounding box, the four sides of the QR code bounding box are discretized and sampled using a linear interpolation method to obtain the set of pixels of the QR code boundary.

[0064] Step 4: Calculate the depth gradient;

[0065] The Sobel operator is used to calculate the gradient of the depth map, and the gradient magnitude of the extracted pixels is calculated to obtain the set of boundary gradients.

[0066] Step 5: Obtain the depth gradient at the QR code boundary:

[0067] The maximum value in the set of boundary gradients is obtained as the depth gradient at the boundary of the QR code.

[0068] Preferably, step S223 includes: obtaining the grayscale value of each pixel in the background region of the previous frame and the current frame, and directly calculating the correlation coefficient based on the grayscale value using the corrcoef function to obtain the correlation coefficient of the background region.

[0069] Preferably, the feature vector is a 12-dimensional vector, specifically composed of:

[0070] Six-dimensional features from S21:

[0071] Displacement normalization value: The average displacement is normalized and mapped to the range [0, 1].

[0072] The average area change rate is taken directly.

[0073] The motion consistency index is directly taken.

[0074] 2D motion mark: 1.0 or 0.0;

[0075] Rigid plane marking: 1.0 or 0.0;

[0076] Displacement standard deviation normalized value: mapped to the range [0, 1] by displacement standard deviation normalization;

[0077] 3D features from S22:

[0078] Motion difference normalization value: Motion difference normalization is mapped to the range [0, 1];

[0079] Depth gradient normalization value: The depth gradient normalization at the QR code boundary is mapped to the range [0, 1];

[0080] Background stability is obtained by directly acquiring the correlation coefficient of the background region.

[0081] 3D features from S23:

[0082] High-frequency energy: The high-frequency noise energy of the QR code area is directly taken as the value;

[0083] Moiré pattern intensity: The intensity of the moiré effect in the QR code area is directly taken as the value;

[0084] Edge sharpness: The average gradient of the edge of the QR code area is directly taken.

[0085] Preferably, the classification model is a pre-trained lightweight CNN classifier with 3 convolutional layers, 128 input feature dimensions, ReLU activation function, and output probabilities of three-dimensional physical carriers and two-dimensional planar carriers.

[0086] Preferably, step S4 specifically includes:

[0087] If it is determined to be a "three-dimensional physical carrier", the verification is successful, confirming that the QR code is attached to a real object;

[0088] If the QR code is determined to be a "two-dimensional planar carrier", the verification will fail, indicating that there is suspicion of cheating by using a pre-stored image / screen display. A pop-up window will be displayed on the user interface, prompting "The QR code may have come from an image or screen. Please scan the actual object again to verify."

[0089] If the number of valid frames is insufficient or the QR code cannot be extracted, the verification will fail and the user interface will display the message "QR code not detected, please scan the object again for verification"; if the number of valid frames N is ≥ 5, the verification will fail.

[0090] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Preventing cheating at the source: By forcing real-time video recording and prohibiting access to the local photo album, the possibility of using pre-stored QR code images or videos for deception is fundamentally eliminated.

[0091] 2. High efficiency and real-time performance: The algorithm is lightweight and combines efficient frame sampling and feature extraction strategies, enabling the entire process from video capture to authenticity determination to be completed within 5 seconds on mobile devices or in the cloud, meeting the application requirements of rapid response on mobile devices.

[0092] 3. High identification accuracy: By comprehensively utilizing the spatiotemporal dynamic characteristics of the QR code area itself (position / posture changes, surface deformation consistency), the correlation characteristics between the QR code and the surrounding background (motion consistency, depth continuity), and image anti-spoofing features for multi-dimensional fusion analysis, the accuracy of distinguishing between three-dimensional physical carriers and two-dimensional planar carriers is significantly improved.

[0093] 4. Low resource consumption: It only requires the capture of 2-3 seconds of short video (instead of the 10-30 seconds of long video required by traditional solutions), and the processing is optimized, significantly reducing data transmission volume (traffic consumption) and cloud computing overhead. At the same time, the shorter video length and processing time make it more robust in weak network environments.

[0094] 5. Wide applicability: This method can be applied to various scenarios that require the identification of the authenticity of physical objects, such as anti-counterfeiting inspection of goods, verification of the authenticity of documents, and traceability of industrial parts. Attached Figure Description

[0095] Figure 1 This is a flowchart illustrating the steps of a QR code-attached video recognition and analysis method based on object perception according to the present invention.

[0096] Figure 2 This is an overall flowchart of a QR code-attached video recognition and analysis method based on object perception according to the present invention. Detailed Implementation

[0097] To provide a further understanding of the purpose, structure, features, and functions of the present invention, detailed descriptions are provided below with reference to specific embodiments.

[0098] like Figure 1 and Figure 2 A QR code-attached video recognition and analysis method based on object perception, comprising a mobile terminal equipped with a QR code scanning and verification module, including the following steps:

[0099] S1: Real-time video acquisition and preprocessing;

[0100] S11: Startup and constraints of the QR code scanning verification module;

[0101] This method activates a QR code scanning verification module on a mobile device, restricts its access to the rear camera, and blocks access to the photo album. Users are required to use the module to photograph an object with a QR code to be recognized. This restriction, limiting access to the rear camera and blocking the photo album, precisely addresses the vulnerability in existing technologies that allow uploading pre-stored images / videos. In sales auditing scenarios, this prevents auditors from using pre-saved QR code images to pass inspections, ensuring data originates from genuine object scanning and blocking fraudulent activities at the source. The rear camera offers higher resolution and stability on mobile devices, making it more suitable for clearly capturing QR code details. The permission-blocking design requires no additional user intervention, improving usability, reducing learning costs, and enhancing operational efficiency in auditing and verification scenarios.

[0102] S12: Short video recording;

[0103] After the user triggers the recording, the app controls the camera to continuously record a short video of fixed duration (preferably 2-3 seconds). This process requires the user to keep the phone relatively stable (allowing for slight shaking from normal hand operation), and the camera to be at a suitable distance from the QR code (ensuring the QR code is clearly identifiable and occupies a certain proportion of the frame). This ensures the accuracy of subsequent frame decoding and feature analysis, and reduces invalid verifications caused by poor shooting conditions (such as hand tremors or excessive distance).

[0104] S13: Keyframe extraction and decoding;

[0105] From the recorded video, samples are taken at fixed intervals, such as 0.2 seconds, to extract valid frames that have successfully recognized the QR code (the number of valid frames N≥5; if less than 5, the verification is considered a failure). If no valid frames can be extracted, the verification is considered a failure. Then, each valid frame undergoes image preprocessing (such as noise reduction and brightness / contrast adjustment) to eliminate environmental interference and ensure the accuracy of QR code bounding boxes, corner coordinates, and other information. A fast QR code recognition library is used to locate and decode the QR code region, recording the bounding box coordinates, center point coordinates, and area of ​​the QR code in each frame. Simultaneously, the frame images are converted to grayscale for subsequent processing. Blurry or unrecognized invalid frames are filtered out to reduce the computational load of subsequent feature extraction.

[0106] Specifically, the video is loaded and all frames are extracted. For each frame, a lightweight QR code detection library (such as pyzbar's decode function) is used to quickly detect the presence of a QR code. Valid frames containing QR codes (at least 5 frames) are selected, and their timestamps are recorded. For the valid frames, image preprocessing is performed, and a fast QR code recognition library (such as zxing) is used to detect the QR code, obtaining the bounding box ((x, y, width, height)). Here, x and y are the coordinates of the top-left corner of the QR code rectangle. Width and height are the width and height of the QR code rectangle. The center point coordinates are obtained by calculating (x+w / 2, y+h / 2) using the bounding box. The area of ​​the bounding box (width×height) is calculated, and the bounding box coordinates, center point coordinates, and area of ​​the QR code in each frame are recorded.

[0107] S2: Feature extraction;

[0108] S21: Spatiotemporal feature extraction of QR code region;

[0109] S211: Position / Attitude Change Analysis;

[0110] Calculate the movement trajectory of the center point coordinates of the QR code bounding box between adjacent frames, the change in the bounding box area, and the motion vector of the adjacent frames (i.e., the displacement vector of the center point). Analyze the magnitude of the displacement and the rate of change of the area. Determine whether the QR code exhibits two-dimensional planar motion or three-dimensional motion.

[0111] Based on the bounding box coordinates, center point coordinates, and area of ​​the QR code in adjacent frames (t-1 and t), calculate the motion features:

[0112] Let the center point coordinates of the effective frame t be... The area of ​​the bounding box ;

[0113] Center point coordinates of effective frame t-1 The area of ​​the bounding box ;

[0114] 1) Obtain the displacement vector of the center point of the QR code bounding box in adjacent frames, and calculate the displacement magnitude;

[0115] Based on the displacement vector of the center point of the QR code bounding box in adjacent frames: Calculate the magnitude of the displacement (Euclidean norm). ;in This is the displacement vector of the center point of the bounding box of the QR code in the adjacent frame. For vectors The modulus. Traverse all adjacent valid frames to obtain the displacement vector and displacement magnitude of the bounding box center point of the entire frame sequence;

[0116] 2) Obtain the rate of change of the area of ​​the bounding boxes of adjacent frames;

[0117] According to the formula for the rate of change of the bounding box area: The bounding box area change rate is obtained by traversing all adjacent frames.

[0118] 3) Calculate the average displacement of the entire frame sequence;

[0119] Obtain the displacement of the center point of the bounding box for the entire frame sequence and calculate the average value;

[0120] Average displacement of the entire frame sequence: Where N is the number of valid frames, This represents the displacement of the center point of the bounding box.

[0121] 4) Calculate the maximum displacement of the entire frame sequence;

[0122] Compare the displacements of the bounding box center points throughout the entire frame sequence to obtain the maximum value;

[0123]

[0124] 5) Calculate the standard deviation of the displacement for the entire frame sequence;

[0125] Obtain the displacement magnitude of the bounding box center point of the entire frame sequence and calculate the standard deviation;

[0126] Displacement standard deviation:

[0127] 6) Calculate the average area change rate of the entire frame sequence;

[0128] Obtain the area change rate of the entire frame sequence and calculate the average value;

[0129] Average area change rate:

[0130] 7) Calculate the maximum area change rate for the entire frame sequence;

[0131] Compare the area change rate of the entire frame sequence to obtain the maximum value;

[0132] Maximum area change rate:

[0133] S212: Surface Deformation Consistency Analysis;

[0134] Calculate the cosine similarity (i.e., the cosine of the angle between the vectors) between all pairs of non-zero motion vectors, and then calculate the average cosine similarity of all inter-frame motion vectors in the entire sequence as a motion consistency index.

[0135] S212a: Obtain a non-zero motion vector;

[0136] Obtain the displacement vectors of the center points of the bounding boxes of all adjacent QR codes to obtain the set of motion vectors. Then remove vectors whose magnitude is approximately zero (e.g., ...). To avoid interference with consistency calculations, valid motion vectors are obtained, and these valid vectors are grouped into a set of non-zero motion vectors { , ,..., };

[0137] S212b: Calculate the cosine similarity between any two distinct vectors in a set of non-zero motion vectors;

[0138] Use a double loop to traverse the set of non-zero motion vectors and filter out all those that satisfy the condition. i ≠ j Valid vector pairs ( , ); Calculate the cosine similarity of all valid vector pairs;

[0139] The calculation formula is:

[0140]

[0141] S212c: Obtain motion consistency index;

[0142] Calculate the average cosine similarity of all vector pairs to obtain the motion consistency index.

[0143] Obtain the cosine values ​​of all vector pairs to form the set of motion vector cosine similarity { , ,..., };

[0144] according to Calculate and obtain the motion consistency index;

[0145] Where m is the number of valid vector pairs. The cosine similarity of the k-th motion vector in the set of motion vector cosine similarities;

[0146] Valid vector pairs , where n is the number of vectors in the set of non-zero motion vectors.

[0147] S213: Motion mode determination;

[0148] S213a: Determine whether it is a 2D motion marker;

[0149] To determine whether the movement of the QR code is two-dimensional planar motion: if it meets the following conditions: average displacement ≤ 15 pixels, displacement standard deviation < 10 pixels, average area change rate ≤ 5%, and motion consistency > 0.7, then it is determined to be two-dimensional planar motion. If these conditions are not met, then it is determined not to be two-dimensional planar motion.

[0150] The determination result is converted to 1.0 or 0.0 using a Boolean value. If the determination result is a two-dimensional planar motion, it is converted to 1.0; otherwise, it is converted to 0.0.

[0151] S213b: Determine whether it is a rigid planar marker;

[0152] Determine if the QR code is attached to a rigid plane: If the following conditions are met: average area change rate ≤ 3%, maximum area change rate ≤ 10%, and displacement standard deviation < 8 pixels, then it is determined to be attached to a rigid plane (rigid plane flag is true). If these conditions are not met, then it is determined not to be attached to a rigid plane.

[0153] The determination result is converted to 1.0 or 0.0 via a Boolean value. If the determination result is that the object is attached to a rigid plane, it is converted to 1.0; otherwise, it is converted to 0.0.

[0154] S214: Eigenvalue output;

[0155] The characteristic values ​​are summarized and output, including: average displacement, standard deviation of displacement, average area change rate, maximum area change rate, motion consistency index, 2D motion marker, and rigid plane marker.

[0156] S22: Extraction of background features around the QR code;

[0157] S221: Background motion consistency analysis;

[0158] A background region is selected outside the QR code bounding box, and the optical flow fields of the background region and the QR code region are calculated using the Farneback optical flow method. The difference (i.e., motion difference) between the average optical flow vector of the background region and the average optical flow vector of the QR code region is compared.

[0159] On three-dimensional objects, the background and the QR code area typically exhibit a coherent or related motion pattern (small motion difference); however, on two-dimensional images, the QR code area may exhibit independent motion relative to the background (large motion difference). Using the feature vector of this motion pattern ensures accurate judgment and is suitable for handling complex scenarios.

[0160] S221a: Calculate the optical flow field of the background region and the QR code region;

[0161] First, obtain the QR code region and background region: extract the grayscale images of a set of adjacent frames (t and t-1) of the middle valid frame, and crop out the rectangular region as the QR code region according to the coordinates of the QR code's bounding box; select a background region outside the QR code's bounding box (such as the right or bottom, with an area similar to the QR code region and not overlapping, and an area 0.5 to 1 times that of the QR code region), and crop it as the background region.

[0162] Next, feature points are extracted: feature points are extracted from the QR code region and background region of the valid frame t-1 respectively to obtain the feature point set of the QR code region and the feature point set of the background region of the valid frame t-1.

[0163] Feature points are extracted from the QR code region and background region of the effective frame t using feature point matching, thus obtaining the feature point set of the QR code region and the feature point set of the background region of the effective frame t.

[0164] The set of feature points of the QR code region in the effective frame t-1 is denoted as {p j} (j=1 to M, where M is the number of feature points in the QR code region) and the feature point set of the background region {p k (k=1 to N, where N is the number of feature points in the background region);

[0165] The set of feature points of the QR code region in a valid frame t is denoted as {p} j {} (j=1 to M, where M is the number of feature points in the QR code region) and the feature point set of the background region {p} k  (k=1 to N, where N is the number of feature points in the background region);

[0166] Finally, the optical flow vectors of the feature points in the QR code region and the background region are calculated separately.

[0167] Let the optical flow vector of the feature points in the QR code region be denoted as... The optical flow vector of the feature points in the background region is denoted as ;

[0168] Obtained from the difference of matching feature points in the QR code area Obtained from the difference of feature points matching the background region ;

[0169] Right now ; .

[0170] Based on feature point and optical flow field analysis, the computation is smaller and faster, without relying on high-performance hardware. The optical flow field calculation uses local feature point analysis instead of full-screen calculation, further reducing computing power consumption and achieving real-time response.

[0171] S221b: Calculate the average optical flow vector of all feature points in the QR code region and the background region respectively;

[0172] Average optical flow vector of the QR code region: ;

[0173] Average optical flow vector of background region: ;

[0174] in: Indicates the QR code area number 1 Optical flow vectors of feature points Indicates the background area. Optical flow vectors of feature points This indicates the number of feature points in the QR code area. This indicates the number of feature points in the background region.

[0175] S221c: Calculate motion difference;

[0176] The motion difference is obtained by calculating the magnitude of the difference between the average optical flow vector of the QR code region and the average optical flow vector of the background region.

[0177] according to: Obtain the degree of motion difference.

[0178] By incorporating background region motion comparison, background motion consistency analysis, through region association verification, can reduce the limitations of single features and improve the algorithm's adaptability to complex scenes. For example, some high-quality printed images (two-dimensional carriers) may simulate three-dimensional motion patterns, but the motion difference between the QR code region and the background region is higher, while the motion of the two is more coherent in three-dimensional objects, further eliminating advanced cheating behaviors.

[0179] S222: Obtain background depth cues: Use a lightweight depth estimation model to output a depth map and obtain the depth gradient of the QR code boundary;

[0180] Includes the following steps:

[0181] Step 1: Preprocessing of valid frame images;

[0182] Obtain grayscale images of valid frames (intermediate valid frames can be selected), and generate images that meet the input requirements of the lightweight MiDaS model through resizing and normalization. The normalization process includes pixel value normalization and format conversion.

[0183] Step 2: Depth map generation;

[0184] The preprocessed valid frame images are input into a lightweight MiDaS model. The MiDaS model infers and outputs a depth map D(x,y), where (x,y) represents the image pixel coordinates. .

[0185] Step 3: Extract the QR code boundaries;

[0186] Based on the obtained bounding box, the four sides of the QR code bounding box are discretized and sampled using a linear interpolation method to obtain the set of pixels Γ of the QR code boundary.

[0187] Step 4: Calculate the depth gradient;

[0188] The Sobel operator is used to calculate the gradient of the depth map D(x,y), resulting in the horizontal gradient map Gx(x,y) and the vertical gradient map Gy(x,y). The coordinates of all pixels on the QR code boundary are extracted, and the gradient magnitude of the extracted pixels is calculated. We obtain the set of boundary gradients {‖∇D(x,y)‖|(x,y)∈Γ};

[0189] Step 5: Obtain the depth gradient at the QR code boundary:

[0190] The maximum value in the set of boundary gradients is taken as the depth gradient at the boundary of the QR code.

[0191] ;

[0192] Depth map analysis based on a lightweight MiDaS model can capture the depth relationship between QR codes and the background. In 3D objects, the depths of the two are continuous (depth gradient at the boundary ≤ 0.3), while in 2D objects, depth abrupt changes occur (depth gradient at the boundary > 0.3), significantly improving the ability to identify advanced cheating techniques (such as high-quality synthetic images). Choosing the lightweight MiDaS model allows for the rapid output of coarse depth maps on mobile devices without relying on cloud computing; simultaneously, it analyzes the depth gradient at the QR code boundary rather than the entire image depth, further reducing computational load and ensuring real-time algorithm performance.

[0193] S223: Background stability analysis;

[0194] The Pearson correlation coefficient is used to calculate the linear correlation of grayscale values ​​in the background region between two adjacent frames, i.e., the correlation coefficient of the background region: ,in and The grayscale values ​​of the background regions in the previous and current frames are respectively obtained. The cropped background regions are obtained separately for the background region of the previous frame (effective frame t-1) and the background region of the current frame (effective frame t). The grayscale value of each pixel in the background regions of the previous and current frames is obtained. Based on the grayscale values, the corrcoef function is called to directly calculate the correlation coefficient of the background region. This achieves fast calculation, improves the real-time responsiveness of the algorithm, reduces waiting time, reduces processing time, and ensures that the total analysis time is less than 5 seconds.

[0195] S23: Extracting image quality and anti-spoofing features;

[0196] Print / Screen Characteristic Detection: The system selects the QR code region within the middle valid frame for analysis, examining its high-frequency noise energy (calculated via Fourier transform), moiré pattern effect (variance calculated via Laplacian operator), and edge sharpness (average gradient calculated via Sobel operator). It also provides color channel statistical characteristics (variance of each channel in the HSV space is calculated) and texture complexity (standard deviation of the grayscale image).

[0197] Selecting the middle effective frames avoids quality fluctuations in edge effective frames, ensuring the reliability of the basic detection data. The middle effective frames are in the stable stage of video recording, providing accurate and interference-free basic data for printing / screen feature detection, avoiding detection errors caused by frame quality fluctuations, ensuring feature representativeness, and avoiding random errors from a single frame.

[0198] It supplements multi-dimensional identification criteria, can quickly identify cheating behaviors such as taking pictures of QR codes on the screen with a mobile phone, makes up for the lack of spatiotemporal features and background features, responds to new cheating methods, responds to the upgrading of printed and synthesized QR code technology, and simulates three-dimensional motion or depth features for shooting scenarios.

[0199] S3: Feature fusion and classification;

[0200] S31: Extract feature vectors;

[0201] The three types of features extracted in step S2 are combined into a 12-dimensional feature vector, specifically structured as follows:

[0202] 6-dimensional features derived from S21 (spatial-temporal features of the QR code region):

[0203] Displacement normalization value: derived from average displacement Obtained through normalization: The maximum expected displacement is preset, derived from the average displacement. Divide by the preset maximum expected displacement value to obtain the displacement normalization value, and then average the displacement. The mapping is within the range [0, 1].

[0204] Average area change rate: (Direct value);

[0205] Motion consistency index: (Direct value);

[0206] 2D sports logo: (Convert Boolean value to 1.0 or 0.0);

[0207] Rigid planar markings: (Convert Boolean value to 1.0 or 0.0);

[0208] Normalized value of displacement standard deviation: derived from displacement standard deviation Obtained through normalization: The expected value of the maximum standard deviation is preset, derived from the displacement standard deviation. Divide by the preset maximum expected value of the standard deviation to map the displacement standard deviation within the range of [0, 1].

[0209] 3D features derived from S22 (background features surrounding the QR code):

[0210] Motion difference normalized value: optical flow difference between the background area and the QR code area ( The normalization process is as follows: the maximum expected value of difference is preset, the optical flow difference between the background area and the QR code area is divided by the preset maximum expected value of difference, and the optical flow difference is mapped in the range of [0, 1].

[0211] Depth gradient normalized value: Depth gradient at the QR code boundary ( Normalized to obtain: Preset maximum gradient expectation value, depth gradient at the QR code boundary ( Divide the depth gradient by the preset maximum expected value to map the depth gradient in the range of [0, 1].

[0212] Background stability: the correlation coefficient of the background region ( (direct value retrieval)

[0213] 3D features from S23 (Image Quality and Anti-Spoofing Features):

[0214] High-frequency energy: High-frequency noise energy in the QR code area (direct value);

[0215] Moiré pattern intensity: The intensity of the moiré effect in the QR code area (direct value);

[0216] Edge sharpness: Average gradient of the edges of the QR code area (direct value);

[0217] After multi-feature fusion, the algorithm can comprehensively analyze from three dimensions: motion pattern, depth relationship, and physical properties, significantly improving the accuracy of distinguishing between three-dimensional objects and two-dimensional carriers.

[0218] Normalization is a simple process requiring only basic division. Its straightforward logic minimizes computational burden and reduces computational cost, thus not increasing the time spent on feature processing. It ensures that all feature vectors are confined to the range [0, 1], preventing feature magnitude differences from dominating model judgments. This ensures fair weighting of each feature's contribution to model decisions, accelerates model training convergence, and results in more balanced gradient changes and stable weight updates. It significantly shortens model training time, facilitates rapid deployment of lightweight models, and enhances model generalization capabilities. When facing feature data from new scenarios, prediction results are more stable. For example, feature distributions may differ between product inspection and document verification scenarios; normalization allows the model to maintain consistent recognition logic in both scenarios, preventing accuracy drops due to scenario differences. It is also resistant to outlier interference, preventing model bias caused by extreme data, making it particularly suitable for complex scenarios such as product inspection and document verification.

[0219] S32: Feature vector input classification model;

[0220] The feature vector is input into a classification model, which is a pre-trained lightweight CNN classifier (3 convolutional layers, 128-dimensional input features, ReLU activation function). The output is the probability of a 3D physical object or a 2D planar object. The 3-layer convolutional structure avoids the high computational cost of deep networks, the 128-dimensional feature vector reduces data processing, and the ReLU activation function is fast, enabling classification to be completed within 5 seconds on mobile devices while ensuring classification accuracy (after prior feature optimization, the classification accuracy is close to that of deep networks).

[0221] Pre-training involves collecting samples from product anti-counterfeiting inspections and document verification, including three-dimensional physical carrier samples (QR codes on real products and physical documents, 2-3 second short videos recorded, and 12-dimensional feature vectors extracted) and two-dimensional planar carrier samples (printed QR code images, QR codes displayed on screens, also with 12-dimensional feature vectors extracted). The total number of samples is no less than 200 sets, and each set is labeled with a true category label. A lightweight CNN classifier model is pre-trained using these samples to obtain a pre-trained model. The training of the classification model is existing technology in this field and is not an inventive solution of this application; therefore, it will not be elaborated upon here.

[0222] The feature vector is input into the model for forward propagation. The input layer passes a 12-dimensional vector to the first hidden layer, resulting in a 24-dimensional hidden layer output. The output of the first hidden layer is then passed to the second hidden layer, resulting in a 12-dimensional hidden layer output, which is then passed to the output layer. Using the Softmax() function, a 2-dimensional probability vector [P1, P2] is calculated, where P1 represents the probability of a 3D physical carrier, and P2 represents the probability of a 2D planar carrier; P1 + P2 = 1. The classification logic is clear, and the results are intuitive and reliable, facilitating subsequent determination of carrier type based on probability without complex interpretation.

[0223] The classification model of this invention is lightweight and can be quickly loaded and computed on mobile devices, avoiding excessive memory and computing power consumption. The model training and deployment are also easy.

[0224] S4: Result output and anti-cheating judgment;

[0225] Receive the output of the classification model and determine whether it is a three-dimensional physical carrier or a two-dimensional planar carrier;

[0226] According to the probability vector, if P1>P2, it is determined to be a three-dimensional physical carrier; otherwise, it is determined to be a two-dimensional planar carrier.

[0227] If it is determined to be a "three-dimensional physical carrier", the verification is successful, confirming that the QR code is attached to a real object.

[0228] If the QR code is identified as a "two-dimensional planar carrier", the verification fails, indicating that there is suspicion of cheating by using a pre-stored image / screen display. A pop-up window will appear in the user interface, prompting "The QR code may have come from an image or screen. Please scan the actual object again to verify."

[0229] Furthermore, if there are insufficient valid frames or the QR code cannot be extracted, the verification will fail, and the user interface will display the message "QR code not detected. Please scan the physical object again to verify."

[0230] By providing clear feedback on the results through user interface prompts and guiding users to perform the correct operations, we can avoid user confusion caused by "verification failure without knowing the reason" and improve the user experience.

[0231] This invention overcomes the shortcomings of existing QR code inspection technologies, such as the inability to identify the physical properties of the carrier, slow response speed, and susceptibility to being deceived by pre-stored images / videos. It forces the mobile device to capture a short video (usually 2-3 seconds) containing the target QR code in real time, and quickly analyzes the dynamic visual features of the QR code area and its surrounding background in the video frame sequence (total time ≤ 5 seconds) to determine whether the QR code is attached to a three-dimensional physical surface or exists only on a two-dimensional planar medium (image or screen), thereby preventing cheating behavior using pre-stored QR code images / videos from the source.

[0232] The present invention has been described in the above-described embodiments; however, these embodiments are merely examples for implementing the present invention. It must be noted that the disclosed embodiments do not limit the scope of the present invention. Conversely, any modifications and refinements made without departing from the spirit and scope of the present invention are within the scope of patent protection of the present invention.

Claims

1. A QR code-attached video recognition and analysis method based on object perception, characterized in that: Includes a mobile terminal, on which a QR code scanning and verification module is installed; It also includes the following steps: S1: Real-time video acquisition and preprocessing; S11: Startup and constraints of the QR code scanning verification module; On the mobile terminal, start the QR code scanning verification module, set the permissions of the QR code scanning verification module, restrict it to only calling the rear camera and block the album access interface, and the user uses the QR code scanning verification module to take a picture of the object with the QR code to be recognized attached. S12: Short video recording; After the user triggers the recording, a short video of a certain duration is recorded continuously. S13: Keyframe extraction and decoding; From the recorded video, samples are taken at fixed intervals to extract valid frames that have been successfully recognized by QR codes. Each frame is preprocessed, and the QR code region is located and decoded using a fast QR code recognition library. The bounding box coordinates, center point coordinates, and area of ​​the QR code in each frame are recorded. The frame image is then converted into a grayscale image. S2: Feature extraction; S21: Spatiotemporal feature extraction of QR code region; S211: Position / attitude change analysis, obtaining average displacement, standard deviation of displacement, average area change rate, and maximum area change rate; S212: Surface deformation consistency analysis to obtain motion consistency index; S213: Motion mode determination, determining whether the QR code's motion is a 2D motion mark and whether the QR code is a rigid planar mark; S214: Feature value output; Summarize and output the feature values, including: average displacement, standard deviation of displacement, average area change rate, maximum area change rate, motion consistency index, 2D motion indicator, and rigid plane indicator. S22: Extraction of background features around the QR code; S221: Background motion consistency analysis to obtain the motion difference between the background area and the QR code area; S222: Obtain background depth cues: Use a lightweight depth estimation model to output a depth map and obtain the depth gradient of the QR code boundary; S223: Background stability analysis; S23: Extracting image quality and anti-spoofing features; The QR code region of the middle valid frame is selected for analysis, and the high-frequency noise energy, moiré effect and edge sharpness of the QR code region are analyzed. S3: Feature fusion and classification; S31: Extract feature vectors; The feature vector is a 12-dimensional vector, specifically composed of: Six-dimensional features from S21: Displacement normalization value: The average displacement is normalized and mapped to the range [0, 1]. The average area change rate is taken directly. The motion consistency index is directly taken. 2D motion mark: 1.0 or 0.0; Rigid plane marking: 1.0 or 0.0; Displacement standard deviation normalized value: mapped to the range [0, 1] by displacement standard deviation normalization; 3D features from S22: Motion difference normalization value: Motion difference normalization is mapped to the range [0, 1]; Depth gradient normalization value: The depth gradient normalization at the QR code boundary is mapped to the range [0, 1]; Background stability is directly obtained by assessing the correlation coefficient of the background region. 3D features from S23: High-frequency energy: The high-frequency noise energy of the QR code area is directly taken as the value; Moiré pattern intensity: The intensity of the moiré effect in the QR code area is directly taken as the value; Edge sharpness: The average gradient of the edge of the QR code area is directly taken as the value; S32: Feature vector input classification model; S4: Result output and anti-cheating judgment; Receive the output of the classification model and determine whether it is a three-dimensional physical carrier or a two-dimensional planar carrier.

2. The QR code-attached video recognition and analysis method based on object perception as described in claim 1, characterized in that: Step S211 is to calculate motion features based on the bounding box coordinates, center point coordinates, and area of ​​the QR code in adjacent frames. The motion features include: the displacement vector and displacement magnitude of the bounding box center point of the entire frame sequence, the average displacement, the maximum displacement, the displacement standard deviation, the bounding box area change rate, the average area change rate, and the maximum area change rate.

3. The QR code-attached video recognition and analysis method based on object perception as described in claim 2, characterized in that: Step S212 is to calculate the cosine similarity between all pairs of non-zero motion vectors, and then calculate the average cosine similarity of all inter-frame motion vectors in the entire sequence as the motion consistency index. S212a: Obtain a non-zero motion vector; Obtain the displacement vectors of the center points of the bounding boxes of all adjacent QR codes to obtain the motion vector set, and then remove the magnitude vectors. From the motion vectors, we obtain the effective motion vectors, and then form a set of non-zero motion vectors from the effective vectors. S212b: Calculate the cosine similarity between any two distinct vectors in a set of non-zero motion vectors; Select valid vector pairs and calculate the cosine similarity of all valid vector pairs; S212c: Obtain motion consistency index; Calculate the average cosine similarity of all valid vector pairs to obtain the motion consistency index.

4. The QR code-attached video recognition and analysis method based on object perception as described in claim 3, characterized in that: Also includes: S213a: Determine whether it is a 2D motion marker; Determine whether the movement of the QR code is a two-dimensional planar motion: If it meets the following conditions: average displacement ≤ 15 pixels, displacement standard deviation < 10 pixels, average area change rate ≤ 5% and motion consistency > 0.7, it is determined to be a two-dimensional planar motion; if it does not meet these conditions, it is determined not to be a two-dimensional planar motion, and the determination result is converted to 1.0 or 0.0 via Boolean value. S213b: Determine whether it is a rigid planar marker; Determine if the QR code is attached to a rigid plane: If the following conditions are met: average area change rate ≤ 3%, maximum area change rate ≤ 10%, and displacement standard deviation < 8 pixels, then it is determined to be attached to a rigid plane. If these conditions are not met, then it is determined not to be attached to a rigid plane. The determination result is converted to 1.0 or 0.0 via Boolean value.

5. The QR code-attached video recognition and analysis method based on object perception as described in claim 1, characterized in that: Step S221: Select a background region outside the QR code bounding box, use the Farneback optical flow method to calculate the optical flow fields of the background region and the QR code region, and compare the difference between the average optical flow vector of the background region and the average optical flow vector of the QR code region. S221a: Calculate the optical flow field of the background region and the QR code region; First, obtain the QR code region and background region: extract the grayscale images of a set of adjacent frames from the middle valid frame, and crop out a rectangular region as the QR code region according to the coordinates of the QR code's bounding box; select a background region outside the QR code's bounding box. Next, feature points are extracted: first, feature points of the QR code region and background region of a valid frame are extracted, and then feature point matching is used to extract feature points of the QR code region and background region of another frame. Finally, the optical flow vectors of the feature points in the QR code region and the background region are calculated separately. The optical flow vector of the QR code region is obtained by comparing the difference between the matching feature points in the QR code region, and the optical flow vector of the background region is obtained by comparing the difference between the matching feature points in the background region. S221b: Calculate the average optical flow vector of all feature points in the QR code region and the background region respectively; S221c: Calculate motion difference; The motion difference is obtained by calculating the magnitude of the difference between the average optical flow vector of the QR code region and the average optical flow vector of the background region.

6. The QR code-attached video recognition and analysis method based on object perception as described in claim 1, characterized in that: Step S222 includes: Step 1: Preprocessing of valid frame images; Obtain grayscale images of valid frames, and generate images that meet the input requirements of a lightweight MiDaS model through resizing and normalization. Step 2: Depth map generation; The preprocessed valid frame images are input into the lightweight MiDaS model, and the MiDaS model infers and outputs a depth map. Step 3: Extract the QR code boundaries; Based on the obtained bounding box, the four sides of the QR code bounding box are discretized and sampled using a linear interpolation method to obtain the set of pixels of the QR code boundary. Step 4: Calculate the depth gradient; The Sobel operator is used to calculate the gradient of the depth map, and the gradient magnitude of the extracted pixels is calculated to obtain the set of boundary gradients. Step 5: Obtain the depth gradient at the QR code boundary: The maximum value in the set of boundary gradients is obtained as the depth gradient at the boundary of the QR code.

7. The QR code-attached video recognition and analysis method based on object perception as described in claim 1, characterized in that: Step S223 includes: obtaining the grayscale value of each pixel in the background region of the previous frame and the current frame, and directly calculating the correlation coefficient based on the grayscale value using the corrcoef function to obtain the correlation coefficient of the background region.

8. The QR code-attached video recognition and analysis method based on object perception as described in claim 1, characterized in that: The classification model is a pre-trained lightweight CNN classifier with 3 convolutional layers, 128 input feature dimensions, ReLU activation function, and output probabilities of three-dimensional physical carriers and two-dimensional planar carriers.

9. The QR code-attached video recognition and analysis method based on object perception as described in claim 1, characterized in that: Step S4 is as follows: If it is determined to be a "three-dimensional physical carrier", the verification is successful, confirming that the QR code is attached to a real object; If the QR code is determined to be a "two-dimensional planar carrier", the verification will fail, indicating that there is suspicion of cheating by using a pre-stored image / screen display. A pop-up window will be displayed on the user interface, prompting "The QR code may have come from an image or screen. Please scan the actual object again to verify". If the number of valid frames is insufficient or the QR code cannot be extracted, the verification will fail and the user interface will display the message "QR code not detected, please scan the object again for verification"; if the number of valid frames N is ≥ 5, the verification will fail.

Images