A method and system for detecting unauthorized use of screen display information based on visual recognition

Abstract

This invention discloses a method and system for detecting covert filming of screen display information based on visual recognition. The method includes: collecting data on multiple preset covert filming actions to form a dataset; training a target detection model using the dataset to obtain a server-side model; fusing a behavior fusion algorithm and multi-view target localization into the server-side model to obtain a multi-view collaborative server-side model; deploying the multi-view collaborative server-side model to an edge device and converting it into a terminal model; the edge device then uses the terminal model to detect covert filming of received video stream data. This invention significantly reduces the computational load of the terminal model and effectively solves the problem of incomplete observation caused by viewpoint occlusion or unclear viewpoint images, thereby improving the accuracy and robustness of target behavior detection and recognition technology on edge devices.

Description

Technical Field

[0001] This invention relates to the field of detection technology, and in particular to a method and system for detecting unauthorized photographing of screen display information based on visual recognition. Background Technology

[0002] Vision, as one of the most important human senses, generates images, which are the most commonly used information carriers in human social activities. Image processing and computer vision technologies have long been a major area of ​​in-depth research and development in computer technology. Thanks to the booming development of deep learning technology, image processing and computer vision technologies have advanced rapidly. Among these, the detection and recognition of target object behavior is currently a hot research topic. Simultaneously, in practical production and daily life, people urgently need a low-cost and practical target detection and recognition method. Currently, mainstream target detection and recognition technologies are server-side based, meaning that both the algorithm training and recognition phases are deployed on servers. This leverages powerful computing resources to maximize the accuracy of target recognition. However, this inevitably leads to some problems, such as high server usage and maintenance costs; the need to consider communication issues between the server and terminal devices; and resource allocation issues among terminal devices. To address these issues and adapt to more complex real-world scenarios, the algorithm training and recognition execution phases of target detection and recognition technologies can be deployed on the server and terminal respectively. This two-stage deployment solution can better balance the contradiction between computing power requirements and operating costs. However, at the same time, there may be problems such as insufficient computing power of terminal devices and a single perspective on a single device, which may lead to a decrease in the accuracy of target recognition. Summary of the Invention

[0003] To address the problems of insufficient computing power of terminal devices and limited viewing angles on single devices in the original two-stage deployment technology solution, this invention provides a method and system for detecting hidden camera footage of screen display information based on visual recognition, in order to solve the above-mentioned technical problems.

[0004] This invention discloses a method for detecting unauthorized photographing of screen display information based on visual recognition, which includes the following steps:

[0005] Step 1: Collect data from multiple preset surreptitious shooting actions to form a dataset;

[0006] Step 2: Train the object detection model using the dataset to obtain the server-side model;

[0007] Step 3: Integrate the behavior fusion algorithm and multi-view target localization into the server-side model to obtain the server-side model after multi-view collaboration;

[0008] Step 4: Deploy the server-side model after multi-view collaboration to the edge device, and convert the server-side model after multi-view collaboration into a terminal model. The edge device uses the terminal model to perform hidden camera detection on the received video stream data.

[0009] Furthermore, the preset multiple surreptitious shooting actions include surreptitious shooting with a mobile phone at close range, surreptitious shooting with a mobile phone at normal distance, and surreptitious shooting with the front camera of a mobile phone.

[0010] Further, step 2 includes:

[0011] Step 21: Randomly select 80% of the dataset each time as the training set;

[0012] Step 22: Input the preprocessed training set into the object detection model;

[0013] Step 23: The object detection model extracts features from the preprocessed training set.

[0014] Step 24: The extracted features are fed into the prediction layer through the FPN+PAN feature extraction pyramid. The prediction layer then infers the bounding box and action category information. Each surreptitious action is treated as a category of data.

[0015] Step 25: Extract 10% of the dataset for model validation, adjust the model hyperparameters, and repeat steps 21 to 25 until the optimal model is obtained; wherein, the optimal model is the server-side model.

[0016] Further, step 3 includes:

[0017] Step 31: Use the optimal model and re-identification technology to identify targets detected by different cameras, and calibrate the perspectives between each camera to determine specific parameters of each perspective and obtain the relationship between the perspectives of different cameras; wherein, the specific parameters include angle, height and distance; when the distance parameter is fixed, all perspectives form a perspective sphere, and each point on the perspective sphere corresponds to a specific perspective of the target; the relationship between different perspectives is determined by the positional relationship of the points on the perspective sphere.

[0018] Step 32: By using positional constraints between viewpoints, the relationship between classifiers corresponding to the viewpoints is indirectly characterized to obtain multiple behavioral representations of the same target under different viewpoints, thereby effectively fusing information from classifiers of different viewpoints to obtain a server-side model after multi-view collaboration.

[0019] Further, step 4 includes:

[0020] Step 41: Convert the server-side model after multi-view collaboration into an intermediate ONNX model;

[0021] Step 42: Import the intermediate ONNX model into the RKNN development toolkit to convert the server-side model after multi-view collaboration into an RKNN model, i.e., the terminal model;

[0022] Step 43: The development board receives video stream data from multiple perspectives and performs model inference on the target behavior in the video stream data to determine whether it is a covert recording.

[0023] Furthermore, it also includes:

[0024] The performance of the terminal model was evaluated using 10% of the dataset.

[0025] This invention also discloses a screen display information covert photography detection system based on visual recognition, which includes:

[0026] The data acquisition module is used to collect data from multiple preset covert shooting actions and form a dataset;

[0027] The target training module is used to train the target detection model using the dataset to obtain the server-side model.

[0028] The multi-view collaboration module is used to integrate the behavior fusion algorithm and multi-view target localization into the server-side model to obtain the server-side model after multi-view collaboration.

[0029] The detection module is used to deploy the server-side model after multi-view collaboration to the edge device and convert the server-side model after multi-view collaboration into a terminal model. The edge device uses the terminal model to perform hidden camera detection on the received video stream data.

[0030] Furthermore, the target training module includes:

[0031] An extraction unit is used to randomly extract 80% of the dataset each time as the training set;

[0032] The input unit is used to input the preprocessed training set into the target detection model;

[0033] The feature extraction unit is used by the target detection model to extract features from the preprocessed training set.

[0034] The inference unit extracts features that are fed into the prediction layer via the FPN+PAN feature extraction pyramid. The prediction layer then infers the bounding box and action category information. Each surreptitious action is treated as a category of data.

[0035] The repeated training unit is used to extract 10% of the dataset for model validation, adjust the model hyperparameters, and repeatedly enter the extraction unit, input unit, feature extraction unit, inference unit, and repeated training unit until the optimal model is obtained; wherein, the optimal model is the server-side model.

[0036] Furthermore, the multi-view collaboration module includes:

[0037] The determination unit is used to identify targets detected by different cameras by means of an optimal model and re-identification technology, and to determine specific parameters of each viewpoint by calibrating the viewpoints between the cameras, thereby obtaining the relationship between the viewpoints of different cameras; wherein, the specific parameters include angle, height and distance; when the distance parameter is fixed, all viewpoints form a viewpoint sphere, and each point on the viewpoint sphere corresponds to a specific viewpoint of the target; the relationship between different viewpoints is determined by the positional relationship of the points on the viewpoint sphere.

[0038] The acquisition unit is used to indirectly characterize the relationship between classifiers corresponding to different viewpoints through positional constraints between viewpoints, so as to obtain multiple behavioral representations of the same target under different viewpoints, thereby effectively fusing information from classifiers of different viewpoints to obtain a server-side model after multi-view collaboration.

[0039] Furthermore, the detection module includes:

[0040] The first conversion unit is used to convert the server-side model after multi-view collaboration into an intermediate ONNX model;

[0041] The second conversion unit is used to import the intermediate ONNX model into the RKNN development toolkit and convert the server-side model after multi-view collaboration into an RKNN model, i.e., the terminal model.

[0042] The first conversion development board receives video stream data from multiple perspectives and performs model inference on the target behavior in the video stream data to determine whether it is a covert recording.

[0043] Due to the adoption of the above technical solution, the present invention has the following advantages: the present invention greatly reduces the computational load of the terminal model, and at the same time effectively solves the problem of incomplete observation caused by viewpoint occlusion or unclear viewpoint image, thereby improving the accuracy and robustness of target behavior detection and recognition technology on edge devices. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in the embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0045] Figure 1 This is a schematic diagram illustrating a scenario where the target captured by some cameras is obstructed, according to an embodiment of the present invention.

[0046] Figure 2 This is a flowchart illustrating a method for detecting unauthorized photographing of screen display information based on visual recognition, according to an embodiment of the present invention.

[0047] Figure 3 This is a schematic diagram illustrating information fusion from different camera perspectives according to an embodiment of the present invention;

[0048] Figure 4 This is a schematic diagram illustrating the construction process of the simulation verification platform on the edge device according to an embodiment of the present invention. Detailed Implementation

[0049] The present invention will be further described in conjunction with the accompanying drawings and embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art should fall within the protection scope of the present invention.

[0050] This invention provides an embodiment of a method for detecting unauthorized filming of screen display information based on visual recognition. First, the algorithm is implemented using the PyTorch machine learning framework, building a target behavior analysis framework based on image recognition. Multi-view sample data is used for collaborative learning and model training. During deployment, the model is converted into an RKNN model using the RKNN toolkit provided by Rockchip through model lightweighting techniques. This model is then embedded into an AI processing development board, connected to a multi-view high-definition device. The development board receives video stream data, analyzes the target behavior, and returns results in real time, ultimately completing the performance evaluation. For details, see [link to relevant documentation]. Figure 2 This embodiment includes the following steps:

[0051] S1: Collect data from multiple preset surreptitious shooting actions to form a dataset;

[0052] S2: Train the object detection model using the dataset to obtain the server-side model;

[0053] S3: Integrate the behavior fusion algorithm and multi-view target localization into the server-side model to obtain a server-side model after multi-view collaboration; this eliminates the problem of incomplete observation caused by view occlusion or unclear view images and improves the robustness of the server-side model.

[0054] S4: Deploy the server-side model, after multi-view collaboration, to the edge device, and convert the server-side model into a terminal model. The edge device then uses the terminal model to perform surreptitious filming detection on the received video stream data. See details. Figure 4 .

[0055] In this embodiment, the preset multiple surreptitious shooting actions include surreptitious shooting with a mobile phone at close range, surreptitious shooting with a mobile phone at normal distance, and surreptitious shooting with the front camera of a mobile phone.

[0056] In this embodiment, S2 includes:

[0057] S21: Randomly select 80% of the dataset each time as the training set;

[0058] S22: Input the preprocessed training set into the object detection model;

[0059] Specifically, the object detection model can be the YOLOv5 object detection network.

[0060] S23: The object detection model extracts features from the preprocessed training set as input;

[0061] Specifically, preprocessing includes Mosaic data augmentation, adaptive anchor box calculation, and adaptive image scaling. The baseline network for the YOLOv5 object detection network is mainly composed of CBL, CSPNet, Focus, and SPP modules. The preprocessed training set is input into the baseline network for feature extraction.

[0062] S24: The extracted features are fed into the prediction layer through the FPN+PAN feature extraction pyramid. The prediction layer then infers the bounding box and action category information. Each surreptitious action is treated as a category of data.

[0063] S25: Extract 10% of the dataset for model validation, adjust the model hyperparameters, and repeat S21 to S25 until the optimal model is obtained; the optimal model is the server-side model.

[0064] In real-world scenarios, multiple cameras at different locations are often needed to cover the complete motion area of ​​a target, and these cameras typically have different perspectives. However, existing methods cannot perform collaborative learning across multiple perspectives. Figure 1As shown, the ABG camera can observe the target behavior normally, but the CDF camera cannot observe the target due to viewpoint occlusion. By jointly learning from multiple viewpoints, the incompleteness of observation caused by viewpoint occlusion can be effectively eliminated, improving the robustness of the algorithm model for target behavior analysis. Multi-view scenario algorithmic collaborative analysis technology can collaboratively represent and analyze the behavioral characteristics of the same target from different viewpoints, obtaining more comprehensive analysis results compared to behavioral feature analysis from a single viewpoint. In this embodiment of the invention, considering the need to use multiple surveillance cameras to eliminate the influence of blind spots in behavior detection, the method proposed in this embodiment is adopted. For example, S3 includes:

[0065] S31: Using the optimal model and re-identification technology, targets detected by different cameras are identified, and the specific parameters of each viewpoint are determined by calibrating the viewpoints between the cameras to obtain the relationship between the viewpoints of different cameras; the specific parameters include angle, height and distance; when the distance parameter is fixed, all viewpoints form a viewpoint sphere, and each point on the viewpoint sphere corresponds to a specific viewpoint of the target; the relationship between different viewpoints is determined by the positional relationship of the points on the viewpoint sphere.

[0066] S32: By using positional constraints between viewpoints, the relationship between classifiers corresponding to those viewpoints is indirectly characterized, thereby obtaining multiple behavioral representations of the same target from different viewpoints. This effectively fuses information from classifiers of different viewpoints, such as... Figure 3 As shown, the server-side model after multi-view collaboration is obtained.

[0067] Collaborative target behavior detection from multiple perspectives can compensate for incomplete image information or inconsistent data quality obtained from certain perspectives. Furthermore, by fusing information from different cameras at different perspectives, the system's behavior can be identified, thus improving its accuracy.

[0068] In this embodiment, S4 includes:

[0069] S41: Convert the server-side model after multi-view collaboration into an intermediate ONNX model;

[0070] S42: Import the intermediate ONNX model into the RKNN development toolkit to convert the server-side model after multi-view collaboration into an RKNN model, i.e., the terminal model.

[0071] S43: The development board receives video stream data from multiple perspectives and performs model inference on the target behavior in the video stream data to determine whether it is a covert recording.

[0072] This embodiment also includes:

[0073] The performance of the terminal model was evaluated using 10% of the dataset.

[0074] The present invention also provides an embodiment of a screen display information covert photography detection system based on visual recognition, which includes:

[0075] The data acquisition module is used to collect data from multiple preset covert shooting actions and form a dataset;

[0076] The target training module is used to train the target detection model using the dataset to obtain the server-side model;

[0077] The multi-view collaboration module is used to integrate the behavior fusion algorithm and multi-view target localization into the server-side model to obtain the server-side model after multi-view collaboration.

[0078] The detection module is used to deploy the server-side model after multi-view collaboration to the edge device and convert the server-side model after multi-view collaboration into a terminal model. The edge device uses the terminal model to perform hidden camera detection on the received video stream data.

[0079] In this embodiment, the target training module includes:

[0080] Extraction unit, used to randomly select 80% of the dataset each time as the training set;

[0081] The input unit is used to input the preprocessed training set into the object detection model;

[0082] The feature extraction unit is used by the target detection model to extract features from the preprocessed training set.

[0083] The inference unit extracts features that are fed into the prediction layer via the FPN+PAN feature extraction pyramid. The prediction layer then infers the bounding box and action category information. Each surreptitious action is treated as a category of data.

[0084] The repeated training unit is used to extract 10% of the dataset for model validation, adjust the model hyperparameters, and repeatedly enter the extraction unit, input unit, feature extraction unit, inference unit, and repeated training unit until the optimal model is obtained; the optimal model is the server-side model.

[0085] In this embodiment, the multi-view collaboration module includes:

[0086] The determination unit is used to identify targets detected by different cameras by using the optimal model and re-identification technology, and to determine specific parameters of each viewpoint by calibrating the viewpoints between the cameras, thereby obtaining the relationship between the viewpoints of different cameras; the specific parameters include angle, height and distance; when the distance parameter is fixed, all viewpoints form a viewpoint sphere, and each point on the viewpoint sphere corresponds to a specific viewpoint of the target; the relationship between different viewpoints is determined by the positional relationship of the points on the viewpoint sphere.

[0087] The acquisition unit is used to indirectly characterize the relationship between classifiers corresponding to different viewpoints through positional constraints between viewpoints, so as to obtain multiple behavioral representations of the same target under different viewpoints, thereby effectively fusing information from classifiers of different viewpoints to obtain a server-side model after multi-view collaboration.

[0088] In this embodiment, the detection module includes:

[0089] The first conversion unit is used to convert the server-side model after multi-view collaboration into an intermediate ONNX model;

[0090] The second conversion unit is used to import the intermediate ONNX model into the RKNN development toolkit and convert the server-side model after multi-view collaboration into an RKNN model, i.e., the terminal model.

[0091] The first conversion development board receives video stream data from multiple perspectives and performs model inference on the target behavior in the video stream data to determine whether it is a covert recording.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A visual recognition-based detection method for screen information stealthy shooting, characterized in that, The method comprises the following steps: Step 1: data collection of preset multiple sneaking actions to form a data set; Step 2: training a target detection model through the data set to obtain a server-side model; Step 3: fusing a behavior fusion algorithm and multi-view target positioning into the server-side model to obtain a multi-view collaborative server-side model; Step 4: deploying the multi-view collaborative server-side model to an edge device and converting the multi-view collaborative server-side model into a terminal model, and the edge device performing sneaking detection on received video stream data through the terminal model; The step 2 comprises: Step 21: randomly extracting a first preset proportion from the data set as a training set each time; Step 22: inputting the preprocessed training set into a target detection model; Step 23: the target detection model performing feature extraction on the input preprocessed training set; Step 24: the extracted features being input into a prediction layer through an FPN+PAN feature extraction pyramid to obtain a prediction frame and action category information through inference of the prediction layer; wherein each sneaking action is regarded as a category of data; Step 25: extracting a second preset proportion of the data set for model verification, adjusting model hyperparameters, and repeating steps 21 to 25 until an optimal model is obtained; wherein the optimal model is a server-side model; and the second preset proportion is less than the first preset proportion; The step 3 comprises: Step 31: identifying targets detected by different cameras through the optimal model and a re-identification technology, and determining specific parameters of each view angle by calibrating the view angles between the cameras to obtain the relationship between different camera view angles; wherein the specific parameters include angle, height and distance; when the distance parameter is fixed, all view angles form a view angle sphere, and each point on the view angle sphere corresponds to a specific view angle of a target; the relationship between different view angles is determined by the positional relationship of the points on the view angle sphere; Step 32: indirectly depicting the relationship between the classifiers corresponding to the view angles through the positional constraints between the view angles to obtain multiple behavior representations of the same target under different view angles, thereby effectively fusing information from different view angle classifiers to obtain a multi-view collaborative server-side model.

2. The method of claim 1, wherein, The preset multiple sneaking actions include sneaking with a mobile phone at close range, sneaking with a mobile phone at a normal distance, and sneaking with a mobile phone front camera.

3. The method of claim 1, wherein, The step 4 comprises: Step 41: converting the multi-view collaborative server-side model into an intermediate ONNX model; Step 42: importing the intermediate ONNX model into an RKNN development kit to convert the multi-view collaborative server-side model into an RKNN model, i.e., a terminal model; Step 43: a development board receiving video stream data under multiple view angles and performing model inference on target behaviors in the video stream data to determine whether it is sneaking.

4. The method of claim 3, wherein, Further comprising: Performance evaluation of the terminal model using the data set of the second preset proportion.

5. A system for detecting unauthorized photographing of screen information based on visual recognition, characterized in that, Comprise: A data collection module for collecting data of preset multiple sneaking actions to form a data set; The target training module is configured to train a target detection model through the data set to obtain a server-side model; The multi-view cooperation module is configured to fuse a behavior fusion algorithm and multi-view target positioning into the server-side model to obtain a multi-view cooperated server-side model; The detection module is configured to deploy the multi-view cooperated server-side model to an edge device and convert the multi-view cooperated server-side model into a terminal model, so that the edge device performs pickpocketing detection on received video stream data through the terminal model. The target training module comprises: The extraction unit is configured to randomly extract a first preset proportion of the data set as a training set each time; The input unit is configured to input the preprocessed training set into a target detection model; The feature extraction unit is configured to perform feature extraction on the input preprocessed training set by the target detection model; The inference unit is configured to input the extracted features to a prediction layer through an FPN+PAN feature extraction pyramid, and obtain a prediction box and action category information through the prediction layer inference; wherein each pickpocketing action is regarded as a category of data; The repeated training unit is configured to extract a second preset proportion of the data set for model verification, adjust model hyperparameters, and repeatedly enter the extraction unit, the input unit, the feature extraction unit, the inference unit, and the repeated training unit until an optimal model is obtained; wherein the optimal model is a server-side model; and the second preset proportion is less than the first preset proportion; The multi-view cooperation module comprises: The determination unit is configured to identify targets detected by different cameras by means of the optimal model and re-identification technology, and determine specific parameters of each view angle by calibrating the view angles between the cameras to obtain the relationship between different camera view angles; wherein the specific parameters include angle, height, and distance; when the distance parameter is fixed, all view angles form a view angle sphere, and each point on the view angle sphere corresponds to a certain specific view angle of the target; the relationship between different view angles is determined by the positional relationship of the points on the view angle sphere; The acquisition unit is configured to indirectly depict the relationship between the classifiers corresponding to the view angles through the positional constraints between the view angles to acquire multiple behavior representations of the same target under different view angles, so as to effectively fuse information from different view angle classifiers to obtain a multi-view cooperated server-side model.

6. The system of claim 5, wherein, The detection module comprises: The first conversion unit is configured to convert the multi-view cooperated server-side model into an intermediate state ONNX model; The second conversion unit is configured to import the intermediate state ONNX model into an RKNN development kit, and convert the multi-view cooperated server-side model into an RKNN model, i.e., a terminal model; The first conversion development board is configured to receive video stream data under multiple view angles and perform model inference on target behaviors in the video stream data to determine whether it is pickpocketing.

Images