Screen candid shooting prevention monitoring method and system based on multi-modal model fusion

Abstract

The invention discloses a screen candid shooting prevention monitoring method and system based on multi-mode model fusion, and relates to screen privacy protection.The method comprises the steps that firstly, multi-source data from a video mode, a wireless positioning mode, an environment light intensity mode, an audio mode and a peripheral behavior mode are acquired through a data acquisition module; and the data enhancement module is used for enhancing the source data. Then, feature extraction is carried out on the enhanced multi-modal data, and corresponding multi-modal feature representation is obtained; and inhibiting inter-modal redundant information through an adaptive feature enhancement module, highlighting significant features related to candid photographing behaviors, and generating optimized fusion features. And on the basis of the optimized fusion features, a multi-task learning architecture module is adopted to simultaneously predict a device category, a user behavior category and whether a secret photographing behavior exists or not. According to the method, multi-modal data can be fully fused, the accuracy and robustness of screen candid shooting prevention detection are improved, and the method is suitable for real-time security monitoring of multiple scenes and multiple types of terminals.

Description

TECHNICAL FIELD

[0001] The present specification relates to screen privacy protection, more particularly, the present application relates to a screen anti-stealing monitoring method and system based on multi-modal model fusion. BACKGROUND

[0002] With the popularity of mobile terminals and miniaturized camera equipment, the behavior of stealing by using mobile phones, miniature cameras and other equipment is increasingly covert and diversified, which has brought serious threats to conference scenes, office environments and personal privacy protection. The existing anti-stealing technology mostly relies on single modal data for recognition, such as detecting suspicious actions through video monitoring or judging the distance between the user and the screen through wireless positioning.

[0003] Such single modal methods often have the following shortcomings when facing complex environments: first, the information of different modalities has limitations, for example, only video may be affected by light, angle and obstruction; second, the complementary relationship between modalities is not utilized, which easily leads to false positives or false negatives; third, the generalization ability of the model is insufficient in different devices and different scenes, and often needs to be retrained for specific scenes; fourth, after recognizing the risk, there is a lack of effective real-time protection measures, which cannot dynamically desensitize the screen content in time.

[0004] Therefore, it is urgent to propose a screen anti-stealing monitoring method and system based on multi-modal model fusion to at least solve some of the above problems. SUMMARY

[0005] A series of simplified concepts are introduced in the summary section, which will be further described in detail in the specific embodiment section. The summary section of the present application does not mean to attempt to limit the key features and necessary technical features of the claimed technical solutions, nor to attempt to determine the protection scope of the claimed technical solutions.

[0006] In a first aspect, the present application proposes a screen anti-stealing monitoring method and system based on multi-modal model fusion, the method comprising:

[0007] obtaining a data set from different acquisition devices and different acquisition scenes through a data acquisition module, wherein the data set includes video modal data, wireless positioning modal data, ambient light intensity modal data, audio modal data and peripheral behavior modal data;

[0008] performing data enhancement on the source data of the above data set through a data enhancement module to obtain enhanced data;

[0009] inputting the above enhanced data into a MixNet module for feature extraction to obtain corresponding multi-modal feature representation;

[0010] The multi-modal feature representation is encoded by the adaptive feature enhancement module to suppress inter-modal redundant information and highlight significant features related to the behavior of taking pictures, so as to obtain an optimized fusion feature.

[0011] Based on the optimized fusion feature, a multi-task learning architecture module is used to simultaneously predict the device category, the user behavior category and whether there is the behavior of taking pictures.

[0012] In a feasible implementation, the data acquisition module includes a RotMix sub-module and a RotCom sub-module.

[0013] The data set from different acquisition devices and different acquisition scenes is obtained by the data acquisition module, including:

[0014] The Rot Mix sub-module performs rotation disturbance and feature mixing within the single modal of the source data of the data set to generate intra-class diversified samples.

[0015] The RotCom sub-module combines modalities between different samples of the source data of the data set and generates cross-modal enhanced samples based on consistency constraints.

[0016] The intra-class diversified samples and the cross-modal enhanced samples are added to the data set to obtain enhanced data.

[0017] In a feasible implementation, during the training of the behavior of taking pictures recognition model composed of the data acquisition module, the data enhancement module, the MixNet module, the adaptive feature enhancement module and the multi-task learning architecture module, the method further includes:

[0018] A gradient reversal layer is introduced between the feature extractor and the domain discriminator, and the source domain feature and the target domain feature are made indistinguishable in the representation space through adversarial training.

[0019] The covariance matrix difference of the source domain feature and the target domain feature is calculated, and optimization is performed based on a correlation alignment loss function to reduce the second-order statistical distribution difference between the two;

[0020] The mean and variance parameters of the source domain and the target domain are maintained respectively in the batch normalization process to realize adaptive adjustment of the feature distribution under different devices and different scenes.

[0021] In a feasible implementation, the adaptive feature enhancement module includes a modal-by-modal encoding sub-module, a redundancy suppression sub-module, a gated attention sub-module and a fusion sub-module,

[0022] The above multi-modal feature representation is encoded by the adaptive feature enhancement module, the inter-modal redundant information is suppressed, and the significant features related to the paparazzi behavior are highlighted to obtain the optimized fusion feature, including:

[0023] Each modal feature is mapped into a feature vector of a unified dimension by the above modal-by-modal encoding submodule;

[0024] The orthogonal constraint and mutual information minimization constraint are applied between the feature vectors of different modalities by the above redundancy suppression submodule to reduce the inter-modal redundant information;

[0025] The feature vectors of each modality are assigned weight coefficients by the above gated attention submodule, the features related to the paparazzi behavior are weighted and the irrelevant features are weakened;

[0026] The weighted modal features are summed by the fusion submodule to generate the above optimized fusion feature.

[0027] In a feasible implementation, the above redundancy suppression submodule applies orthogonal constraint and mutual information minimization constraint between the feature vectors of different modalities to reduce the inter-modal redundant information, including:

[0028] Orthogonal constraint is applied between the feature vectors of different modalities, and the cosine similarity square term is minimized to promote the feature vectors to be approximately orthogonal in the representation space, so as to improve the complementarity between modalities;

[0029] Mutual information minimization constraint is applied between the feature vectors of different modalities, and the inter-modal mutual information is calculated based on the Hilbert-Schmidt independence criterion and minimized to reduce the inter-modal redundant information;

[0030] The orthogonal constraint and the mutual information minimization constraint are combined and weighted, and the weight coefficient is dynamically adjusted during the training process, so as to reduce the redundancy while maintaining the discriminative ability for paparazzi behavior recognition.

[0031] In a feasible implementation, the multi-task learning architecture module includes a bottom shared feature layer, a first task submodule, a second task submodule, and a third task submodule;

[0032] Based on the above optimized fusion feature, the multi-task learning architecture module is used to simultaneously predict the device category, the user behavior category, and whether there is a paparazzi behavior, including:

[0033] The above optimized fusion feature is received by the bottom shared feature layer and a shared representation is generated;

[0034] Whether there is a paparazzi behavior is predicted by the first task submodule based on the shared representation;

[0035] predicting the device category based on the shared representation by the second task sub-module;

[0036] predicting the user behavior category based on the shared representation by the third task sub-module; wherein the optimization objective of the multi-task learning architecture is a joint loss function, and the joint loss function is a weighted combination of the Focal Loss, the cross-entropy loss and the triplet loss.

[0037] In an available implementation, the first task sub-module is configured to obtain a pickpocketing behavior discrimination score by fusing a fragment-level full connection network and a time-series-level one-dimensional convolution in parallel based on the shared representation, and the discrimination score is mapped to a probability by Sigmoid and optimized by Focal Loss with an unbalanced adjustment factor.

[0038] In an available implementation, the second task sub-module is configured to perform hierarchical prediction on the device category, the second task sub-module first judges the device category and then judges the fine-grained model, and introduces a label smoothing and a class balance term weighted by the number of effective samples in the cross-entropy loss, and adds a distillation loss based on the soft label of the teacher model.

[0039] In an available implementation, the third task sub-module is configured to perform time-series prediction on the user behavior category, generate frame-level logits by using a bidirectional recurrent unit, and perform sequence decoding by using a conditional random field, and introduce a triplet loss in the behavior embedding space to enhance the fine-grained distinguishing ability.

[0040] In a second aspect, the present application provides a screen pickpocketing monitoring system based on multi-modal model fusion, comprising:

[0041] a data acquisition module configured to acquire a data set from different acquisition devices and different acquisition scenes, wherein the data set comprises video modal data, wireless positioning modal data, ambient light intensity modal data, audio modal data and peripheral device behavior modal data;

[0042] a data enhancement module configured to perform data enhancement on source data of the data set to obtain enhanced data;

[0043] a MixNet module configured to perform feature extraction on the enhanced data to obtain corresponding multi-modal feature representation;

[0044] an adaptive feature enhancement module configured to perform modal-by-modal encoding on the multi-modal feature representation, suppresses redundant information between modalities and highlights significant features related to pickpocketing behavior to obtain optimized fusion features;

[0045] A multi-task learning architecture module is configured to simultaneously predict the device category, the user behavior category, and whether the behavior is a surreptitious shooting based on the optimized fused features.

[0046] In summary, the present application proposes a screen anti-spying monitoring method based on multi-modal model fusion, which has the following beneficial effects: First, the present embodiment integrates video, wireless positioning, ambient light intensity, audio, and peripheral behavior and other multi-source information through the data acquisition module to construct a more comprehensive input data set, which can obtain multi-dimensional features related to user behavior and device use in different environments, avoiding the limitations brought by relying on a single modality. Second, the RotMix and RotCom two types of enhancement strategies are introduced in the data enhancement stage, which not only increases the intra-class diversity of samples within a single modality, but also generates new combined samples at the level of cross-modality and cross-sample, thereby improving the robustness and generalization ability of the model under different scenarios and different device conditions. Third, the present embodiment sets up an adaptive feature enhancement module in the feature processing process to perform modal-by-modal encoding, redundancy suppression, and attention weighting on multi-modal features, which can highlight the significant features highly related to the behavior of spying and suppress irrelevant or repetitive information between modalities, thereby improving the discriminability of feature fusion. Further, the device category, user behavior category, and whether the behavior is a surreptitious shooting are simultaneously predicted through multi-task learning architecture, which realizes multi-target recognition in a unified framework, improves the accuracy of the behavior of spying, and provides more rich auxiliary information for subsequent protection strategies. The present embodiment also introduces an unbalanced adjustment loss and a fine-grained distinction constraint in the training, which further enhances the adaptability to actual complex scenarios. Finally, the present application not only gives the recognition result and confidence of the behavior of spying in the inference stage, but also performs the same style camouflage desensitization processing on the suspicious attention area of the screen according to the result, which can implement protection in real time when the risk is found, avoid sensitive information leakage, and form a complete closed loop from detection to protection. In summary, the present embodiment significantly improves the accuracy, robustness, and practicality of screen anti-spying monitoring through a series of innovative designs such as multi-modal data fusion, data enhancement, feature optimization, and multi-task recognition, and solves the problems of relying on a single modality, insufficient generalization, and lack of real-time protection in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0047] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not intended to limit the scope of the present application. Moreover, the same reference numerals are used throughout the same figures. In the drawings: Figure 1 A flowchart of a screen anti-spying monitoring method based on multi-modal model fusion provided by the present application; Figure 2A flowchart of a data acquisition module collecting a data set provided by an embodiment of the present application is shown in the figure; Figure 3 A flowchart of a recognition model training provided by an embodiment of the present application is shown in the figure; Figure 4 A flowchart of a self-adaptive feature enhancement module obtaining optimized fusion features provided by an embodiment of the present application is shown in the figure; Figure 5 A flowchart of a redundancy suppression sub-module reducing inter-modal redundancy information provided by an embodiment of the present application is shown in the figure; Figure 6 A flowchart of a multi-task learning architecture module predicting a process provided by an embodiment of the present application is shown in the figure; Figure 7 A structural diagram of a screen anti-stealing monitoring system based on multi-modal model fusion provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0054] The terms "first", "second", "third", "fourth" and the like in the technical solutions of the present application and the above figures (if any) are used to distinguish similar objects and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device. The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments.

[0055] Figure 1 A flowchart of a screen anti-stealing monitoring method based on multi-modal model fusion provided by an embodiment of the present application is shown in the figure. The method can specifically include:

[0056] S110, acquiring data sets from different acquisition devices and different acquisition scenes through a data acquisition module, wherein the data sets include video modal data, wireless positioning modal data, ambient light intensity modal data, audio modal data and peripheral behavior modal data;

[0057] S120, data augmentation is performed on the source data of the data set by a data augmentation module to obtain augmented data;

[0058] S130, the augmented data is input into a MixNet module for feature extraction to obtain corresponding multi-modal feature representations;

[0059] S140, the multi-modal feature representations are encoded by an adaptive feature enhancement module to suppress inter-modal redundant information and highlight significant features related to the behavior of taking pictures, so as to obtain optimized fusion features;

[0060] S150, based on the optimized fusion features, a multi-task learning architecture module is used to simultaneously predict the device category, the user behavior category and whether there is a behavior of taking pictures.

[0061] For example, in step S110, the system obtains multi-modal data from different collection devices and different collection scenes through a data collection module. Specifically, the video modality is collected by a high-speed camera at a fixed frame rate to obtain a sequence of pictures, the wireless positioning modality is collected by a UWB or Bluetooth base station to obtain distance information between the user and the screen, the ambient light intensity modality is collected by a light sensor to obtain ambient brightness changes, the audio modality is collected by a microphone to obtain environmental sound features, and the peripheral behavior modality records the interaction events of the user operating the keyboard, mouse or touch screen. The data of the above different modalities are all provided with a unified time stamp, and after time alignment processing, they are divided into several time segments, each of which is used as a training or identification sample.

[0062] In step S120, the sample data enters the data augmentation module. The module includes a RotMix sub-module and a RotCom sub-module. The RotMix sub-module introduces diversification disturbance within a single modality, such as rotating and brightness disturbance of video frames, time axis translation or frequency band shielding of audio signals, so as to generate samples with differentiation within the same class. The RotCom sub-module combines multi-modal features between different samples, cross-pairs the environment-independent modalities with the modalities directly related to user behavior, and applies consistency constraints, so that the generated combined samples maintain the behavior semantics and can improve the generalization ability across scenes. After processing by the two sub-modules, the system obtains augmented multi-modal training samples.

[0063] In step S130, the augmented multi-modal data is input into the MixNet module for feature extraction. The MixNet module extracts local detail features and global semantic features through different scale convolution operations, so as to capture spatial changes of actions in the video modality, extract rhythm and frequency information in the audio modality, extract trend changes in the positioning and light intensity modalities, and extract rhythm patterns of user operations in the peripheral device modality. After unified processing of each modality, corresponding feature representations are generated.

[0064] In step S140, the feature representation is further input into an adaptive feature enhancement module for processing. The module first performs per-modality encoding on each modality feature, mapping it to a unified representation space. Then, a redundancy suppression unit is used to reduce the overlap and interference between different modalities, making the inter-modality information more complementary. Next, a gated attention unit gives higher weights to features with high relevance to the behavior of taking pictures, and lower weights to noise or irrelevant features. Finally, a fusion unit integrates the weighted features to obtain an optimized fused feature representation.

[0065] In step S150, based on the optimized fused features, the system uses a multi-task learning architecture module for recognition. The architecture includes a shared feature layer and multiple task sub-modules: the first task sub-module is used to determine whether there is a behavior of taking pictures in the current segment, and adjusts the loss function to cope with the class imbalance problem; the second task sub-module is used to predict the category of the device taking pictures, such as distinguishing between different types of mobile phones, cameras, etc.; the third task sub-module is used to identify the specific behavior category of the user, such as raising hands, staring at the screen, or operating peripherals, etc. The above multiple sub-tasks are jointly optimized under the same architecture, where the task of recognizing the behavior of taking pictures is given a higher weight to ensure that the accuracy of this task is prioritized.

[0066] The application provides a screen anti-stealing photograph monitoring method based on multi-modal model fusion, which has the following beneficial effects: first, the embodiment integrates video, wireless positioning, environmental light intensity, audio and peripheral behavior and other multi-source information through a data acquisition module to construct a more comprehensive input data set, which can obtain multi-dimensional features related to user behavior and device use in different environments, avoiding the limitations brought by relying on a single mode. Second, RotMix and RotCom two types of enhancement strategies are introduced in the data enhancement stage, not only increasing the intra-class diversity of samples within a single mode, but also generating new combined samples at the level of cross-modal and cross-sample, thereby improving the robustness and generalization ability of the model under different scenarios and different device conditions. Third, the embodiment sets an adaptive feature enhancement module in the feature processing process, which encodes multi-modal features, suppresses redundancy and weights attention, can highlight significant features highly related to the stealing behavior, and suppress irrelevant or repetitive information between modes, thereby improving the discriminability of feature fusion. Further, through a multi-task learning architecture, the device category, user behavior category and whether there is a stealing behavior are simultaneously predicted, realizing multi-target recognition in a unified framework, which not only improves the accuracy of stealing behavior recognition, but also provides more rich auxiliary information for subsequent protection strategies. The embodiment also introduces an unbalanced regulation loss and a fine-grained distinction constraint in the training, further enhancing the adaptability to actual complex scenes. Finally, the application not only gives the recognition result and confidence of the stealing behavior in the inference stage, but also performs the same style camouflage desensitization processing on the suspicious attention area of the screen according to the result, which can implement protection in real time when the risk is found, avoid sensitive information leakage, and form a complete closed loop from detection to protection. In summary, the embodiment significantly improves the accuracy, robustness and practicality of screen anti-stealing photograph monitoring through a series of innovative designs such as multi-modal data fusion, data enhancement, feature optimization and multi-task recognition, solving the problems of relying on a single mode, insufficient generalization and lack of real-time protection in the prior art.

[0067] In a feasible implementation, the data acquisition module comprises a RotMix sub-module and a RotCom sub-module.

[0068] As shown in the figure, Figure 2 Step S110 obtains a data set from different acquisition devices and different acquisition scenes through the data acquisition module, including:

[0069] S1101, the Rot Mix sub-module is used to perform rotation disturbance and feature mixing within the single mode of the source data of the data set to generate intra-class diversified samples;

[0070] S1102, the RotCom sub-module is used to combine modes between different samples of the source data of the data set and generate cross-modal enhanced samples based on consistency constraints.

[0071] S1103. Add the above-mentioned intra-class diversified samples and the above-mentioned cross-modal enhanced samples to the above-mentioned dataset to obtain the enhanced data.

[0072] For example, this embodiment provides a set of data acquisition and enhancement procedures that can be directly implemented for steps S1101 to S1103, and provides optional mathematical representations and parameter meanings without limiting the scope of protection, so as to facilitate reproduction and comparison experiments.

[0073] In step S110, the system first acquires multimodal raw data from different acquisition devices and scenarios, and completes alignment and slicing according to a unified time base. (The window length is used as a reference.) (e.g., 4s) and step size (e.g., 0.5s) sliding segmentation to obtain samples With the corresponding tags , respectively, represent whether there is any unauthorized filming, the device category, and the atomic behavior category. Here, video modality is the frame sequence or feature frame within a time segment, UWB / Bluetooth positioning is the distance or angle of arrival time sequence, ambient light intensity is the illuminance time sequence, audio is the spectrogram or frame-level feature, and peripheral behavior is the keyboard / mouse / touch event time sequence.

[0074] In step S1101, the RotMix submodule deforms and blends each sample within a single modality to generate diverse samples within the class, thereby enhancing the robustness of the same class under conditions such as angle, lighting, and noise. For image / video modalities, a "rotation + two-level enhancement + blending" approach can be used.

[0075] ;

[0076] in, This indicates that an angle is applied to the frame or feature map. rotation (such as) ), These are mild and moderate image enhancement operators (such as brightness / contrast perturbation, mild sharpening, Gaussian noise, or blurring). Mixing coefficient (commonly used) For audio modes, a combination of time axis shifting and frequency band blocking can be used, for example:

[0077]

[0078] in It is time shift (such as) ), This involves random frequency band / temporal occlusion (SpecAugment approach). For peripheral behavior modalities, a small phase shift or random frame dropping followed by interpolation can be applied to the event timestamps. The RotMix labels remain unchanged. To encourage consistent representation to rotation / disturbance, rotation consistency regularization can be added during training:

[0079] ;

[0080] where is the intermediate representation of the feature extraction network, which is used to guide the network to learn features that are not sensitive to geometric / spectral domain disturbance.

[0081] In step S1102, combination enhancement is performed on the cross-sample and cross-modal level by the RotCom submodule, aiming to break the coupling of scene appearance-behavior semantics and let the model learn more structured signals related to the nature of behavior. The specific method is to select two samples from the same batch or the same category pool , keep the modalities (such as video, peripheral, and part of audio) that are strongly related to behavior from sample , replace the modalities (such as UWB distance and ambient light intensity) that are more biased towards environment / scene from sample , and get the combined sample:

[0082] ;

[0083] To maintain semantic consistency, combination consistency constraints are applied to the original sample and the combined sample in the fusion representation space:

[0084] ;

[0085] where is the common embedding mapping before and after cross-modal fusion (for example, the feature vector output by the fusion layer). The label inheritance rule of RotCom focuses on maintaining the main semantic consistency: the pickpocketing behavior label and the atomic behavior label are inherited from sample ( ), and the device category is usually also inherited from .

[0086] ( ), because the replaced is the environment-related modality, not the device imaging modality. To avoid unreasonable combination, samples and are required to satisfy the coarse-grained compatibility condition (such as the same scene type or time period), so as to improve the authenticity of the combination.

[0087] In step S1103, the enhanced samples generated by RotMix and RotCom are added to the original dataset: the new training set is denoted as

[0088] ;

[0089] and the corresponding labels are inherited or generated according to the above rules. During training, the augmented samples can be proportionally sampled (e.g., augmented samples: original samples or , and and are assigned weights to form a joint optimization target with the main task loss, so as to achieve the synergistic effect of "intra-class diversity promotion + cross-domain generalization enhancement".

[0090] : window length and step, determine the time resolution and coverage (e.g. ). : image / video rotation angle, covering common mobile phone deflection (e.g. ). : two-level enhancement operators, the former is a light perturbation, and the latter is a medium perturbation, which are used to construct strong and weak views. : mixing coefficient and Beta distribution shape parameter, The smaller the mixing is, the more biased it is to the extreme; empirically . : time shift amount of audio / event sequence, used to simulate slight alignment deviation or action fluctuation (e.g. ). : intermediate feature extraction mapping and fusion embedding mapping, respectively, which are implemented by subsequent networks (e.g., MixNet backbone and fusion layer). , : consistency class regularization terms, the former constrains the representations of different rotations / perturbations of the same instance to be close, and the latter constrains the representation after combining across samples to be consistent with the original behavior sample; the corresponding weights can be optimized in [ ]. Sampling compatibility condition: limit the replaced modalities (e.g., light intensity, distance) and the main sample to be approximately in the scene type / time period, to reduce the training noise caused by unreasonable splicing.

[0091] Through the above processes of S1101-S1103, the training set is systematically expanded in the two dimensions of single-modal morphing and cross-modal combination. RotMix enhances the intra-class morphing coverage, making the model more robust to angle, light, spectrum, and time sequence perturbations. RotCom breaks the binding relationship between scene appearance and behavior semantics, guiding the model to focus on the behavior structure, so that it can still maintain a high recognition rate for the stolen behavior when changing devices and scenes. Such a data foundation provides a more representative training distribution for subsequent feature extraction, feature enhancement, and multi-task recognition.

[0092] In a feasible implementation, as Figure 3As shown, in the process of training the above-mentioned data collection module, the above-mentioned data enhancement module, the above-mentioned Mix Net module, the above-mentioned adaptive feature enhancement module and the above-mentioned multi-task learning architecture module, the above-mentioned method further comprises:

[0093] S210, a gradient reversal layer is introduced between the feature extractor and the domain discriminator, and the source domain feature and the target domain feature are made indistinguishable in the representation space through adversarial training;

[0094] S220, the covariance matrix difference of the source domain feature and the target domain feature is calculated, and optimization is performed based on the correlation alignment loss function to reduce the second-order statistical distribution difference of the two;

[0095] S230, the mean and variance parameters of the source domain and the target domain are maintained respectively in the batch normalization process to realize the adaptive adjustment of the feature distribution under different devices and different scenes.

[0096] Exemplarily, in the present embodiment, for the paparazzi behavior recognition model composed of the data collection module, the data enhancement module, the MixNet module, the adaptive feature enhancement module and the multi-task learning architecture module, three types of complementary "cross-domain distribution alignment" mechanisms are introduced in the training stage, corresponding to steps S210-S230. During training, a hybrid mini-batch method is used, and source domain (existing labeled, data set from existing devices and scenes) and target domain (new device / new scene, usually unlabeled or weakly labeled data set) samples are mixed in a certain proportion to form a batch data. Through joint optimization, the model learns to be insensitive to domain changes and highly discriminative to whether it is paparazzi.

[0097] First, in step S210, an adversarial domain confusion training is adopted: the fused feature after adaptive feature enhancement is denoted as , wherein represents a feature extractor composed of a MixNet backbone and an adaptive feature enhancement module, and the parameter is ; a domain discriminator is connected in series after it, which outputs the probability that the sample comes from the source domain. To realize the training mechanism of forward invariance and backward confrontation, a gradient reversal layer (GRL) is inserted between and , which is an identity mapping in forward propagation, and the gradient from the domain discriminator back to the feature extractor is multiplied by in backward propagation. The training target adopts an adversarial loss of binary cross entropy: the source domain feature is encouraged to be 1, and the target domain feature is encouraged to be 0; and under the action of GRL, the feature extractor will be forced to learn to make the two domain features indistinguishable in the representation space. Among them can be changed with the training progress (normalized steps from start to end) adopts an increasing schedule, like , The growth steepness hyperparameter (which can take values from 0 to 1) helps stabilize convergence, with weak confrontation in early stage and strong confrontation in later stage. The domain discriminator can be composed of two or three fully connected layers, with layer normalization and leaky activation to avoid training oscillation caused by too strong discriminator. Optionally, a slight label smoothing or gradient penalty can be introduced to suppress overfitting.

[0098] Then in step S220, statistical distribution alignment is adopted to refine the second-order statistics consistency of the two domains. Let the feature set of the source domain in a batch be , and that of the target domain be , the covariance matrices of the two domains are calculated on the feature dimension , and the difference in their Frobenius norms is taken as the correlation alignment loss: . Where are estimated from the batch-decentered features, denotes the norm of the square root of the sum of the squares of the matrix elements. To take into account the first-order statistics, mean alignment (minimizing the Euclidean distance between the feature means of the two domains) can be added to the same branch, and when stronger distribution matching is needed, the maximum mean difference (MMD) loss can be introduced in parallel: the distance between the two domains is calculated in the reproducing kernel Hilbert space of the Gaussian kernel , and the bandwidth can be adaptively set by the median heuristic from the median distance of the sample pairs in the batch. This statistical alignment term and the adversarial term in S210 form a complement: the adversarial term makes the two domains globally confusable, and the statistical term gradually brings the two domains closer in the sense of covariance and kernel mean, so that it does not rely on the target domain label and can effectively reduce the distribution difference.

[0099] Subsequently in step S230, domain-aware statistical re-estimation is implemented for the normalization layers of the network. The specific approach is: for each layer, two independent sets of running statistics and learnable affine parameters are maintained for the source domain and the target domain . During training, if the current mini-batch comes from the source domain, the mean and variance of the source domain are updated and normalized using ; if it comes from the target domain, the target domain's set of parameters is updated and used. This way, the difference between the two domains in the activation distribution can be explicitly modeled, avoiding the use of a single statistic to pull both domains. To improve adaptability to unknown scenarios during deployment, short-term statistical re-estimation (such as online estimation using exponential moving average) is performed on real-time data of the target domain during inference, or the "target domain confidence" output by the domain discriminator is used to adaptively weight the fusion of the two sets of statistics: . Where​ The probability output of the domain discriminator on the segment of input or its temperature calibrated value. This can provide a smoother transition in the early stage of new device online (target domain statistics are not stable yet).

[0100] The total goal of training is generally written as the weighted sum of "task loss + domain alignment loss": . Where is the joint loss of the aforementioned multi-task learning (including the Focal loss of the behavior of taking pictures, the cross-entropy of the device category, the CRF / triplet of the behavior timing, etc.), is the adversarial term of S210, is the covariance alignment term of S220, is the kernel mean alignment term; is the corresponding weight, generally in the order of , and can be segmented or cosine adjusted according to the training progress or the performance of the validation set. When optimizing, the alternating update is adopted: the feature extractor and the task head are updated at the same time in one forward and one backward, and the pace of the multi-step discriminator update / generator update of the domain discriminator can be selected to enhance the stability of the confrontation; the learning rate and the weight decay can be adjusted with AdamW, linear warmup and cosine annealing. In order to prevent the main task accuracy from decreasing due to unstable confrontation, it is recommended to only enable statistical alignment in the first few rounds, and then gradually increase and .

[0101] Through the adversarial confusion of S210, the statistical alignment of S220 and the domain-aware normalization of S230, the embodiment significantly reduces the representation difference between the source domain and the target domain without the need for target domain labels or only a small amount of weak annotation, so that the model can still stably output a high-confidence judgment of whether there is a behavior of taking pictures in a new device, a new perspective and a new lighting environment, while maintaining the reliability of device category and user behavior recognition.

[0102] In a feasible embodiment, the adaptive feature enhancement module includes a modal-by-modal encoding submodule, a redundancy suppression submodule, a gated attention submodule, and a fusion submodule,

[0103] As shown in Figure 4 , step S140 performs modal-by-modal encoding on the multi-modal feature representation through the adaptive feature enhancement module, suppresses the redundant information between the modalities and highlights the significant features related to the behavior of taking pictures, to obtain optimized fusion features, including:

[0104] S1401, map each modal feature to a feature vector of a unified dimension through the aforementioned modal-by-modal encoding submodule;

[0105] S1402. By applying orthogonal constraints and mutual information minimization constraints to the feature vectors of different modes through the above-mentioned redundancy suppression submodule, the redundancy information between modes is reduced.

[0106] S1403. By assigning weight coefficients to the feature vectors of each modality of the above-mentioned gated attention submodule, the features related to the surreptitious shooting behavior are weighted up and irrelevant features are weakened.

[0107] S1404. The weighted modal features are summed by the fusion submodule to generate the optimized fusion features mentioned above.

[0108] In one feasible implementation, such as Figure 5 The above-described step S1402 applies orthogonal constraints and mutual information minimization constraints between feature vectors of different modalities through the aforementioned redundancy suppression submodule, in order to reduce intermodal redundancy information, including:

[0109] S14021. Apply orthogonality constraints to the feature vectors of different modes, and minimize the cosine similarity square term to make the feature vectors approximately orthogonal in the representation space, so as to improve the complementarity between modes.

[0110] S14022. Apply mutual information minimization constraints to the feature vectors of different modes, calculate and minimize the mutual information between modes based on the Hilbert-Schmidt independence criterion, so as to reduce redundant information between modes.

[0111] S14023. The above orthogonal constraints and mutual information minimization constraints are weighted and combined, and the weight coefficients are dynamically adjusted during training to reduce redundancy while maintaining the ability to identify surreptitious behavior.

[0112] For example, in this embodiment, the adaptive feature enhancement module operates continuously according to S1401 to S1404, and is used to complete modal encoding, cross-modal redundancy suppression, gated weighting, and efficient fusion at a unified time and feature scale. Assume that within the same time window there are... There are *n* modalities, and the input feature sequence for each modality is denoted as […]. ,in The time steps for this window, The modal input dimension.

[0113] In S1401, each mode first passes through its dedicated encoder. Representations are extracted (including 1D convolutions and residual blocks), and then linearly projected onto a common dimension. ,get

[0114]

[0115] in, For point-wise nonlinearity (e.g. ReLU / GELU), LN is layer normalization, If some modalities are more sparse, aligning to a uniform time grid (linear interpolation or down-sampling) makes the time dimension of all modalities .

[0116] In step S1402, to reduce cross-modality redundancy without loss of discriminability, the embodiment adopts a global vector and double constraint approach. First, pool each modality in the time dimension to obtain a global vector The pooling can be average pooling,

[0117]

[0118] where and are attention parameters.

[0119] In step S14021, for the global vectors of any two modalities, minimize the square term of the cosine similarity:

[0120] ;

[0121] This term encourages different modalities to separate in the representation space direction, reduces redundant information, and thus improves the complementarity and fusion efficiency between modalities.

[0122] In step S14022, the Hilbert-Schmidt independence criterion (HSIC) is used to estimate the statistical dependence between different modalities in the batch and to weaken it. Let the batch size be , and the Gram kernel matrix of modality is

[0123] , whose elements are calculated by the Gaussian kernel:

[0124] ;

[0125] The calculation result is: is the centered matrix, then

[0126] ;

[0127] By minimizing , the redundant statistical dependence between modalities can be effectively reduced. The kernel bandwidth is adaptively set according to the median of the Euclidean distance of the sample pairs in the batch, taking into account stability and computational complexity.

[0128] In step S14023, the total loss of redundancy suppression is:

[0129] ;

[0130] and set the training progress Dynamic growing weights:

[0131]

[0132] Avoid early over-strong regularization task convergence in a weak-to-strong manner. If necessary, gradient normalization (e.g. adaptive scaling according to the norm of each gradient) can be added to balance the influence of the two regularization terms.

[0133] Formula parameter explanation: is the number of time steps, determined by the window length and the sampling rate; is the modal input dimension and the unified public dimension, respectively; is the structure and parameters of the modal encoder; is the parameter and weight of attention pooling; is the batch size; is the kernel matrix of ; is the bandwidth of the Gaussian kernel; is the static / maximum weight of the redundancy suppression regularization; is the normalized training progress (e.g. iteration number / total iteration number).

[0134] In step S1403, the global vectors of all modalities are spliced into the context , and two layers of perceptron are used to obtain the unnormalized gate value , and the modal weight is obtained by using Softmax with temperature

[0135] ;

[0136] where controls the sharpness of the weight distribution. For each time step , multiply after broadcasting to obtain the weighted sequence . When a modality is temporarily missing, introduce the visibility mask

[0137] and normalize the weight to to ensure the robustness of inference. is the unnormalized gate value, is the attention weight of the th modality; is the temperature coefficient; is the modality visibility mask.

[0138] ​In step S1404, the modal weighted sequence is aggregated in time dimension per time to obtain the optimized fusion feature:

[0139] ;

[0140] It can be understood that a lightweight time sequence modeling (such as a single-layer self-attention) can be superimposed on the above, and attention pooling is performed to generate a segment vector as the input of the downstream multi-task head.

[0141] The adaptive feature enhancement module is trained end-to-end with the downstream task, and the comprehensive loss is:

[0142]

[0143] Wherein is the joint loss of multi-task learning (including Focal loss of whether to take pictures, cross-entropy of device category, sequence loss of user behavior, etc.), and is the gate sparse regularization weight (suppressing invalid multi-modal while avoiding overfitting).

[0144] Through unified encoding of S1401, orthogonal and HSIC double de-redundancy of S1402, gate attention focusing of S1403 and efficient fusion of S1404, the embodiment not only weakens the repetition and noise coupling of cross-modal, but also adaptively highlights the modal and time related to the behavior of taking pictures, thereby significantly improving the discriminability, robustness and generalizability of the fusion feature without increasing the reasoning time delay.

[0145] In a feasible embodiment, the multi-task learning architecture module described above includes a bottom shared feature layer, a first task sub-module, a second task sub-module and a third task sub-module.

[0146] The above step S150, as shown in Figure 6 , based on the optimized fusion feature, a multi-task learning architecture module is used to simultaneously predict the device category, the user behavior category and whether there is a behavior of taking pictures, including:

[0147] S1501, receiving the optimized fusion feature through the bottom shared feature layer and generating a shared representation;

[0148] S1502, predicting whether there is a behavior of taking pictures based on the shared representation through the first task sub-module;

[0149] S1503, predicting the device category based on the shared representation through the second task sub-module;

[0150] ​S1504. Predict user behavior categories based on the shared representation through the third task submodule; wherein, the optimization objective of the multi-task learning architecture is a joint loss function, which is a weighted combination of Focal Loss, cross-entropy loss and triplet loss.

[0151] In one feasible implementation, the first task submodule is used to obtain a surreptitious behavior discrimination score by fusion of a fragment-level fully connected network and a temporal-level one-dimensional convolution based on the shared representation. The discrimination score is mapped to a probability by a Sigmoid and optimized with Focal Loss with an imbalance adjustment factor.

[0152] In one feasible implementation, the second task submodule is used to perform hierarchical prediction of equipment categories. The second task submodule first determines the major equipment category and then the fine-grained model. It introduces label smoothing and class balance terms weighted by the number of effective samples per class into the cross-entropy loss, and adds distillation loss based on the soft label of the teacher model.

[0153] In one feasible implementation, the aforementioned third task submodule is used to perform temporal prediction of user behavior categories, generate frame-level logits using bidirectional cyclic units, and perform sequence decoding using conditional random fields. At the same time, triplet loss is introduced into the behavior embedding space to enhance fine-grained discrimination capability.

[0154] For example, step S150 uses a multi-task learning architecture with a shared base and three task sub-modules to jointly predict whether there is surreptitious filming, the device category, and the user behavior category. Let the optimized fusion feature obtained after adaptive feature enhancement be... , For time steps, This represents the unified feature dimensions. First, the underlying shared feature layer... Temporal modeling and aggregation are performed to obtain a shared representation, which is then fed into three task sub-modules for output, and end-to-end training is performed using joint loss.

[0155] In S1501, the underlying shared feature layer pairs Time coding is performed to obtain (Enc()• can be a single-layer bidirectional GRU, TCN, or timing Transformer, (for the encoded dimension), while attention pooling is used to generate fragment-level shared vectors.

[0156]

[0157] in for At any moment The vector, With as attention parameters, as interpretable temporal weights. for time-series sub-module, for segment-level discrimination and device classification.

[0158] In S1502, the first task sub-module outputs a binary classification result indicating whether there is a shooting behavior based on the shared representation. To balance the segment-level and time-series level criteria, two parallel branches are adopted:

[0159] The segment-level branch maps the shared representation to a score ; through attention weighting or temporal pooling operation, with dimension as the weight matrix of the coarse-grained device classification layer, with size , where is the number of coarse classification categories, used to map device features to a coarse classification space. is the bias vector corresponding to the linear mapping, used to correct the reference position of the feature mapping output. φ(•) represents a nonlinear activation function, used to enhance the nonlinear representation ability of the model, represents the output layer weight vector, used to further map the nonlinear transformed features to a scalar output, is the bias term of the final classification layer, used to adjust the output balance of the predicted score.

[0160] The time-series level branch applies one-dimensional convolution (convolution kernel size ) and global maximum pooling to obtain a score

[0161] .

[0162] H represents the time series feature matrix obtained through the pre-sequence feature extraction module, with dimension , where T is the number of time steps, is the feature dimension. represents the one-dimensional convolution operation on the input feature along the time axis, with convolution kernel size k, used to extract local time series features. represents a nonlinear activation function, which can be ReLU or tanh function, used to enhance the model's representation ability for nonlinear features; GMP(•) represents the global maximum pooling (GlobalMaxPooling) operation, used to extract the most significant global time series features from the convolution result; is the weight vector of the behavior branch output layer, used to map the pooled features to a scalar space; ​This is the corresponding bias term, used to adjust the baseline value of the output score; The output score for this branch represents the prediction result of the existence of the target behavior, which is subsequently mapped to the probability of behavior existence using the Sigmoid function. It is used for binary classification.

[0163] Two-way fractions with coefficients merge into

[0164] The probability is mapped to Sigmoid. During training, FocalLoss with imbalance adjustment is used:

[0165] ;

[0166] in The true label indicating whether or not the film was taken secretly. For positive and negative sample weights, Focusing factor for difficult examples; temperature scaling can be used during inference. Perform confidence level calibration and use a threshold decision making.

[0167] In S1503, the second task submodule performs hierarchical prediction of device categories. First, it performs... Classify the equipment into coarse-grained categories to obtain Then in the selected major category The following fine-grained model classification is performed to obtain .enter This represents the global device feature vector obtained after attention-weighted or temporal pooling operations, with dimension d; The weight matrix of the fine-grained classification layer is represented by the gating vector. After modulation, dynamically adjustable mapping weights are formed, where The gate vector is used to adjust the weight allocation of the current fine-classification features based on the coarse classification results of the previous layer, thereby achieving hierarchical feature classification. This indicates the corresponding bias term, also represented by... Modulation is used to correct the center position of the output distribution during dynamic mapping; softmax(•) represents the probability normalization function, which is used to map the linear output to the probability distribution of each sub-category; This is the fine-grained device category prediction probability vector output by the model, with dimension . ,in This represents the number of categories for the fine-grained classification. Both the coarse and fine-grained levels employ cross-entropy loss, and label smoothing is introduced into the target distribution. To mitigate overfitting; for class imbalance, a weighted average based on the number of effective samples is used. The number of samples in the category . At the same time, introduce knowledge distillation loss to enhance the generalization of new devices: let the soft target of the teacher model under the temperature , the student output is , then:

[0168] ;

[0169] The device task loss is obtained by integrating, where and are weights.

[0170] In S1504, the third task sub-module performs time sequence prediction of user behavior categories. Take as input, use bidirectional recurrent units to obtain context-enhanced sequence , and generate frame-level logits at each time step: . To ensure the time sequence consistency and prior transition constraint of the behavior sequence, use linear chain conditional random field (CRF) to model transition matrix , obtain negative log-likelihood loss by maximum likelihood training, and obtain by Viterbi decoding in the inference stage. To improve the fine-grained distinguishing ability, also construct behavior embedding on the shared vector, and impose a triplet loss on the semi-hard negative samples:

[0171] ;

[0172] Where are the needle, positive, and negative sample embeddings, respectively, is the interval. The total loss of this sub-module is denoted as .

[0173] The joint optimization objective of the whole multi-task learning is

[0174] ;

[0175] Where is used to strengthen the main task of whether to take a photo. The optimizer can be AdamW, and the learning rate adopts linear warm-up and cosine annealing. To stabilize the multi-task weighting, use uncertainty weighting or adaptive weight based on gradient norm.

[0176] The parameters and symbol meanings in the above formula are as follows: is the number of time steps in the time window, ​​​input to the shared layer and the dimension of the encoding; Enc(•) denotes a temporal encoder; is a nonlinear activation; Conv1D is a is a kernel size; is a two-branch fusion coefficient; is a class weight and a focal factor of FocalLoss, is a decision threshold; is a label smoothing coefficient, is the number of classes; and are a smoothing factor and a class sample size of the class balance weight, respectively; is a distillation temperature, is a distillation loss weight; is a CRF transition matrix; is a triplet loss interval, is a triplet loss weight; is the weight of the three tasks in the joint target.

[0177] Through the shared multi-task architecture, the system cooperatively learns three types of targets in the same representation space: the dominant optimization of the clip-level sneak shot discrimination, the structured prior and auxiliary supervision provided by the device and behavior tasks, so as to realize stable, interpretable and high-precision screen anti-sneak shot monitoring under complex scenes and new device conditions.

[0178] In a second aspect, as shown in Figure 7 , the application proposes a screen anti-sneak shot monitoring system based on multi-modal model fusion, comprising:

[0179] A data acquisition module 21 is configured to acquire a data set from different acquisition devices and different acquisition scenes, wherein the data set comprises video modal data, wireless positioning modal data, ambient light intensity modal data, audio modal data and peripheral behavior modal data.

[0180] A data enhancement module 22 is configured to perform data enhancement on source data of the data set to obtain enhanced data.

[0181] A MixNet module 23 is configured to perform feature extraction on the enhanced data to obtain corresponding multi-modal feature representations.

[0182] An adaptive feature enhancement module 24 is configured to perform modal-by-modal encoding on the multi-modal feature representations, suppresses redundant information between modalities and highlights significant features related to the sneak shot behavior, to obtain optimized fusion features.

[0183] A multi-task learning architecture module 25 is configured to simultaneously predict the device category, the user behavior category and whether there is a sneak shot behavior based on the optimized fusion features.

[0184] It can be understood that the system can also perform the method as claimed in any one of the first aspect.

[0185] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A screen anti-stealing monitoring method based on multi-modal model fusion, characterized in that, The method comprises the following steps: acquiring a dataset from different acquisition devices and different acquisition scenes through a data acquisition module, wherein the dataset comprises video modal data, wireless positioning modal data, ambient light intensity modal data, audio modal data and peripheral device behavior modal data; performing data enhancement on source data of the dataset through a data enhancement module to obtain enhanced data; inputting the enhanced data into a MixNet module for feature extraction to obtain corresponding multi-modal feature representation; performing modal-by-modal encoding on the multi-modal feature representation through an adaptive feature enhancement module to suppress inter-modal redundant information and highlight significant features related to the behavior of secretly taking pictures to obtain optimized fusion features; based on the optimized fusion features, simultaneously predicting the device category, user behavior category and whether there is the behavior of secretly taking pictures through a multi-task learning architecture module.

2. The screen anti-stealing monitoring method based on multi-modal model fusion according to claim 1, characterized in that, The data acquisition module comprises a RotMix sub-module and a RotCom sub-module; acquiring a dataset from different acquisition devices and different acquisition scenes through a data acquisition module, comprising: performing rotation disturbance and feature mixing within a single modality of source data of the dataset through the RotMix sub-module to generate intra-class diversified samples; performing modal combination between different samples of source data of the dataset through the RotCom sub-module and generating cross-modal enhanced samples based on consistency constraints; adding the intra-class diversified samples and the cross-modal enhanced samples to the dataset to obtain enhanced data.

3. The screen anti-stealing monitoring method based on multi-modal model fusion according to claim 1 or 2, characterized in that, In the process of training the secretly taking pictures behavior recognition model composed of the data acquisition module, the data enhancement module, the MixNet module, the adaptive feature enhancement module and the multi-task learning architecture module, the method further comprises: introducing a gradient reversal layer between a feature extractor and a domain discriminator to make source domain features and target domain features indistinguishable in a representation space through adversarial training; calculating the covariance matrix difference of the source domain features and the target domain features, and optimizing based on a correlation alignment loss function to reduce the second-order statistical distribution difference between the two; respectively maintaining the mean and variance parameters of the source domain and the target domain in the batch normalization process to realize adaptive adjustment of the feature distribution under different devices and different scenes.

4. The screen anti-stealing monitoring method based on multi-modal model fusion according to claim 1, characterized in that, The adaptive feature enhancement module comprises a modal-by-modal encoding sub-module, a redundancy suppression sub-module, a gated attention sub-module and a fusion sub-module, the adaptive feature enhancement module comprises a modal-by-modal encoding sub-module, a redundancy suppression sub-module, a gated attention sub-module and a fusion sub-module, the adaptive feature enhancement module comprises a modal-by-modal encoding sub-module, a redundancy suppression sub-module, a gated attention sub-module and a fusion sub-module, the adaptive feature enhancement module comprises a modal-by-modal encoding sub-module, a redundancy suppression sub-module, a gated attention sub-module and a fusion sub-module, the adaptive feature enhancement module comprises a modal-by-modal encoding sub-module, a redundancy suppression sub-module, a gated attention sub-module and a fusion sub-module, the adaptive feature enhancement module comprises a modal-by-modal encoding sub-module, a redundancy suppression sub-module, a gated attention sub-module and a fusion sub-module.

5. The screen anti-stealing monitoring method based on multi-modal model fusion according to claim 4, characterized in that, The orthogonal constraint and the mutual information minimization constraint are imposed between the feature vectors of different modalities by the redundancy suppression sub-module to reduce inter-modal redundant information, including: The orthogonal constraint is imposed between the feature vectors of different modalities, and a cosine similarity square term is minimized to promote the feature vectors to remain approximately orthogonal in the representation space, thereby improving the complementarity between modalities; The mutual information minimization constraint is imposed between the feature vectors of different modalities, and the inter-modal mutual information is calculated based on the Hilbert-Schmidt independence criterion and minimized to reduce inter-modal redundant information; The orthogonal constraint and the mutual information minimization constraint are combined and the weight coefficient is dynamically adjusted during the training process to reduce redundancy while maintaining the discriminative ability for recognizing the behavior of taking pictures.

6. The screen anti-stealing monitoring method based on multi-modal model fusion according to claim 1, characterized in that, The multi-task learning architecture module includes a bottom layer shared feature layer, a first task sub-module, a second task sub-module, and a third task sub-module; Based on the optimized fusion feature, a multi-task learning architecture module is used to simultaneously predict the device category, the user behavior category, and whether there is a behavior of taking pictures, including: The bottom layer shared feature layer receives the optimized fusion feature and generates a shared representation; The first task sub-module predicts whether there is a behavior of taking pictures based on the shared representation; The second task sub-module predicts the device category based on the shared representation; The third task sub-module predicts the user behavior category based on the shared representation; wherein the optimization target of the multi-task learning architecture is a joint loss function, and the joint loss function is a weighted combination of Focal Loss, cross-entropy loss, and triplet loss.

7. The screen anti-stealing monitoring method based on multi-modal model fusion according to claim 6, characterized in that, The first task sub-module is used to obtain a behavior of taking pictures discriminant score by parallel fusion of a segment-level fully connected network and a time-series one-dimensional convolution based on the shared representation, and the discriminant score is mapped to a probability by Sigmoid and optimized by Focal Loss with an imbalance adjustment factor.

8. The screen anti-stealing monitoring method based on multi-modal model fusion according to claim 6, characterized in that, The second task sub-module is used to perform hierarchical prediction on the device category, which first judges the device category and then judges the fine-grained model, and introduces label smoothing and class balance terms weighted by the number of effective samples in the cross-entropy loss, and adds a distillation loss based on the soft label of the teacher model.

9. The screen anti-stealing monitoring method based on multi-modal model fusion according to claim 6, characterized in that, The third task sub-module is used to perform time-series prediction on the user behavior category, generates frame-level logits using a bidirectional recurrent unit, and performs sequence decoding using a conditional random field, while introducing a triplet loss in the behavior embedding space to enhance the fine-grained discrimination ability.

10. A screen anti-stealing monitoring system based on multi-modal model fusion, characterized in that, including: A data acquisition module is configured to acquire a dataset from different acquisition devices and different acquisition scenarios, wherein the dataset includes video modal data, wireless positioning modal data, ambient light intensity modal data, audio modal data, and peripheral device behavior modal data; A data enhancement module is configured to perform data enhancement on source data of the dataset to obtain enhanced data; A MixNet module is configured to extract features from the enhanced data to obtain corresponding multi-modal feature representations; An adaptive feature enhancement module is configured to encode the multi-modal feature representation modality by modality, suppress inter-modal redundant information and highlight salient features related to the surreptitious behavior to obtain an optimized fusion feature; A multi-task learning architecture module is configured to simultaneously predict the device category, the user behavior category and whether the surreptitious behavior exists based on the optimized fusion feature.

Images