A screen anti-stealing monitoring method and system based on multi-modal model fusion

Abstract

The application discloses a screen anti-stealing monitoring method and system based on multi-modal model fusion, relates to screen privacy protection, and the method first acquires multi-source data from a video mode, a wireless positioning mode, an ambient light intensity mode, an audio mode and a peripheral behavior mode through a data acquisition module, and enhances the source data by using a data enhancement module. Then, the enhanced multi-modal data is subjected to feature extraction to obtain corresponding multi-modal feature representations. By using an adaptive feature enhancement module, redundant information between modes is suppressed and significant features related to the stealing behavior are highlighted to generate optimized fusion features. Based on the optimized fusion features, a multi-task learning architecture module is used to simultaneously predict the device category, the user behavior category and whether the stealing behavior exists. The method can fully fuse multi-modal data, improve the accuracy and robustness of screen anti-stealing detection, and is suitable for real-time safety monitoring of multi-scene and multi-type terminals.

Description

Technical Field

[0001] This specification relates to screen privacy protection; more specifically, this application relates to a screen anti-spy camera monitoring method and system based on multimodal model fusion. Background Technology

[0002] With the widespread use of mobile devices and miniaturized recording equipment, the act of secretly filming using smartphones, miniature cameras, and other devices is becoming increasingly covert and diverse, posing a serious threat to meeting settings, office environments, and personal privacy. Most existing anti-spy camera technologies rely on single-modal data for identification, such as detecting suspicious behavior through video surveillance or determining the distance between the user and the screen through wireless positioning.

[0003] These single-modal methods often have the following shortcomings when facing complex environments: First, the information from different modalities is limited, for example, video may be affected by lighting and occlusion angles; second, they lack the utilization of complementary relationships between modalities, which can easily lead to false positives or false negatives; third, the generalization ability of the model is insufficient under different devices and different scenarios, and often requires retraining for specific scenarios; fourth, after identifying risks, there is a lack of effective real-time protection measures, and it is impossible to dynamically desensitize the screen content in a timely manner.

[0004] Therefore, there is an urgent need to propose a screen anti-spy camera monitoring method and system based on multimodal model fusion to at least solve some of the above problems. Summary of the Invention

[0005] The summary section introduces a series of simplified concepts, which will be further explained in detail in the detailed description section. This summary section is not intended to limit the key and essential technical features of the claimed technical solution, nor is it intended to determine the scope of protection of the claimed technical solution.

[0006] Firstly, this application proposes a screen anti-spy camera monitoring method and system based on multimodal model fusion, the method comprising:

[0007] Data sets from different acquisition devices and different acquisition scenarios are acquired through the data acquisition module. These data sets include video modal data, wireless positioning modal data, ambient light intensity modal data, audio modal data, and peripheral behavior modal data.

[0008] The data augmentation module is used to augment the source data of the above dataset to obtain augmented data.

[0009] The enhanced data is then input into the MixNet module for feature extraction to obtain the corresponding multimodal feature representation.

[0010] The above multimodal feature representations are encoded modally by an adaptive feature enhancement module to suppress redundant information between modalities and highlight significant features related to surreptitious photography, so as to obtain optimized fusion features.

[0011] Based on the optimized fusion features mentioned above, a multi-task learning architecture module is adopted to simultaneously predict device category, user behavior category, and whether there is any surreptitious filming behavior.

[0012] In one feasible implementation, the data acquisition module includes a RotMix submodule and a RotCom submodule;

[0013] Data sets from different acquisition devices and different acquisition scenarios are acquired through the data acquisition module, including:

[0014] The Rot Mix submodule described above performs rotational perturbation and feature mixing within the single modality of the source data in the dataset to generate diverse intra-class samples.

[0015] The RotCom submodule described above performs modal combination among different samples of the source data in the above dataset and generates cross-modal augmented samples based on consistency constraints.

[0016] The above-mentioned intra-class diversified samples and cross-modal augmented samples are added to the above dataset to obtain augmented data.

[0017] In one feasible implementation, during the training of the covert photography behavior recognition model composed of the aforementioned data acquisition module, data augmentation module, MixNet module, adaptive feature enhancement module, and multi-task learning architecture module, the method further includes:

[0018] A gradient inversion layer is introduced between the feature extractor and the domain discriminator, and adversarial training is used to make the source domain features and the target domain features indistinguishable in the representation space.

[0019] The difference in covariance matrices between source domain features and target domain features is calculated and optimized based on the relevant alignment loss function to reduce the difference in their second-order statistical distributions.

[0020] During batch normalization, the mean and variance parameters of the source and target domains are maintained respectively to achieve adaptive adjustment of feature distribution under different devices and scenarios.

[0021] In one feasible implementation, the adaptive feature enhancement module includes a modal coding submodule, a redundancy suppression submodule, a gated attention submodule, and a fusion submodule.

[0022] The above-mentioned multimodal feature representation is encoded modally by an adaptive feature enhancement module to suppress redundant information between modalities and highlight salient features related to surreptitious photography, thereby obtaining optimized fusion features, including:

[0023] The modal coding submodule described above maps each modal feature into a feature vector of a uniform dimension.

[0024] The above-mentioned redundancy suppression submodule applies orthogonal constraints and mutual information minimization constraints to the feature vectors of different modes in order to reduce redundancy information between modes.

[0025] By assigning weight coefficients to the feature vectors of each modality in the above-mentioned gated attention submodule, the features related to the voyeuristic behavior are weighted up while irrelevant features are weakened.

[0026] The weighted modal features are summed using the fusion submodule to generate the optimized fusion features described above.

[0027] In one feasible implementation, the redundancy suppression submodule applies orthogonal constraints and mutual information minimization constraints between feature vectors of different modalities to reduce intermodal redundancy, including:

[0028] Orthogonality constraints are imposed on the feature vectors of different modalities. By minimizing the cosine similarity squared term, the feature vectors are made approximately orthogonal in the representation space, thereby improving the complementarity between modalities.

[0029] Mutual information minimization constraints are imposed on the feature vectors of different modes. The mutual information between modes is calculated and minimized based on the Hilbert-Schmidt independence criterion to reduce redundant information between modes.

[0030] The orthogonal constraints and mutual information minimization constraints mentioned above are weighted and combined, and the weight coefficients are dynamically adjusted during training to maintain the ability to identify surreptitious filming behavior while reducing redundancy.

[0031] In one feasible implementation, the above-mentioned multi-task learning architecture module includes a bottom shared feature layer, a first task sub-module, a second task sub-module, and a third task sub-module.

[0032] Based on the optimized fusion features described above, a multi-task learning architecture module is used to simultaneously predict device category, user behavior category, and the presence of surreptitious filming behavior, including:

[0033] The optimized fusion features are received through the aforementioned underlying shared feature layer, and a shared representation is generated.

[0034] The first task submodule predicts whether there is any surreptitious filming based on the shared representation mentioned above.

[0035] The second task submodule described above predicts the device category based on the shared representation.

[0036] The third task submodule predicts user behavior categories based on the shared representation. The optimization objective of the multi-task learning architecture is a joint loss function, which is a weighted combination of Focal Loss, cross-entropy loss, and triplet loss.

[0037] In one feasible implementation, the first task submodule is used to obtain a surreptitious behavior discrimination score by fusion of a fragment-level fully connected network and a temporal-level one-dimensional convolution based on the shared representation. The discrimination score is mapped to a probability by a Sigmoid and optimized with Focal Loss with an imbalance adjustment factor.

[0038] In one feasible implementation, the second task submodule is used to perform hierarchical prediction of equipment categories. The second task submodule first determines the major equipment category and then the fine-grained model. It introduces label smoothing and class balance terms weighted by the number of effective samples per class into the cross-entropy loss, and adds distillation loss based on the soft label of the teacher model.

[0039] In one feasible implementation, the aforementioned third task submodule is used to perform temporal prediction of user behavior categories, generate frame-level logits using bidirectional cyclic units, and perform sequence decoding using conditional random fields. At the same time, triplet loss is introduced into the behavior embedding space to enhance fine-grained discrimination capability.

[0040] Secondly, this application proposes a screen anti-spy camera monitoring system based on multimodal model fusion, including:

[0041] The data acquisition module is used to acquire datasets from different acquisition devices and different acquisition scenarios. These datasets include video modal data, wireless positioning modal data, ambient light intensity modal data, audio modal data, and peripheral behavior modal data.

[0042] The data augmentation module is used to augment the source data of the above dataset to obtain augmented data;

[0043] The MixNet module is used to extract features from the enhanced data to obtain the corresponding multimodal feature representations.

[0044] An adaptive feature enhancement module is used to encode the above multimodal feature representations modally, suppress redundant information between modalities and highlight significant features related to the surreptitious behavior, so as to obtain optimized fusion features.

[0045] The multi-task learning architecture module is used to simultaneously predict device category, user behavior category, and whether there is any surreptitious filming based on the optimized fusion features mentioned above.

[0046] In summary, this invention proposes a screen anti-spy camera monitoring method based on multimodal model fusion, which has the following beneficial effects: First, this embodiment integrates multi-source information such as video, wireless positioning, ambient light intensity, audio, and peripheral device behavior through a data acquisition module, constructing a more comprehensive input dataset. This enables the acquisition of multi-dimensional features related to user behavior and device usage in different environments, avoiding the limitations of relying on a single modality. Second, the introduction of two enhancement strategies, RotMix and RotCom, during the data augmentation stage not only increases the intra-class diversity of samples within a single modality but also generates new combined samples at the cross-modal and cross-sample levels, thereby improving the robustness and generalization ability of the model under different scenarios and device conditions. Third, this embodiment sets up an adaptive feature enhancement module during feature processing, performing modal-wise encoding, redundancy suppression, and attention weighting on multimodal features. This highlights significant features highly correlated with spy camera behavior while suppressing irrelevant or repetitive information between modalities, thereby improving the discriminative power of feature fusion. Furthermore, by employing a multi-task learning architecture to simultaneously predict device category, user behavior category, and the presence of surreptitious filming, multi-target recognition is achieved within a unified framework. This not only improves the accuracy of surreptitious filming recognition but also provides richer auxiliary information for subsequent protection strategies. This embodiment also introduces imbalanced adjustment loss and fine-grained discrimination constraints during training, further enhancing its adaptability to complex real-world scenarios. Finally, during the inference phase, this invention not only provides the recognition results and confidence levels for surreptitious filming but also performs style-based desensitization processing on suspicious areas of interest on the screen based on the results. This enables immediate protection upon risk detection, preventing the leakage of sensitive information and forming a complete closed loop from detection to protection. In summary, this embodiment, through a series of innovative designs such as multimodal data fusion, data augmentation, feature optimization, and multi-task recognition, significantly improves the accuracy, robustness, and practicality of screen anti-surreptitious filming monitoring, solving the problems of reliance on single-modality methods, insufficient generalization, and lack of real-time protection in existing technologies. Attached Figure Description

[0047] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit this specification. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0048] Figure 1 This is a flowchart illustrating a screen anti-spy camera monitoring method based on multimodal model fusion, provided in an embodiment of this application.

[0049] Figure 2 This is a flowchart illustrating the process of a data acquisition module acquiring a dataset, as provided in an embodiment of this application.

[0050] Figure 3 A flowchart illustrating the training process of a recognition model is provided for an embodiment of this application.

[0051] Figure 4 A flowchart illustrating the process of an adaptive feature enhancement module obtaining optimized fused features, provided in an embodiment of this application;

[0052] Figure 5 This application provides a flowchart illustrating how a redundancy suppression submodule reduces intermodal redundancy information.

[0053] Figure 6 A flowchart illustrating the prediction process of a multi-task learning architecture module provided in an embodiment of this application;

[0054] Figure 7 This is a structural schematic diagram of a screen anti-spy camera monitoring system based on multimodal model fusion, provided as an embodiment of this application. Detailed Implementation

[0055] The terms "first," "second," "third," "fourth," etc. (if present) in the technical solutions of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices. The technical solutions of the embodiments of this application will now be clearly and completely described in conjunction with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0056] Figure 1 This application provides a flowchart illustrating a screen anti-spy camera monitoring method based on multimodal model fusion, which may specifically include:

[0057] S110. Acquire datasets from different acquisition devices and different acquisition scenarios through the data acquisition module. The datasets include video modal data, wireless positioning modal data, ambient light intensity modal data, audio modal data, and peripheral behavior modal data.

[0058] S120. The source data of the above dataset is augmented using the data augmentation module to obtain augmented data;

[0059] S130. Input the enhanced data into the MixNet module for feature extraction to obtain the corresponding multimodal feature representation;

[0060] S140. The above multimodal feature representations are encoded modally by an adaptive feature enhancement module to suppress redundant information between modalities and highlight significant features related to the act of secretly filming, so as to obtain optimized fusion features.

[0061] S150. Based on the above-mentioned optimized fusion features, a multi-task learning architecture module is adopted to simultaneously predict the device category, user behavior category, and whether there is any surreptitious filming behavior.

[0062] For example, in step S110, the system acquires multimodal data from different acquisition devices and different acquisition scenarios through the data acquisition module. Specifically, the video modality is captured by a high-speed camera at a fixed frame rate to capture image sequences; the wireless positioning modality is captured by a UWB or Bluetooth base station to collect distance information between the user and the screen; the ambient light intensity modality is captured by a light sensor to collect changes in ambient brightness; the audio modality is captured by a microphone to collect ambient sound characteristics; and the peripheral behavior modality records user interaction events using the keyboard, mouse, or touchscreen. All the data from these different modalities have a unified timestamp and are divided into several time segments after time alignment processing, with each segment serving as a training or recognition sample.

[0063] In step S120, the sample data enters the data augmentation module. This module includes the RotMix submodule and the RotCom submodule. The RotMix submodule introduces diverse perturbations within a single modality, such as rotating and brightness perturbing video frames, and shifting the time axis or occluding the frequency band of audio signals, thereby generating samples with intra-class differentiation. The RotCom submodule combines multimodal features among different samples, cross-pairing environment-independent modalities with modalities directly related to user behavior, and applying consistency constraints so that the generated combined samples retain behavioral semantics while improving generalization ability across scenarios. After processing by these two submodules, the system obtains augmented multimodal training samples.

[0064] In step S130, the enhanced multimodal data is input into the MixNet module for feature extraction. The MixNet module simultaneously extracts local detail features and global semantic features through convolutional operations at different scales, thereby capturing spatial variations of actions in the video modality, extracting rhythm and frequency information in the audio modality, extracting trend changes in the localization and light intensity modality, and extracting rhythmic patterns of user operations in the peripheral modality. Each modality is then processed uniformly to generate corresponding feature representations.

[0065] In step S140, the feature representation is further input into the adaptive feature enhancement module for processing. This module first performs modal-wise encoding on each modality feature, mapping it to a unified representation space. Subsequently, a redundancy suppression unit reduces overlap and interference between different modalities, making the information between modalities more complementary. Next, a gated attention unit assigns higher weights to features highly relevant to the surreptitious behavior, while assigning lower weights to noisy or irrelevant features. Finally, a fusion unit integrates the weighted features to obtain an optimized fused feature representation.

[0066] In step S150, based on the optimized fusion features, the system employs a multi-task learning architecture module for recognition. This architecture includes a shared feature layer and multiple task sub-modules: the first task sub-module is used to determine whether there is any surreptitious filming behavior in the current segment and to address the class imbalance problem by adjusting the loss function; the second task sub-module is used to predict the category of the surreptitious filming device, such as distinguishing between different types like mobile phones and cameras; the third task sub-module is used to identify the specific category of the user's behavior, such as raising a hand, looking at the screen, or operating peripherals. These multiple sub-tasks are jointly optimized under the same architecture, with the surreptitious filming behavior recognition task given a higher weight to ensure its accuracy is prioritized.

[0067] This invention proposes a screen anti-spy camera monitoring method based on multimodal model fusion, which has the following beneficial effects: First, this embodiment integrates multi-source information such as video, wireless positioning, ambient light intensity, audio, and peripheral device behavior through a data acquisition module, constructing a more comprehensive input dataset. This enables the acquisition of multi-dimensional features related to user behavior and device usage in different environments, avoiding the limitations of relying on a single modality. Second, in the data augmentation stage, two augmentation strategies, RotMix and RotCom, are introduced. This not only increases the intra-class diversity of samples within a single modality but also generates new combined samples at the cross-modal and cross-sample levels, thereby improving the robustness and generalization ability of the model under different scenarios and device conditions. Third, this embodiment sets up an adaptive feature augmentation module in the feature processing process, performing modal-wise encoding, redundancy suppression, and attention weighting on multimodal features. This highlights significant features highly correlated with spy camera behavior while suppressing irrelevant or repetitive information between modalities, thereby improving the discriminative power of feature fusion. Furthermore, by employing a multi-task learning architecture to simultaneously predict device category, user behavior category, and the presence of surreptitious filming, multi-target recognition is achieved within a unified framework. This not only improves the accuracy of surreptitious filming recognition but also provides richer auxiliary information for subsequent protection strategies. This embodiment also introduces imbalanced adjustment loss and fine-grained discrimination constraints during training, further enhancing its adaptability to complex real-world scenarios. Finally, during the inference phase, this invention not only provides the recognition results and confidence levels for surreptitious filming but also performs style-based desensitization processing on suspicious areas of interest on the screen based on the results. This enables immediate protection upon risk detection, preventing the leakage of sensitive information and forming a complete closed loop from detection to protection. In summary, this embodiment, through a series of innovative designs such as multimodal data fusion, data augmentation, feature optimization, and multi-task recognition, significantly improves the accuracy, robustness, and practicality of screen anti-surreptitious filming monitoring, solving the problems of reliance on single-modality methods, insufficient generalization, and lack of real-time protection in existing technologies.

[0068] In one feasible implementation, the data acquisition module includes a RotMix submodule and a RotCom submodule;

[0069] like Figure 2 As shown, step S110 acquires datasets from different acquisition devices and different acquisition scenarios through the data acquisition module, including:

[0070] S1101. The Rot Mix submodule described above performs rotational perturbation and feature mixing within the single modality of the source data in the dataset to generate diverse intra-class samples.

[0071] S1102. Using the RotCom submodule described above, modal combinations are performed between different samples of the source data in the dataset, and cross-modal augmented samples are generated based on consistency constraints.

[0072] S1103. Add the above-mentioned intra-class diversified samples and the above-mentioned cross-modal enhanced samples to the above-mentioned dataset to obtain enhanced data.

[0073] For example, this embodiment provides a set of data acquisition and enhancement procedures that can be directly implemented for steps S1101 to S1103, and provides optional mathematical representations and parameter meanings without limiting the scope of protection, so as to facilitate reproduction and comparison experiments.

[0074] In step S110, the system first acquires multimodal raw data from different acquisition devices and scenarios, and completes alignment and slicing according to a unified time base. (The window length is used as a reference.) (e.g., 4s) and step size (e.g., 0.5s) sliding segmentation to obtain samples With the corresponding tags , respectively, represent whether there is any unauthorized filming, the device category, and the atomic behavior category. Here, video modality is the frame sequence or feature frame within a time segment, UWB / Bluetooth positioning is the distance or angle of arrival time sequence, ambient light intensity is the illuminance time sequence, audio is the spectrogram or frame-level feature, and peripheral behavior is the keyboard / mouse / touch event time sequence.

[0075] In step S1101, the RotMix submodule deforms and blends each sample within a single modality to generate diverse samples within the class, thereby enhancing the robustness of the same class under conditions such as angle, lighting, and noise. For image / video modalities, a "rotation + two-level enhancement + blending" approach can be used.

[0076] ;

[0077] in, This indicates that an angle is applied to the frame or feature map. rotation (such as) ), These are mild and moderate image enhancement operators (such as brightness / contrast perturbation, mild sharpening, Gaussian noise, or blurring). Mixing coefficient (commonly used) For audio modes, a combination of time axis shifting and frequency band blocking can be used, for example:

[0078]

[0079] in It is time shift (such as) ), This involves random frequency band / temporal occlusion (SpecAugment approach). For peripheral behavior modalities, a small phase shift or random frame dropping followed by interpolation can be applied to the event timestamps. The RotMix labels remain unchanged. To encourage consistent representations with respect to rotations / perturbations, a rotation consistency regularization can be added during training:

[0080] ;

[0081] in This is an intermediate representation for the feature extraction network, used to guide the network to learn features that are insensitive to geometric / spectral perturbations.

[0082] In step S1102, the RotCom submodule performs combined enhancements at the cross-sample and cross-modal levels. The aim is to break the coupling between scene appearance and behavioral semantics, allowing the model to learn structured signals that are more relevant to the essence of behavior. Specifically, two samples are selected from the same batch or the same category pool. and Preserve modalities strongly correlated with behavior (such as video, peripherals, and some audio) from the samples. Replace the more environment / scene-oriented modalities (such as UWB distance and ambient light intensity) with the samples. The corresponding modalities are used to obtain combined samples:

[0083] ;

[0084] To maintain semantic consistency, a combinatorial consistency constraint is imposed on the original samples and the combined samples in the fusion representation space:

[0085] ;

[0086] in This refers to the common embedding mapping before and after cross-modal fusion (e.g., the feature vector output by the fusion layer). RotCom's label inheritance rule focuses on maintaining consistency of the main semantics: the label for the act of secretly filming and the label for atomic acts are derived from the sample. inherit( ), the equipment categories are usually also from inherit

[0087] ( This is because the replacement is of environment-related modalities rather than device imaging modalities. To avoid inappropriate combinations, the sample size can be limited. and Satisfying coarse-grained compatibility conditions (such as the same scene type or time period) enhances the realism of the combination.

[0088] In step S1103, the augmented samples generated by RotMix and RotCom are added to the original dataset: the new training set is denoted as...

[0089] ;

[0090] And inherit or generate corresponding labels according to the above rules. During training, augmented samples can be sampled proportionally (e.g., augmented samples: original samples). or and to and Assign weights This forms a joint optimization objective with the loss of the main task, in order to achieve a synergistic effect of "intra-class diversity enhancement + cross-domain generalization enhancement".

[0091] Window length and stride determine timing resolution and coverage (e.g., ). Image / video rotation angle, covering common handheld tilt angles (such as...) ). Two-level enhancement operators, the former with light perturbation and the latter with medium perturbation, are used to construct strong and weak views. Mixing coefficient and shape parameter of Beta distribution The smaller the mixture, the more extreme it tends to be; empirically... . The time shift of an audio / event sequence, used to simulate slight alignment discrepancies or motion fluctuations (such as...). ). These are the intermediate feature extraction mapping and the fusion embedding mapping, respectively, which are implemented by subsequent networks (such as the MixNet backbone and fusion layer). , Consistency-type regularization terms: the former constrains that the representations of the same instance under different rotations / perturbations are similar, while the latter constrains that the representations after cross-sample combination are consistent with the original behavioral samples; corresponding weights. Available in [ Internal optimization. Sampling compatibility conditions: Limit the replacement modality (such as light intensity, distance) to be similar to the main sample in scene type / time period to reduce training noise caused by unreasonable splicing.

[0092] Through the processes described in S1101–S1103, the training set is systematically expanded in both single-modal deformation and cross-modal combination dimensions. RotMix enhances intra-class deformation coverage, making the model more robust to angle, illumination, spectral, and temporal perturbations. RotCom breaks down the binding relationship between scene appearance and behavioral semantics, guiding the model to focus on behavioral structure, thus maintaining a high recognition rate for surreptitious photography even when changing devices and scenes. This data foundation provides a more representative training distribution for subsequent feature extraction, feature enhancement, and multi-task recognition.

[0093] In one feasible implementation, such as Figure 3As shown, during the training of the surreptitious photography behavior recognition model composed of the aforementioned data acquisition module, data augmentation module, Mix Net module, adaptive feature enhancement module, and multi-task learning architecture module, the method further includes:

[0094] S210. Introduce a gradient inversion layer between the feature extractor and the domain discriminator, and make the source domain features and target domain features indistinguishable in the representation space through adversarial training.

[0095] S220. Calculate the difference in covariance matrix between the source domain features and the target domain features, and optimize it based on the relevant alignment loss function to reduce the difference in their second-order statistical distributions.

[0096] S230. During batch normalization, the mean and variance parameters of the source and target domains are maintained respectively to achieve adaptive adjustment of feature distribution under different devices and different scenarios.

[0097] For example, in this embodiment, for the surreptitious photography behavior recognition model composed of a data acquisition module, a data augmentation module, a MixNet module, an adaptive feature augmentation module, and a multi-task learning architecture module, three complementary "cross-domain distribution alignment" mechanisms are introduced during the training phase, corresponding to steps S210 to S230. During training, a mixed mini-batch approach is adopted, mixing samples from the source domain (already labeled datasets from existing devices and scenes) and the target domain (new devices / new scenes, typically unlabeled or weakly labeled datasets) in a certain proportion to form batch data. Through joint optimization, the model learns feature representations that are insensitive to domain changes but highly discriminative of whether or not surreptitious photography has occurred.

[0098] First, in step S210, adversarial domain confusion training is employed: the fused features after adaptive feature enhancement are denoted as... ,in This represents a feature extractor consisting of a MixNet backbone and an adaptive feature enhancement module, with parameters as follows: A domain discriminator is then connected in series. The output sample is the probability that it comes from the source domain. To achieve a forward-invariant, backward-adversarial training mechanism, in and A gradient inversion layer (GRL) is inserted between these layers. This layer performs an identity mapping during forward propagation and, during backward propagation, multiplies the gradients back from the domain discriminator to the feature extractor by a uniform multiplication factor. The training objective employs adversarial loss based on binary cross-entropy: encouraging the input of source domain features. A score of 1 indicates encouragement for the target domain features. The result is set to 0; however, under the influence of GRL, the feature extractor is forced to learn to make the features from the two domains indistinguishable in the representation space. It can be adjusted according to the training progress. (Normalized number of steps from start to finish) uses an incremental schedule, such as , To increase the steepness hyperparameter (which can be taken) Early-stage weak adversarial interaction and later-stage strong adversarial interaction contribute to stable convergence. Domain discriminator It can consist of two to three fully connected layers, coupled with layer normalization and leaky activation to prevent the discriminator from becoming too strong and causing training oscillations. Optionally, it can be used to... Introduce slight label smoothing or gradient penalty to suppress overfitting.

[0099] Next, in step S220, statistical distribution alignment is used to refine the second-order statistical consistency between the two domains. Let the feature set of the source domain in a batch be... The target domain is In the feature dimension The covariance matrix of the two domains is calculated, and the difference in their Frobenius norms is used as the relevant alignment loss: .in The features were obtained from the decentralized features within the batch. This represents the norm of the square root of the sum of squares of the matrix elements. To account for first-order statistics, mean alignment (minimizing the Euclidean distance between the means of features from two domains) can be added to the same branch. When a stronger distribution matching is required, maximum mean difference (MMD) loss can be introduced in parallel: in the Gaussian kernel... Calculate the distance and bandwidth between two domains in the regenerating kernel Hilbert space. The median heuristic can be adaptively set based on the median distance between sample pairs within a batch. This statistical alignment term complements the adversarial term of S210: the adversarial term makes the two domains globally confoundable, while the statistical term gradually brings the two domains closer in terms of covariance and kernel mean, thus effectively reducing distributional differences without relying on the target domain label.

[0100] Subsequently, in step S230, domain-aware statistical reestimation is performed on the normalization layer of the network. Specifically, for each batch normalization layer (BatchNorm), two independent sets of operational statistics and learnable affine parameter source domains are maintained. With the target domain During training, if the current mini-batch comes from the source domain, then update the mean and variance of the source domain and use... Normalization is performed; if the data originates from the target domain, a set of parameters from the target domain is updated and used. This explicitly models the difference in activation distribution between the two domains, avoiding the bias caused by a single statistic. To improve adaptability to unknown scenarios during deployment, a brief statistical reestimation of the target domain's real-time data is performed during the inference phase (e.g., online estimation using exponential moving averages), or based on the "target domain confidence score" output by the domain discriminator. Adaptive weighted fusion of the two sets of statistics: .in The probability output of the acceptable domain discriminator on this input segment, or its temperature-calibrated value, is used. This provides a smoother transition during the initial deployment of new equipment (when the target domain statistics are still unstable).

[0101] Combining the above three steps, the overall training objective is usually expressed as a weighted sum of "task loss + domain alignment loss": .in It is the joint loss of the aforementioned multi-task learning (including Focal loss for surreptitious photography, cross-entropy of device categories, CRF / triples for behavior time sequence, etc.). As a countermeasure to S210, For the covariance alignment term of S220, For kernel mean alignment; For the corresponding weight, the value is generally taken as... The magnitude of the algorithm is significant and can be piecewise or cosine-adjusted based on training progress or validation set performance. Optimization employs alternating updates: one forward and one backward pass simultaneously updates the feature extractor and task head. The domain discriminator can be updated using a multi-step discriminator / generator update rhythm to enhance adversarial stability. The learning rate and weight decay can be achieved using AdamW combined with linear warmup and cosine annealing. To prevent adversarial instability from causing a decrease in main task accuracy, it is recommended to enable only statistical alignment in the first few rounds, gradually increasing it only after the main task converges to a stable range. and .

[0102] Through adversarial obfuscation in S210, statistical alignment in S220, and domain-aware normalization in S230, this implementation significantly reduces the representational differences between the source and target domains without requiring target domain labels or with only a very small number of weak labels. This enables the model to stably output high-confidence judgments on the presence or absence of surreptitious filming under new devices, new perspectives, and new lighting conditions, while maintaining the reliability of device category and user behavior recognition.

[0103] In one feasible implementation, the adaptive feature enhancement module includes a modal coding submodule, a redundancy suppression submodule, a gated attention submodule, and a fusion submodule.

[0104] like Figure 4 As shown, step S140 uses an adaptive feature enhancement module to perform modal-wise encoding on the above multimodal feature representation, suppressing redundant information between modalities and highlighting significant features related to the surreptitious photography behavior, in order to obtain optimized fusion features, including:

[0105] S1401. The modal features are mapped to feature vectors of a uniform dimension through the above modal coding submodule.

[0106] S1402. By applying orthogonal constraints and mutual information minimization constraints to the feature vectors of different modes through the above-mentioned redundancy suppression submodule, the redundancy information between modes is reduced.

[0107] S1403. By assigning weight coefficients to the feature vectors of each modality of the above-mentioned gated attention submodule, the features related to the surreptitious behavior are weighted up and irrelevant features are weakened.

[0108] S1404. The weighted modal features are summed by the fusion submodule to generate the optimized fusion features mentioned above.

[0109] In one feasible implementation, such as Figure 5 The above-described step S1402 applies orthogonal constraints and mutual information minimization constraints between feature vectors of different modes through the redundancy suppression submodule, thereby reducing inter-modal redundancy information, including:

[0110] S14021. Apply orthogonality constraints to the feature vectors of different modes, and minimize the cosine similarity square term to make the feature vectors approximately orthogonal in the representation space, so as to improve the complementarity between modes.

[0111] S14022. Apply mutual information minimization constraints to the feature vectors of different modes, calculate and minimize the mutual information between modes based on the Hilbert-Schmidt independence criterion, so as to reduce redundant information between modes.

[0112] S14023. The above orthogonal constraints and mutual information minimization constraints are weighted and combined, and the weight coefficients are dynamically adjusted during training to reduce redundancy while maintaining the ability to identify surreptitious behavior.

[0113] For example, in this embodiment, the adaptive feature enhancement module operates continuously according to S1401 to S1404, and is used to complete modal encoding, cross-modal redundancy suppression, gated weighting, and efficient fusion at a unified time and feature scale. Assume that within the same time window there are... There are *n* modalities, and the input feature sequence for each modality is denoted as […]. ,in The time steps for this window, The modal input dimension.

[0114] In S1401, each mode first passes through its dedicated encoder. Representations are extracted (including 1D convolutions and residual blocks), and then linearly projected onto a common dimension. ,get

[0115]

[0116] in, For pointwise nonlinearity (such as ReLU / GELU), LN is for layer normalization. If a modality's sampling is sparser, then align it to a uniform time grid (linear interpolation or downsampling) so that the time dimension of all modes is equal. .

[0117] In step S1402, to reduce cross-modal redundancy without sacrificing discriminative power, this embodiment employs a global vector and dual constraints approach. First, pooling is performed on each mode in the time dimension to obtain the global vector. Pooling can be done using average pooling.

[0118]

[0119] in and For attention parameters.

[0120] In step S14021, for any two modalities' global vectors, minimize the squared term of the cosine similarity:

[0121] ;

[0122] This approach encourages the separation of different modalities in the representation space, reducing redundant information and thereby improving the complementarity and fusion efficiency between modalities.

[0123] In step S14022, the Hilbert-Schmidt independence criterion (HSIC) is used to estimate and weaken the statistical dependencies between different modes within a batch. Let the batch size be... Modality The Gram kernel matrix is

[0124] Its elements consist of a Gaussian kernel:

[0125] ;

[0126] The calculation yielded: For a centered matrix, then

[0127] ;

[0128] By minimizing This can effectively reduce cross-modal redundant statistical dependencies. Kernel bandwidth The "median distance heuristic" is adopted to adaptively set the median Euclidean distance of sample pairs within a batch, balancing stability and computational cost.

[0129] In step S14023, the total loss for redundancy suppression is:

[0130] ;

[0131] And set it to follow the training progress Dynamically increasing weights:

[0132]

[0133] To avoid excessively strong regularization in the early stages interfering with the convergence of the main task, a weak-to-strong approach is adopted. If necessary, gradient normalization (e.g., adaptive scaling based on the norm of each gradient term) can be added to balance the influence of the two regularization terms.

[0134] The formula parameters are explained as follows: The number of time steps is determined by the window duration and the sampling rate; These are the modal input dimension and the unified common dimension, respectively. The structure and parameters of the modal encoder; The parameters and weights for attention pooling; Batch size; For modality The kernel matrix; Gaussian kernel bandwidth; Static / maximum weights for redundancy suppression regularization; To normalize the training progress (e.g., number of iterations / total number of iterations).

[0135] In step S1403, the global vectors of all modalities are concatenated into a context. The unnormalized threshold value is obtained through two layers of perceptron. Then, the modal weights are obtained using temperature-controlled Softmax.

[0136] ;

[0137] in Control the sharpness of the weight distribution. For each time step... ,Will After the broadcast and Multiplication yields a weighted sequence When a certain modality is temporarily missing, a visibility mask is introduced.

[0138] And renormalize the weights to This ensures the robustness of the reasoning. For unnormalized gate values, For the first Attention weights for each modality; Temperature coefficient; This is a modal visibility mask.

[0139] In step S1404, the modal weighted sequence is aggregated time-by-time along the time dimension to obtain optimized fusion features:

[0140] ;

[0141] Understandably, it is possible Overlay a lightweight timing modeling layer (such as a single layer) Self-attention and attention pooling are then used to generate fragment vectors. , as input to the downstream multitasking head.

[0142] The adaptive feature enhancement module is jointly trained end-to-end with the downstream task, and the comprehensive loss is:

[0143]

[0144] in The joint loss for multi-task learning includes Focal loss for whether the photo was taken secretly, cross-entropy of device type, and sequence loss of user behavior. This is a gated sparse regularized weight (to suppress ineffective multimodal data while avoiding overfitting).

[0145] Through unified encoding in S1401, orthogonal and HSIC dual redundancy removal in S1402, gated attention focusing in S1403, and efficient fusion in S1404, this implementation reduces cross-modal repetition and noise coupling, while adaptively highlighting modalities and times highly related to the surreptitious behavior. Thus, without increasing inference latency, it significantly improves the discriminative power, robustness, and generalizability of the fused features.

[0146] In one feasible implementation, the above-mentioned multi-task learning architecture module includes a bottom shared feature layer, a first task sub-module, a second task sub-module, and a third task sub-module.

[0147] The above step S150, as follows Figure 6 As shown, based on the optimized fusion features described above, a multi-task learning architecture module is used to simultaneously predict device category, user behavior category, and whether there is any surreptitious filming behavior, including:

[0148] S1501. Receive the optimized fusion features through the aforementioned underlying shared feature layer and generate a shared representation;

[0149] S1502. Based on the shared representation, the first task submodule predicts whether there is any surreptitious filming behavior.

[0150] S1503. Predict the device category based on the shared representation using the second task submodule described above;

[0151] S1504. Predict user behavior categories based on the shared representation through the third task submodule; wherein, the optimization objective of the multi-task learning architecture is a joint loss function, which is a weighted combination of Focal Loss, cross-entropy loss and triplet loss.

[0152] In one feasible implementation, the first task submodule is used to obtain a surreptitious behavior discrimination score by fusion of a fragment-level fully connected network and a temporal-level one-dimensional convolution based on the shared representation. The discrimination score is mapped to a probability by a Sigmoid and optimized with Focal Loss with an imbalance adjustment factor.

[0153] In one feasible implementation, the second task submodule is used to perform hierarchical prediction of equipment categories. The second task submodule first determines the major equipment category and then the fine-grained model. It introduces label smoothing and class balance terms weighted by the number of effective samples per class into the cross-entropy loss, and adds distillation loss based on the soft label of the teacher model.

[0154] In one feasible implementation, the aforementioned third task submodule is used to perform temporal prediction of user behavior categories, generate frame-level logits using bidirectional cyclic units, and perform sequence decoding using conditional random fields. At the same time, triplet loss is introduced into the behavior embedding space to enhance fine-grained discrimination capability.

[0155] For example, step S150 uses a multi-task learning architecture with a shared base and three task sub-modules to jointly predict whether there is surreptitious filming, the device category, and the user behavior category. Let the optimized fusion feature obtained after adaptive feature enhancement be... , For time steps, This represents the unified feature dimensions. First, the underlying shared feature layer... Temporal modeling and aggregation are performed to obtain a shared representation, which is then fed into three task sub-modules for output, and end-to-end training is performed using joint loss.

[0156] In S1501, the underlying shared feature layer pairs Time coding is performed to obtain (Enc()• can be a single-layer bidirectional GRU, TCN, or timing Transformer, (for the encoded dimension), while attention pooling is used to generate fragment-level shared vectors.

[0157]

[0158] in for At any moment The vector, and For attention parameters, Interpretable time weights. Used for sequential submodules Used for fragment-level discrimination and device classification.

[0159] In S1502, the first task submodule outputs a binary classification result based on the shared representation to determine whether surreptitious filming has occurred. To balance segment-level and time-series-level criteria, two parallel branches are used:

[0160] Fragment-level branch pairs Score is obtained by performing a fully connected mapping. ; This represents the global device feature representation vector obtained through attention-weighted or temporal pooling operations, with dimension 1. This is the weight matrix for the coarse-grained equipment classification layer, with a size of [size missing]. ,in The number of coarse classification categories is used to map device features to the coarse classification space. This is the bias vector corresponding to the linear mapping, used to correct the baseline position of the feature mapping output. φ(•) represents the nonlinear activation function, used to enhance the nonlinear expressive power of the model. This represents the output layer weight vector, used to further map the nonlinearly transformed features into a scalar output. This is the bias term for the final classification layer, used to adjust the output balance of the predicted scores.

[0161] Time-level branch pairs Apply one-dimensional convolution (kernel size) The score is obtained by combining global max pooling.

[0162] .

[0163] H represents the time series feature matrix obtained by the preceding feature extraction module, and its dimension is 1. Where T is the number of time steps, For feature dimensions. Indicates input features The result of performing a one-dimensional convolution operation along the time axis, with a kernel size of k, is used to extract local temporal features. The represents a non-linear activation function, which can be ReLU or tanh, used to enhance the model's ability to express non-linear features; GMP(•) represents Global Max Pooling, used to extract the most significant global temporal features from the convolution results; The weight vector of the behavior branch output layer is used to map the pooled features to the scalar space; This is the corresponding bias term, used to adjust the baseline value of the output score; The output score for this branch represents the prediction result of the existence of the target behavior, which is subsequently mapped to the probability of behavior existence using the Sigmoid function. It is used for binary classification.

[0164] Two-way fractions with coefficients merge into

[0165] The probability is mapped to Sigmoid. During training, FocalLoss with imbalance adjustment is used:

[0166] ;

[0167] in The true label indicating whether or not the film was taken secretly. For positive and negative sample weights, Focusing factor for difficult examples; temperature scaling can be used during inference. Perform confidence level calibration and use a threshold decision making.

[0168] In S1503, the second task submodule performs hierarchical prediction of device categories. First, it performs... Classify the equipment into coarse-grained categories to obtain Then in the selected major category The following fine-grained model classification is performed to obtain .enter This represents the global device feature vector obtained after attention-weighted or temporal pooling operations, with dimension d; The weight matrix of the fine-grained classification layer is represented by the gating vector. After modulation, dynamically adjustable mapping weights are formed, where The gate vector is used to adjust the weight allocation of the current fine-classification features based on the coarse classification results of the previous layer, thereby achieving hierarchical feature classification. This indicates the corresponding bias term, also represented by... Modulation is used to correct the center position of the output distribution during dynamic mapping; softmax(•) represents the probability normalization function, which is used to map the linear output to the probability distribution of each sub-category; This is the fine-grained device category prediction probability vector output by the model, with dimension . ,in This represents the number of categories for the fine-grained classification. Both the coarse and fine-grained levels employ cross-entropy loss, and label smoothing is introduced into the target distribution. To mitigate overfitting; for class imbalance, a weighted average based on the number of effective samples is used. ( For category The number of samples, Simultaneously, knowledge distillation loss is introduced to enhance the generalization of the new device: assuming the teacher model is at a certain temperature... The following soft targets are Student output is ,but:

[0169] ;

[0170] Overall equipment task losses ,in and For weights.

[0171] In S1504, the third task submodule performs time-series prediction of user behavior categories. As input, a bidirectional cyclic unit is used to obtain the context-enhanced sequence. And generate frame-level logits for each time step: To ensure the temporal consistency of the behavioral sequence and the prior transition constraints, a linear chain conditional random field (CRF) is used to model the transition matrix. Negative log-likelihood loss is obtained through maximum likelihood training. The inference phase uses Viterbi decoding to obtain... To enhance fine-grained discrimination capabilities, behavioral embeddings are also constructed on shared vectors. And apply triplet loss to the semi-hard negative samples:

[0172] ;

[0173] in Embeddings for needles, positive samples, and negative samples, respectively. The interval is denoted as . The total loss of this submodule is denoted as . .

[0174] The joint optimization objective of the entire multi-task learning is

[0175] ;

[0176] in This is used to reinforce the primary task of detecting whether someone is secretly filming. The optimizer can be AdamW, with a learning rate employing linear warm-up and cosine annealing. To stabilize the multi-task tradeoffs, uncertainty-weighted or gradient-norm-based adaptive weights are used.

[0177] The meanings of the parameters and symbols in the above formulas are as follows: The number of time steps within the time window. The shared layer input and encoded dimensions; Enc(•) represents the temporal encoder; Non-linear activation; Conv1D of The kernel size; The fusion coefficient between the two branches; For FocalLoss, class weights and focus factors, This is the decision threshold; The label smoothing coefficient. Number of categories; and These are the smoothing factor for class balance weights and the sample size of each class, respectively. The distillation temperature. Weighting for distillation loss; This is the CRF transition matrix; For triplet loss interval, Weights for triplet loss; The weights of the three tasks in the joint objective.

[0178] Through this shared multi-task architecture, the system collaboratively learns three types of objectives within the same representation space: fragment-level surreptitious filming detection is optimized, and device and behavioral tasks provide structured priors and auxiliary supervision, thereby achieving stable, interpretable, and high-precision screen anti-surreptitious filming monitoring in complex scenarios and under new device conditions.

[0179] Secondly, such as Figure 7 As shown, this application proposes a screen anti-spy camera monitoring system based on multimodal model fusion, comprising:

[0180] The data acquisition module 21 is used to acquire datasets from different acquisition devices and different acquisition scenarios. The datasets include video modal data, wireless positioning modal data, ambient light intensity modal data, audio modal data, and peripheral behavior modal data.

[0181] Data augmentation module 22 is used to augment the source data of the above dataset to obtain augmented data;

[0182] MixNet module 23 is used to extract features from the enhanced data to obtain the corresponding multimodal feature representation;

[0183] The adaptive feature enhancement module 24 is used to perform modal encoding on the above multimodal feature representation, suppress redundant information between modalities and highlight significant features related to the surreptitious behavior, so as to obtain optimized fusion features;

[0184] The multi-task learning architecture module 25 is used to simultaneously predict device category, user behavior category, and whether there is any surreptitious filming behavior based on the above-mentioned optimized fusion features.

[0185] It is understood that this system can also perform the methods described in any of the first aspects.

[0186] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A screen anti-spy camera monitoring method based on multimodal model fusion, characterized in that, include: The data acquisition module acquires datasets from different acquisition devices and different acquisition scenarios. The datasets include video modal data, wireless positioning modal data, ambient light intensity modal data, audio modal data, and peripheral behavior modal data. The data augmentation module is used to augment the source data of the dataset to obtain augmented data. The enhanced data is input into the MixNet module for feature extraction to obtain the corresponding multimodal feature representation; The multimodal feature representation is encoded modally by an adaptive feature enhancement module to suppress redundant information between modalities and highlight significant features related to surreptitious photography, thereby obtaining optimized fusion features. Based on the optimized fusion features, a multi-task learning architecture module is used to simultaneously predict device category, user behavior category, and whether there is any surreptitious filming behavior. The adaptive feature enhancement module includes a modal coding submodule, a redundancy suppression submodule, a gated attention submodule, and a fusion submodule. The step of performing modal-wise encoding on the multimodal feature representation through an adaptive feature enhancement module to suppress redundant information between modalities and highlight significant features related to the surreptitious photography behavior, in order to obtain optimized fusion features, includes: The modal coding submodule maps each modal feature into a feature vector of a uniform dimension. The redundancy suppression submodule applies orthogonal constraints and mutual information minimization constraints to the feature vectors of different modes to reduce redundancy information between modes. By assigning weight coefficients to the feature vectors of each modality in the gated attention submodule, the features related to the act of secretly filming are weighted and irrelevant features are weakened. The weighted modal features are summed by the fusion submodule to generate the optimized fusion features.

2. The screen anti-spy camera monitoring method based on multimodal model fusion according to claim 1, characterized in that, The data acquisition module includes a RotMix submodule and a RotCom submodule; Data sets from different acquisition devices and different acquisition scenarios are acquired through the data acquisition module, including: The RotMix submodule performs rotational perturbation and feature mixing within a single modality of the source data in the dataset to generate in-class diverse samples. The RotCom submodule performs modal combination among different samples of the source data in the dataset and generates cross-modal enhanced samples based on consistency constraints. The intra-class diversified samples and the cross-modal augmented samples are added to the dataset to obtain augmented data.

3. The screen anti-spy camera monitoring method based on multimodal model fusion according to claim 1 or 2, characterized in that, During the training of the covert photography behavior recognition model composed of the data acquisition module, the data augmentation module, the MixNet module, the adaptive feature enhancement module, and the multi-task learning architecture module, the method further includes: A gradient inversion layer is introduced between the feature extractor and the domain discriminator, and adversarial training is used to make the source domain features and the target domain features indistinguishable in the representation space. The difference in covariance matrices between source domain features and target domain features is calculated and optimized based on the relevant alignment loss function to reduce the difference in their second-order statistical distributions. During batch normalization, the mean and variance parameters of the source and target domains are maintained respectively to achieve adaptive adjustment of feature distribution under different devices and scenarios.

4. The screen anti-spy camera monitoring method based on multimodal model fusion according to claim 1, characterized in that, The step of applying orthogonality constraints and mutual information minimization constraints to the feature vectors of different modalities through the redundancy suppression submodule to reduce intermodal redundancy includes: Orthogonality constraints are imposed on feature vectors of different modalities, and the feature vectors are made approximately orthogonal in the representation space by minimizing the cosine similarity square term, thereby improving the complementarity between modalities. Mutual information minimization constraints are imposed on the feature vectors of different modes. The mutual information between modes is calculated and minimized based on the Hilbert-Schmidt independence criterion to reduce redundant information between modes. The orthogonal constraints and the mutual information minimization constraints are weighted and combined, and the weight coefficients are dynamically adjusted during training to reduce redundancy while maintaining the ability to identify surreptitious filming behavior.

5. The screen anti-spy camera monitoring method based on multimodal model fusion according to claim 1, characterized in that, The multi-task learning architecture module includes a bottom shared feature layer, a first task submodule, a second task submodule, and a third task submodule; The method based on the optimized fusion features, employing a multi-task learning architecture module, simultaneously predicts device category, user behavior category, and the presence of surreptitious filming behavior, including: The optimized fusion features are received through the underlying shared feature layer, and a shared representation is generated. The first task submodule predicts whether there is any surreptitious filming based on the shared representation. The second task submodule predicts the device category based on the shared representation; The third task submodule predicts user behavior categories based on the shared representation; wherein the optimization objective of the multi-task learning architecture is a joint loss function, which is a weighted combination of Focal Loss, cross-entropy loss, and triplet loss.

6. The screen anti-spy camera monitoring method based on multimodal model fusion according to claim 5, characterized in that, The first task submodule is used to obtain a surreptitious behavior discrimination score by using a fragment-level fully connected network and a temporal-level one-dimensional convolution in parallel based on the shared representation. The discrimination score is mapped to a probability by a Sigmoid and optimized with Focal Loss with an imbalance adjustment factor.

7. The screen anti-spy camera monitoring method based on multimodal model fusion according to claim 5, characterized in that, The second task submodule is used to perform hierarchical prediction of equipment categories. The second task submodule first determines the major category of equipment and then determines the fine-grained model. It introduces label smoothing and class balance terms weighted by the number of effective samples per class into the cross-entropy loss, and adds distillation loss based on the soft label of the teacher model.

8. The screen anti-spy camera monitoring method based on multimodal model fusion according to claim 5, characterized in that, The third task submodule is used to perform temporal prediction of user behavior categories. It uses bidirectional cyclic units to generate frame-level logits and performs sequence decoding through conditional random fields. At the same time, it introduces triplet loss in the behavior embedding space to enhance fine-grained discrimination capability.

9. A screen anti-spy camera monitoring system based on multimodal model fusion, used to execute the screen anti-spy camera monitoring method based on multimodal model fusion as described in any one of claims 1 to 8, characterized in that, include: The data acquisition module is used to acquire datasets from different acquisition devices and different acquisition scenarios. The datasets include video modal data, wireless positioning modal data, ambient light intensity modal data, audio modal data, and peripheral behavior modal data. The data augmentation module is used to augment the source data of the dataset to obtain augmented data; The MixNet module is used to extract features from the enhanced data to obtain corresponding multimodal feature representations; An adaptive feature enhancement module is used to perform modal-wise encoding on the multimodal feature representation, suppress redundant information between modalities and highlight significant features related to the surreptitious behavior, so as to obtain optimized fusion features; A multi-task learning architecture module is used to simultaneously predict device category, user behavior category, and whether there is any surreptitious filming behavior based on the optimized fusion features.

Images