Semantically constrained multimodal personality feature anti-detection method and device
By extracting frames and text information on the original video, combining multimodal personality detection model and large language model, a high-natural resistance personality detection adversarial sample is generated, which solves the problem of privacy protection of personality characteristics in the existing technology and realizes multiple Effective privacy protection for modal personality detection.
Patent Information
- Application Number
- CN202411102426.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-12
AI Technical Summary
Existing multimodal personality detection technology is difficult to effectively protect the privacy of personality traits, especially on video platforms, which cannot effectively generate high-natural resistance to personality detection and adversarial samples, resulting in personality traits being easily leaked and utilized.
Using a sequence annotation method, visual adversarial samples are generated by drawing frames on the original video, and text adversarial samples with semantic similarity are generated by combining the LLM model to generate anti-detection video data, and pre-trained multimodal personality detection is used. The model and DAN/D-DGCN network perform sample quality evaluation and iterative updates to generate multimodal fusion information.
Effectively retain the semantic information of the video, generate high-natural resistance to personality detection and adversarial samples, improve the privacy protection of personality traits, and is suitable for multimodal personality detection scenarios, and enhance the privacy and security of video personality detection.
Smart Images

Figure CN118627015B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of personality trait anti-detection, and in particular to a semantically constrained multimodal personality trait anti-detection method and device. Background Art
[0002] Personality, as a person's high-level privacy, contains deep information such as individual character traits, emotional tendencies, and behavioral patterns. It is one of the most sensitive and private personal data. In the Internet environment, due to personal privacy protection, the threshold for obtaining basic personal information is high, and the impact is relatively direct and limited. As a high-level privacy, personality traits are hidden in all the information released by individuals, and the threshold for obtaining them is relatively low. They not only contain the deep psychological characteristics of individuals, but also can accurately predict a person's actions and decisions through behavioral patterns and emotional tendencies, and are of extremely high value. In recent years, with the continuous development of Internet technology, social platforms and video platforms have gradually emerged. According to statistics from CNNIC, Mob Research Institute and BACKLINKO, by 2023, the scale of Internet users in my country will be 1.092 billion, and the proportion of short video users has reached 95.2%; by 2024, there will be 4.74 billion active video users among the world's 8.06 billion people; and there will be more than 4.95 billion online social media users. The super-large number of users has created a massive amount of data from Internet applications such as social platforms and video platforms, which contain rich personality traits.
[0003] A series of multimodal personality detection studies have shown that: based on massive video data from the Internet, people's apparent personality is analyzed through their movements, facial changes, language and other features, and personality privacy is easily leaked and exploited. In the field of apparent personality analysis, related detection can detect the personality traits of users based on multimodal information such as pictures, text and audio. Related research can analyze individual behaviors, such as facial expressions, voice features, body language, text content and social interaction patterns, to predict personality traits, analyze individual thinking patterns, personality, emotional state and social relationships. Through multimodal personality detection technology, attackers can easily obtain the personality characteristics of the target object, which seriously threatens the privacy security of personality. Existing anti-detection methods for videos also ignore the attention to personality characteristics, and it is difficult to migrate and apply them to the defense process of multimodal personality detection, which brings huge challenges to the privacy protection of video personality detection.
[0004] At present, multimodal personality detection technology basically relies on deep neural network structures. Some studies have shown that in models based on deep neural networks, network parameters cannot completely cover the entire semantic information space, and the mapping between input and output of parameter learning is not continuous. By constructing adversarial samples containing slight perturbations, the results may have a greater impact on the results. These adversarial samples as perturbations provide an important technical basis for personality protection. However, existing multimodal anti-detection methods for videos have ignored the focus on personality characteristics and cannot be directly transferred to the multimodal personality privacy protection process. The exposure points of personality characteristics in different modalities are different, and the perturbations for different modalities are also different. Targeted privacy protection is required for different modalities, which further increases the difficulty of fusing different modalities and is difficult to apply to the defense process of multimodal personality detection. At the same time, it brings huge challenges to the privacy protection of video personality detection.
[0005] Therefore, how to retain the semantics of the original video, generate anti-personality detection adversarial samples of different modalities, synthesize generated videos that are resistant to personality detection, and improve the naturalness of the generated videos are important issues facing the field of anti-personality detection of social network videos. Summary of the invention
[0006] Based on this, it is necessary to provide a method, device and computer equipment for locating web page text based on sequence annotation to address the above technical problems.
[0007] A semantically constrained multimodal personality feature anti-detection method, the method comprising:
[0008] Extract frames from the original video to obtain picture frames, generate adversarial frames and mixed frames of the picture frames, and generate visual adversarial samples from the picture frames, adversarial frames and mixed frames;
[0009] The audio stream in the original video is recognized to obtain text information, and according to the language styles corresponding to different personality characteristics contained in the text information, the LLM model is guided to generate text adversarial samples with semantic similarity to the text information;
[0010] The text adversarial sample and the visual adversarial sample are fused to obtain anti-detection video data.
[0011] In one of the embodiments, it also includes: fusing the picture frame and the text information to obtain multimodal fusion information; inputting the multimodal fusion information into a pre-trained multimodal personality detection model, and outputting the personality characteristics corresponding to the characters in the original video data.
[0012] In one of the embodiments, the method further includes: inputting the picture frame into a pre-trained DAN network, comparing the personality characteristics output by the multimodal personality detection model with the personality characteristics output by the DAN network, and evaluating the sample quality of the picture frame.
[0013] In one of the embodiments, the method further includes: inputting the text information into a D-DGCN network, comparing the personality features output by a multimodal personality detection model with the personality features output by the D-DGCN network, and evaluating the sample quality of the text information.
[0014] In one of the embodiments, it also includes: for picture frames whose sample quality meets preset conditions, using the MI-FGSM algorithm and the DAN network to perform T rounds of iterative updates to generate adversarial frames and mixed frames, and generating visual adversarial samples from the picture frames, adversarial frames and mixed frames.
[0015] In one of the embodiments, it also includes: generating a prompt template according to system instructions, knowledge prompts and task requirements; guiding the LLM model to generate text adversarial samples that are semantically similar to the text information according to the prompt template, a preset semantic similarity detection algorithm and a preset quality threshold.
[0016] In one of the embodiments, the method further includes: fusing the text adversarial sample and the visual adversarial sample according to a pre-trained SyncNet network to obtain anti-detection video data.
[0017] A semantically constrained multimodal personality feature anti-detection device, the device comprising:
[0018] A visual adversarial sample generation module is used to extract frames from the original video to obtain picture frames, generate adversarial frames and mixed frames of the picture frames, and generate visual adversarial samples from the picture frames, adversarial frames and mixed frames;
[0019] A text adversarial sample generation module is used to identify the audio stream in the original video to obtain text information, and guide the LLM model to generate text adversarial samples with semantic similarity to the text information according to the language styles corresponding to different personality characteristics contained in the text information;
[0020] The anti-detection video generation module is used to fuse the text adversarial sample and the visual adversarial sample to obtain anti-detection video data.
[0021] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a method when executing the computer program:
[0022] Extract frames from the original video to obtain picture frames, generate adversarial frames and mixed frames of the picture frames, and generate visual adversarial samples from the picture frames, adversarial frames and mixed frames;
[0023] The audio stream in the original video is recognized to obtain text information, and according to the language styles corresponding to different personality characteristics contained in the text information, the LLM model is guided to generate text adversarial samples with semantic similarity to the text information;
[0024] The text adversarial sample and the visual adversarial sample are fused to obtain anti-detection video data.
[0025] The above-mentioned semantically constrained multimodal personality trait anti-detection method and device are divided into two parts: visual personality detection adversarial sample generation and text personality detection adversarial sample generation. When generating visual personality detection adversarial samples, visual adversarial samples are generated by extracting frames from the original video, generating adversarial frames, and synthesizing mixed frames; when generating text personality detection adversarial samples, a prompt text framework of system roles, knowledge prompts, and task descriptions is constructed to guide the large language model to understand the language styles of different personality traits, and stimulate the LLM to generate adversarial samples under semantic constraints. Through the iterative generation of the detection model during adversarial sample generation and the fine-tuning of indicators after the final adversarial sample generation, the error in the adversarial sample generation process is reduced, and adversarial samples with better effects are obtained. Thereby, a video with multimodal personality trait anti-detection is generated. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 is a schematic flow chart of a semantically constrained multimodal personality feature anti-detection method in one embodiment;
[0027] Figure 2 A pseudo code schematic diagram of generating adversarial samples for visual personality detection in one embodiment;
[0028] Figure 3 A flowchart of optimizing prompt text based on semantic similarity according to an embodiment;
[0029] Figure 4 A pseudo code schematic diagram of a prompt text optimization strategy based on semantic detection in one embodiment;
[0030] Figure 5 is a structural block diagram of a semantically constrained multimodal personality feature anti-detection device in one embodiment;
[0031] Figure 6 The figure is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0032] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0033] In one embodiment, if Figure 1 As shown, a semantically constrained multimodal personality feature anti-detection method is provided, comprising:
[0034] Step 102, extract frames from the original video to obtain picture frames, generate adversarial frames and mixed frames of the picture frames, and generate visual adversarial samples by combining the picture frames, adversarial frames and mixed frames.
[0035] Step 104, the audio stream in the original video is recognized to obtain text information, and the LLM model is guided to generate text adversarial samples with semantic similarity to the text information according to the language styles corresponding to different personality traits contained in the text information.
[0036] Step 106: fuse the text adversarial sample and the visual adversarial sample to obtain detection-resistant video data.
[0037] The above-mentioned semantically constrained multimodal personality trait anti-detection method is divided into two parts: visual personality detection adversarial sample generation and text personality detection adversarial sample generation. When generating visual personality detection adversarial samples, visual adversarial samples are generated by extracting frames from the original video, generating adversarial frames, and synthesizing mixed frames; when generating text personality detection adversarial samples, a prompt text framework of system roles, knowledge prompts, and task descriptions is constructed to guide the large language model to understand the language style of different personality traits and stimulate the LLM to generate adversarial samples under semantic constraints. Through the iterative generation of the detection model during adversarial sample generation and the fine-tuning of indicators after the final adversarial sample generation, the error in the adversarial sample generation process is reduced, and adversarial samples with better effects are obtained. Thus, a video with multimodal personality trait anti-detection is generated.
[0038] In one embodiment, a video file generally includes two main parts: a video stream and an audio stream. A video stream is a series of pictures (frames) played in a specific order to create a dynamic image effect. An audio stream is a series of sound signals played synchronously with the video stream to provide sound effects and dialogues. For the processing of video files with multimodal attributes, after considering extracting its different modes, the modes it has are processed separately according to different methods of different modes. For the video stream of the input video, all its frames are extracted, and the video stream is separated into a picture stream; for the audio stream, the audio attribute of the video file is used to extract the sound part in the video, and for the extracted audio stream, its language content and language type are identified, and the audio stream is converted into a text stream. Through the separation processing of the video stream and the audio stream, the multimodal input video with multiple features is separated into pictures and texts that are easy to process, and the processing of the video stream and the audio stream is indirectly realized by processing the pictures and texts, thereby realizing the processing of the video file integrating multiple features.
[0039] In one of the embodiments, in the process of generating visual and textual personality privacy detection adversarial samples, in order to obtain better results, it is first necessary to evaluate the effect of the generated samples based on the visual and textual personality detection model and provide feedback, and select the generated samples with better effects through multiple rounds of iterations. In addition, in the process of generating text adversarial samples, in order to ensure semantic constraints, it is necessary to introduce an evaluation system based on semantic similarity and discard text generation samples whose semantic similarity does not meet the requirements. Therefore, in order to obtain better visual and textual personality privacy adversarial samples, first of all, better visual, textual personality detection models and semantic similarity evaluation systems are required.
[0040] In one of the embodiments, the image frame and the text information are fused to obtain multimodal fusion information; the multimodal fusion information is input into a pre-trained multimodal personality detection model to output the personality characteristics corresponding to the characters in the original video data.
[0041] Specifically, the multimodal personality assessment task can be formalized as a video-based multi-feature regression problem. From a mathematical point of view, given a user-generated video Contains three modes: visual flow , audio stream , and text streams ,in represents the sequence length, represents the embedding dimension. The goal of the model is to score the user based on multimodal signals of The strength of personality traits.
[0042] Based on this, the model uses the PCENet model, which simultaneously models modality and feature cues. PCENet consists of two parts: a multimodal hierarchical decoupled encoder that decomposes the modality and obtains stable and variable representations of each modality; and a feature-guided decoder that applies feature correlation learning to guide model optimization.
[0043] Finally, the output of the last layer of the decoder is Used to estimate personality through a linear layer and a sigmoid layer:
[0044] ;
[0045] in Represents the transformation matrix of the assessment component, with an output dimension of 4 (MBTI personality trait dimensions).
[0046] In one of the embodiments, the picture frame is input into a pre-trained DAN network, and the sample quality of the picture frame is evaluated by comparing the personality features output by the multimodal personality detection model with the personality features output by the DAN network.
[0047] Specifically, the visual personality detection task is formally defined as follows: given a user's unimodal video ,in , that is, a frame, which is classified as the true label .
[0048] In order to perform image processing, images need to be extracted from the video. However, if all images are extracted, the computational cost and memory cost will be very high, and considering that the video has spatial continuity and close frames are very similar, downsampling is used to extract images from the video, and about 100 images are extracted from each video.
[0049] The model uses the description aggregation network DAN, which has three parts in total.
[0050] In the first part, we use the convolutional neural network (CNN) to extract features from the image. After processing, the feature vector is
[0051] ;
[0052] In the second part, the feature vector is pooled using the global maximum pooling layer and the global average pooling layer. Feature screening is performed to obtain and , and go through After regularization, they are merged into a single visual feature .
[0053] ;
[0054] In the third part, the fully connected layer FC and Sigmoid activation function are used to Classify and obtain the predicted probability of each personality category, and select the category with the maximum probability as the predicted value.
[0055] ;
[0056] During training, the loss function is defined as the mean squared error:
[0057] ;
[0058] in, To predict the label, is the true label.
[0059] In another embodiment, the text information is input into the D-DGCN network, and the sample quality of the text information is evaluated by comparing the personality features output by the multimodal personality detection model with the personality features output by the D-DGCN network.
[0060] Specifically, a graph-based dynamic deep graph convolutional network (D-DGCN) is used to implement the multi-classification task of personality traits. A graph is constructed to represent the user's posts, where each post is represented as a node and initialized by embedding the pre-trained BERT model. In addition, a special node representing the user is added to facilitate the classification of personality traits. In addition, considering that the connection between two nodes is established non-deterministically, a multi-hop-based learning connection (L2C) module is added. This module has a differentiable threshold function and can automatically build basic edges layer by layer to better adapt and expand to different samples. In view of the problem that traditional graph neural networks (GCNs) cannot stack multiple layers due to over-smoothing problems, a deep graph convolutional network (DGCN) is applied to collect useful information from posts in a larger receptive field to remedy smoothing.
[0061] In one of the embodiments, for picture frames whose sample quality meets preset conditions, the MI-FGSM algorithm and the DAN network are used to perform T rounds of iterative updates to generate adversarial frames and mixed frames, and the picture frames, adversarial frames and mixed frames are combined to generate visual adversarial samples.
[0062] Specifically, the original video is divided into different frames. For each frame, the MI-FGSM algorithm and the visual personality detection model are used to perform T rounds of iterative updates. Then all updated frames are synthesized into video adversarial samples. , use the downsampling method to extract the sequence equally . Let the personality characteristics of the video be , the personality characteristics of the adversarial generated samples are , the perturbation size is , the number of iterations is , the attenuation coefficient is , the loss function of DAN is , use the MI-FGSM algorithm for each frame of image Perform iterative update: Let the update step length be , direction vector Initialized to 0. Pseudo code is as follows Figure 2 As shown. The updated frame sequence , and the original video Synthesize and obtain visual adversarial samples .
[0063] In one of the embodiments, a prompt template is generated according to system instructions, knowledge prompts and task requirements; based on the prompt template, a preset semantic similarity detection algorithm and a preset quality threshold, the LLM model is guided to generate text adversarial samples that are semantically similar to the text information.
[0064] Specifically, we use the advantages of LLM in the field of text generation and use knowledge-enhanced prompts to improve the performance of the model in few-shot or zero-shot learning scenarios. By designing prompts based on personality semantics, we guide LLM to generate corresponding adversarial texts while keeping the semantics of the original input text unchanged.
[0065] There are usually three roles in the prompt text: System, Assistant, and User:
[0066] The User role refers to the user itself, and the content is the information entered by the user;
[0067] The Assistant role refers to the big language model, and the content is the answer of the big language model;
[0068] The System role is equivalent to a special User and appears at the beginning of the Prompt.
[0069] According to the attention mechanism of the large language model, the beginning and ending sentences are more likely to be valued. Therefore, the system generally defines the role of the large language model and inputs some basic and security-enhancing instructions.
[0070] The optimization process of prompt text based on semantic similarity is as follows Figure 3As shown in the figure, the main part of the prompt text consists of three parts: system instructions, knowledge prompts, and task requirements, which respectively define the role of LLM, provide knowledge, and describe the task, so as to standardize the input of LLM and improve the understanding of the task by the large language model. At the same time, the user input information is adjusted to the specified format and the results are output in the specified format.
[0071] The prompt text optimization technology for adversarial sample evaluation is a technology further developed on the basis of adversarial text generation. Its core goal is to maximize the potential of LLM. The present invention designs an evaluation index based on semantic constraints, aiming to accurately extract and analyze the subtle differences between the generated adversarial samples and the original text, and further constrain the semantic features of the generated text.
[0072] The present invention uses four detection methods, namely cosine distance, LSA, Word2Vec and IDA, to quantitatively evaluate the semantic similarity of texts. The specific process pseudo code is as follows: Figure 4 shown.
[0073] In one of the embodiments, text adversarial samples and visual adversarial samples are fused according to a pre-trained SyncNet network to obtain anti-detection video data.
[0074] Specifically, after completing the frame, text, and voice modification, the lip movements of the video need to be modified and synthesized to better match the audio and avoid the problem of mismatch between voice and image movements. The Wav2Lip model is used to synthesize the image. The specific implementation is divided into two stages:
[0075] Training the lip shape expert model SyncNet
[0076] SyncNet Input One A window of consecutive facial frames with only the lower half and a size of Voice clips ,in and are the time steps of the video and audio, respectively. The audio is obtained by randomly sampling an audio window that is aligned with the video (in sync) or from a different time step (out of sync) To distinguish the synchronization between audio and video, SyncNet is trained. It consists of a face encoder and an audio encoder, both of which consist of a series of 2D convolutions. The embedding vectors generated by these encoders are calculated distance, and train the model to maximize the boundary loss to minimize the distance between synchronized pairs.
[0077] Generate accurate lip sync by learning from lip sync experts
[0078] Based on an accurate lip sync discriminator SyncNet, it penalizes the generator for inaccurate generation during training.
[0079] Generator It consists of three blocks: (i) identity encoder, (ii) speech encoder, and (iii) face decoder. The identity encoder is a series of residual convolutional layers that train on random reference frames. Encode, with posture prior (The lower half of the target face is covered) connected along the channel axis. The speech encoder is also a series of 2D convolutional layers that are applied to the input speech segment. The decoder is also a series of convolutional layers, with transposed convolutions for upsampling.
[0080] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0081] In one embodiment, Figure 5 As shown, a semantically constrained multimodal personality feature anti-detection device is provided, comprising: a visual adversarial sample generation module 502, a text adversarial sample generation module 504 and an anti-detection video generation module 506, wherein:
[0082] A visual adversarial sample generation module 502 is used to extract frames from the original video to obtain picture frames, generate adversarial frames and mixed frames of the picture frames, and generate visual adversarial samples from the picture frames, adversarial frames and mixed frames;
[0083] A text adversarial sample generation module 504 is used to identify the audio stream in the original video to obtain text information, and guide the LLM model to generate text adversarial samples with semantic similarity to the text information according to the language styles corresponding to different personality characteristics contained in the text information;
[0084] The anti-detection video generation module 506 is used to fuse the text adversarial sample and the visual adversarial sample to obtain anti-detection video data.
[0085] For the specific definition of the semantically constrained multimodal personality trait anti-detection device, please refer to the definition of the semantically constrained multimodal personality trait anti-detection method above, which will not be repeated here. Each module in the above-mentioned semantically constrained multimodal personality trait anti-detection device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0086] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for locating the body of a web page based on sequence annotation is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.
[0087] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0088] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.
[0089] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0090] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0091] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A semantically constrained multimodal personality feature anti-detection method, characterized in that: The method comprises: Extract frames from the original video to obtain picture frames, generate adversarial frames and mixed frames of the picture frames, and generate visual adversarial samples from the picture frames, adversarial frames and mixed frames; The audio stream in the original video is recognized to obtain text information, and according to the language styles corresponding to different personality characteristics contained in the text information, the LLM model is guided to generate text adversarial samples with semantic similarity to the text information; Fusing the text adversarial sample and the visual adversarial sample to obtain anti-detection video data; The text adversarial sample and the visual adversarial sample are fused to obtain anti-detection video data, including: The text adversarial sample and the visual adversarial sample are fused according to a pre-trained SyncNet network to obtain anti-detection video data; The method further comprises: fusing the picture frame and the text information to obtain multimodal fusion information; Inputting the multimodal fusion information into a pre-trained multimodal personality detection model, and outputting personality characteristics corresponding to the characters in the original video; The method further comprises: The picture frame is input into a pre-trained DAN network, and the sample quality of the picture frame is evaluated by comparing the personality characteristics output by the multimodal personality detection model with the personality characteristics output by the DAN network.
2. The method according to claim 1, characterized in that The method further comprises: The text information is input into the D-DGCN network, and the personality features output by the multimodal personality detection model are compared with the personality features output by the D-DGCN network to evaluate the sample quality of the text information.
3. The method according to claim 1, characterized in that Generating an adversarial frame and a mixed frame of the picture frame, and generating a visual adversarial sample from the picture frame, the adversarial frame and the mixed frame, including: For picture frames whose sample quality meets the preset conditions, the MI-FGSM algorithm and the DAN network are used to perform T rounds of iterative updates to generate adversarial frames and mixed frames, and the picture frames, adversarial frames and mixed frames are combined to generate visual adversarial samples.
4. The method according to claim 2, characterized in that: According to the language styles corresponding to the different personality traits contained in the text information, guiding the LLM model to generate text adversarial samples with semantic similarity to the text information, including: Generate prompt templates based on system instructions, knowledge prompts and task requirements; According to the prompt template, the preset semantic similarity detection algorithm and the preset quality threshold, the LLM model is guided to generate text adversarial samples that are semantically similar to the text information.
5. A semantically constrained multimodal personality feature anti-detection device, characterized in that: The device comprises: A visual adversarial sample generation module is used to extract frames from the original video to obtain picture frames, generate adversarial frames and mixed frames of the picture frames, and generate visual adversarial samples from the picture frames, adversarial frames and mixed frames; A text adversarial sample generation module is used to identify the audio stream in the original video to obtain text information, and guide the LLM model to generate text adversarial samples with semantic similarity to the text information according to the language styles corresponding to different personality characteristics contained in the text information; An anti-detection video generation module, used for fusing the text adversarial sample and the visual adversarial sample to obtain anti-detection video data; The anti-detection video generation module is further used to fuse the text adversarial sample and the visual adversarial sample according to the pre-trained SyncNet network to obtain anti-detection video data; The text adversarial sample generation module is also used to fuse the picture frame and the text information to obtain multimodal fusion information; input the multimodal fusion information into a pre-trained multimodal personality detection model to output the personality characteristics corresponding to the person in the original video; The text adversarial sample generation module is also used to input the picture frame into a pre-trained DAN network, compare the personality features output by the multimodal personality detection model with the personality features output by the DAN network, and evaluate the sample quality of the picture frame.
