Screen content detection method and system based on multi-modal large model and storage medium

Abstract

The invention provides a multi-modal large model-based screen content detection method and system and a storage medium, and the method comprises the steps: carrying out the image preprocessing of a screen content image, and obtaining a to-be-detected image and a to-be-detected text; inputting a to-be-detected image into the pre-trained visual detection large model for visual risk detection to obtain a visual risk value, and inputting a to-be-detected text into the pre-trained text detection large model for text risk detection to obtain a text risk value; performing weighted operation on the visual risk value and the text risk value according to the image type of the screen content image to obtain a target risk value; and if the target risk value is greater than the first risk threshold, controlling the target screen to start the virtual private network, and controlling the target screen to display preset content according to the virtual private network. According to the embodiment of the invention, a multi-modal content detection mode combining visual risk detection and text risk detection is adopted, so that the screen content detection accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of data detection technology, and in particular to a screen content detection method, system and storage medium based on a multimodal large model. Background Art

[0002] With the explosive growth of Internet content, the interfaces of public large-screen devices may display illegal content (such as pornography, violence, fraudulent information, etc.) due to domain name hijacking, malicious application installation or user misoperation, affecting the physical and mental health of netizens, especially minors, and also bringing tremendous pressure to network administrators. Therefore, the issue of screen content detection for public large-screen displays has received increasing attention.

[0003] In the existing screen content detection process, text keyword filtering is generally used for detection, resulting in low screen content detection accuracy. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide a screen content detection method, system and storage medium based on a multimodal large model to solve the problem of low screen content detection accuracy in the prior art.

[0005] The embodiment of the present invention is implemented as follows: a screen content detection method based on a multimodal large model, the method comprising:

[0006] Taking a screenshot of the target screen to obtain a screen content image, and performing image preprocessing on the screen content image to obtain an image to be detected and text to be detected;

[0007] Inputting the image to be detected into the pre-trained visual detection model to perform visual risk detection to obtain a visual risk value, and inputting the text to be detected into the pre-trained text detection model to perform text risk detection to obtain a text risk value;

[0008] Acquiring an image type of the screen content image, and performing a weighted operation on the visual risk value and the text risk value according to the image type to obtain a target risk value;

[0009] If the target risk value is greater than a first risk threshold, performing forensic packaging on the screen content image to obtain violation information, and issuing a content warning based on the violation information;

[0010] The target screen is controlled to open a virtual private network, and the target screen is controlled to display preset content according to the virtual private network.

[0011] Preferably, before inputting the image to be detected into the pre-trained visual detection model for visual risk detection, the method further includes:

[0012] Obtaining a visual sample, and inputting the visual sample into the visual detection large model to perform multi-head self-attention mechanism calculation to obtain a sample feature map;

[0013] Performing object recognition on the sample feature map to obtain a feature object, and performing type prediction on the feature object to obtain an object prediction type;

[0014] Loss calculation is performed based on the object prediction type and the visual label of the visual sample to obtain visual loss, and parameters of the visual detection model are updated based on the visual loss until the visual detection model converges, thereby obtaining the pre-trained visual detection large model.

[0015] Preferably, before inputting the text to be detected into the pre-trained text detection model for text risk detection, the method further includes:

[0016] Obtain a text sample, and input the text sample into the text detection model for word segmentation to obtain sample word segmentation;

[0017] Performing vector conversion on the sample word segmentation to obtain a sample vector, and performing feature encoding on the sample vector to obtain a sample encoding feature;

[0018] Performing semantic prediction on the sample coding features to obtain sample semantics, and determining predicted risk words based on the sample semantics;

[0019] Combining the sample semantics to obtain combined semantics, and determining semantically contradictory words based on the combined semantics;

[0020] Loss is calculated based on the predicted risk words, the semantically contradictory words and the text labels of the text samples to obtain text loss, and parameters of the text detection model are updated based on the text loss until the text detection model converges to obtain the pre-trained text detection large model.

[0021] Preferably, obtaining the image type of the screen content image includes:

[0022] When it is detected that there is an overlap of text and images in the screen content image, determining that the image type of the screen content image is an advertisement pop-up type;

[0023] When it is detected that only text information exists in the screen content image, determining that the image type of the screen content image is a text message type;

[0024] In the weighted calculation corresponding to the advertisement pop-up window type, the weighted coefficient of the visual risk value is greater than the weighted coefficient of the text risk value;

[0025] In the weighted calculation corresponding to the SMS type, the weighted coefficient of the text risk value is greater than the weighted coefficient of the visual risk value.

[0026] Preferably, after controlling the target screen to display preset content according to the virtual private network, the method further includes:

[0027] Performing screen locking processing on the target screen and obtaining the display progress of the screen content image;

[0028] Turning on a freezer of a system in the target screen, creating a freeze group according to the freezer, and adding the display process to the freeze group to freeze the process;

[0029] Obtaining the application installation package on the target screen and performing authorization detection on the application installation package;

[0030] If any of the application installation packages fails the authorization check, the application installation package is subjected to risk isolation processing.

[0031] Preferably, after performing a weighted operation on the visual risk value and the text risk value according to the image type to obtain a target risk value, the method further includes:

[0032] If the target risk value is greater than the second risk threshold and less than or equal to the first risk threshold, reducing the screenshot time interval;

[0033] According to the reduced screenshot time interval, return to the step of taking a screenshot of the target screen and subsequent steps until the target risk value is less than or equal to the second risk threshold, or the target risk value is greater than the first risk threshold.

[0034] Preferably, performing forensic packaging on the screen content image to obtain illegal packaging data includes:

[0035] Performing image encapsulation on the screen content image to obtain image encapsulation, and obtaining a timestamp of the screen content image to obtain a risk timestamp;

[0036] Acquire the display process of the screen content image, and acquire the process identifier of the display process to obtain a risk identifier;

[0037] The image package, the target risk value, the risk timestamp and the risk identifier are combined to obtain the illegal package data.

[0038] Another object of an embodiment of the present invention is to provide a screen content detection system based on a multimodal large model, the system comprising:

[0039] A preprocessing module is used to take a screenshot of the target screen to obtain a screen content image, and perform image preprocessing on the screen content image to obtain an image to be detected and text to be detected;

[0040] A risk detection module is configured to input the image to be detected into a pre-trained visual detection model to perform visual risk detection to obtain a visual risk value, and input the text to be detected into a pre-trained text detection model to perform text risk detection to obtain a text risk value;

[0041] a weight adaptation module, configured to obtain an image type of the screen content image, and perform a weighted operation on the visual risk value and the text risk value according to the image type to obtain a target risk value;

[0042] a forensic packaging module, configured to perform forensic packaging on the screen content image to obtain violation information if the target risk value is greater than a first risk threshold, and issue a content warning based on the violation information;

[0043] The active defense module is used to control the target screen to open a virtual private network, and control the target screen to display preset content according to the virtual private network.

[0044] Preferably, the risk detection module is further configured to: obtain visual samples, and input the visual samples into the visual detection large model to perform multi-head self-attention mechanism calculation to obtain a sample feature map;

[0045] Performing object recognition on the sample feature map to obtain a feature object, and performing type prediction on the feature object to obtain an object prediction type;

[0046] Loss calculation is performed based on the object prediction type and the visual label of the visual sample to obtain visual loss, and parameters of the visual detection model are updated based on the visual loss until the visual detection model converges, thereby obtaining the pre-trained visual detection large model.

[0047] The embodiments of the present invention can effectively analyze the visual risk value of the image and the text risk value of the text in the screen content image by performing visual risk detection on the image to be detected and performing text risk detection on the text to be detected. By performing weighted operations on the visual risk value and the text risk value according to the image type, the target risk value can be effectively adaptively adjusted in terms of weight, thereby improving the accuracy of the target risk value. If the target risk value is greater than the first risk threshold, it is determined that there is illegal content in the screen content image. By performing forensic packaging on the screen content image, the illegal screen content image can be effectively collected forensics. By controlling the target screen to display preset content through a virtual private network, the display of illegal content generated by domain name hijacking or malicious applications can be effectively blocked, thereby effectively achieving an active defense effect on illegal content. By adopting a multimodal content detection method that combines visual risk detection and text risk detection, the accuracy of screen content detection is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flowchart of a screen content detection method based on a multimodal large model provided by the first embodiment of the present invention;

[0049] Figure 2 2 is a schematic structural diagram of a screen content detection system based on a multimodal large model provided by a second embodiment of the present invention;

[0050] Figure 3 2 is a schematic diagram of a specific implementation of a screen content detection system based on a multimodal large model provided by a second embodiment of the present invention;

[0051] Figure 4 It is a structural diagram of a terminal device provided by the third embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0053] In order to illustrate the technical solution of the present invention, specific embodiments are provided below.

[0054] Example 1

[0055] See also Figure 1 , is a flow chart of a screen content detection method based on a multimodal large model provided by a first embodiment of the present invention. The screen content detection method based on a multimodal large model can be applied to any device or system. The screen content detection method based on a multimodal large model includes the following steps:

[0056] Step S10, taking a screenshot of the target screen to obtain a screen content image, and performing image preprocessing on the screen content image to obtain an image to be detected and text to be detected;

[0057] The target screen can be captured using real-time triggering. The trigger mechanism can be set to capture the original screen image at a fixed frequency (default 0.5 seconds) or dynamically (when a new window or network fluctuation is detected) to obtain the screen content image. In this step, the GPU hardware accelerator (such as Android SurfaceFlinger) in the target screen can be called to directly read the frame buffer data to obtain the screen content image, thus avoiding malware interception.

[0058] Optionally, in this step, image preprocessing and text extraction are performed on the screen content image in parallel. The screen content image is subjected to image normalization to obtain the image to be detected. The image normalization includes adjusting the size of the screen content image to a specified size (for example, 512×512) and performing normalized pixel value processing to suppress screen glare noise. By adopting the lightweight PaddleOCR engine, all text areas in the screen content image are located, and structured data is output to obtain the text to be detected.

[0059] Step S20: Input the image to be detected into a pre-trained visual detection model to perform visual risk detection to obtain a visual risk value, and input the text to be detected into a pre-trained text detection model to perform text risk detection to obtain a text risk value;

[0060] Among them, visual risk detection is performed by inputting the image to be detected into the pre-trained visual detection model to evaluate the risk probability of the existence of illegal images in the image to be detected, and text risk detection is performed by inputting the text to be detected into the pre-trained text detection model to evaluate the risk probability of the existence of illegal text in the image to be detected.

[0061] Optionally, before inputting the image to be detected into the pre-trained visual detection model for visual risk detection, the method further includes:

[0062] Obtaining a visual sample and inputting the visual sample into the visual detection large model for multi-head self-attention mechanism calculation to obtain a sample feature map; wherein the multi-head self-attention mechanism calculation is performed on the visual sample by the 16-layer Transformer block in the visual detection large model to obtain the sample feature map, which is a multi-scale feature map;

[0063] Performing object recognition on the sample feature graph to obtain a feature object, and performing type prediction on the feature object to obtain an object prediction type; wherein, by performing contour recognition on the sample feature graph, determining the feature object based on the contour recognition result, and based on the object prediction type, it can be effectively judged whether the object corresponding to the object prediction type is a violation object, and the violation object includes gambling tools (dice / chips), etc. Optionally, in this step, the sample feature graph is also analyzed for operational interface anomalies to detect whether there are unauthorized floating windows, interface layout mutations (such as a full-screen advertisement suddenly appearing on a bank APP), and other phenomena;

[0064] Loss calculation is performed based on the object prediction type and the visual label of the visual sample to obtain visual loss, and parameters of the visual detection model are updated based on the visual loss until the visual detection model converges, thereby obtaining the pre-trained visual detection large model.

[0065] Furthermore, before inputting the text to be detected into the pre-trained text detection model for text risk detection, the method further includes:

[0066] Obtain a text sample, and input the text sample into the text detection model for word segmentation to obtain sample word segmentation;

[0067] Performing vector conversion on the sample word segmentation to obtain a sample vector, and performing feature encoding on the sample vector to obtain a sample encoding feature; wherein, the sample word segmentation is mapped into a 128-dimensional vector through the embedding layer in the text detection large model to obtain a sample vector, and the sample vector is feature encoded through a 6-layer Transformer encoder in the text detection large model to obtain a sample encoding feature;

[0068] Performing semantic prediction on the sample coding features to obtain sample semantics, and determining predicted risk words based on the sample semantics; wherein, calculating the feature similarity between the sample coding features and the specified features, determining the semantics corresponding to the maximum feature similarity as the sample semantics, and determining predicted risk words in the sample segmentation based on the semantic coding of the sample semantics, wherein the predicted risk words include words corresponding to gambling inducement, pornographic suggestion, or fraudulent rhetoric;

[0069] The sample semantics are combined to obtain a combined semantics, and a semantically contradictory word is determined based on the combined semantics; wherein the combined semantics is matched with a semantic contradiction list, and if the combined semantics successfully matches the contradiction list, the combined semantics is determined to be a semantically contradictory word, and a preset semantically contradictory word exists in the semantic contradiction list;

[0070] Loss is calculated based on the predicted risk words, the semantically contradictory words and the text labels of the text samples to obtain text loss, and parameters of the text detection model are updated based on the text loss until the text detection model converges to obtain the pre-trained text detection large model.

[0071] Step S30, obtaining the image type of the screen content image, and performing a weighted operation on the visual risk value and the text risk value according to the image type to obtain a target risk value;

[0072] Among them, weighted calculation of visual risk value and text risk value by image type can effectively play the effect of weight adaptive adjustment of target risk value, thereby improving the accuracy of target risk value.

[0073] Optionally, obtaining the image type of the screen content image includes:

[0074] When it is detected that there is an overlap of text and images in the screen content image, determining that the image type of the screen content image is an advertisement pop-up type;

[0075] When it is detected that only text information exists in the screen content image, determining that the image type of the screen content image is a text message type;

[0076] In the weighted calculation corresponding to the ad pop-up type, the weighting coefficient of the visual risk value is greater than the weighting coefficient of the text risk value. For example, when image and text overlap (such as ad pop-up) is detected, the weighting coefficient α of the visual risk value is set to 0.6, and the weighting coefficient β of the text risk value is set to 0.4 (focusing on visual evidence);

[0077] In the weighted calculation corresponding to the SMS type, the weighting coefficient of the text risk value is greater than the weighting coefficient of the visual risk value. For example, when it is detected that the text exists independently (such as SMS), the weighting coefficient α of the visual risk value is set to 0.4, and the weighting coefficient β of the text risk value is set to 0.6 (focusing on semantic analysis).

[0078] Step S40: If the target risk value is greater than a first risk threshold, the screen content image is forensically packaged to obtain violation information, and a content warning is issued based on the violation information;

[0079] Among them, the first risk threshold can be set according to needs. If the target risk value is greater than the first risk threshold, it is determined that there is illegal content in the screen content image. By forensically packaging the screen content image, the illegal screen content image can be effectively collected.

[0080] Optionally, after performing a weighted operation on the visual risk value and the text risk value according to the image type to obtain a target risk value, the method further includes:

[0081] If the target risk value is greater than the second risk threshold and less than or equal to the first risk threshold, the screenshot interval is reduced; wherein, if the target risk value is greater than the second risk threshold and less than or equal to the first risk threshold, it is determined that the content displayed on the target screen needs to be verified twice, the second risk threshold can be set as needed, and the second risk threshold is less than the first risk threshold. The screenshot interval is used to control the interval length of image capture of the target screen;

[0082] According to the reduced screenshot time interval, return to the step of taking a screenshot of the target screen and subsequent steps until the target risk value is less than or equal to the second risk threshold, or the target risk value is greater than the first risk threshold.

[0083] Furthermore, the screen content image is forensically packaged to obtain illegal packaged data, including:

[0084] Performing image encapsulation on the screen content image to obtain image encapsulation, and obtaining a timestamp of the screen content image to obtain a risk timestamp;

[0085] Obtaining a display process of the screen content image and a process identifier of the display process to obtain a risk identifier; wherein the risk identifier is used to locate a corresponding malicious application;

[0086] Combining the image package, the target risk value, the risk timestamp, and the risk identifier to obtain the violation package data;

[0087] Among them, the illegally packaged data will be pushed through multiple channels, for example, through local pop-up windows on the device, remote API reporting (enterprise security platform), or SMS backup channels (sent through baseband when the network is disconnected).

[0088] Step S50, controlling the target screen to open a virtual private network, and controlling the target screen to display preset content according to the virtual private network;

[0089] Among them, by turning on a virtual private network, domain name hijacking can be effectively bypassed, and the target screen can be controlled through the virtual private network to display preset content. It can effectively block the display of illegal content generated by domain name hijacking or malicious applications, and effectively achieve the effect of active defense against illegal content.

[0090] Optionally, after controlling the target screen to display preset content according to the virtual private network, the method further includes:

[0091] Performing screen locking processing on the target screen and obtaining the display process of the screen content image; wherein, by performing screen locking processing on the target screen, a physical intervention effect on the target screen is achieved, thereby preventing malicious applications from tampering with the currently displayed preset content;

[0092] Activating a freezer of the system in the target screen, creating a freeze group according to the freezer, and adding the display process to the freeze group to freeze the process; wherein, by adding the display process to the freeze group to freeze the process, suspicious processes can be effectively frozen to prevent the suspicious processes from tampering with screen content;

[0093] Obtaining an application installation package in the target screen and performing an authorization check on the application installation package; wherein the authorization check on the application installation package is performed to scan unauthorized applications in the target screen;

[0094] If any of the application installation packages fails the authorization check, the application installation package is subjected to risk isolation processing.

[0095] Furthermore, in this embodiment, a continuous learning mechanism is provided, which sets the incorrectly identified screen content images marked by the user as false alarm samples, and fine-tunes the parameters of the pre-trained visual detection large model and the pre-trained text detection model based on the false alarm samples to achieve a continuous learning effect of the model, thereby improving the accuracy of screen content detection. Furthermore, a dynamic threshold adjustment mechanism is provided, which can automatically calibrate the first risk threshold according to the scenario of the target screen display content. For example, in the financial content scenario, the first risk threshold is set lower.

[0096] In this embodiment, by performing visual risk detection on the image to be detected and text risk detection on the text to be detected, the visual risk value of the image and the text risk value of the text in the screen content image can be effectively analyzed. By performing weighted operations on the visual risk value and the text risk value according to the image type, the target risk value can be effectively adaptively adjusted in terms of weight, thereby improving the accuracy of the target risk value. If the target risk value is greater than the first risk threshold, it is determined that there is illegal content in the screen content image. By performing forensic packaging on the screen content image, the illegal screen content image can be effectively collected forensics. By controlling the target screen to display preset content through a virtual private network, the display of illegal content generated by domain name hijacking or malicious applications can be effectively blocked, thereby effectively achieving an active defense effect on illegal content. By adopting a multimodal content detection method that combines visual risk detection and text risk detection, the accuracy of screen content detection is improved.

[0097] Example 2

[0098] See also Figure 2, is a schematic structural diagram of a screen content detection system 100 based on a multimodal large model provided in a second embodiment of the present invention, comprising:

[0099] The preprocessing module 10 is used to take a screenshot of the target screen to obtain a screen content image, and perform image preprocessing on the screen content image to obtain an image to be detected and a text to be detected.

[0100] The risk detection module 11 is used to input the image to be detected into the pre-trained visual detection model for visual risk detection to obtain a visual risk value, and input the text to be detected into the pre-trained text detection model for text risk detection to obtain a text risk value.

[0101] Optionally, the risk detection module 11 is further configured to: obtain a visual sample, and input the visual sample into the visual detection large model to perform multi-head self-attention mechanism calculation to obtain a sample feature map;

[0102] Performing object recognition on the sample feature map to obtain a feature object, and performing type prediction on the feature object to obtain an object prediction type;

[0103] Loss calculation is performed based on the object prediction type and the visual label of the visual sample to obtain visual loss, and parameters of the visual detection model are updated based on the visual loss until the visual detection model converges, thereby obtaining the pre-trained visual detection large model.

[0104] Furthermore, the risk detection module 11 is further configured to: obtain a text sample, and input the text sample into the text detection model for word segmentation to obtain a sample word segmentation;

[0105] Performing vector conversion on the sample word segmentation to obtain a sample vector, and performing feature encoding on the sample vector to obtain a sample encoding feature;

[0106] Performing semantic prediction on the sample coding features to obtain sample semantics, and determining predicted risk words based on the sample semantics;

[0107] Combining the sample semantics to obtain combined semantics, and determining semantically contradictory words based on the combined semantics;

[0108] Loss is calculated based on the predicted risk words, the semantically contradictory words and the text labels of the text samples to obtain text loss, and parameters of the text detection model are updated based on the text loss until the text detection model converges to obtain the pre-trained text detection large model.

[0109] The weight adaptation module 12 is configured to obtain the image type of the screen content image, and perform a weighted operation on the visual risk value and the text risk value according to the image type to obtain a target risk value.

[0110] Optionally, the weight adaptive module 12 is further configured to: when detecting that there is an overlap of text and images in the screen content image, determine that the image type of the screen content image is an advertisement pop-up type;

[0111] When it is detected that only text information exists in the screen content image, determining that the image type of the screen content image is a text message type;

[0112] In the weighted calculation corresponding to the advertisement pop-up window type, the weighted coefficient of the visual risk value is greater than the weighted coefficient of the text risk value;

[0113] In the weighted calculation corresponding to the SMS type, the weighted coefficient of the text risk value is greater than the weighted coefficient of the visual risk value.

[0114] The evidence collection and packaging module 13 is configured to perform evidence collection and packaging on the screen content image if the target risk value is greater than a first risk threshold, obtain violation information, and issue a content warning based on the violation information.

[0115] Optionally, the evidence collection and packaging module 13 is further configured to: if the target risk value is greater than a second risk threshold and less than or equal to the first risk threshold, reduce the screenshot time interval;

[0116] According to the reduced screenshot time interval, return to the step of taking a screenshot of the target screen and subsequent steps until the target risk value is less than or equal to the second risk threshold, or the target risk value is greater than the first risk threshold.

[0117] Furthermore, the evidence encapsulation module 13 is further configured to: perform image encapsulation on the screen content image to obtain an image encapsulation, and obtain a timestamp of the screen content image to obtain a risk timestamp;

[0118] Acquire the display process of the screen content image, and acquire the process identifier of the display process to obtain a risk identifier;

[0119] The image package, the target risk value, the risk timestamp and the risk identifier are combined to obtain the illegal package data.

[0120] The active defense module 14 is used to control the target screen to open a virtual private network, and control the target screen to display preset content according to the virtual private network.

[0121] Optionally, the active defense module 14 is further configured to: lock the target screen and obtain the display process of the screen content image;

[0122] Turning on a freezer of a system in the target screen, creating a freeze group according to the freezer, and adding the display process to the freeze group to freeze the process;

[0123] Obtaining the application installation package on the target screen and performing authorization detection on the application installation package;

[0124] If any of the application installation packages fails the authorization check, the application installation package is subjected to risk isolation processing.

[0125] See also Figure 3 , start the security detection service, capture the target screen, obtain the screen content image, store the screen content image in a multi-frame buffer pool, perform multi-module reasoning on the screen content image in the multi-frame buffer pool, obtain the visual risk value and the text risk value, perform weighted operation on the visual risk value and the text risk value according to the image type, obtain the target risk value, compare the target risk value with the first risk threshold to determine whether the screen content image contains any illegal content, if the screen content image contains any illegal content, start active defense, lock the target screen and issue an alarm, and generate a security log, if the screen content image is normal, clean up the cache of the screen content image.

[0126] In this embodiment, by performing visual risk detection on the image to be detected and text risk detection on the text to be detected, the visual risk value of the image and the text risk value of the text in the screen content image can be effectively analyzed. By performing weighted operations on the visual risk value and the text risk value according to the image type, the target risk value can be effectively adaptively adjusted in terms of weight, thereby improving the accuracy of the target risk value. If the target risk value is greater than the first risk threshold, it is determined that there is illegal content in the screen content image. By performing forensic packaging on the screen content image, the illegal screen content image can be effectively collected forensics. By controlling the target screen to display preset content through a virtual private network, the display of illegal content generated by domain name hijacking or malicious applications can be effectively blocked, thereby effectively achieving an active defense effect on illegal content. By adopting a multimodal content detection method that combines visual risk detection and text risk detection, the accuracy of screen content detection is improved.

[0127] Example 3

[0128] Figure 4 This is a block diagram of a terminal device 2 provided in the third embodiment of the present application. Figure 4As shown, the terminal device 2 of this embodiment includes: a processor 20, a memory 21, and a computer program 22 stored in the memory 21 and executable on the processor 20, such as a program for a method for detecting screen content based on a multimodal large model. When the processor 20 executes the computer program 22, the steps of each embodiment of the method for detecting screen content based on a multimodal large model are implemented.

[0129] Exemplarily, the computer program 22 may be divided into one or more modules, which are stored in the memory 21 and executed by the processor 20 to implement the present application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program 22 in the terminal device 2. The terminal device may include, but is not limited to, a processor 20 and a memory 21.

[0130] The processor 20 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0131] The memory 21 may be an internal storage unit of the terminal device 2, such as a hard disk or memory of the terminal device 2. The memory 21 may also be an external storage device of the terminal device 2, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 2. Furthermore, the memory 21 may include both an internal storage unit of the terminal device 2 and an external storage device. The memory 21 is used to store the computer program and other programs and data required by the terminal device. The memory 21 may also be used to temporarily store data that has been output or is about to be output.

[0132] In addition, the functional modules in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0133] If the integrated module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Among them, the computer-readable storage medium can be non-volatile or volatile. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable storage medium may include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in computer-readable storage media can be appropriately increased or decreased according to the requirements of legislation and patent practices in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practices, computer-readable storage media do not include electrical carrier signals and telecommunications signals.

[0134] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A screen content detection method based on a multimodal large model, characterized in that: The method comprises: Taking a screenshot of the target screen to obtain a screen content image, and performing image preprocessing on the screen content image to obtain an image to be detected and text to be detected; Inputting the image to be detected into the pre-trained visual detection model to perform visual risk detection to obtain a visual risk value, and inputting the text to be detected into the pre-trained text detection model to perform text risk detection to obtain a text risk value; Acquiring an image type of the screen content image, and performing a weighted operation on the visual risk value and the text risk value according to the image type to obtain a target risk value; If the target risk value is greater than a first risk threshold, performing forensic packaging on the screen content image to obtain violation information, and issuing a content warning based on the violation information; The target screen is controlled to open a virtual private network, and the target screen is controlled to display preset content according to the virtual private network.

2. The screen content detection method based on a multimodal large model according to claim 1, wherein: Before inputting the image to be detected into the pre-trained visual detection model for visual risk detection, the method further includes: Obtaining a visual sample, and inputting the visual sample into the visual detection large model to perform multi-head self-attention mechanism calculation to obtain a sample feature map; Performing object recognition on the sample feature map to obtain a feature object, and performing type prediction on the feature object to obtain an object prediction type; Loss calculation is performed based on the object prediction type and the visual label of the visual sample to obtain visual loss, and parameters of the visual detection model are updated based on the visual loss until the visual detection model converges, thereby obtaining the pre-trained visual detection large model.

3. The screen content detection method based on a multimodal large model according to claim 1, characterized in that: Before inputting the to-be-detected text into the pre-trained text detection model for text risk detection, the method further includes: Obtain a text sample, and input the text sample into the text detection model for word segmentation to obtain sample word segmentation; Performing vector conversion on the sample word segmentation to obtain a sample vector, and performing feature encoding on the sample vector to obtain a sample encoding feature; Performing semantic prediction on the sample coding features to obtain sample semantics, and determining predicted risk words based on the sample semantics; Combining the sample semantics to obtain combined semantics, and determining semantically contradictory words based on the combined semantics; Loss is calculated based on the predicted risk words, the semantically contradictory words and the text labels of the text samples to obtain text loss, and parameters of the text detection model are updated based on the text loss until the text detection model converges to obtain the pre-trained text detection large model.

4. The screen content detection method based on a multimodal large model according to claim 1, wherein: Obtaining the image type of the screen content image includes: When it is detected that there is an overlap of text and images in the screen content image, determining that the image type of the screen content image is an advertisement pop-up type; When it is detected that only text information exists in the screen content image, determining that the image type of the screen content image is a text message type; In the weighted calculation corresponding to the advertisement pop-up window type, the weighted coefficient of the visual risk value is greater than the weighted coefficient of the text risk value; In the weighted calculation corresponding to the SMS type, the weighted coefficient of the text risk value is greater than the weighted coefficient of the visual risk value.

5. The screen content detection method based on a multimodal large model according to claim 1, wherein: After controlling the target screen to display preset content according to the virtual private network, the method further includes: Performing screen locking processing on the target screen and obtaining the display progress of the screen content image; Turning on a freezer of a system in the target screen, creating a freeze group according to the freezer, and adding the display process to the freeze group to freeze the process; Obtaining the application installation package on the target screen and performing authorization detection on the application installation package; If any of the application installation packages fails the authorization check, the application installation package is subjected to risk isolation processing.

6. The screen content detection method based on a multimodal large model according to claim 1, wherein: After performing a weighted operation on the visual risk value and the text risk value according to the image type to obtain a target risk value, the method further includes: If the target risk value is greater than the second risk threshold and less than or equal to the first risk threshold, reducing the screenshot time interval; According to the reduced screenshot time interval, return to the step of taking a screenshot of the target screen and subsequent steps until the target risk value is less than or equal to the second risk threshold, or the target risk value is greater than the first risk threshold.

7. The screen content detection method based on a multimodal large model according to claim 1, wherein: The screen content image is forensically packaged to obtain illegal packaged data, including: Performing image encapsulation on the screen content image to obtain image encapsulation, and obtaining a timestamp of the screen content image to obtain a risk timestamp; Acquire the display process of the screen content image, and acquire the process identifier of the display process to obtain a risk identifier; The image package, the target risk value, the risk timestamp and the risk identifier are combined to obtain the illegal package data.

8. A screen content detection system based on a multimodal large model, characterized in that: The system comprises: A preprocessing module is used to take a screenshot of the target screen to obtain a screen content image, and perform image preprocessing on the screen content image to obtain an image to be detected and text to be detected; A risk detection module is configured to input the image to be detected into a pre-trained visual detection model to perform visual risk detection to obtain a visual risk value, and input the text to be detected into a pre-trained text detection model to perform text risk detection to obtain a text risk value; a weight adaptive module, configured to obtain an image type of the screen content image, and perform a weighted operation on the visual risk value and the text risk value according to the image type to obtain a target risk value; a forensic packaging module, configured to perform forensic packaging on the screen content image to obtain violation information if the target risk value is greater than a first risk threshold, and issue a content warning based on the violation information; The active defense module is used to control the target screen to open a virtual private network, and control the target screen to display preset content according to the virtual private network.

9. The screen content detection system based on a multimodal large model according to claim 8, characterized in that: The risk detection module is also used to: Obtaining a visual sample, and inputting the visual sample into the visual detection large model to perform multi-head self-attention mechanism calculation to obtain a sample feature map; Performing object recognition on the sample feature map to obtain a feature object, and performing type prediction on the feature object to obtain an object prediction type; Loss calculation is performed based on the object prediction type and the visual label of the visual sample to obtain visual loss, and parameters of the visual detection model are updated based on the visual loss until the visual detection model converges, thereby obtaining the pre-trained visual detection large model.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images