Application program security control method and device, equipment, medium and product

Abstract

The invention discloses an application program safety control method and device, equipment, a medium and a product. The method comprises the steps of obtaining a screenshot image and corresponding user interface structure information when a user operates a current application program; based on the screenshot image and the user interface structure information, using a preset first large language model to perform user behavior analysis to determine whether the user has a high-risk behavior; and if the user has the high-risk behavior, triggering safety early warning. According to the invention, the high-risk behavior of the user can be accurately identified and early warned, and the risk identification precision and adaptability are effectively improved.

Description

Technical Field

[0001] This application relates to the field of information security technology, and in particular to an application security control method, apparatus, device, medium and product. Background Technology

[0002] Currently, mobile app security control methods typically employ static blacklist-based prevention techniques to combat fraudulent activities. This mechanism establishes a list or blacklist of financially relevant apps; when a user launches these apps, the system automatically triggers corresponding preventative measures, such as two-factor authentication, automatic blocking, or alerts. However, existing technologies have the following drawbacks: Existing static prevention mechanisms rely on pre-set rule bases, and the static nature of these rule bases leads to high false positive rates and significant risks of missed detections; existing solutions are based on fixed scripts and rule matching, lacking dynamic evolution capabilities and unable to cope with rapidly changing fraud methods. Therefore, existing app security technologies have low risk identification accuracy and lack adaptability. Summary of the Invention

[0003] To address the above technical issues, this application provides an application security control method, apparatus, device, medium, and product. Through real-time dynamic analysis, it can accurately identify and issue warnings for high-risk user behaviors, effectively improving the accuracy and adaptability of risk identification.

[0004] This application provides an application security control method, including: Obtain screenshots of the user interacting with the current application and the corresponding user interface structure information; Based on the screenshot image and the user interface structure information, a preset first language model is used to perform user behavior analysis in order to determine whether the user has high-risk behavior. If the user exhibits high-risk behavior, a security alert will be triggered.

[0005] As an improvement to the above solution, before obtaining the screenshot image of the user operating the current application and the corresponding user interface structure information, the method further includes: Based on the user interface text of the current application, it is confirmed that the current application is a financial application.

[0006] As an improvement to the above solution, the step of confirming that the current application is a financial application based on the user interface text of the current application includes: Get the text of the current application's user interface; Based on a pre-set financial keyword library, keyword matching is performed on the user interface text to obtain keyword matching results; Based on the keyword matching results, determine whether the current application is a financial application; if so, confirm that the current application is a financial application.

[0007] As an improvement to the above solution, obtaining the user interface text of the current application includes: Retrieve the Extensible Markup Language (XML) file corresponding to the user interface hierarchy of the current application; Based on the XML file, extract the text information of the user interface elements to obtain the user interface text.

[0008] As an improvement to the above solution, the step of performing keyword matching on the user interface text based on a preset financial keyword library to obtain keyword matching results includes: The user interface text is matched with keywords in the financial keyword library using regular expressions to obtain the regular expression matching results. The user interface text is semantically matched with the keywords to obtain semantic matching results; The keyword matching result is determined based on the regular expression matching result and the semantic matching result.

[0009] As an improvement to the above solution, the step of performing regular expression matching between the user interface text and keywords in the financial keyword library to obtain the regular expression matching result includes: Construct a regular expression based on the keywords in the aforementioned financial keyword library; The user interface text is traversed according to the regular expression to perform text matching and obtain the regular expression matching result.

[0010] As an improvement to the above solution, the step of semantically matching the user interface text with the keywords to obtain semantic matching results includes: Convert the keywords in the financial keyword library and the user interface text into word vectors; The word vectors corresponding to the keywords are semantically compared with the word vectors corresponding to the user interface text to obtain semantic matching results.

[0011] As an improvement to the above solution, the step of determining whether the current application is a financial application based on the keyword matching result includes: If the keyword matching result is successful, then obtain the current page information of the current application; Based on the current page information, the category of the current application is determined using a preset second language model, and the category includes financial and non-financial applications.

[0012] As an improvement to the above solution, the current page information includes: the package name of the currently running application and the user interface structure information of the current page.

[0013] As an improvement to the above scheme, the training process of the second major language model includes: Retrieve historical page data from financial and non-financial applications; Based on the reinforcement learning framework, the historical page data is used as training data to train the basic large language model, thereby obtaining the second large language model.

[0014] As an improvement to the above solution, the step of performing user behavior analysis based on the screenshot image and the user interface structure information using a preset first language model to determine whether the user exhibits high-risk behavior includes: Based on the user interface structure information, extract user interface elements and user interaction events; Based on a preset prompt template, input data is constructed according to the screenshot image, the user interface elements, and the user interaction events; The input data is fed into a preset first large language model to identify the user's high-risk behavior; wherein, the first large language model is a multimodal large language model.

[0015] As an improvement to the above solution, after triggering the security warning, the method further includes: Pause the current user's operation and verify the user's identity; When identity verification is successful, the system determines whether the user belongs to the preset target group based on the user's identity information; if not, the user's operation continues; if yes, the system enters the collaborative verification process to initiate a remote confirmation request to the preset associated users for security verification.

[0016] This application also provides an application security control device, including: The data acquisition module is used to acquire screenshots of the user's operation of the current application and the corresponding user interface structure information; The risk identification module is used to perform user behavior analysis based on the screenshot image and the user interface structure information, using a preset first language model, to determine whether the user has high-risk behavior. The security alert module is used to trigger a security alert if the user engages in high-risk behavior.

[0017] Furthermore, the device also includes: The application identification module is used to identify whether the current application is a financial application based on the user interface text of the current application.

[0018] Furthermore, the application identification module is specifically used for: Get the text of the current application's user interface; Based on a pre-set financial keyword library, keyword matching is performed on the user interface text to obtain keyword matching results; Based on the keyword matching results, determine whether the current application is a financial application; if so, confirm that the current application is a financial application.

[0019] Furthermore, obtaining the user interface text of the current application includes: Retrieve the Extensible Markup Language (XML) file corresponding to the user interface hierarchy of the current application; Based on the XML file, extract the text information of the user interface elements to obtain the user interface text.

[0020] Furthermore, the keyword matching of the user interface text based on a preset financial keyword library to obtain keyword matching results includes: The user interface text is matched with keywords in the financial keyword library using regular expressions to obtain the regular expression matching results. The user interface text is semantically matched with the keywords to obtain semantic matching results; The keyword matching result is determined based on the regular expression matching result and the semantic matching result.

[0021] Further, the step of performing regular expression matching between the user interface text and keywords in the financial keyword library to obtain the regular expression matching result includes: Construct a regular expression based on the keywords in the aforementioned financial keyword library; The user interface text is traversed according to the regular expression to perform text matching and obtain the regular expression matching result.

[0022] Further, the step of semantically matching the user interface text with the keywords to obtain semantic matching results includes: Convert the keywords in the financial keyword library and the user interface text into word vectors; The word vectors corresponding to the keywords are semantically compared with the word vectors corresponding to the user interface text to obtain semantic matching results.

[0023] Further, determining whether the current application is a financial application based on the keyword matching results includes: If the keyword matching result is successful, then obtain the current page information of the current application; Based on the current page information, the category of the current application is determined using a preset second language model, and the category includes financial and non-financial applications.

[0024] Furthermore, the current page information includes: the package name of the currently running application and the user interface structure information of the current page.

[0025] Furthermore, the training process of the second major language model includes: Retrieve historical page data from financial and non-financial applications; Based on the reinforcement learning framework, the historical page data is used as training data to train the basic large language model, thereby obtaining the second large language model.

[0026] Furthermore, the step of performing user behavior analysis based on the screenshot image and the user interface structure information using a preset first language model to determine whether the user exhibits high-risk behavior includes: Based on the user interface structure information, extract user interface elements and user interaction events; Based on a preset prompt template, input data is constructed according to the screenshot image, the user interface elements, and the user interaction events; The input data is fed into a preset first large language model to identify the user's high-risk behavior; wherein, the first large language model is a multimodal large language model.

[0027] Furthermore, the device is also used for: After the security alert is triggered, the current user's operation is suspended, and the user's identity is verified. When identity verification is successful, the system determines whether the user belongs to the preset target group based on the user's identity information; if not, the user's operation continues; if yes, the system enters the collaborative verification process to initiate a remote confirmation request to the preset associated users for security verification.

[0028] This application also provides a computer device, including a processor and a memory, wherein the memory stores a computer program and the computer program is configured to be executed by the processor, wherein the processor, when executing the computer program, implements the application security control method described in any of the preceding claims.

[0029] This application also provides a computer-readable storage medium storing a computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the application security control method described above.

[0030] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the application security control method described in any of the above claims.

[0031] Compared to existing technologies, the beneficial effects of the application security control method, apparatus, device, medium, and product provided in this application are as follows: By analyzing user behavior based on screenshot images and user interface structure information using a preset first large language model, it is determined whether the user exhibits high-risk behavior. If the user exhibits high-risk behavior, a security warning is triggered, improving the ability to identify complex fraud scenarios and enhancing the accuracy and efficiency of high-risk behavior identification. By first confirming that the current application is a financial application based on the current user interface text before identifying high-risk behavior, the efficiency and accuracy of application security control are further improved. By performing keyword matching based on the current user interface text and then combining it with the large language model for secondary judgment, the system can accurately determine whether the current application is a financial application, effectively reducing false positives and false negatives and enhancing the system's financial security protection capabilities. Attached Figure Description

[0032] Figure 1 This is a flowchart illustrating an application security control method provided in an embodiment of this application; Figure 2 This is a schematic diagram of a keyword matching process provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an application security control device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0034] Please see Figure 1 , Figure 1 This is a flowchart illustrating an application security control method provided in an embodiment of this application. The application security control method includes: S1: Obtain a screenshot of the user operating the current application and the corresponding user interface structure information; S2: Based on the screenshot image and the user interface structure information, user behavior analysis is performed using a preset first language model to determine whether the user has high-risk behavior; S3: If the user exhibits high-risk behavior, a security alert will be triggered.

[0035] Specifically, the embodiments of this application are applied to mobile phone systems capable of running APPs (i.e., applications), including but not limited to mobile phones or cloud phones. In a cloud phone, the mobile phone system is deployed in a virtualized manner in the cloud, and the cloud server is responsible for the computation. Users can remotely access and operate this system through the network.

[0036] To detect potential fraudulent activities and prevent high-risk operations, this application analyzes screenshots of the current user interface and the user interface structure information of the current application. The user interface structure information refers to the UI (User Interface) structure information of the app. These two types of data are input into a first language model, which performs semantic fusion analysis and outputs a judgment result indicating whether the user is performing a high-risk operation. When a high-risk behavior is determined, a security alert is triggered, interrupting the user's high-risk behavior and initiating a security verification process to ensure the user's information and property security.

[0037] High-risk behaviors refer to transactions and account changes that are easily used to defraud funds or cooperate with telecommunications and network fraud, such as large-amount transfers, withdrawals, and sensitive operations such as modifying account information.

[0038] This application embodiment analyzes screenshot images and user interface structure information to accurately identify high-risk user behaviors and issue warnings, thereby improving the ability to identify complex fraud scenarios and effectively reducing false alarm and false negative rates.

[0039] As one optional embodiment, the step of performing user behavior analysis based on the screenshot image and the user interface structure information using a preset first language model to determine whether the user exhibits high-risk behavior includes: Based on the user interface structure information, extract user interface elements and user interaction events; Based on a preset prompt template, input data is constructed according to the screenshot image, the user interface elements, and the user interaction events; The input data is fed into a preset first large language model to identify the user's high-risk behavior; wherein, the first large language model is a multimodal large language model.

[0040] Specifically, this application embodiment employs a multimodal large language model to understand and analyze structural and image information. MLLM (Multimodal Large Language Models) combines the advantages of large language models (LLM) and large visual models (LVM), enabling it to process and understand multimodal data such as text, images, and audio. The multimodal large language model in this application embodiment is trained based on historical screenshots and corresponding historical user interface elements and historical user interaction events.

[0041] This application embodiment analyzes the UI elements of the current page in real time, extracting user interface elements and user interaction events from the user interface structure information to accurately capture specific operations and identify high-risk user behaviors. Furthermore, it analyzes screenshots of the current interface to further confirm whether the user is in a sensitive process, such as entering an amount, selecting a recipient, or confirming payment. The screenshots, after image processing, are concatenated with preset prompt templates, user interface elements, and user interaction events to form input data, which is then input into a multimodal large language model for semantic fusion analysis. The multimodal large language model comprehensively judges whether the user is performing a high-risk behavior. When the model determines that the user is performing a high-risk behavior, the system immediately triggers a warning to remind the user that the current operation is risky.

[0042] This application embodiment performs fine-grained monitoring and analysis of users' real-time operations in applications based on a multimodal large language model, which can effectively identify fraudulent behavior, reduce false alarms and missed detections, and enhance the security protection capabilities of mobile phones.

[0043] As one optional embodiment, before obtaining the screenshot image of the user operating the current application and the corresponding user interface structure information, the method further includes: Based on the user interface text of the current application, it is confirmed that the current application is a financial application.

[0044] This application embodiment addresses the security control of financial applications. By first identifying the type of the current application and then detecting high-risk user behavior when the application is identified as a financial application, the accuracy of identification is improved, while also saving computing resources.

[0045] Specifically, financial applications can be identified by performing text recognition and rule matching on the user interface text of the current application. Alternatively, the user interface text can be encoded into vectors using a pre-trained language model, and then classified using a classifier to determine whether it is a financial application.

[0046] This application embodiment confirms that the current application is a financial application before detecting the user's high-risk behavior. It accurately matches the user's high-risk behavior with the user's operation when using financial applications, which further improves the recognition accuracy and efficiency, reduces interference with the user's operation, and optimizes the user experience.

[0047] As one optional embodiment, confirming that the current application is a financial application based on the user interface text of the current application includes: Get the text of the current application's user interface; Based on a pre-set financial keyword library, keyword matching is performed on the user interface text to obtain keyword matching results; Based on the keyword matching results, determine whether the current application is a financial application; if so, confirm that the current application is a financial application.

[0048] Specifically, this application embodiment first obtains the user interface text of the current application. This can be done by obtaining a screenshot of the current application and performing text detection on the screenshot. This method can recognize all visual text, but the recognition error is relatively large, and the operation is complex and inefficient. Alternatively, the user interface text can be obtained by parsing the text file corresponding to the UI hierarchy structure of the current application. Although this method cannot obtain text drawn in images, it can accurately and conveniently handle dynamic elements, with higher efficiency and accuracy.

[0049] Furthermore, after obtaining the user interface text, the user interface text is matched against preset financial keywords. The keyword matching methods include, but are not limited to, direct matching, regular expression matching, semantic matching, and fuzzy matching. Then, based on the keyword matching results, if the user interface text successfully matches a financial keyword, the current application is determined to be a financial application.

[0050] This application embodiment uses a pre-set financial keyword library to perform keyword matching on the user interface text, which can effectively identify and confirm financial applications, improve the accuracy and efficiency of financial application identification, and facilitate more accurate identification of high-risk user behavior in the future, thus protecting property security.

[0051] As one optional embodiment, obtaining the user interface text of the current application includes: Retrieve the Extensible Markup Language (XML) file corresponding to the user interface hierarchy of the current application; Based on the XML file, extract the text information of the user interface elements to obtain the user interface text.

[0052] Specifically, the XML (Extensible Markup Language) file corresponding to the user interface hierarchy is a view tree representation exported by the system or automation framework, with each UI element represented by an XML node. Text information from the UI elements can be extracted from the XML file, including the text directly displayed on the screen, as well as text attributes, position, interactivity, and other textual information, ultimately yielding the user interface text.

[0053] The embodiments of this application obtain semantically rich structured text from XML files, which is beneficial to improving recognition accuracy. At the same time, parsing XML requires less resources and is more efficient than parsing an entire screenshot.

[0054] As one optional embodiment, the step of performing keyword matching on the user interface text based on a preset financial keyword library to obtain keyword matching results includes: The user interface text is matched with keywords in the financial keyword library using regular expressions to obtain the regular expression matching results. The user interface text is semantically matched with the keywords to obtain semantic matching results; The keyword matching result is determined based on the regular expression matching result and the semantic matching result.

[0055] Specifically, this application embodiment combines regular expression matching and semantic matching of keywords to perform financial keyword matching. Regular expression matching uses rule-based string matching, employing regular expressions to describe patterns and matching whether the text satisfies the pattern. Semantic matching maps the text to vectors and then matches keywords based on vector similarity.

[0056] Please see Figure 2 , Figure 2 This is a schematic diagram of a keyword matching process provided in an embodiment of this application. On one hand, based on a preset financial keyword library, regular expression matching is performed on the user interface text to obtain a regular expression matching result. For example, financial keywords include, but are not limited to, foreign exchange, payment, wealth management, top-up, balance, funds, deposit, and transfer. On the other hand, based on a word vector model (i.e., Embedding Model), phrases in the keyword library and the user interface text are vectorized to obtain a vectorized keyword library and user interface text. Then, semantic matching is performed based on the vectorized keyword library and user interface text to obtain a semantic matching result. Further, based on the regular expression matching result and the semantic matching result, it is determined whether the user interface text matches the preset keyword library to obtain a keyword matching result, including successful keyword matching and failed keyword matching.

[0057] The embodiments of this application combine regular expression matching and semantic matching for keyword matching, which can balance the accuracy of pattern recognition and the flexibility of semantic understanding, effectively improving the coverage and accuracy of financial keyword recognition.

[0058] As one optional embodiment, the step of performing regular expression matching between the user interface text and keywords in the financial keyword library to obtain the regular expression matching result includes: Construct a regular expression based on the keywords in the aforementioned financial keyword library; The user interface text is traversed according to the regular expression to perform text matching and obtain the regular expression matching result.

[0059] Specifically, in this embodiment, a regular expression is constructed based on keywords from a financial keyword database. The regular expression is used to retrieve text from the user interface text that matches a specific pattern. By constructing the regular expression using the keyword database, financial keywords are written into the regular expression. Therefore, when traversing the user interface text using the regular expression, each word in the regular expression is compared one by one with each keyword in the keyword database to be matched, completing the regular expression matching. Finally, the regular expression matching result returns the successfully matched keywords. The success or failure of the regular expression matching result can be determined based on the number of successfully matched keywords, or the result can be quantified into a matching degree based on the number of successfully matched keywords for subsequent calculations.

[0060] The embodiments of this application can quickly complete keyword matching, have high interpretability, low resource consumption, and are conducive to improving computing efficiency.

[0061] As one optional embodiment, the step of semantically matching the user interface text with the keywords to obtain semantic matching results includes: Convert the keywords in the financial keyword library and the user interface text into word vectors; The word vectors corresponding to the keywords are semantically compared with the word vectors corresponding to the user interface text to obtain semantic matching results.

[0062] Specifically, based on a pre-defined word vector model, keywords in a financial keyword database are vectorized to obtain corresponding word vectors; user interface text is also vectorized to obtain corresponding word vectors. The word vector model encodes natural language words into vectors, where semantically similar words also have similar vector representations.

[0063] Then, the similarity between the word vectors corresponding to the keywords and the word vectors corresponding to the user interface text is calculated for semantic comparison, and the semantic matching result is determined based on the similarity of the word vectors. For example, if the calculated similarity is greater than a preset similarity threshold, the semantic matching is considered successful; otherwise, the semantic matching is considered unsuccessful.

[0064] The embodiments of this application, through semantic matching, can identify expressions that are different but semantically similar. It has strong adaptability, can reduce the false negative rate, has stronger generalization ability, and further improves the accuracy of financial application identification.

[0065] Further, the keyword matching result is determined based on the regular expression matching result and the semantic matching result. In one embodiment, when both the regular expression matching result and the semantic matching result are successful, the keyword matching result is determined to be successful, and the current application is determined to be a financial application; when either the regular expression matching result or the semantic matching result fails, the keyword matching result is determined to be unsuccessful. In another embodiment, when either the regular expression matching result or the semantic matching result is successful, the keyword matching result is determined to be successful, and the current application is determined to be a financial application; when both the regular expression matching result and the semantic matching result are unsuccessful, the keyword matching result is determined to be unsuccessful, and the current application is determined to be a non-financial application.

[0066] As one optional embodiment, determining whether the current application is a financial application based on the keyword matching result includes: If the keyword matching result is successful, then obtain the current page information of the current application; Based on the current page information, the category of the current application is determined using a preset second language model, and the category includes financial and non-financial applications.

[0067] Specifically, to further improve the accuracy of application type identification, this embodiment of the application performs a secondary determination based on a second language model after successful keyword matching. First, the current page information of the current application is obtained, and the second language model is used for semantic analysis and judgment to more accurately identify whether the current application is a financial application.

[0068] Among them, the task of judging financial applications is essentially a classification problem. By designing prompt words for the large language model, the second large language model is comprehensively guided to refer to the current page information and contextual semantic information. Through semantic analysis and judgment, the application is classified as either financial or non-financial.

[0069] This application embodiment improves the accuracy of financial application identification by using a large language model combined with current page information for semantic analysis after initially identifying financial keywords, effectively reducing false positives and false negatives.

[0070] As one optional embodiment, the current page information includes: the package name of the currently running application and the user interface structure information of the current page.

[0071] Specifically, in this embodiment, after successful keyword matching, the current page information is obtained, including the package name of the current application and the user interface structure information of the current page. This information is then input into a preset second language model to determine the application type. The package name is the unique package identifier for an Android application, such as "com.eg.android.AlipayGphone"; the user interface structure information refers to the UI structure information of the current page.

[0072] In practical applications, operation pages containing the word "payment" may appear in various scenarios such as games, finance, and e-commerce. However, in a game scenario, this might refer to in-game purchases, while in a financial scenario, it involves real transactions. In these cases, keyword matching alone cannot accurately identify financial applications. Therefore, this embodiment of the application combines the application's package name and its corresponding UI structure information to achieve highly accurate determination of application attributes, thereby improving recognition precision.

[0073] As one optional embodiment, the training process of the second large language model includes: Retrieve historical page data from financial and non-financial applications; Based on the reinforcement learning framework, the historical page data is used as training data to train the basic large language model, thereby obtaining the second large language model.

[0074] Specifically, the second major language model is built upon the existing major language model. It incorporates a reinforcement learning framework and employs lightweight parameter fine-tuning techniques such as LoRA (Low-Rank Adaptation) to perform targeted fine-tuning training on the model, forming a second major language model used to determine the classification of applications.

[0075] The training process for the second large language model involves first collecting a large amount of historical page data from both financial and non-financial applications, including the application package names and their corresponding user interface structure information, as training data. The corresponding application types are then used as label data for training the large language model. This application embodiment continuously optimizes the performance of the large language model on the financial application recognition task, thereby improving the overall system's robustness and recognition accuracy in complex scenarios.

[0076] Furthermore, after keyword matching and financial attribute determination to confirm that the current application is a financial application, fine-grained monitoring and analysis of the user's real-time operations within the financial application are performed based on a multimodal large language model. When high-risk user behavior is identified, a security alert is triggered, requiring the user to re-authenticate to ensure security.

[0077] As one optional embodiment, after the security alert is triggered, the method further includes: Pause the current user's operation and verify the user's identity; When identity verification is successful, the system determines whether the user belongs to the preset target group based on the user's identity information; if not, the user's operation continues; if yes, the system enters the collaborative verification process to initiate a remote confirmation request to the preset associated users for security verification.

[0078] Specifically, when high-risk behavior is identified, the user's current operation is paused, and a security verification process begins. This verification process dynamically adjusts based on the user's identity and behavioral patterns. First, user identity is verified using conventional methods such as fingerprint recognition, SMS verification codes, or dynamic facial recognition. Once identity verification is successful, the system determines if the user belongs to a pre-defined target group, which includes groups requiring guardianship, such as the elderly or children. If the current user is confirmed to be part of the target group, a collaborative verification process is initiated, sending a remote confirmation request to a pre-defined associated user (the current user's guardian). The guardian then performs a secondary confirmation to complete the security verification. A notification is then sent via SMS or app push notification to ensure the safety of funds.

[0079] This application embodiment dynamically adjusts the security verification method based on the operational risk level corresponding to the user's identity, constructing an adaptive and differentiated personalized anti-fraud prevention and control system. In normal scenarios, identity verification methods such as fingerprint recognition, SMS verification codes, or dynamic facial recognition are enabled; in user scenarios requiring a guardian, secondary confirmation by the guardian is required for security verification. This embodiment balances security and user experience, achieving more precise and flexible application security control.

[0080] Accordingly, this application also provides an application security control device capable of implementing all the processes of the application security control method in the above embodiments.

[0081] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of an application security control device provided in an embodiment of this application. The application security control device includes: The data acquisition module 301 is used to acquire screenshots of the user's operation of the current application and the corresponding user interface structure information. Risk identification module 302 is used to perform user behavior analysis based on the screenshot image and the user interface structure information, using a preset first language model, to determine whether the user has high-risk behavior; The security warning module 303 is used to trigger a security warning if the user engages in high-risk behavior.

[0082] Preferably, the device further includes: The application identification module is used to identify whether the current application is a financial application based on the user interface text of the current application.

[0083] Preferably, the application identification module is specifically used for: Get the text of the current application's user interface; Based on a pre-set financial keyword library, keyword matching is performed on the user interface text to obtain keyword matching results; Based on the keyword matching results, determine whether the current application is a financial application; if so, confirm that the current application is a financial application.

[0084] Preferably, obtaining the user interface text of the current application includes: Retrieve the Extensible Markup Language (XML) file corresponding to the user interface hierarchy of the current application; Based on the XML file, extract the text information of the user interface elements to obtain the user interface text.

[0085] Preferably, the step of performing keyword matching on the user interface text based on a preset financial keyword library to obtain keyword matching results includes: The user interface text is matched with keywords in the financial keyword library using regular expressions to obtain the regular expression matching results. The user interface text is semantically matched with the keywords to obtain semantic matching results; The keyword matching result is determined based on the regular expression matching result and the semantic matching result.

[0086] Preferably, the step of performing regular expression matching between the user interface text and keywords in the financial keyword library to obtain the regular expression matching result includes: Construct a regular expression based on the keywords in the aforementioned financial keyword library; The user interface text is traversed according to the regular expression to perform text matching and obtain the regular expression matching result.

[0087] Preferably, the step of semantically matching the user interface text with the keywords to obtain semantic matching results includes: Convert the keywords in the financial keyword library and the user interface text into word vectors; The word vectors corresponding to the keywords are semantically compared with the word vectors corresponding to the user interface text to obtain semantic matching results.

[0088] Preferably, determining whether the current application is a financial application based on the keyword matching result includes: If the keyword matching result is successful, then obtain the current page information of the current application; Based on the current page information, the category of the current application is determined using a preset second language model, and the category includes financial and non-financial applications.

[0089] Preferably, the current page information includes: the package name of the currently running application and the user interface structure information of the current page.

[0090] Preferably, the training process of the second large language model includes: Retrieve historical page data from financial and non-financial applications; Based on the reinforcement learning framework, the historical page data is used as training data to train the basic large language model, thereby obtaining the second large language model.

[0091] Preferably, the step of performing user behavior analysis based on the screenshot image and the user interface structure information, using a preset first language model, to determine whether the user exhibits high-risk behavior includes: Based on the user interface structure information, extract user interface elements and user interaction events; Based on a preset prompt template, input data is constructed according to the screenshot image, the user interface elements, and the user interaction events; The input data is fed into a preset first large language model to identify the user's high-risk behavior; wherein, the first large language model is a multimodal large language model.

[0092] Preferably, the device is further used for: After the security alert is triggered, the current user's operation is suspended, and the user's identity is verified. When identity verification is successful, the system determines whether the user belongs to the preset target group based on the user's identity information; if not, the user's operation continues; if yes, the system enters the collaborative verification process to initiate a remote confirmation request to the preset associated users for security verification.

[0093] In specific implementation, the working principle, control process and technical effects of the application security control device provided in this application embodiment are the same as those of the application security control method in the above embodiment, and will not be repeated here.

[0094] See Figure 4 , Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device includes: a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor 401. When the processor 401 executes the computer program, it implements the steps in the above-described application security control method embodiment. Alternatively, when the processor 401 executes the computer program, it implements the functions of each module / unit in the above-described device embodiments.

[0095] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 402 and executed by the processor 401 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device.

[0096] The computer device may include, but is not limited to, processor 401 and memory 402. Those skilled in the art will understand that the schematic diagram is merely an example of a computer device and does not constitute a limitation on the computer device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0097] The processor 401 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 401 is the control center of the computer device, connecting various parts of the entire computer device through various interfaces and lines.

[0098] The memory 402 can be used to store the computer programs and / or modules. The processor 401 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 402 and calling the data stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc.; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0099] Wherein, if the modules / units integrated into the computer device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 401, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms.

[0100] This application also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the application security control method described in any of the above embodiments.

[0101] This application also provides a computer program product, including a computer program / instruction, which, when executed by the computer program / instruction processor, implements the application security control method described in any of the above embodiments.

[0102] The methods in this application can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in this application are performed, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, network equipment, user equipment, core network equipment, or other programmable devices.

[0103] The computer program or instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions may be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; or an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0104] The computer programs / instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper cables, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and forwards those instructions to computer-readable storage media within the respective computing / processing device.

[0105] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​and conventional procedural programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of this application.

[0106] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0107] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0108] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0109] Each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0110] This application provides an application security control method, apparatus, device, medium, and product, the beneficial effects of which are as follows: By analyzing user behavior based on screenshot images and user interface structure information using a preset first large language model, it is possible to determine whether the user exhibits high-risk behavior. If the user exhibits high-risk behavior, a security warning is triggered, improving the ability to identify complex fraud scenarios and enhancing the accuracy and efficiency of high-risk behavior identification; By first confirming that the current application is a financial application based on the current user interface text before identifying high-risk behavior, the efficiency and accuracy of application security control are further improved; By performing keyword matching based on the current user interface text and then combining it with the large language model for secondary judgment, the system can accurately determine whether the current application is a financial application, effectively reducing false positives and false negatives and enhancing the system's financial security protection capabilities.

[0111] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. An application security control method, characterized in that, include: Obtain screenshots of the user interacting with the current application and the corresponding user interface structure information; Based on the screenshot image and the user interface structure information, a preset first language model is used to perform user behavior analysis in order to determine whether the user has high-risk behavior. If the user exhibits high-risk behavior, a security alert will be triggered.

2. The application security control method as described in claim 1, characterized in that, Before acquiring the screenshot image of the user operating the current application and the corresponding user interface structure information, the method further includes: Based on the user interface text of the current application, it is confirmed that the current application is a financial application.

3. The application security control method as described in claim 2, characterized in that, The step of confirming that the current application is a financial application based on the user interface text of the current application includes: Get the text of the current application's user interface; Based on a pre-set financial keyword library, keyword matching is performed on the user interface text to obtain keyword matching results; Based on the keyword matching results, determine whether the current application is a financial application; if so, confirm that the current application is a financial application.

4. The application security control method as described in claim 3, characterized in that, The step of obtaining the user interface text of the current application includes: Retrieve the Extensible Markup Language (XML) file corresponding to the user interface hierarchy of the current application; Based on the XML file, extract the text information of the user interface elements to obtain the user interface text.

5. The application security control method as described in claim 3, characterized in that, The method of performing keyword matching on the user interface text based on a preset financial keyword library to obtain keyword matching results includes: The user interface text is matched with keywords in the financial keyword library using regular expressions to obtain the regular expression matching results. The user interface text is semantically matched with the keywords to obtain semantic matching results; The keyword matching result is determined based on the regular expression matching result and the semantic matching result.

6. The application security control method as described in claim 5, characterized in that, The step of performing regular expression matching between the user interface text and keywords in the financial keyword library to obtain the regular expression matching result includes: Construct a regular expression based on the keywords in the aforementioned financial keyword library; The user interface text is traversed according to the regular expression to perform text matching and obtain the regular expression matching result.

7. The application security control method as described in claim 5, characterized in that, The step of semantically matching the user interface text with the keywords to obtain semantic matching results includes: Convert the keywords in the financial keyword library and the user interface text into word vectors; The word vectors corresponding to the keywords are semantically compared with the word vectors corresponding to the user interface text to obtain semantic matching results.

8. The application security control method as described in claim 3, characterized in that, The step of determining whether the current application is a financial application based on the keyword matching results includes: If the keyword matching result is successful, then obtain the current page information of the current application; Based on the current page information, the category of the current application is determined using a preset second language model, and the category includes financial and non-financial applications.

9. The application security control method as described in claim 8, characterized in that, The current page information includes: the package name of the currently running application and the user interface structure information of the current page.

10. The application security control method as described in claim 8, characterized in that, The training process of the second major language model includes: Retrieve historical page data from financial and non-financial applications; Based on the reinforcement learning framework, the historical page data is used as training data to train the basic large language model, thereby obtaining the second large language model.

11. The application security control method as described in claim 1, characterized in that, The step of analyzing user behavior using a preset first language model based on the screenshot image and the user interface structure information to determine whether the user exhibits high-risk behavior includes: Based on the user interface structure information, extract user interface elements and user interaction events; Based on a preset prompt template, input data is constructed according to the screenshot image, the user interface elements, and the user interaction events; The input data is fed into a preset first large language model to identify the user's high-risk behavior; wherein, the first large language model is a multimodal large language model.

12. The application security control method as described in claim 1, characterized in that, After triggering the security alert, the method further includes: Pause the current user's operation and verify the user's identity; When identity verification is successful, the system determines whether the user belongs to the preset target group based on the user's identity information; if not, the user's operation continues; if yes, the system enters the collaborative verification process to initiate a remote confirmation request to the preset associated users for security verification.

13. An application security control device, characterized in that, include: The data acquisition module is used to acquire screenshots of the user's operation of the current application and the corresponding user interface structure information; The risk identification module is used to perform user behavior analysis based on the screenshot image and the user interface structure information, using a preset first language model, to determine whether the user has high-risk behavior. The security alert module is used to trigger a security alert if the user engages in high-risk behavior.

14. The application security control device as described in claim 13, characterized in that, The device further includes: The application identification module is used to identify whether the current application is a financial application based on the user interface text of the current application.

15. The application security control device as described in claim 14, characterized in that, The application identification module is specifically used for: Get the text of the current application's user interface; Based on a pre-set financial keyword library, keyword matching is performed on the user interface text to obtain keyword matching results; Based on the keyword matching results, determine whether the current application is a financial application; if so, confirm that the current application is a financial application.

16. The application security control device as described in claim 15, characterized in that, The step of obtaining the user interface text of the current application includes: Retrieve the Extensible Markup Language (XML) file corresponding to the user interface hierarchy of the current application; Based on the XML file, extract the text information of the user interface elements to obtain the user interface text.

17. The application security control device as described in claim 15, characterized in that, The method of performing keyword matching on the user interface text based on a preset financial keyword library to obtain keyword matching results includes: The user interface text is matched with keywords in the financial keyword library using regular expressions to obtain the regular expression matching results. The user interface text is semantically matched with the keywords to obtain semantic matching results; The keyword matching result is determined based on the regular expression matching result and the semantic matching result.

18. The application security control device as described in claim 17, characterized in that, The step of performing regular expression matching between the user interface text and keywords in the financial keyword library to obtain the regular expression matching result includes: Construct a regular expression based on the keywords in the aforementioned financial keyword library; The user interface text is traversed according to the regular expression to perform text matching and obtain the regular expression matching result.

19. The application security control device as described in claim 17, characterized in that, The step of semantically matching the user interface text with the keywords to obtain semantic matching results includes: Convert the keywords in the financial keyword library and the user interface text into word vectors; The word vectors corresponding to the keywords are semantically compared with the word vectors corresponding to the user interface text to obtain semantic matching results.

20. The application security control device as described in claim 15, characterized in that, The step of determining whether the current application is a financial application based on the keyword matching results includes: If the keyword matching result is successful, then obtain the current page information of the current application; Based on the current page information, the category of the current application is determined using a preset second language model, and the category includes financial and non-financial applications.

21. The application security control device as described in claim 20, characterized in that, The current page information includes: the package name of the currently running application and the user interface structure information of the current page.

22. The application security control device as described in claim 20, characterized in that, The training process of the second major language model includes: Retrieve historical page data from financial and non-financial applications; Based on the reinforcement learning framework, the historical page data is used as training data to train the basic large language model, thereby obtaining the second large language model.

23. The application security control device as described in claim 13, characterized in that, The step of analyzing user behavior using a preset first language model based on the screenshot image and the user interface structure information to determine whether the user exhibits high-risk behavior includes: Based on the user interface structure information, extract user interface elements and user interaction events; Based on a preset prompt template, input data is constructed according to the screenshot image, the user interface elements, and the user interaction events; The input data is fed into a preset first large language model to identify the user's high-risk behavior; wherein, the first large language model is a multimodal large language model.

24. The application security control device as described in claim 13, characterized in that, The device is also used for: After the security alert is triggered, the current user's operation is suspended, and the user's identity is verified. When identity verification is successful, the system determines whether the user belongs to the preset target group based on the user's identity information; if not, the user's operation continues; if yes, the system enters the collaborative verification process to initiate a remote confirmation request to the preset associated users for security verification.

25. A computer device, characterized in that, The device includes a processor and a memory, wherein the memory stores a computer program and the computer program is configured to be executed by the processor, wherein the processor, when executing the computer program, implements the application security control method as described in any one of claims 1 to 12.

26. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the application security control method as described in any one of claims 1 to 12.

27. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the application security control method as described in any one of claims 1 to 12.

Images