Operation behavior identification method and system, computing device and storage medium

Abstract

The embodiment of the invention relates to the technical field of computers, and provides an operation behavior recognition method and system, computing equipment and a storage medium, and the operation behavior recognition method comprises the steps that a target object server determines a first page image corresponding to a current operation behavior in response to the current operation behavior sent by a target object client, determining a first page image corresponding to a current operation behavior, determining a second page image corresponding to a previous operation behavior of the current operation behavior, determining a difference page image between the first page image and the second page image according to the first page image and the second page image, and sending the difference page image to a risk identification server; and the risk identification server performs risk identification on the data of the at least two modalities included in the difference page image according to the risk identification model, obtains an image risk identification result of the difference page image, and determines a behavior risk identification result of the current operation behavior according to the image risk identification result.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of computer technology, and particularly relate to an operation behavior identification method and system, a computing device, and a storage medium. BACKGROUND

[0002] With the development of computer technology, users can usually access resources and services provided by a server through a client, such as browsing a webpage or accessing an application through a client. In order to ensure the safety of user access, it is usually necessary to identify the risk of the content accessed by the user.

[0003] However, when identifying the risk of the content accessed by the user, the cost of identifying the risk of the content accessed by the user is high, which leads to high occupation of computing resources and storage resources of the server, and also easily causes missed detection and misjudgment, affecting the safety of user access and the data security of the client and the server. Therefore, an effective technical solution is needed to solve the above problems. SUMMARY

[0004] Therefore, the embodiments of the present specification provide three operation behavior identification methods. One or more embodiments of the present specification simultaneously relate to two operation behavior identification devices, an operation behavior identification system, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects of high cost of identifying the risk of the content accessed by the user in the prior art.

[0005] According to a first aspect of the embodiments of the present specification, an operation behavior identification method is provided, applied to an operation behavior identification system, the operation behavior identification system comprising a target object server and a risk identification server, wherein the method comprises:

[0006] The target object server determines a first page image corresponding to a current operation behavior in response to the current operation behavior sent by a target object client, and determines a second page image corresponding to a previous operation behavior of the current operation behavior,

[0007] According to the first page image and the second page image, a difference page image between the first page image and the second page image is determined, and the difference page image is sent to the risk identification server;

[0008] The risk identification server identifies the risk of data of at least two modalities included in the difference page image according to a risk identification model, and obtains an image risk identification result of the difference page image,

[0009] According to the image risk identification result, a behavior risk identification result of the current operation behavior is determined.

[0010] According to a second aspect of the embodiments of the present specification, an operation behavior identification system is provided, the operation behavior identification system comprises a target object server and a risk identification server, wherein,

[0011] The target object server is configured to, in response to a current operation behavior sent by a target object client, determine a first page image corresponding to the current operation behavior, and determine a second page image corresponding to a previous operation behavior of the current operation behavior,

[0012] According to the first page image and the second page image, a difference page image between the first page image and the second page image is determined, and the difference page image is sent to the risk identification server;

[0013] The risk identification server is configured to, according to a risk identification model, perform risk identification on data of at least two modalities included in the difference page image, to obtain an image risk identification result of the difference page image,

[0014] According to the image risk identification result, a behavior risk identification result of the current operation behavior is determined.

[0015] According to a third aspect of the embodiments of the present specification, an operation behavior identification method applied to a target object server is provided, wherein the method comprises:

[0016] In response to a current operation behavior sent by a target object client, a first page image corresponding to the current operation behavior is determined, and a second page image corresponding to a previous operation behavior of the current operation behavior is determined;

[0017] According to the first page image and the second page image, a difference page image between the first page image and the second page image is determined, and the difference page image is sent to a risk identification server;

[0018] Receive a behavior risk identification result of the current operation behavior sent by the risk identification server, wherein the behavior risk identification result is determined according to an image risk identification result, and the image risk identification result is obtained by the risk identification server according to a risk identification model, performing risk identification on data of at least two modalities included in the difference page image.

[0019] According to a fourth aspect of the embodiments of the present specification, an operation behavior identification device applied to a target object server is provided, wherein the method comprises:

[0020] The first determining module is configured to determine a first page image corresponding to the current operation behavior and determine a second page image corresponding to a previous operation behavior of the current operation behavior in response to the current operation behavior sent by the target object client;

[0021] The second determining module is configured to determine a difference page image between the first page image and the second page image according to the first page image and the second page image, and send the difference page image to a risk identification server;

[0022] The receiving module is configured to receive a behavior risk identification result of the current operation behavior sent by the risk identification server, wherein the behavior risk identification result is determined according to an image risk identification result, and the image risk identification result is obtained by the risk identification server according to a risk identification model by performing risk identification on data of at least two modalities included in the difference page image.

[0023] According to a fifth aspect of an embodiment of the present specification, a method for identifying an operation behavior is provided, which is applied to a risk identification server, and the method comprises:

[0024] The receiving module is configured to receive a difference page image sent by a target object server in response to a current operation behavior sent by a target object client, wherein the difference page image is determined by the target object server according to a first page image corresponding to the current operation behavior and a second page image corresponding to a previous operation behavior of the current operation behavior;

[0025] According to a risk identification model, risk identification is performed on data of at least two modalities included in the difference page image to obtain an image risk identification result of the difference page image;

[0026] According to the image risk identification result, a behavior risk identification result of the current operation behavior is determined.

[0027] According to a sixth aspect of an embodiment of the present specification, an operation behavior identification device is provided, which is applied to a risk identification server, and the method comprises:

[0028] The receiving module is configured to receive a difference page image sent by a target object server in response to a current operation behavior sent by a target object client, wherein the difference page image is determined by the target object server according to a first page image corresponding to the current operation behavior and a second page image corresponding to a previous operation behavior of the current operation behavior;

[0029] The identification module is configured to perform risk identification on data of at least two modalities included in the difference page image according to a risk identification model, to obtain an image risk identification result of the difference page image.

[0030] The determination module is configured to determine a behavior risk identification result of the current operation behavior according to the image risk identification result.

[0031] According to a seventh aspect of an embodiment of the present specification, a computing device is provided, comprising:

[0032] a memory and a processor;

[0033] The memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions, which implement the steps of the operation behavior identification method when executed by the processor.

[0034] According to an eighth aspect of an embodiment of the present specification, a computer readable storage medium is provided, which stores computer programs / instructions, which implement the steps of the operation behavior identification method when executed by the processor.

[0035] According to a ninth aspect of an embodiment of the present specification, a computer program product is provided, comprising computer programs / instructions, which implement the steps of the operation behavior identification method when executed by the processor.

[0036] In the operation behavior identification method provided by one embodiment of the present specification, the target object server determines a first page image corresponding to the current operation behavior in response to the target object client sending the current operation behavior, and determines a second page image corresponding to a previous operation behavior of the current operation behavior, and determines a difference page image between the first page image and the second page image according to the first page image and the second page image, thereby obtaining a difference page image that appears different from the previous operation behavior based on the current operation behavior of the target object client, and sending the difference page image to the risk identification server. The risk identification server performs risk identification on the difference page image according to a risk identification model, without performing risk identification on the entire first page image, thereby reducing the use cost of the risk identification model and saving the resource occupation of the risk identification server. Moreover, the risk identification model can also perform risk identification on data of at least two modalities included in the difference page image, further ensuring the comprehensiveness and accuracy of risk identification, so that the behavior risk identification result determined according to the image risk identification result is more accurate, and the risk operation behavior is avoided to be missed and misjudged. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1is a schematic diagram of an application scenario of an operation behavior recognition method provided by an embodiment of the present specification;

[0038] Figure 2 is a flowchart of an operation behavior recognition method provided by an embodiment of the present specification;

[0039] Figure 3 is a structural schematic diagram of an operation behavior recognition system in an operation behavior recognition method provided by an embodiment of the present specification;

[0040] Figure 4 is a process flowchart of an operation behavior recognition method provided by an embodiment of the present specification;

[0041] Figure 5 is a structural schematic diagram of an operation behavior recognition system provided by an embodiment of the present specification;

[0042] Figure 6 is a flowchart of a second operation behavior recognition method provided by an embodiment of the present specification;

[0043] Figure 7 is a structural schematic diagram of a second operation behavior recognition device provided by an embodiment of the present specification;

[0044] Figure 8 is a flowchart of a third operation behavior recognition method provided by an embodiment of the present specification;

[0045] Figure 9 is a structural schematic diagram of a third operation behavior recognition device provided by an embodiment of the present specification;

[0046] Figure 10 is a structural block diagram of a computing device provided by an embodiment of the present specification. DETAILED DESCRIPTION

[0047] In the following description, a lot of specific details are set forth in order to facilitate a thorough understanding of the present specification. However, the present specification can be implemented in many different ways than described herein, and those skilled in the art can make similar extensions without departing from the connotation of the present specification, so the present specification is not limited to the specific implementation disclosed below.

[0048] The terminology used in this disclosure of one or more embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of this disclosure of one or more embodiments. As used in this disclosure and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0049] It will be understood that, although the terms first, second, etc. can be used herein to describe various information, these terms are not intended to denote a temporal sequence, but are used only to distinguish one piece of information from another. For example, without departing from the scope of the disclosure of one or more embodiments, first can be termed second, and similarly, second can be termed first. Depending on the context, the word "if' as used herein can be interpreted to mean "when" or "in response to determining."

[0050] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the disclosure of one or more embodiments are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0051] In the disclosure of one or more embodiments, a large model refers to a deep learning model with a large number of model parameters, usually containing hundreds of millions, tens of billions, hundreds of billions, thousands of billions or even tens of billions of model parameters. The large model can also be called a foundation model. Through large-scale unlabeled corpus pre-training of the large model, a pre-trained model with hundreds of millions of parameters is produced. Such a model can adapt to a wide range of downstream tasks and has good generalization ability. For example, a large language model (LLM) and a multi-modal pre-training model.

[0052] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0053] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0054] Cloud PC: An operating system environment that runs on the cloud, which clients connect to and use via a streaming protocol.

[0055] Multimodal large models are a machine learning technique based on deep learning that integrates different media data (such as text, images, audio, and video) and learns the relationships between different modalities to achieve more intelligent information processing and understanding.

[0056] Screen recording audit: Conduct behavioral analysis and monitoring of employees' screen recording using cloud computers, and audit the compliance of their computer use.

[0057] Session: Each time a user connects to and uses a cloud computer, the activities and operations that occur constitute a session.

[0058] This specification provides three methods for recognizing operational behaviors, and also relates to two devices for recognizing operational behaviors, an operational behavior recognition system, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0059] See Figure 1 , Figure 1 The illustration shows an application scenario diagram of an operation behavior recognition method provided according to an embodiment of this specification. The method specifically includes the following steps.

[0060] The target object server, in response to the current operation sent by the target object client, determines the first page image corresponding to the current operation and the second page image corresponding to the previous operation.

[0061] According to the first page image and the second page image, a difference page image between the first page image and the second page image is determined, and the difference page image is sent to the risk identification server end;

[0062] The risk identification server end performs risk identification on data of at least two modalities included in the difference page image according to a risk identification model, to obtain an image risk identification result of the difference page image,

[0063] According to the image risk identification result, a behavior risk identification result of the current operation behavior is determined.

[0064] In an embodiment of the present specification, the target object server end determines a first page image corresponding to the current operation behavior in response to the current operation behavior sent by the target object client end, and determines a second page image corresponding to a previous operation behavior of the current operation behavior. According to the first page image and the second page image, a difference page image between the first page image and the second page image is determined, so as to obtain a difference page image that appears different from the previous operation behavior based on the current operation behavior of the target object client end. The difference page image is sent to the risk identification server end, and the risk identification server end performs risk identification on the difference page image according to a risk identification model, without performing risk identification on the entire first page image, thereby reducing the use cost of the risk identification model, so as to save the resource occupation of the risk identification server end. Moreover, the risk identification model can also perform risk identification on data of at least two modalities included in the difference page image, so as to further ensure the comprehensiveness and accuracy of risk identification, thereby making the behavior risk identification result determined according to the image risk identification result more accurate, and avoiding missed detection and misjudgment of the risk operation behavior.

[0065] As shown in FIG. 1, Figure 1 FIG. 1 shows a system architecture of a risk identification method according to an embodiment of the present specification. Figure 1 The system architecture shown in FIG. 1 includes a target object client end 102, a target object server end 104, and a risk identification server end 106.

[0066] In specific implementation, the user can send a current operation behavior to the target object server 104 through the target object client 102, the target object server can determine a first page image corresponding to the current operation behavior, and determine a second page image corresponding to a previous operation behavior of the current operation behavior, determine a difference page image between the first page image and the second page image, the target object server 104 sends the difference page image to the risk identification server 106, the risk identification server 106 can perform risk identification on the difference page image according to the risk identification model, obtain an image risk identification result of the difference page image, and determine a behavior risk identification result of the current operation behavior according to the image risk identification result, the target object server 104 can determine whether to execute a control instruction of the current operation behavior according to the behavior risk identification result, and perform interactive control on the target object client 102 according to the control instruction, to ensure the security of the user accessing the target object server.

[0067] Referring to Figure 2 , Figure 2 A flowchart of an operation behavior identification method according to an embodiment of the present specification is shown, which is applied to an operation behavior identification system including a target object server and a risk identification server, and specifically includes the following steps.

[0068] Step 202: The target object server determines a first page image corresponding to a current operation behavior and a second page image corresponding to a previous operation behavior of the current operation behavior in response to the current operation behavior sent by the target object client.

[0069] Specifically, the operation behavior identification method provided by the embodiments of the present specification can be applied to the field of cloud computers. In the field of cloud computers, the cloud computer is an operating system environment running in the cloud, and the client can connect and use the cloud computer through the push stream protocol. The cloud computer can transfer the functions and data of the personal computer to the cloud server for running. The user can connect to the virtual computer in the cloud through the network, use the computing resources and storage services provided by the cloud computer, and the user is not limited by the hardware configuration of the local computer. Only a simple terminal device (such as a notebook computer, a tablet computer, a smart phone, etc.) is needed to access the cloud computer service. Alternatively, the operation behavior identification method provided by the embodiments of the present specification can also be applied to the field of remote control. In this field, the user can remotely control another client through a client, and the operation behavior identification method can be used to identify the risk of the operation behavior of the user remotely controlling.

[0070] For ease of understanding, the operation behavior identification method is applied to the field of cloud computers in the embodiments of the present specification.

[0071] The target object can be understood as a cloud computer, and the target object server is a server of the cloud computer, and the target object client is a client of the cloud computer. The current operation behavior sent by the target object client can be understood as an operation behavior performed by a user at a current time on the target object client, and the operation behavior can be, for example, a webpage browsing behavior, a game login behavior, a video download behavior, an application startup behavior, and the like. In actual application, the current operation behavior sent by the target object client can be a session initiated by the user. The previous operation behavior can be understood as an operation behavior performed by the user at a previous time on the target object client, for example, after the user performs a webpage entering behavior on the target object client, the user performs a webpage page turning behavior, and the webpage page turning behavior is the current operation behavior at the current time, and the webpage entering behavior is the previous operation behavior at the previous time.

[0072] The first page image corresponding to the current operation behavior can be understood as a display image formed by page data corresponding to the current operation behavior, for example, the current operation behavior is a webpage browsing behavior, and the page data corresponding to the webpage browsing behavior is webpage data that the webpage browsing behavior wants to browse, which can include text data, video data, and image data, and the first page image corresponding to the webpage browsing behavior can be understood as a display image rendered according to the webpage data. The second page image corresponding to the previous operation behavior can be understood as a display image formed by page data corresponding to the previous operation behavior.

[0073] Therefore, the server of the cloud computer can respond to the current operation behavior sent by the user through the client of the cloud computer, determine the page data corresponding to the current operation behavior, obtain the first page image corresponding to the current operation behavior, and obtain the second page image corresponding to the previous operation behavior of the current operation behavior.

[0074] In specific implementation, the server of the cloud computer can respond to the current operation behavior sent by the client of the cloud computer, determine the page data corresponding to the current operation behavior, obtain the first page image corresponding to the current operation behavior, and obtain the second page image corresponding to the previous operation behavior of the current operation behavior.

[0075] In specific implementation, the server of the cloud computer can respond to the current operation behavior sent by the client of the cloud computer, determine the page data corresponding to the current operation behavior, obtain the first page image corresponding to the current operation behavior, and obtain the second page image corresponding to the previous operation behavior of the current operation behavior.

[0076] In specific implementation, the server of the cloud computer can respond to the current operation behavior sent by the client of the cloud computer, determine the page data corresponding to the current operation behavior, obtain the first page image corresponding to the current operation behavior, and obtain the second page image corresponding to the previous operation behavior of the current operation behavior.

[0077] Specifically, the target object client responds to a point selection instruction of the user, determines a current operation behavior of the user, and sends the current operation behavior to the target object server. The target object server responds to the current operation behavior sent by the target object client, obtains page data corresponding to the current operation behavior from a database, and renders the page data to obtain a first page image corresponding to the current operation behavior.

[0078] Referring to Figure 3 , Figure 3 A structural schematic diagram of an operation behavior recognition system in an operation behavior recognition method according to one embodiment of the present specification is shown. As Figure 3 shown, the operation recognition behavior recognition system includes a target object server and a target object client, and the target object client is a client of the cloud computer. The client can be understood as a client connected to the cloud computer through a streaming protocol, and can include a soft terminal or a hardware terminal. The target object server can be understood as an environment running the cloud computer. The target object server can include an operating system environment, a streaming server and an interactive control process. The target object client can send a current operation behavior to the target object server. The target object server determines page data corresponding to the current operation behavior in response to the current operation behavior, and renders the page data in the operating system environment. The target object server obtains a first page image corresponding to the current operation behavior by taking a screenshot of the operating system environment. The streaming server can send the first page image to the target object client according to the streaming protocol, so as to facilitate subsequent display of the first page image on the display interface of the target object client.

[0079] In summary, the target object server renders the page data corresponding to the current operation behavior, thereby obtaining the first page image corresponding to the current operation behavior, which facilitates subsequent risk identification and client streaming.

[0080] In one embodiment of the present specification, after obtaining the first page image corresponding to the current operation behavior, the method further includes:

[0081] sending the first page image to the target object client for display on the display interface of the target object client.

[0082] Specifically, after obtaining the first page image, the first page image can be transmitted to the target object client according to a network transmission protocol, so as to be displayed on the display interface of the target object client.

[0083] In specific implementation, before sending the first page image to the target object client, the first page image can be analyzed, encoded and compressed to reduce the amount of data to be transmitted.

[0084] In summary, by sending the first page image to the target object client, a streaming service to the target object client is realized, and the user experience of using the cloud computer is ensured.

[0085] In addition, after determining the first page image corresponding to the current operation behavior, the method further includes:

[0086] store the first page image to a database, so that the first page image is acquired from the database to identify a behavior risk of a next operation behavior in response to a case of the next operation behavior of the current operation behavior.

[0087] Specifically, after obtaining the first page image corresponding to the current operation behavior, the first page image can be stored to a database, so that the target object server acquires the first page image from the database to identify a behavior risk of a next operation behavior in response to a case of the next operation behavior of the current operation behavior. It can be understood that the process of identifying the behavior risk of the next operation behavior is similar to the process of identifying the behavior risk of the current operation behavior, and the embodiments of the present specification will not repeat the description.

[0088] Then, when determining the second page image corresponding to the previous operation behavior of the current operation behavior, the second page image corresponding to the previous operation behavior can be acquired from the database.

[0089] For example, the user sends a webpage browsing behavior to the target object server through the target object client, then the first page image corresponding to the webpage browsing behavior can be determined, the first page image is stored to the database, and the second page image corresponding to the previous operation behavior (for example, a webpage login behavior) of the webpage browsing behavior is acquired from the database, and the subsequent behavior risk identification is performed according to the first page image and the second page image. In response to the next operation behavior (for example, a webpage page turning behavior) after the webpage browsing behavior sent by the target object client, the target object server can determine the page image corresponding to the webpage page turning behavior, and acquire the first page image corresponding to the previous operation behavior (the webpage browsing behavior) of the webpage page turning behavior from the database, so that the behavior risk identification of the webpage page turning behavior is performed subsequently.

[0090] In practical application, in combination with the above Figure 3The operation behavior recognition system further includes a bypass server, which can be understood as a server in the cloud computer that accepts real-time desktop screen streaming. The risk identification server can run in the bypass server as a risk identification process. The target object server can push the first page image to the bypass stream through the streaming service and store the first page image in the database through the bypass stream. It can be understood that the first page image can be understood as a screenshot of the operating system environment in the cloud computer, which is used to represent the real-time changes of the desktop screen. In addition, the database can be understood as an object storage system on the cloud. The database can be based on object storage and can be used to store screen recording files and screen recording audit files in a specific format, such as an extension format. Accordingly, the first page image can be a screen recording video frame, and the first page image can be stored in the form of a screen recording file in the database. In practical applications, the operation behavior recognition method can be applied to the operation monitoring of enterprise employees. The database can also store an enterprise knowledge base in the form of a vector database.

[0091] In summary, by setting the database to store the page image corresponding to each operation behavior, it is convenient to obtain the page image from the database for behavior risk identification, further improving the efficiency of risk identification. Moreover, the risk identification server runs in the bypass server, i.e., in the streaming server in the cloud, avoiding malicious destruction or bypassing of the user environment, and ensuring the stability of the user environment and behavior risk identification.

[0092] Step 204: According to the first page image and the second page image, determine the difference page image between the first page image and the second page image, and send the difference page image to the risk identification server.

[0093] The difference page image can be understood as the image that is different between the first page image and the second page image. For example, the first page image includes text 1, text 2, and image 1, and the second page image includes text 1, text 2, and image 2. Then, the image 2 in the second page image is the difference page image.

[0094] In practical applications, the difference page image between the first page image and the second page image is the difference point between the first page image and the second page image. The difference point can be at the pixel level, and the pixel-level difference between the first page image and the second page image can be calculated according to the streaming protocol. The pixel-level difference can be used as the difference page image between the first page image and the second page image. By determining the difference page image, the subsequent risk identification of the multi-modal data contained in the difference page image can be met, and the number of training labels for the subsequent multi-modal risk identification model can also be saved.

[0095] In particular implementations, the determining the difference page image between the first page image and the second page image according to the first page image and the second page image comprises:

[0096] calculating an image similarity between the first page image and the second page image;

[0097] determining the difference page image between the first page image and the second page image according to the image similarity.

[0098] In particular, the image similarity between the first page image and the second page image can be calculated according to an image similarity calculation algorithm, and the image similarity satisfying a preset similarity threshold can be determined as the difference page image between the first page image and the second page image according to the image similarity.

[0099] In summary, by determining the difference page image according to the image similarity, the full-screen content does not need to be submitted for subsequent risk identification, and only the screen content with differences needs to be submitted, so that the part to be identified by the risk identification model is less, thereby saving the number of calls to the risk identification model and the number of labels of the training labels, and reducing the use cost.

[0100] In particular implementations, the difference page image between the first page image and the second page image can be determined by a hash algorithm, a histogram algorithm, or a pixel comparison method. For example, the first page image and the second page image can be reduced and grayed, the difference between each pixel and the average value in the first page image and the second page image is calculated respectively, the difference is converted into a binary value by a hash function, and the difference page image is determined according to the comparison result between the hash value of the first page image and the hash value of the second page image. Alternatively, the difference between adjacent pixels in the first page image and the second page image can be calculated respectively, a hash value is generated based on the difference, and the difference page image is determined according to the comparison result between the hash value of the first page image and the hash value of the second page image. Alternatively, the difference page image can also be determined according to parameters such as a structural similarity index, a cosine similarity, or an Euclidean distance between the first page image and the second page image, which are not limited in the embodiments of the present specification.

[0101] In an embodiment of the present specification, the calculating the image similarity between the first page image and the second page image comprises:

[0102] determining a plurality of first pixel points in the first page image, and determining a plurality of second pixel points in the second page image at the same positions as the plurality of first pixel points;

[0103] calculating a pixel similarity between the first pixel points and the second pixel points;

[0104] The determining the difference page image between the first page image and the second page image according to the image similarity comprises:

[0105] The second pixel point with the pixel similarity less than the preset similarity threshold is determined as a target pixel point, and the difference page image is determined according to the target pixel point.

[0106] The preset similarity threshold can be understood as a preset similarity threshold. If the pixel similarity is less than the preset similarity threshold, it indicates that the first pixel point and the second pixel point are not similar and there is a difference.

[0107] Based on this, a plurality of first pixel points in the first page image can be determined, and a plurality of second pixel points in the second page image with the same position as the plurality of first pixel points can be determined. According to a similarity calculation algorithm, the pixel similarity between the first pixel points and the second pixel points with the same position is calculated. The second pixel point with the pixel similarity less than the preset similarity threshold is determined as a target pixel point, and the difference page image is formed according to the target pixel point.

[0108] For example, the first pixel point A corresponding to the position 1 in the first page image and the second pixel point B corresponding to the position 1 in the second page image can be determined. Then, the image positions of the first pixel point A and the second pixel point B are the same, and the pixel similarity between the first pixel point A and the second pixel point B can be calculated. In the case where the pixel similarity is less than the preset similarity threshold, the second pixel point B is determined as a target pixel point. Similarly, the same calculation is performed on other pixel points with the same position in the first page image and the second page image, and the difference page image is formed according to the target pixel point.

[0109] In summary, by calculating the pixel similarity, the determination of the difference page image is realized, and the accuracy of the determined difference page image is ensured.

[0110] In practical applications, before the difference page image is sent to the risk identification server, the method further comprises:

[0111] Determining data of at least two modalities included in the difference page image;

[0112] The sending of the difference page image to the risk identification server comprises:

[0113] The difference page image and the data of the at least two modalities are encapsulated into a data frame and sent to the risk identification server.

[0114] The at least two modalities can include a text modality, an image modality, and a video modality, etc. The difference page image and the data of the at least two modalities can be understood as content for updating the screen of the target object client in real time. The data frame can be a stream data frame.

[0115] Specifically, the data of the at least two modalities included in the difference page image can be determined, and the difference page data and the data of the at least two modalities can be encapsulated into a data frame and sent to the risk identification server.

[0116] In actual application, in combination with the above Figure 3 The difference page image and the data of the at least two modalities can be encapsulated by a bypass stream, encapsulated into a stream data frame, and sent to the risk identification server. In specific implementation, the difference page image and the data of the at least two modalities can be converted into a target transmission format, and an encapsulation format can be determined according to a network transmission protocol. The encapsulation format can be, for example, an Ethernet frame format. A frame header and a frame trailer can be added before and after the difference page image and the data of the at least two modalities. The frame header can include control information, such as synchronization information and address information, etc. The frame trailer can include check information for detecting whether an error occurs in the data during transmission, so as to complete encapsulation of the difference page image and the data of the at least two modalities.

[0117] In summary, by encapsulating into a data frame and sending to the risk identification server, the amount of data transmitted each time is reduced, the integrity of each data transmission is ensured, and subsequent behavior risk identification of the data frame is facilitated.

[0118] Step 206: The risk identification server identifies the data of the at least two modalities included in the difference page image according to a risk identification model, and obtains an image risk identification result of the difference page image.

[0119] The image risk identification result can be understood as a result of whether the difference page image has a risk. For example, if there is abnormal information in the difference page image, the image risk identification result is that the difference page image has a risk. The image risk identification result can also include a risk type of the difference page image. The risk type can include a data leakage risk, a browsing of an abnormal website risk, an access to an application that is not allowed to access risk, etc. The risk identification model can be understood as a multi-modal risk identification model. The multi-modal risk identification model can identify risks of text data, image data, and video data, etc. The multi-modal risk identification model has text understanding capability, image understanding capability, and video understanding capability.

[0120] Specifically, the risk identification server can perform risk identification on the difference page image and data of at least two modalities included in the difference page image according to the risk identification model to obtain an image risk identification result of the difference page.

[0121] For example, the difference page image includes text data, image data, and video data. The difference page image can be input into the risk identification model to obtain an image risk identification result of the difference page image. Alternatively, the text data, image data, and video data included in the difference page image can be input into the risk identification model to obtain an image risk identification result of the difference page image. Alternatively, the difference page image and the text data, image data, and video data included in the difference page image can be input into the risk identification model to obtain an image risk identification result of the difference page image.

[0122] In an embodiment of the present specification, the risk identification model can be multiple, such as a text risk identification model, an image risk identification model, and a video risk identification model. The multiple risk identification models can be used to perform risk identification on data of at least two modalities respectively to obtain an image risk identification result of the difference page image. For example, when it is determined that the difference page image includes text data and image data, the text risk identification model can be used to perform risk identification on the text data, and the image risk identification model can be used to perform risk identification on the image data to obtain an image risk identification result of the difference page image.

[0123] In an embodiment of the present specification, the risk identification model is a multi-modal machine learning model. In another embodiment of the present specification, the risk identification model can be a multi-modal large model. It can be understood that the risk identification model can be supervised or unsupervised before risk identification according to the risk identification model.

[0124] In a specific implementation, the risk identification server performs risk identification on data of at least two modalities included in the difference page image according to the risk identification model to obtain an image risk identification result of the difference page image, including:

[0125] The risk identification server calls the risk identification model through a model calling interface, performs risk identification on data of at least two modalities included in the difference page image according to the called risk identification model, and obtains an image risk identification result of the difference page image.

[0126] Specifically, the risk identification server can call the risk identification model through a model calling interface deployed by the risk identification server, input the difference page image and the data of at least two modalities included in the difference page image into the risk identification model for risk identification, and obtain the image risk identification result of the difference page image output by the risk identification model. In addition, the risk identification model can also be deployed on the risk identification server, and the risk identification server can directly use the risk identification model.

[0127] In summary, the risk identification process is introduced into the push server, the multi-modal large model capability is called, and the understanding and identification capability of the user's conversation content is enhanced, which does not rely on user environment to collect operation sequence or application software by burying points.

[0128] Step 208: determining the behavior risk identification result of the current operation behavior according to the image risk identification result.

[0129] The behavior risk identification result can be understood as the result of whether the current operation behavior has a risk, and the behavior risk identification result can also include the risk type of the current operation behavior.

[0130] In specific implementation, the determination of the behavior risk identification result of the current operation behavior according to the image risk identification result includes:

[0131] According to the image risk identification result, the image risk identification result and the reference behavior rule are matched to obtain a matching result, and the behavior risk identification result of the current operation behavior is determined according to the matching result.

[0132] The reference behavior rule is a pre-defined risk behavior rule.

[0133] Specifically, the image risk identification result and the pre-defined risk behavior rule can be matched according to the image risk identification result to obtain a matching result, and the image risk identification result is determined whether it hits the pre-defined risk behavior rule according to the matching result, so as to determine the behavior risk identification result of the current operation behavior.

[0134] In actual application, referring to Figure 3 The risk identification server can be accessed through a three-party configuration system to realize the configuration of the reference behavior rule. Specifically, the reference behavior rule can include content audit, action rule and enterprise knowledge base. The enterprise can customize content audit, prompt words, action and enterprise knowledge base and other configurations according to project requirements. The content audit can include monitoring whether the enterprise employee installs an application that is not allowed to be installed, sending data files to the outside, and having abnormal files, etc. The action rule can include risk behaviors defined by the enterprise, such as deleting files.

[0135] In summary, by accessing the three-party configuration system, allowing enterprises to customize configuration actions and update enterprise knowledge base, risk identification of illegal operations, data leakage, illegal software installation and other actions of enterprise employees can be performed, which can integrate the understanding of enterprise knowledge on the basis of general knowledge understanding of large models, realize the retrieval enhancement of external knowledge base in large models, and have strong scalability. Moreover, the cloud server directly interfaces the large model and the enterprise knowledge base, and the demand for computing power, desktop performance overhead of the target object client is small, reducing the impact on user experience.

[0136] In practical applications, after determining the behavior risk identification result of the current operation behavior, the method further includes:

[0137] The risk identification server determines a control instruction for the current operation behavior according to the behavior risk identification result of the current operation behavior, and sends the control instruction to the target object server;

[0138] The target object server processes the current operation behavior according to the control instruction.

[0139] The control instruction can include an alarm instruction, an execution instruction, and an interception instruction. When the control instruction is an alarm instruction, the current operation behavior is not executed according to the alarm instruction, and alarm information is generated and sent to the target object client. When the control instruction is an execution instruction, the current operation behavior can be directly executed. When the control instruction is an interception instruction, the current operation behavior can be intercepted.

[0140] In practical applications, in combination with the above Figure 3 The risk identification server can determine a control instruction for the current operation behavior according to the behavior risk identification result, send the control instruction to the target object server, and process the current operation behavior according to the control instruction by the interaction control process in the target object server. In addition, the risk identification server can also access a monitoring and alarm system, which can generate and send alarm information for risk behaviors with high risk levels. The monitoring and alarm system can access a project system and send a short message or an email to project personnel in the case of monitoring a risk behavior with a high risk level.

[0141] In summary, by intervening and controlling the current operation behavior, the security of using the cloud computer by the user is ensured.

[0142] In an embodiment of the present specification, after determining the first page image corresponding to the current operation behavior in response to the target object client sending the current operation behavior, the method further includes:

[0143] The target object server sends the first page image to the risk identification server in a case where it is determined that there is no previous operation behavior of the current operation behavior.

[0144] The risk identification server identifies risks of data of at least two modalities included in the first page image according to a risk identification model, to obtain a first image risk identification result of the first page image.

[0145] The behavior risk identification result of the current operation behavior is determined according to the first image risk identification result.

[0146] The first image risk identification result of the first page image can be understood as an image risk identification result of the first page image.

[0147] Specifically, the target object server determines that the current operation behavior is the first operation behavior initiated by the user in a case where it is determined that there is no previous operation behavior of the current operation behavior. At this time, the target object server can directly send the first page image corresponding to the current operation behavior to the risk identification server. The risk identification server can identify risks of data of at least two modalities included in the first page image according to a risk identification model, to obtain a first image risk identification result of the first page image. The behavior risk identification result of the current operation behavior is determined according to the first image risk identification result.

[0148] In an embodiment of the present specification, the target object server determines a first page image corresponding to the current operation behavior and a second page image corresponding to a previous operation behavior of the current operation behavior in response to the current operation behavior sent by the target object client. The target object server determines a difference page image between the first page image and the second page image according to the first page image and the second page image, thereby obtaining a difference page image that appears different from the previous operation behavior based on the current operation behavior of the target object client. The target object server sends the difference page image to the risk identification server. The risk identification server identifies risks of the difference page image according to a risk identification model, without identifying risks of the entire first page image, thereby reducing the use cost of the risk identification model and saving the resource occupation of the risk identification server. In addition, the risk identification model can also identify risks of data of at least two modalities included in the difference page image, thereby further ensuring the comprehensiveness and accuracy of risk identification, so that the behavior risk identification result determined according to the image risk identification result is more accurate, and the risk operation behavior is avoided to be missed and misjudged.

[0149] The following will be described in combination with the accompanying drawings. Figure 4 The operation behavior identification method provided in the present specification is taken as an example to further illustrate the operation behavior identification method. Wherein, Figure 4A processing process flow chart of an operation behavior recognition method provided by one embodiment of the present specification is shown, and specifically includes the following steps.

[0150] Step 402: The target object server determines the first page image corresponding to the current operation behavior and determines the second page image corresponding to the previous operation behavior of the current operation behavior in response to the current operation behavior sent by the target object client.

[0151] Specifically, the target object client is the client of the cloud computer, which can be understood as the client connected to the cloud computer through the streaming protocol, and can include a soft terminal or a hardware terminal. The target object server can be understood as an environment running the cloud computer, which can include an operating system environment, a streaming server and an interactive control process. The target object client can send the current operation behavior to the target object server, and the target object server determines the page data corresponding to the current operation behavior in response to the current operation behavior, and renders the page data in the operating system environment, and obtains the first page image corresponding to the current operation behavior by taking a screenshot of the operating system environment. The streaming server can send the first page image to the target object client according to the streaming protocol, so as to facilitate the display of the first page image on the display interface of the target object client.

[0152] The server of the cloud computer can respond to the current operation behavior sent by the user through the client of the cloud computer, determine the page data corresponding to the current operation behavior, obtain the first page image corresponding to the current operation behavior, and obtain the second page image corresponding to the previous operation behavior of the current operation behavior.

[0153] Step 404: The target object server determines the difference page image between the first page image and the second page image according to the first page image and the second page image.

[0154] Specifically, a plurality of first pixel points in the first page image can be determined, and a plurality of second pixel points in the second page image corresponding to the positions of the plurality of first pixel points can be determined. The pixel similarity between the first pixel points and the second pixel points at the same position is calculated. The second pixel points with a pixel similarity less than a preset similarity threshold are determined as target pixel points, and the difference page image is formed according to the target pixel points.

[0155] For example, the first pixel point A corresponding to the position 1 in the first page image and the second pixel point B corresponding to the position 1 in the second page image can be determined. Then, the image positions of the first pixel point A and the second pixel point B are the same, the pixel similarity between the first pixel point A and the second pixel point B can be calculated, and in a case where the pixel similarity is less than a preset similarity threshold, the second pixel point B is determined as a target pixel point. Similarly, other pixel points at the same positions in the first page image and the second page image are calculated in the same manner, and the difference page image is formed according to the target pixel points.

[0156] Step 406: The target object server encapsulates the difference page image and the data of at least two modalities included in the difference page image into a data frame and sends the data frame to the risk identification server.

[0157] Step 408: The risk identification server performs risk identification on the data of at least two modalities included in the difference page image according to a risk identification model to obtain an image risk identification result of the difference page image.

[0158] Step 410: The risk identification server determines a behavior risk identification result of the current operation behavior according to the image risk identification result.

[0159] Step 412: The risk identification server determines a control instruction for the current operation behavior according to the behavior risk identification result and sends the control instruction to the target object server.

[0160] In an embodiment of the present specification, the target object server determines a first page image corresponding to a current operation behavior in response to the current operation behavior sent by the target object client, determines a second page image corresponding to a previous operation behavior of the current operation behavior, determines a difference page image between the first page image and the second page image according to the first page image and the second page image, thereby obtaining a difference page image that appears different from the previous operation behavior based on the current operation behavior of the target object client, and sends the difference page image to the risk identification server. The risk identification server performs risk identification on the difference page image according to a risk identification model, without performing risk identification on the entire first page image, thereby reducing the use cost of the risk identification model and saving the resource occupation of the risk identification server. In addition, the risk identification model can also perform risk identification on the data of at least two modalities included in the difference page image, thereby further ensuring the comprehensiveness and accuracy of risk identification, so that the behavior risk identification result determined according to the image risk identification result is more accurate, and the risk operation behavior is avoided to be missed and misjudged.

[0161] Corresponding to the method embodiments described above, the present specification also provides operation behavior identification system embodiments,Figure 5 An embodiment of the present specification provides a structural schematic diagram of an operation behavior recognition system. As shown in the figure, Figure 5 The system 500 includes:

[0162] a target object server 502 and a risk identification server 504, wherein,

[0163] The target object server 502 is configured to determine a first page image corresponding to a current operation behavior in response to the current operation behavior sent by a target object client, and determine a second page image corresponding to a previous operation behavior of the current operation behavior,

[0164] According to the first page image and the second page image, a difference page image between the first page image and the second page image is determined, and the difference page image is sent to the risk identification server 504;

[0165] The risk identification server 504 is configured to perform risk identification on data of at least two modalities included in the difference page image according to a risk identification model, to obtain an image risk identification result of the difference page image,

[0166] According to the image risk identification result, a behavior risk identification result of the current operation behavior is determined.

[0167] In an optional embodiment, the target object server is further configured to send the first page image to the risk identification server in the case where it is determined that there is no previous operation behavior of the current operation behavior;

[0168] The risk identification server is further configured to perform risk identification on data of at least two modalities included in the first page image according to a risk identification model, to obtain a first image risk identification result of the first page image,

[0169] According to the first image risk identification result, a behavior risk identification result of the current operation behavior is determined.

[0170] In an optional embodiment, the risk identification server is further configured to:

[0171] According to the image risk identification result, the image risk identification result and a reference behavior rule are matched to obtain a matching result, and the matching result is used to determine the behavior risk identification result of the current operation behavior;

[0172] The reference behavior rule is a pre-defined risk behavior rule.

[0173] In an optional embodiment, the risk identification server is further configured to determine a control instruction for the current operation behavior according to the behavior risk identification result of the current operation behavior, and send the control instruction to the target object server.

[0174] The target object server is further configured to process the current operation behavior according to the control instruction.

[0175] In an optional embodiment, the target object server is further configured to:

[0176] calculate an image similarity between the first page image and the second page image;

[0177] determine the difference page image between the first page image and the second page image according to the image similarity.

[0178] In an optional embodiment, the target object server is further configured to:

[0179] determine a plurality of first pixel points in the first page image, and determine a plurality of second pixel points in the second page image at the same positions as the plurality of first pixel points;

[0180] calculate a pixel similarity between the first pixel points and the second pixel points;

[0181] The determining the difference page image between the first page image and the second page image according to the image similarity comprises:

[0182] determining a target pixel point from the second pixel points with a pixel similarity less than a preset similarity threshold, and determining the difference page image according to the target pixel point.

[0183] In an optional embodiment, the target object server is further configured to:

[0184] determine data of at least two modalities included in the difference page image;

[0185] encapsulate the difference page image and the data of the at least two modalities into a data frame and send the data frame to the risk identification server.

[0186] In an optional embodiment, the risk identification server is further configured to call the risk identification model through a model calling interface, perform risk identification on the data of the at least two modalities included in the difference page image according to the called risk identification model, and obtain an image risk identification result of the difference page image.

[0187] In an optional embodiment, the target object server is further configured to:

[0188] determine page data corresponding to the current operation behavior in response to the current operation behavior sent by the target object client;

[0189] render the page data to obtain the first page image corresponding to the current operation behavior.

[0190] In an optional embodiment, the target object server is further configured to:

[0191] send the first page image to the target object client for display through a display interface of the target object client.

[0192] In an optional embodiment, the target object server is further configured to:

[0193] store the first page image in a database, so that in response to a next operation behavior of the current operation behavior, the first page image is obtained from the database to identify the behavior risk of the next operation behavior.

[0194] In an optional embodiment, the risk identification model is a multi-modal machine learning model.

[0195] In an optional embodiment, the target object server is a server of a cloud computer, and the target object client is a client of the cloud computer.

[0196] In an embodiment of the present specification, the target object server determines a first page image corresponding to a current operation behavior of a target object client in response to the current operation behavior sent by the target object client, and determines a second page image corresponding to a previous operation behavior of the current operation behavior, determines a difference page image between the first page image and the second page image according to the first page image and the second page image, thereby obtaining a difference page image that appears different from the previous operation behavior based on the current operation behavior of the target object client, sends the difference page image to a risk identification server, and the risk identification server identifies the risk of the difference page image according to a risk identification model, without identifying the risk of the entire first page image, thereby reducing the use cost of the risk identification model, thereby saving the resource occupation of the risk identification server. Moreover, the risk identification model can also identify the risk of at least two modal data included in the difference page image, further ensuring the comprehensiveness and accuracy of risk identification, so that the behavior risk identification result determined according to the image risk identification result is more accurate, avoiding missed detection and misjudgment of risk operation behavior.

[0197] The above is a schematic scheme of an operation behavior identification system according to an embodiment. It should be noted that the technical scheme of the operation behavior identification system and the technical scheme of the operation behavior identification method described above belong to the same concept, and the details of the technical scheme of the operation behavior identification system that are not described in detail can be referred to the description of the technical scheme of the operation behavior identification method.

[0198] Referring to Figure 6 , Figure 6 A flowchart of a second operation behavior identification method according to an embodiment of the present specification is shown, which is applied to a target object server and specifically includes the following steps.

[0199] Step 602: In response to the current operation behavior sent by the target object client, determining a first page image corresponding to the current operation behavior, and determining a second page image corresponding to a previous operation behavior of the current operation behavior;

[0200] Step 604: According to the first page image and the second page image, determining a difference page image between the first page image and the second page image, and sending the difference page image to a risk identification server;

[0201] Step 606: Receive the behavior risk identification result of the current operation behavior sent by the risk identification server, wherein the behavior risk identification result is determined according to an image risk identification result, and the image risk identification result is obtained by the risk identification server according to a risk identification model, and the risk identification of at least two modal data included in the difference page image.

[0202] In an embodiment of the present specification, the target object server determines the first page image corresponding to the current operation behavior and the second page image corresponding to the previous operation behavior of the current operation behavior in response to the current operation behavior sent by the target object client, and determines the difference page image between the first page image and the second page image according to the first page image and the second page image, thereby obtaining the difference page image that appears different from the previous operation behavior based on the current operation behavior of the target object client. The difference page image is sent to the risk identification server, and the risk identification server performs risk identification on the difference page image according to the risk identification model, without performing risk identification on the entire first page image, reducing the use cost of the risk identification model, thereby saving the resource occupation of the risk identification server. Moreover, the risk identification model can also perform risk identification on at least two modal data included in the difference page image, further ensuring the comprehensiveness and accuracy of risk identification, so that the behavior risk identification result determined according to the image risk identification result is more accurate, avoiding missed detection and misjudgment of risk operation behavior.

[0203] The above is a schematic scheme of the operation behavior recognition method of the embodiment. It should be noted that the technical scheme of the operation behavior recognition method belongs to the same concept as the technical scheme of the operation behavior recognition method described above, and the details of the operation behavior recognition method which are not described in detail can be referred to the description of the technical scheme of the operation behavior recognition method.

[0204] Corresponding to the method embodiment, the specification also provides an operation behavior recognition device embodiment, applied to a target object server, Figure 7 A structural schematic diagram of a second operation behavior recognition device provided by an embodiment of the specification is shown.

[0205] As Figure 7 shown, the device comprises:

[0206] A first determination module 702 is configured to determine a first page image corresponding to a current operation behavior in response to the current operation behavior sent by a target object client, and determine a second page image corresponding to a previous operation behavior of the current operation behavior;

[0207] A second determination module 704 is configured to determine a difference page image between the first page image and the second page image according to the first page image and the second page image, and send the difference page image to a risk identification server;

[0208] A receiving module 706 is configured to receive a behavior risk identification result of the current operation behavior sent by the risk identification server, wherein the behavior risk identification result is determined according to an image risk identification result, and the image risk identification result is obtained by the risk identification server according to a risk identification model, by performing risk identification on data of at least two modalities included in the difference page image.

[0209] In an embodiment of the present specification, the target object server determines a first page image corresponding to a current operation behavior in response to the target object client sending the current operation behavior, and determines a second page image corresponding to a previous operation behavior of the current operation behavior, determines a difference page image between the first page image and the second page image according to the first page image and the second page image, thereby obtaining a difference page image that appears different from the previous operation behavior based on the current operation behavior of the target object client, and sends the difference page image to the risk identification server. The risk identification server identifies the risk according to the risk identification model. The risk identification model does not need to identify the risk of the entire first page image, reduces the use cost of the risk identification model, thereby saving the resource occupation of the risk identification server. Moreover, the risk identification model can also identify the risk of the data of at least two modalities included in the difference page image, further ensuring the comprehensiveness and accuracy of the risk identification, thereby making the behavior risk identification result determined according to the image risk identification result more accurate, avoiding the missed detection and misjudgment of the risk operation behavior.

[0210] The above is a schematic scheme of an operation behavior identification device according to an embodiment of the present specification. It should be noted that the technical scheme of the operation behavior identification device belongs to the same concept as the technical scheme of the operation behavior identification method described above. The technical scheme of the operation behavior identification device is not described in detail, and the description of the technical scheme of the operation behavior identification method described above can be referred to.

[0211] Referring to Figure 8 , Figure 8 A flowchart of a third operation behavior identification method according to an embodiment of the present specification is shown, which is applied to a risk identification server and specifically includes the following steps.

[0212] Step 802: receiving a difference page image sent by a target object server in response to a current operation behavior sent by a target object client, wherein the difference page image is determined by the target object server according to a first page image corresponding to the current operation behavior and a second page image corresponding to a previous operation behavior of the current operation behavior;

[0213] Step 804: identifying the risk of at least two modalities of data included in the difference page image according to a risk identification model, to obtain an image risk identification result of the difference page image;

[0214] Step 806: determining a behavior risk identification result of the current operation behavior according to the image risk identification result.

[0215] In one embodiment of this specification, the target object server, in response to a current operation sent by the target object client, determines a first page image corresponding to the current operation and a second page image corresponding to the previous operation. Based on the first and second page images, it determines a difference page image between the first and second page images, thereby obtaining a difference page image generated based on the target object client's current operation that differs from the previous operation. This difference page image is then sent to a risk identification server. The risk identification server performs risk identification on the difference page image according to a risk identification model, eliminating the need to perform risk identification on the entire first page image, thus reducing the cost of using the risk identification model and saving resources on the risk identification server. Furthermore, the risk identification model can also perform risk identification on at least two modalities included in the difference page image, further ensuring the comprehensiveness and accuracy of risk identification. This makes the subsequent behavioral risk identification results determined based on the image risk identification results more accurate, avoiding missed detections and misjudgments of risky operations.

[0216] The above is an illustrative scheme of an operation behavior recognition method according to this embodiment. It should be noted that the technical solution of this operation behavior recognition method belongs to the same concept as the operation behavior recognition method described above. For details not described in detail in the technical solution of the operation behavior recognition method, please refer to the description of the technical solution of the operation behavior recognition method described above.

[0217] Corresponding to the above method embodiments, this specification also provides embodiments of an operation behavior recognition device, applied to a risk recognition server. Figure 9 A schematic diagram of the structure of a third operation behavior recognition device provided in one embodiment of this specification is shown.

[0218] like Figure 9 As shown, the device includes:

[0219] The receiving module 902 is configured to receive the current operation behavior and the difference page image sent by the target object server in response to the target object client, wherein the difference page image is determined by the target object server based on the first page image corresponding to the current operation behavior and the second page image corresponding to the previous operation behavior of the current operation behavior.

[0220] The identification module 904 is configured to perform risk identification on the data of at least two modalities included in the difference page image according to the risk identification model, and obtain the image risk identification result of the difference page image;

[0221] The determination module 906 is configured to determine the behavior risk identification result of the current operation behavior based on the image risk identification result.

[0222] In an embodiment of the present specification, the target object server determines a first page image corresponding to the current operation behavior in response to the target object client sending the current operation behavior, and determines a second page image corresponding to a previous operation behavior of the current operation behavior, determines a difference page image between the first page image and the second page image according to the first page image and the second page image, thereby obtaining a difference page image that appears different from the previous operation behavior based on the current operation behavior of the target object client, and sends the difference page image to the risk identification server. The risk identification server identifies the risk according to the risk identification model for the difference page image, without identifying the risk for the entire first page image, thereby reducing the use cost of the risk identification model, thereby saving the resource occupation of the risk identification server. Moreover, the risk identification model can also identify the risk of the data of at least two modalities included in the difference page image, further ensuring the comprehensiveness and accuracy of the risk identification, thereby making the behavior risk identification result determined according to the image risk identification result more accurate, avoiding the missed detection and misjudgment of the risk operation behavior.

[0223] The above is a schematic scheme of an operation behavior identification device according to an embodiment of the present specification. It should be noted that the technical scheme of the operation behavior identification device belongs to the same concept as the technical scheme of the operation behavior identification method described above, and the details of the technical scheme of the operation behavior identification device which are not described in detail can be referred to the description of the technical scheme of the operation behavior identification method.

[0224] Figure 10 The structure block diagram of a computing device 1000 according to an embodiment of the present specification is shown. The components of the computing device 1000 include but are not limited to a memory 1010 and a processor 1020. The processor 1020 is connected with the memory 1010 through a bus 1030, and a database 1050 is used to save data.

[0225] The computing device 1000 also includes an access device 1040 that enables the computing device 1000 to communicate via one or more networks 1060. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or combinations of such networks, such as the Internet. The access device 1040 can include one or more of any type of network interface (for example, a network interface card (NIC)) such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, Near Field Communication (NFC).

[0226] In one embodiment of the present specification, the above-mentioned components of the computing device 1000 and other components not shown in the Figure 10 may be connected to each other, for example, through a bus. It should be understood that Figure 10 The computing device structure diagram shown is only for the purpose of example, and is not a limitation on the scope of the present specification. Those skilled in the art can add or replace other components as needed.

[0227] The computing device 1000 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (for example, a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, and the like), a mobile phone (for example, a smartphone), a wearable computing device (for example, a smart watch, smart glasses, and the like), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1000 can also be a mobile or stationary server.

[0228] The processor 1020 is configured to execute computer program / instructions that implement the steps of the above-mentioned operation behavior recognition method when the computer program / instructions are executed by the processor.

[0229] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts among the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, the computing device embodiment is described simply because it is basically similar to the operation behavior recognition method embodiment, and the relevant part can be referred to the part of the operation behavior recognition method embodiment.

[0230] An embodiment of the specification also provides a computer readable storage medium storing computer programs / instructions, which are executed by a processor to implement the steps of the operation behavior recognition method.

[0231] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts among the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, the computer readable storage medium embodiment is described simply because it is basically similar to the operation behavior recognition method embodiment, and the relevant part can be referred to the part of the operation behavior recognition method embodiment.

[0232] An embodiment of the specification also provides a computer program product including computer programs / instructions, which are executed by a processor to implement the steps of the operation behavior recognition method.

[0233] The above is a schematic scheme of the computer program product of the embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the operation behavior recognition method described above belong to the same concept, and the details of the technical scheme of the computer program product which are not described in detail can be referred to the description of the technical scheme of the operation behavior recognition method.

[0234] The above describes specific embodiments of the specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims can be performed in a different order than the order in which they are recited and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0235] The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, etc. It should be noted that the computer readable medium can include appropriate additions or deletions according to the requirements of patent practice, for example, according to the patent practice in some regions, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0236] It should be noted that, for the foregoing method embodiments, in order to facilitate description, they are all expressed as a combination of a series of actions, but those skilled in the art should know that the embodiments of the present specification are not limited by the order of the described actions, because according to the embodiments of the present specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of the present specification.

[0237] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0238] The preferred embodiments of the present specification disclosed above are only used to help explain the present specification. The alternative embodiments do not describe all the details and limit the invention to the specific embodiments described. Obviously, according to the content of the embodiments of the present specification, many modifications and changes can be made. The present specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present specification, so that those skilled in the art can well understand and use the present specification. The present specification is limited only by the claims and their entire scope and equivalents.

Claims

1. An operating behavior recognition method applied to an operating behavior recognition system, the operating behavior recognition system comprising a target object server and a risk identification server, wherein, The method comprises: The target object server determines a first page image corresponding to the current operation behavior and a second page image corresponding to a previous operation behavior of the current operation behavior in response to the current operation behavior sent by the target object client, According to the first page image and the second page image, a difference page image between the first page image and the second page image is determined, and the difference page image is sent to the risk identification server; The risk identification server performs risk identification on data of at least two modalities included in the difference page image according to a risk identification model to obtain an image risk identification result of the difference page image, According to the image risk identification result, a behavior risk identification result of the current operation behavior is determined.

2. The operation behavior identification method according to claim 1, after determining the first page image corresponding to the current operation behavior in response to the current operation behavior sent by the target object client, further comprising: The target object server sends the first page image to the risk identification server in a case where it is determined that there is no previous operation behavior of the current operation behavior; The risk identification server performs risk identification on data of at least two modalities included in the first page image according to a risk identification model to obtain a first image risk identification result of the first page image, According to the first image risk identification result, a behavior risk identification result of the current operation behavior is determined.

3. The operation behavior identification method according to claim 1, wherein determining the behavior risk identification result of the current operation behavior according to the image risk identification result comprises: According to the image risk identification result, the image risk identification result and a reference behavior rule are matched to obtain a matching result, and the behavior risk identification result of the current operation behavior is determined according to the matching result; The reference behavior rule is a pre-defined risk behavior rule.

4. The operation behavior identification method according to claim 1, after determining the behavior risk identification result of the current operation behavior, further comprising: The risk identification server determines a control instruction for the current operation behavior according to the behavior risk identification result of the current operation behavior, and sends the control instruction to the target object server; The target object server processes the current operation behavior according to the control instruction.

5. The operation behavior identification method according to claim 1, wherein determining the difference page image between the first page image and the second page image according to the first page image and the second page image comprises: Calculating the image similarity between the first page image and the second page image; According to the image similarity, the difference page image between the first page image and the second page image is determined.

6. The operation behavior identification method according to claim 5, wherein calculating the image similarity between the first page image and the second page image comprises: determine a plurality of first pixel points in the first page image, and determine a plurality of second pixel points in the second page image, which are at the same positions as the plurality of first pixel points; calculate pixel similarity between the first pixel points and the second pixel points; the determining the difference page image between the first page image and the second page image according to the image similarity comprises: determining a second pixel point with a pixel similarity less than a preset similarity threshold as a target pixel point, and determining the difference page image according to the target pixel point.

7. The operation behavior recognition method according to claim 1, before the sending the difference page image to the risk recognition server, further comprising: determining data of at least two modalities included in the difference page image; the sending the difference page image to the risk recognition server comprises: packaging the difference page image and the data of the at least two modalities into a data frame and sending to the risk recognition server.

8. The operation behavior recognition method according to claim 1, the determining the first page image corresponding to the current operation behavior in response to the current operation behavior sent by the target object client, comprising: determining page data corresponding to the current operation behavior in response to the current operation behavior sent by the target object client; rendering the page data to obtain the first page image corresponding to the current operation behavior.

9. The operation behavior recognition method according to claim 8, after the obtaining the first page image corresponding to the current operation behavior, further comprising: sending the first page image to the target object client for display on a display interface of the target object client.

10. The operation behavior recognition method according to claim 1, after the determining the first page image corresponding to the current operation behavior, further comprising: storing the first page image to a database, so that in response to a next operation behavior of the current operation behavior, the first page image is obtained from the database to perform behavior risk identification on the next operation behavior.

11. The method of claim 1-10, wherein, the risk recognition model is a multi-modal machine learning model.

12. The method of claim 1-10, wherein, the target object server is a server of a cloud computer, and the target object client is a client of the cloud computer.

13. An operation behavior recognition method applied to a target object server, wherein, the method comprises: determining a first page image corresponding to a current operation behavior in response to the current operation behavior sent by a target object client, and determining a second page image corresponding to a previous operation behavior of the current operation behavior; determining a difference page image between the first page image and the second page image according to the first page image and the second page image, and sending the difference page image to a risk recognition server; receive a behavior risk identification result of the current operation behavior sent by the risk identification server, wherein the behavior risk identification result is determined according to an image risk identification result, and the image risk identification result is obtained by the risk identification server according to a risk identification model by performing risk identification on data of at least two modalities included in the difference page image.

14. An operation behavior recognition method applied to a risk identification server, wherein, The method comprises: receiving a difference page image sent by a target object server in response to a current operation behavior sent by a target object client, wherein the difference page image is determined by the target object server according to a first page image corresponding to the current operation behavior and a second page image corresponding to a previous operation behavior of the current operation behavior; performing risk identification on data of at least two modalities included in the difference page image according to a risk identification model to obtain an image risk identification result of the difference page image; determining a behavior risk identification result of the current operation behavior according to the image risk identification result.

15. An operation behavior identification system, comprising a target object server and a risk identification server, wherein the target object server is configured to determine a first page image corresponding to a current operation behavior and determine a second page image corresponding to a previous operation behavior of the current operation behavior in response to the current operation behavior sent by a target object client, determine a difference page image between the first page image and the second page image according to the first page image and the second page image, and send the difference page image to the risk identification server; the risk identification server is configured to perform risk identification on data of at least two modalities included in the difference page image according to a risk identification model to obtain an image risk identification result of the difference page image, determine a behavior risk identification result of the current operation behavior according to the image risk identification result.

16. A computing device, comprising: a memory and a processor; the memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which realize the steps of the method in any one of claims 1 to 14 when executed by the processor.

17. A computer readable storage medium storing computer programs / instructions, which realize the steps of the method in any one of claims 1 to 14 when executed by the processor.

18. A computer program product comprising computer programs / instructions, which realize the steps of the method in any one of claims 1 to 14 when executed by the processor.

Images