Object Behavior Simulation Method and System

Abstract

An embodiment of the present application discloses an object behavior simulation method, the method comprising: obtaining an object behavior data set, the object behavior data set including real object behavior data obtained based on real object behaviors; training a model according to the object behavior data set, and outputting a virtual object behavior sequence based on the trained model; and according to the virtual object behavior sequence, performing simulated object behaviors under a test account through a target host. Compared with manually operating or writing scripts to generate object behaviors, in this embodiment, the model is trained through the object behavior data set to generate a virtual object behavior sequence, and object behavior simulation is performed based on the virtual object behavior sequence, improving the efficiency, scalability and adaptability of object behavior simulation and reducing the development cost.

Description

Technical Field

[0001] This application relates to the field of computers, and particularly to an object behavior simulation method, system, computer device, and computer-readable storage medium. Background Art

[0002] Object behavior simulation is used to simulate real object behaviors and is commonly used in scenarios such as software testing and intelligent recognition. In the prior art, object behavior simulation still stays at the stage of manually determining the behavior operation plan and giving judgment exits, and simulating real object behaviors through program execution operations. For example, real object behaviors are simulated by executing manually determined automation cases or object behavior sequences through automation testing software such as selenium.

[0003] However, generating object behaviors through manual operations or writing scripts has high development costs, poor scalability, and poor adaptability. Summary of the Invention

[0004] The purpose of the embodiments of this application is to provide an object behavior simulation method, system, computer device, and computer-readable storage medium to solve the following technical problems: generating object behaviors through manual operations or writing scripts has high development costs, poor scalability, and poor adaptability.

[0005] One aspect of the embodiments of this application provides an object behavior simulation method, and the method includes:

[0006] Obtain an object behavior data set, where the object behavior data set includes real object behavior data obtained based on real object behaviors;

[0007] Train a model according to the object behavior data set, and output a virtual object behavior sequence based on the trained model; and

[0008] Execute simulated object behaviors under a test account through a target host according to the virtual object behavior sequence.

[0009] Optionally, the executing simulated object behaviors under a test account through a target host according to the virtual object behavior sequence includes:

[0010] Decode the virtual object behavior sequence to obtain multiple virtual object behaviors; and

[0011] Add one or more noise behaviors to the multiple virtual object behaviors to obtain the simulated object behaviors.

[0012] Optionally, the executing simulated object behaviors under a test account through a target host according to the virtual object behavior sequence includes:

[0013] Decode the virtual object behavior sequence to obtain multiple virtual object behaviors;

[0014] Obtain string / parallel operation instructions with a time series according to the multiple virtual object behaviors; and

[0015] According to the string / parallel operation instructions, simulate and execute the virtual object behaviors under the test account through the target host.

[0016] Optionally, the simulating and executing the virtual object behaviors under the test account according to the virtual object behavior sequence through the target host includes:

[0017] Simulate and execute the virtual object behaviors by calling an interface; and / or

[0018] In the browser incognito mode or IP proxy mode, simulate and execute the virtual object behaviors through the browser of the target host.

[0019] Optionally, the training the model according to the object behavior dataset and outputting a virtual object behavior sequence based on the trained model includes:

[0020] Train the model according to the object behavior dataset and preset reward parameters.

[0021] Optionally, it further includes:

[0022] Judge whether the test account triggers a risk control policy; and

[0023] If the test account triggers a risk control policy, update the virtual object behavior data of the virtual object behavior to the object behavior dataset as negative feedback data in the object behavior dataset.

[0024] Optionally, the model includes a D3QN network; the training the model according to the object behavior dataset and outputting a virtual object behavior sequence based on the trained model includes:

[0025] Perform multi-round training on the model, and each round of trained model corresponds to outputting a virtual object behavior sequence;

[0026] Among them, the following operations are performed in each round of training:

[0027] Select m groups of sample behavior data from the object behavior dataset, where m is an integer greater than 1;

[0028] Construct a two-dimensional matrix [X T , Y A for each group of sample behavior data, where X T = X1, X2,..., Xt Indicates actions that are executed sequentially, each action has multiple synchronous execution actions Y t，A =Y t，1 , Y t，2 ,...,Y t，a ;

[0029] Each two-dimensional matrix is converted into a one-dimensional matrix by reducing the dimension of the matrix, wherein the dimension reduction is to aggregate each column action in the corresponding two-dimensional matrix into a single action; wherein the single action represents the action taken at the current time j;

[0030] The Q function is iteratively updated according to each one-dimensional matrix, wherein the update formula of the Q function is as follows:

[0031] Q(S,A,θ,α,β)=V(S,θ,α)+A(S,A,θ,β)-loss(A);

[0032] a max (S′, θ)=arg max Q(S′, θ, a);

[0033] y j =R+γQ′((S j ), arg max Q((S j ′), θ, a);

[0034] Where S is the current state, S′ is the next state, A is the current action, arg max Q(S′, θ, a) is the next action, θ is the network parameter, α is the network parameter of the value function, β is the network parameter of the advantage function, V(S, θ, α) is the value function, A(S, A, θ, β) is the advantage function, loss(A) is the loss function, R is the reward value, γ is the decay value, argmax is the maximum function, y j is the target Q value of the Double Q layer at time j.

[0035] One aspect of an embodiment of the present application further provides an object behavior simulation system, the system comprising:

[0036] an acquisition module, configured to acquire an object behavior dataset, wherein the object behavior dataset includes real object behavior data obtained based on real object behavior;

[0037] a training module, configured to train the model according to the object behavior dataset and output a virtual object behavior sequence based on the trained model; and

[0038] The execution module is used to execute the simulated object behavior under the test account through the target host according to the virtual object behavior sequence.

[0039] One aspect of an embodiment of the present application further provides a computer device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it is used to implement the steps of the object behavior simulation method as described above.

[0040] One aspect of an embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program can be executed by at least one processor so that the at least one processor executes the steps of the object behavior simulation method as described above.

[0041] Compared with artificial operation or writing scripts to generate object behaviors, the object behavior simulation method, system, device, and computer-readable storage medium provided by the embodiments of the present application have the following advantages: training the model through an object behavior data set to generate a virtual object behavior sequence, and simulating object behaviors based on the virtual object behavior sequence, which improves the efficiency, scalability, and adaptability of object behavior simulation and reduces the development cost. Description of the Drawings

[0042] Figure 1 Schematically shows an application environment diagram of the object behavior simulation method according to an embodiment of the present application;

[0043] Figure 2 Schematically shows a flowchart of the object behavior simulation method according to Embodiment 1 of the present application;

[0044] Figure 3 For Figure 2 a sub-step flowchart of step S202 in

[0045] Figures 4 to 6 For Figure 2 a sub-step flowchart of step S204 in

[0046] Figure 7 Schematically shows an additional step flowchart of the object behavior simulation method according to Embodiment 1 of the present application;

[0047] Figure 8 Schematically shows a block diagram of the object behavior simulation system according to Embodiment 2 of the present application;

[0048] Figure 9 Schematically shows a hardware architecture diagram of a computer device suitable for implementing the object behavior simulation method according to Embodiment 3 of the present application. Detailed Embodiments

[0049] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without making creative efforts fall within the scope of protection of the present application.

[0050] It should be noted that in the embodiments of the present application, the descriptions involving "first", "second", etc. are only for descriptive purposes and cannot be construed as indicating or implying their relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between various embodiments can be combined with each other, but it must be based on the fact that those of ordinary skill in the art can implement them. When the combination of technical solutions results in contradictions or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present application.

[0051] In the description of the present application, it should be understood that the numerical labels before the steps do not identify the sequence of execution of the steps, but are only used to facilitate the description of the present application and distinguish each step, and thus cannot be construed as a limitation to the present application.

[0052] After analysis by the inventor, using a program to execute operations to simulate real objects currently only stays at the level of artificially determining the behavior operation plan and giving judgment exits. For example, using automation testing software such as selenium to execute the artificially determined automation cases or the sequence of object behaviors to simulate the behaviors of real objects has the following problems:

[0053] First, when using automation software to execute the artificially determined plan, if there are subsequent changes or updates to the attack strategy, it is necessary to synchronously maintain the code of the object behavior operation plan.

[0054] Second, artificially searching for object behaviors that can bypass risk control strategies and attacks requires a large amount of manpower.

[0055] Third, this kind of simulation of object behaviors has no significant effect on complementing risk control strategies.

[0056] In view of the above problems, the present application aims to simulate the behavior of real objects by avoiding the impact of risk control strategies. The D3QN algorithm is used to quickly generate the behavior of objects that can avoid the impact of risk control strategies. Moreover, as the risk control strategy changes, only a small amount of reward strategy code needs to be modified to complete the synchronous iteration of the simulated object behavior code. By simulating the behavior of real objects with a computer, the manpower investment is reduced, and it helps to fill the loopholes in the risk control strategy. The object can be an authorized user, and the object behavior can be user behavior.

[0057] The following are the term explanations of the present application:

[0058] DRL (Deep Reinforcement Learning): Combines the perception ability of deep learning DL (Deep Learning) and the decision-making ability of reinforcement learning RL (Reinforcement Learning). It can directly perform control based on the input information and is an artificial intelligence algorithm closer to the human thinking mode.

[0059] D3QN (Duelling Double Deep Q Network): A deep reinforcement learning algorithm that uses Q-learning as the network and is used for the training and generation of simulated object behavior in the present invention.

[0060] Figure 1 The environmental architecture diagram of the object behavior simulation method according to an embodiment of the present application is schematically shown. In an exemplary embodiment, as Figure 1 shown, the computer device 2 can be connected to multiple electronic devices 6 through the network 4. [[ID=]16]

[0061] The computer device 2 can collect object behavior data from the electronic devices 6 through one or more networks 4.

[0062] The computer device 2 can be composed of a single or multiple computing devices, such as rack servers, blade servers, tower servers, or cabinet servers (including independent servers or server clusters composed of multiple servers), etc. The one or more computer devices can include virtualized computing instances. The computer device can load virtual machines based on virtual images and / or other data that define specific software (such as operating systems, dedicated applications, servers) for simulation. As the demand for different types of processing services changes, different virtual machines can be loaded and / or terminated on the one or more computer devices.

[0063] Network 4 may include various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or the like. Network 4 may include physical links, such as coaxial cable links, twisted pair cable links, fiber optic links, combinations thereof, and / or the like. Network 4 may include wireless links, such as cellular links, satellite links, Wi-Fi links, and / or the like.

[0064] Electronic device 6 may be configured to access computer device 2. Electronic device 6 may include any type of computer device that may have a game engine, such as: smartphones, tablet devices, laptop computers, smart devices (e.g., smartwatches, smart glasses), virtual reality, gaming devices, set-top boxes, digital streaming devices, in-vehicle terminals, smart TVs, TV boxes, MP4 (Moving Picture Experts Group Audio Layer IV) players, etc.

[0065] Hereinafter, this application will provide an object behavior simulation method, system, computer device, and computer-readable storage medium.

[0066] Embodiment 1

[0067] Figure 2 Schematically shows a flowchart of an object behavior simulation method according to Embodiment 1 of this application. The method may be executed in computer device 2. As Figure 2 shown, the object behavior simulation method may include steps S200 to S204, where:

[0068] Step S200, obtain an object behavior data set, where the object behavior data set includes real object behavior data obtained based on real object behaviors.

[0069] In the above step, the real object behaviors may be one or more of: login, browsing, playing, following, sending dynamics, forwarding dynamics, commenting on dynamics, liking, sending bullet screens, coin - tossing, collecting, etc. The real object behavior data may be offline data and / or online data collected from various electronic devices. The real object behavior data constitutes an initial object behavior data set (experience pool) for training the model.

[0070] Step S202, train a model according to the object behavior data set, and output a virtual object behavior sequence based on the trained model.

[0071] In the above steps, the model is trained in units of "round()". In each round of training, the computer device 2 randomly samples multiple groups of object behavior data from the object behavior dataset and performs model training based on the sampled multiple groups of object behavior data. The model is a D3QN network or other model. After each round of training, the model outputs a virtual object behavior sequence based on the updated network parameters after training.

[0072] In an optional embodiment, as Figure 3 shown, step S202 may include: step S300, training the model according to the object behavior dataset and preset reward parameters. In this embodiment, the model training and output can be adjusted by setting relevant reward parameters, enabling it to iterate in the desired direction to quickly generate a virtual object behavior sequence that meets the requirements. Moreover, as the external environment (risk control strategy) changes, only by changing the reward parameters or a corresponding small amount of reward strategy code can the synchronization iteration of the simulated object behavior code be completed, improving adaptability.

[0073] Step S204, according to the virtual object behavior sequence, execute the simulated object behavior under the test account through the target host.

[0074] In the above steps, the operation instructions in the virtual object behavior sequence can be truly generated on the target host, thereby realizing that the simulated object behavior replaces the real object behavior of the real object on the target host.

[0075] In an optional embodiment, as Figure 4 shown, step S204 may include: step S400, decoding the virtual object behavior sequence to obtain multiple virtual object behaviors; and step S402, adding one or more noise behaviors to the multiple virtual object behaviors to obtain the simulated object behavior. During the decoding process, some noise behaviors are randomly generated to simulate operations that are invalid but often appear in real object behaviors, such as invalid mouse clicks, rapid browsing, dragging the progress bar, etc. Therefore, the authenticity of object behavior simulation can be further improved, reducing the risk of the account being identified as an illegal account.

[0076] In an optional embodiment, as Figure 5As shown, the step S204 may include: step S500, decoding the virtual object behavior sequence to obtain multiple virtual object behaviors; step S502, obtaining string / parallel operation instructions with a time series according to the multiple virtual object behaviors; and step S504, simulating and executing the simulated object behavior under the test account according to the string / parallel operation instructions through the target host. In actual operation, many operations are carried out simultaneously. To accurately simulate the real behavior of the object, a part of the parallel operations can be executed during the simulation of the object behavior to further increase the authenticity.

[0077] The simulated object behaviors shown in Table 1, where there are parallel object behaviors for T2, T4, T6, T7, and T8.

[0078] Time series Parallel operation T1 Login T2 Open the home page and click on the second video in the recommendation column T3 Play the video and drag the progress bar T4 Keep playing the video and scroll down the scroll wheel T5 Perform a long press on the like button T6 Enter a comment and click on the share button T7 Click on the comment send button and copy the share link T8 Close the video playback page and move the cursor to the personal profile picture T9 Logout

[0079] Table 1

[0080] In an optional embodiment, as Figure 6 shown, the step S204 may include: step S600, simulating and executing the simulated object behavior by calling an interface; and / or step S602, simulating and executing the simulated object behavior through the browser of the target host in the browser incognito mode or IP proxy mode. In this embodiment, the object behavior can be simulated by calling an interface. In addition, the inventor found that the simulation of object behavior is easily hit by the risk control engine and is easily recognized as an illegal or zombie account, resulting in account locking or cancellation. And compared with the browser, the anti-strike strategy of the App side is more complete. Therefore, in this embodiment, in order to prevent being hit by the risk control engine, the incognito browser and IP proxy methods can also be used to simulate object login and other behaviors in the browser. The Web side has the advantage of avoiding risk control strikes naturally because it collects less and unstable information.

[0081] It should be noted that the above-mentioned several optional embodiments can be implemented separately or in combination.

[0082] The object behavior simulation method provided by the embodiments of the present application may include the following advantages:

[0083] (1) Compared with manual operation or writing scripts to generate object behaviors, in this embodiment, the model is trained with an object behavior data set to generate a virtual object behavior sequence, and the object behavior is simulated based on the virtual object behavior sequence, which improves the efficiency, scalability, and adaptability of object behavior simulation and reduces the development cost.

[0084] (2) After a new account (test account) logs in normally, to prevent the account from being attacked by the risk control engine, simulate the behavior of normal objects based on the behavior sequence of virtual objects, maintain the activity and authenticity of the account, and reduce the probability of being hit.

[0085] As shown above, the simulation of object behavior is easily subject to being hit by the risk control engine.

[0086] To generate object behaviors for simulation, it is often necessary to manually find object behaviors that can bypass the risk control strategy and be hit, consuming a large amount of manpower, and having no significant effect on patching the loopholes in the risk control strategy.

[0087] In view of this, as Figure 7 shown, the object behavior simulation method may further include: step S700, determining whether the test account triggers a risk control strategy; step S702, if the test account triggers a risk control strategy, updating the simulation object behavior data of the simulation object behavior to the object behavior dataset as negative feedback data in the object behavior dataset. In practical applications, by detecting whether the test account is locked or cancelled, the effectiveness of the virtual object behavior sequence produced is ensured. When the test account is locked or cancelled, another test account needs to be generated for the simulation of object behavior.

[0088] In this embodiment, if it is detected that the test account is cancelled, etc., it means that the simulation object behavior at this time has triggered the risk control strategy of the risk control engine. To reduce the virtual object behavior sequence that the model may continue to output and be hit by the risk control engine, update the simulation object behavior data of the simulation object behavior to the object behavior dataset, and assign a negative reward coefficient to this object behavior data. It can be understood that when a large number of such object behavior data with negative reward coefficients are added to the object behavior dataset, the computer device 2 can effectively find object behaviors that can bypass the risk control strategy and be hit, avoid consuming a large amount of manpower, and have a significant effect on patching the loopholes in the risk control strategy.

[0089] In an optional embodiment, the model includes a D3QN network.

[0090] To obtain the optimal virtual object behavior sequence, the step S202 can be implemented through the following steps:

[0091] Train the model for multiple episodes, and each trained model corresponds to outputting a virtual object behavior sequence; [[ID=**********]]

[0092] Among them, the following operations are performed in each episode training:

[0093] S1: Select m groups of sample behavior data from the object behavior dataset, where m is an integer greater than 1.

[0094] S2: Construct a two-dimensional matrix [X T , Y A for each group of sample behavior data, where X T = X1, X2,..., X t represents actions executed sequentially, and each action has multiple actions Y t，A = Y t，1 , Y t，2 ,..., Y t，a .

[0095] S3: Dimensionally reduce each two-dimensional matrix to a one-dimensional matrix, where the dimensional reduction transformation aggregates each column of actions in the corresponding two-dimensional matrix into a single action; where the single action represents the action taken at the current time j;

[0096] S4: Iteratively update the Q function according to each one-dimensional matrix, where the update formula of the Q function is as follows:

[0097] Q(S, A, θ, α, β) = V(S, θ, α) + A(S, A, θ, β) - loss(A);

[0098] a max (S′, θ) = arg max Q(S′, θ, a);

[0099] y j = R + γQ′((S j ), arg max Q((S j ′), θ, a));

[0100] where S is the current state, S′ is the next state, A is the current action, argmaxQ(S′, θ, a) is the next action, θ is the network parameter, α is the network parameter of the value function, β is the network parameter of the advantage function, V(S, θ, α) is the value function, A(S, A, θ, β) is the advantage function, loss(A) is the loss function, R is the reward value, γ is the decay value, argmax is the maximum value function, and y j is the target Q value of the Double Q layer at time j.

[0101] For ease of understanding, a specific training process is provided below:

[0102] First: Construct a D3QN network.

[0103] The D3QN includes two sub-networks, and the structure of each sub-network (Q and Q') is as follows: input layer → 4 convolutional layers distributed in sequence → two parallel branches, where one branch includes a fully connected layer and a state value function (value function), and the other branch includes a fully connected layer and an advantage function (advantage function). The Q value of each object behavior (hereinafter referred to as "action") is calculated through the state value function and the advantage function.

[0104] The Q function is as follows:

[0105] Q(S, A, θ, α, β) = V(S, θ, α) + A(S, A, θ, β) - loss(A)

[0106] Second: Initialize the current Q network parameters θ, initialize the recurrent network Q' network parameters θ', assign the current network parameters to the recurrent network, set the number of rounds T, the decay factor γ, the exploration rate ∈, and the network update frequency P.

[0107] Third: Initialize the object behavior data set (experience pool D) with a capacity set to N.

[0108] Fourth: Randomly select m groups of sample behavior data from the object behavior data set, where m is an integer greater than 1.

[0109] For the m groups of sample behavior data, they are two-dimensional matrices [X T , Y A . Let X T = X1, X2,..., X t represent the sequential execution of actions. For each column X t there are multiple synchronously executed actions Y t，A = Y t，1 , Y t，2 ,..., Y t，a . This two-dimensional matrix will be dimension-reduced through the convolutional network, that is, aggregating each column of actions X t , and the aggregated action A t is the action taken at time t.

[0110] Fifth: Based on the m groups of sample behavior data, train the Q3DN network. The following loss function can be used for this training:

[0111]

[0112] Specifically as follows:

[0113] (1) Initialize the state S;

[0114] Within time T:

[0115] (2) Use the greedy algorithm to randomly select an action or select an action based on the maximum reward;

[0116] (3) Execute the action, and iteratively obtain the reward R and the next state S';

[0117] (4) Store the current state, the next state, the action, and the action reward in the object behavior data set (experience pool D);

[0118] If the number of rounds reaches the limit, return the output reward. If the number of rounds has not reached the limit, process the sample parameters and the action reward, calculate the loss value (Loss), and update the parameter θ and iterate S by backpropagation according to the update frequency P.

[0119] Parameter settings:

[0120] Set ∈ to 0.2, the network width to 512, and the experience pool size to 2 20 , set the convolution size to 256 and γ to 0.99.

[0121] The object behavior goal is to obtain the most exp within the shortest abscissa matrix and not be blocked by the risk control request strike strategy.

[0122] Among them, additional environmental rewards are set:

[0123] The positive reward is to obtain level experience and view, like, share, and comment on videos and dynamics. The negative reward is to accumulate risk control rule factors or be struck by the risk control strategy, account being blocked or being identified as a false account.

[0124] The negative reward part is as follows:

[0125] The cumulative number of accounts with the IP address, whether the liked objects are the same person, whether the coin-voted video numbers are the same, and the video playback duration.

[0126] Finally, the operation matrix X returned by the model A , which is the virtual object behavior sequence. This virtual object behavior sequence is an aggregated one-dimensional array, so the virtual object behavior sequence needs to be recompiled back to the parallel operation mode.

[0127] Example 2

[0128] Figure 8 Schematically shows a block diagram of the object behavior simulation system according to Example 2 of the present application. This object behavior simulation system can be divided into one or more program modules. One or more program modules are stored in a storage medium and are executed by one or more processors to complete the embodiments of the present application. The program modules referred to in the embodiments of the present application refer to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment.

[0129] As Figure 8 shown, the object behavior simulation system 800 may include an acquisition module 810, a training module 820, and an execution module 830, where:

[0130] The acquisition module 810 is configured to acquire an object behavior data set, and the object behavior data set includes real object behavior data obtained based on real object behaviors;

[0131] The training module 820 is configured to train a model according to the object behavior data set, and output a virtual object behavior sequence based on the trained model; and

[0132] The execution module 830 is configured to execute simulated object behaviors under a test account through a target host according to the virtual object behavior sequence.

[0133] In an optional embodiment, the execution module 830 is further configured to:

[0134] Decode the virtual object behavior sequence to obtain multiple virtual object behaviors; and

[0135] Add one or more noise behaviors to the multiple virtual object behaviors to obtain the simulated object behaviors.

[0136] In an optional embodiment, the execution module 830 is further configured to:

[0137] Decode the virtual object behavior sequence to obtain multiple virtual object behaviors;

[0138] Obtain serial / parallel operation instructions with a time series according to the multiple virtual object behaviors; and

[0139] Simulate and execute the simulated object behaviors under the test account through the target host according to the serial / parallel operation instructions.

[0140] In an optional embodiment, the execution module 830 is further configured to:

[0141] Simulate and execute the simulated object behaviors by way of calling an interface; and / or

[0142] In an incognito mode or an IP proxy mode of a browser, simulate and execute the simulated object behaviors through a browser of the target host.

[0143] In an optional embodiment, the training module 820 is further configured to:

[0144] Train the model according to the object behavior data set and a preset reward parameter.

[0145] In an optional embodiment, the system further includes an update module (not identified) for:

[0146] Determine whether the test account triggers a risk control policy; and

[0147] If the test account triggers a risk control policy, update the simulation object behavior data of the simulation object to the object behavior dataset as negative feedback data in the object behavior dataset.

[0148] In an optional embodiment, the model includes a D3QN network; the training module 820 is further configured to:

[0149] Perform multi-round training on the model, and each round of trained model corresponds to output a virtual object behavior sequence;

[0150] Wherein, the following operations are performed in each round of training:

[0151] Select m groups of sample behavior data from the object behavior dataset, where m is an integer greater than 1;

[0152] Construct a two-dimensional matrix [X T , Y A for each group of sample behavior data, where X T = X1, X2,..., X t represents sequentially executed actions, and each action has multiple synchronously executed actions Y t，A = Y t，1 , Y t，2 ,..., Y t，a ;

[0153] Dimensionality reduce each two-dimensional matrix into a one-dimensional matrix, and the dimensionality reduction conversion is to aggregate each column of actions in the corresponding two-dimensional matrix into a single action; wherein, the single action represents the action taken at the current time j;

[0154] Iteratively update the Q function according to each one-dimensional matrix, where the update formula of the Q function is as follows:

[0155] Q(S, A, θ, α, β) = V(S, θ, α) + A(S, A, θ, β) - logg(A);

[0156] a max (S′, θ) = arg max Q(S′, θ, a);

[0157] y j = R + γQ′((S j ), arg max Q((S j ′), θ, a));

[0158] Where, S is the current state, S′ is the next state, A is the current action, arg max Q(S′, θ, a) is the next action, θ is the network parameter, α is the network parameter of the value function, β is the network parameter of the advantage function, V(S, θ, α) is the value function, A(S, A, θ, β) is the advantage function, loss(A) is the loss function, R is the reward value, γ is the decay value, argmax is the maximum value function, y j is the target Q value of the Double Q layer at time j.

[0159] Embodiment III

[0160] Figure 9 Schematically shows a hardware architecture diagram of a computer device 2 suitable for implementing the object behavior simulation method according to Embodiment III of the present application. In this embodiment, the computer device 2 is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. For example, it can be a rack server, a blade server, a tower server or a cabinet server (including an independent server or a server cluster composed of multiple servers), etc. As Figure 9 shown, the computer device 2 at least includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other through a system bus. Among them:

[0161] The memory 10010 includes at least one type of computer-readable storage medium. The readable storage medium includes flash memory, hard disks, multimedia cards, card-type memories (e.g., SD or DX memories, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memories, magnetic disks, optical disks, etc. In some embodiments, the memory 10010 may be an internal storage module of the computer device 2, such as the hard disk or memory of the computer device 2. In other embodiments, the memory 10010 may also be an external storage device of the computer device 2, such as a plug-in hard disk equipped on the computer device 2, a Smart Media Card (SMC), a Secure Digital (SD) card, a FlashCard, etc. Of course, the memory 10010 may also include both the internal storage module and the external storage device of the computer device 2. In this embodiment, the memory 10010 is generally used to store the operating system installed on the computer device 2 and various application software, such as the program code of the object behavior simulation method. In addition, the memory 10010 may also be used to temporarily store various data that have been output or will be output.

[0162] In some embodiments, the processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 10020 is generally used to control the overall operation of the computer device 2, such as performing control and processing related to data interaction or communication with the computer device 2. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.

[0163] The network interface 10030 may include a wireless network interface or a wired network interface, which is generally used to establish a communication link between the computer device 2 and other computer devices. For example, the network interface 10030 is used to connect the computer device 2 to an external terminal through a network, and establish a data transmission channel and a communication link between the computer device 2 and the external terminal. The network may be a wireless or wired network such as an enterprise intranet (Intranet), the Internet, Global System of Mobile communication (GSM for short), Wideband Code Division Multiple Access (WCDMA for short), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0164] It should be noted that Figure 9 Only the computer device with components 10010 - 10030 is shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively.

[0165] In this embodiment, the object behavior simulation method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (processor 10020 in this embodiment) to complete the embodiments of this application.

[0166] Embodiment 4

[0167] This application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the object behavior simulation method in the embodiment are implemented.

[0168] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a Smart Media Card (SMC for short), a Secure Digital (SD) card, a Flash Card, etc. Of course, the computer-readable storage medium may also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is generally used to store the operating system installed in the computer device and various application software, such as the program code of the object behavior simulation method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various data that have been output or will be output.

[0169] Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the embodiments of the present application can be implemented by a general computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. Optionally, they can be implemented by program codes executable by the computing device, so that they can be stored in the storage device and executed by the computing device. And in some cases, the steps shown or described can be executed in a different order from here, or they can be separately made into individual integrated circuit modules, or multiple modules or steps among them can be made into a single integrated circuit module to implement. Thus, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0170] It should be noted that the above are only the preferred embodiments of the present application, and do not limit the patent protection scope of the present application. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be included in the patent protection scope of the present application by the same token.

Claims

1. A method for simulating object behavior, characterized in that: The method comprises: Acquire an object behavior dataset, wherein the object behavior dataset includes real object behavior data obtained based on real object behavior; Training a model according to the object behavior dataset, and outputting a virtual object behavior sequence based on the trained model; and According to the virtual object behavior sequence, the simulated object behavior under the test account is executed through the target host; The step of executing the simulated object behavior under the test account through the target host according to the virtual object behavior sequence includes: Decoding the virtual object behavior sequence to obtain multiple virtual object behaviors; and One or more noise behaviors are added to the multiple virtual object behaviors to obtain the simulated object behavior; wherein the simulated object behavior is used as negative feedback data in the object behavior data set when the test account triggers the risk control strategy.

2. The object behavior simulation method according to claim 1, characterized in that: The step of executing the simulated object behavior under the test account through the target host according to the virtual object behavior sequence includes: Decoding the virtual object behavior sequence to obtain multiple virtual object behaviors; Obtaining serial / parallel operation instructions with a time sequence according to the behaviors of the multiple virtual objects; and According to the serial / parallel operation instruction, the simulated object behavior under the test account is simulated and executed through the target host.

3. The object behavior simulation method according to claim 1, wherein: The step of executing the simulated object behavior under the test account through the target host according to the virtual object behavior sequence includes: Simulate the behavior of the simulation object by calling the interface; and / or In the browser incognito mode or IP proxy mode, the simulated object behavior is executed through the browser simulation of the target host.

4. The object behavior simulation method according to any one of claims 1 to 3, characterized in that: The step of training the model according to the object behavior dataset and outputting a virtual object behavior sequence based on the trained model includes: The model is trained according to the object behavior dataset and preset reward parameters.

5. The object behavior simulation method according to any one of claims 1 to 3, characterized in that: Also includes: Determining whether the test account triggers a risk control strategy; and If the test account triggers the risk control strategy, the simulated object behavior data of the simulated object behavior is updated to the object behavior dataset as negative feedback data in the object behavior dataset.

6. The object behavior simulation method according to claim 5, characterized in that: The model includes a D3QN network; The step of training the model according to the object behavior dataset and outputting a virtual object behavior sequence based on the trained model includes: The model is trained for multiple rounds, and each round of training corresponds to outputting a virtual object behavior sequence; The following operations are performed in each round of training: Selecting m groups of sample behavior data from the object behavior data set, where m is an integer greater than 1; For each set of sample behavior data, a two-dimensional matrix [X T , Y A ], where X T =X1,X2,...,X t Indicates actions that are executed sequentially, each action has multiple synchronous execution actions Y t，A =Y t，1 , Y t，2 ,...,Y t，a ; Each two-dimensional matrix is converted into a one-dimensional matrix by reducing the dimension of the matrix, wherein the dimension reduction is to aggregate each column action in the corresponding two-dimensional matrix into a single action; wherein the single action represents the action taken at the current time j; The Q function is iteratively updated according to each one-dimensional matrix, wherein the update formula of the Q function is as follows: Q(S,A,θ,α,β)=V(S,θ,α)+A(S,A,θ,β)-loss(A); a max (S′,θ)=argmaxQ(S′,θ,a); y j =R+γQ′((S j ),argmaxQ((S j ′),θ,a)); Among them, S is the current state, S' is the next state, A is the current action, argmaxQ(S', θ, a) is the next action, θ is the network parameter, α is the network parameter of the value function, β is the network parameter of the advantage function, V(S, θ, α) is the value function, A(S, A, θ, β) is the advantage function, loss(A) is the loss function, R is the reward value, γ is the decay value, argmax is the maximum function, y j is the target Q value of the Double Q layer at time j.

7. An object behavior simulation system, characterized in that: The system comprises: an acquisition module, configured to acquire an object behavior dataset, wherein the object behavior dataset includes real object behavior data obtained based on real object behavior; a training module, configured to train the model according to the object behavior dataset and output a virtual object behavior sequence based on the trained model; and An execution module, configured to execute the simulated object behavior under the test account through a target host according to the virtual object behavior sequence; Among them, the execution module is also used to decode the virtual object behavior sequence to obtain multiple virtual object behaviors; and add one or more noise behaviors to the multiple virtual object behaviors to obtain the simulated object behavior; wherein, the simulated object behavior is used as negative feedback data in the object behavior data set when the test account triggers the risk control strategy.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it is used to implement the steps of the object behavior simulation method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which can be executed by at least one processor to enable the at least one processor to perform the steps of the object behavior simulation method according to any one of claims 1 to 6.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the object behavior simulation method according to any one of claims 1 to 6 are implemented.

Images