A method and system for completing user operations of parallel APP software

Abstract

The present invention discloses a method and system for completing user operations of parallel APP software, including: Step 1, real-time capture of content data of user operation events of parallel APP software and response data of parallel APP software; Step 2, extraction of content features of user operations of parallel APP software; Step 3, establishment of contextual semantic relationships of content features of user operations of parallel APP software; Step 4, extraction of temporal context of user operation sequences of parallel APP software; Step 5, completion of user operations of parallel APP software. The present invention helps to mine semantic context and temporal context features from user operation sequences of large-scale parallel APP software, supports completion of user operations under the condition of parallel operation of multiple APP software, thereby efficiently and accurately obtaining complete user operation sequences of parallel APP software.

Description

Technical Field

[0001] The present invention relates to a method and system for completing parallel APP software user operations, and belongs to the field of APP software user operation analysis. Background Art

[0002] In the process of user-centered APP software evolution research, APP software user operation analysis has gradually become the focus of researchers. Some commercial frameworks such as FlurryAnalytics are used to evaluate the usability of mobile devices. These analysis tools can collect statistics on the use of mobile software, such as usage frequency, memory size, etc. However, in order to use these frameworks, developers need to manually add source code for different applications, and the information collected by these tools focuses on the statistical data of mobile software usage, which is difficult to provide effective support for analyzing APP software user operations.

[0003] Lettner et al. proposed a method for automatically collecting and analyzing logs, which collects logs of APP software user operations by embedding the program in the target APP source code. Hilbert et al. extracted the availability information of APP software user operations from the Android system user interface. Ma et al. defined a part of user operation events based on the Android system user interface design, and developed a toolkit that can be embedded in the application. The toolkit only needs to make minimal modifications to the target source program to collect the defined APP user operation information. However, McMillan pointed out that most logs are limited to recording coarse-grained user operations. These log analysis-based methods can answer when the user opened or closed which application, but it is difficult to capture the specific details of the user's operations within the application.

[0004] In recent years, Krieter and others have used screen recording tools to record the process of users using APPs, and used Python scripts to convert high-quality recorded videos into pictures, and then used image processing technology to match these pictures with manually defined event picture templates to analyze user operations in the APP. Since high-quality video files take up a lot of memory space, Xin simplifies the analysis of video files.

[0005] In the actual process of users using APP software, we found that the current methods for analyzing APP software user operations still have the following problems. (1) Due to the great differences in the design of different APP user interfaces, relying on manually defined user operations means that the method can only collect limited, coarse-grained user operations, and the effectiveness of the method will decrease with the update of the Android system and APP software. (2) Experiments show that existing methods often only analyze the user operations of a single APP software, ignoring the user's more realistic usage scenarios, that is, users interact with multiple parallel running APP software within a period of time. (3) Due to the particularity of the Android system user interface framework, when multiple APPs are running in parallel or switching frequently, the APP software user operations recorded by existing methods will inevitably have missing data and errors.

[0006] Based on the above problems, existing research cannot effectively obtain parallel APP software user operations, which also leads to limited applications for APP software user operations. For example, in the study of predicting APP user operation behavior to recommend relevant content, Liu et al. could only rely on screenshots and manually define APP user operations to analyze the user operations of a single APP (SnapChat). Therefore, how to complete parallel APP software user operations is an urgent problem to be solved. Summary of the invention

[0007] The present invention provides a method and system for completing user operations of parallel APP software, so as to solve the problem of completing user operations of parallel APP software.

[0008] The technical solution of the present invention is: a method for completing parallel APP software user operations, including: Step 1, real-time capture of parallel APP software user operation event content data and parallel APP software response data; Step 2, extracting parallel APP software user operation content features; Step 3, establishing contextual semantic relationships of parallel APP software user operation content features; Step 4, extracting the timing context of the parallel APP software user operation sequence; wherein the timing context includes a global timing context and a parallel APP timing context; Step 5, completing the parallel APP software user operations.

[0009] Furthermore, the Step 2 specifically includes: based on the Android user interface framework, initializing the parallel APP software user operation content characteristics, and establishing a parallel APP software user operation event response tree; based on the parallel APP software user operation content characteristics, obtaining the parallel APP software user operation content from the parallel APP software user operation event content data and the parallel APP software response data; initializing a node list based on the parallel APP software user operation event response tree; extracting the parallel APP software response node information from the node list based on the parallel APP software user operation content; determining the parallel APP software user operation content characteristics according to the parallel APP software response node information.

[0010] Furthermore, the method of determining the parallel APP software user operation content characteristics based on the parallel APP software response node information includes: if the parallel APP software response node information is empty and there is no parent node, obtaining the parallel APP software user operation content characteristics based on the parallel APP software user operation content; otherwise, cross-validating the parallel APP software user operation content and the APP software user operation interaction component information to obtain the parallel APP software user operation content characteristics; wherein, extracting the APP software user operation interaction component information in the parallel APP software response node information used to supplement the parallel APP software user operation content.

[0011] Furthermore, the Step 3 specifically includes: obtaining a structured data set D based on the content features of parallel APP software user operations; obtaining a parallel APP user operation sequence after sorting the structured data set D according to time; segmenting each parallel APP software user operation in the parallel APP user operation sequence to obtain a parallel APP user operation segmentation sequence; constructing a neural network training set for the parallel APP user operation segmentation sequence according to a set sequence length; using the neural network training set to train the neural network to establish the contextual semantic relationship of the parallel APP software user operation content features.

[0012] Furthermore, the Step 4 specifically includes: obtaining a global timing sequence and a parallel APP timing sequence; inputting the global timing sequence into a first LSTM-based neural network model, and extracting the model LSTM layer output as a global timing context; inputting the parallel APP timing sequence into a second LSTM-based neural network model, and extracting the model LSTM layer output as a parallel APP timing context.

[0013] Furthermore, the obtaining of the global timing sequence and the parallel APP timing sequence includes: storing the elements in the parallel APP user operation word segmentation sequence into the global timing sequence in chronological order; extracting the user operation sequence of each parallel APP software in the parallel APP user operation word segmentation sequence and storing it into the parallel APP timing sequence.

[0014] Furthermore, the first LSTM-based neural network model and the second LSTM-based neural network model both include an input layer, an LSTM layer, a fully connected layer, and an output layer.

[0015] Furthermore, the Step 5 specifically includes: splicing the global timing context and the parallel APP timing context as the timing context; inputting the timing context into the TextCNN-based neural network model until the network converges to obtain an optimized TextCNN-based neural network model; completing the parallel APP software user operations through the optimized TextCNN-based neural network model to obtain a complete APP software user operation sequence at each time point.

[0016] Furthermore, the TextCNN-based neural network includes an input layer, a concatenation layer, a TextCNN layer, a pooling layer, a fully connected layer, and an output layer.

[0017] According to another aspect of the present invention, a parallel APP software user operation completion system is also provided, including: a capture module for capturing parallel APP software user operation event content data and parallel APP software response data in real time; an extraction module for extracting parallel APP software user operation content features; an establishment module for establishing contextual semantic relationships of parallel APP software user operation content features; an extraction module for extracting the timing context of the parallel APP software user operation sequence; wherein the timing context includes a global timing context and a parallel APP timing context; and a completion module for completing parallel APP software user operations.

[0018] The beneficial effects of the present invention are:

[0019] The present invention firstly acquires parallel APP software user operations in real time based on a monitoring service program, and parses parallel APP software user operations based on a user interface framework to acquire fine-grained APP software user operation content, which is helpful to acquire and analyze parallel running APP software user operation data on a large scale; secondly, the present invention extracts context features between APP software user operation contents through a semantic context extractor, and extracts timing features of a global timing dimension and an APP timing dimension through a timing context detector, and obtains a completed parallel APP software user operation sequence through a fully connected neural network using the timing features.

[0020] In summary, the present invention helps to mine semantic context and temporal context features from large-scale parallel APP software user operation sequences, supports user operation completion when multiple APP software are running in parallel, and thus efficiently and accurately obtains complete parallel APP software user operation sequences. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is the overall flow chart of the present invention;

[0022] Figure 2 is a flow chart of step Step 2 of an optional embodiment of the present invention;

[0023] Figure 3 is a flow chart of step Step 3 of an optional embodiment of the present invention;

[0024] Figure 4 It is the process of completing parallel APP software user operations based on the parallel APP software operation content characteristics. DETAILED DESCRIPTION

[0025] The invention is further described below in conjunction with the accompanying drawings and embodiments, but the content of the invention is not limited to the scope of the embodiments.

[0026] Example 1: Figure 1-4 As shown, a method for completing user operations of parallel APP software includes:

[0027] Step 1: Real-time capture of parallel APP software user operation event content data and parallel APP software response data;

[0028] Step 2, extract the user operation content features of parallel APP software;

[0029] Step 3, establish the contextual semantic relationship of the content features of parallel APP software user operations;

[0030] Step 4, extract the timing context of the parallel APP software user operation sequence; wherein the timing context includes the global timing context and the parallel APP timing context;

[0031] Step 5. Complete the parallel APP software user operations.

[0032] Furthermore, the Step 2 specifically includes: based on the Android user interface framework, initializing the parallel APP software user operation content characteristics, and establishing a parallel APP software user operation event response tree; based on the parallel APP software user operation content characteristics, obtaining the parallel APP software user operation content from the parallel APP software user operation event content data and the parallel APP software response data; initializing a node list based on the parallel APP software user operation event response tree; extracting the parallel APP software response node information from the node list based on the parallel APP software user operation content; determining the parallel APP software user operation content characteristics according to the parallel APP software response node information.

[0033] Furthermore, the method of determining the parallel APP software user operation content characteristics based on the parallel APP software response node information includes: if the parallel APP software response node information is empty and there is no parent node, obtaining the parallel APP software user operation content characteristics based on the parallel APP software user operation content; otherwise, cross-validating the parallel APP software user operation content and the APP software user operation interaction component information to obtain the parallel APP software user operation content characteristics; wherein, extracting the APP software user operation interaction component information in the parallel APP software response node information used to supplement the parallel APP software user operation content.

[0034] Furthermore, the Step 3 specifically includes: obtaining a structured data set D based on the content features of parallel APP software user operations; obtaining a parallel APP user operation sequence after sorting the structured data set D according to time; segmenting each parallel APP software user operation in the parallel APP user operation sequence to obtain a parallel APP user operation segmentation sequence; constructing a neural network training set for the parallel APP user operation segmentation sequence according to a set sequence length; using the neural network training set to train the neural network to establish the contextual semantic relationship of the parallel APP software user operation content features.

[0035] Furthermore, the Step 4 specifically includes: obtaining a global timing sequence and a parallel APP timing sequence; inputting the global timing sequence into a first LSTM-based neural network model, and extracting the model LSTM layer output as a global timing context; inputting the parallel APP timing sequence into a second LSTM-based neural network model, and extracting the model LSTM layer output as a parallel APP timing context.

[0036] Furthermore, the obtaining of the global timing sequence and the parallel APP timing sequence includes: storing the elements in the parallel APP user operation word segmentation sequence into the global timing sequence in chronological order; extracting the user operation sequence of each parallel APP software in the parallel APP user operation word segmentation sequence and storing it into the parallel APP timing sequence.

[0037] Furthermore, the first LSTM-based neural network model and the second LSTM-based neural network model both include an input layer, an LSTM layer, a fully connected layer, and an output layer.

[0038] Furthermore, the Step 5 specifically includes: splicing the global timing context and the parallel APP timing context as the timing context; inputting the timing context into the TextCNN-based neural network model until the network converges to obtain an optimized TextCNN-based neural network model; completing the parallel APP software user operations through the optimized TextCNN-based neural network model to obtain a complete APP software user operation sequence at each time point.

[0039] Furthermore, the TextCNN-based neural network includes an input layer, a concatenation layer, a TextCNN layer, a pooling layer, a fully connected layer, and an output layer.

[0040] Furthermore, the specific steps of the method of the above optional embodiment are as follows:

[0041] Step 1: Real-time capture of parallel APP software user operation data and parallel APP software response data;

[0042] Step 1.1. Monitor the service program on the Android platform, capture the original parallel APP software user operation data in real time, and execute Step 1.2. It should be noted that the Hongmeng platform and the like can also use the method of the present invention to realize parallel APP software user operation completion.

[0043] Step 1.2, the original parallel APP software user operation data contains a large amount of user operation event information and node information of the Android user interface framework; based on the accessibility service interface, the user operation event information related to the parallel APP software user operation is filtered out from the original parallel APP software user operation data as shown in Table 1.

[0044] Table 1 Parallel APP software user operation events

[0045]

[0046] The user operation event information shown in Table 1 includes a portion of parallel APP software user operation content, which is screened again to obtain preliminary parallel APP software user operation event content data, which is stored in operation_data as shown in Table 2.

[0047] Table 2 Preliminary screening of parallel APP software user operation content operation_data

[0048] APP software user operation content name meaning Package APP software name Component APP software interface component name Time The time when the APP software user operation occurs EventType APP software user operation type

[0049] In Table 2, Component represents the name of the APP software user interface component; in the Android system, the user interface framework is stored in a tree structure; based on the Android system user interface framework, an APP software user operation event response tree is established and parallel APP software response data is filtered and obtained and stored in resp_data, as shown in Table 3.

[0050] Table 3 Parallel APP software response data resp_data

[0051] APP software response data name meaning Component The name of the APP software interface component that responds to user operations Text Response text for user actions BoundsInScreen The screen area that responds to user actions ChildNode Respond to user operation on the child nodes of the node ParentNode Responds to the parent node of the user operation node

[0052] Based on Table 3, we can then filter out the parallel APP software response data information that is not related to user operations (not related to user operations, that is, atypical user operation events); among them, Component represents the name of the APP software interface component that responds to user operations. This value corresponds to the Component value in Table 2, which provides a basis for obtaining the user operation content features of parallel APP software.

[0053] Step 2, extract the user operation content features of parallel APP software;

[0054] Step 2.1, based on the Android user interface framework, initialize the parallel APP software user operation content features, recorded as FeatureUI_operation. Among them, FeatureUI_operation is a common user interface feature in APP software based on the Android platform, as shown in Table 4.

[0055] Table 4 FeatureUI_operation

[0056] APP software user operation content characteristics meaning Package APP software name EventType APP software user operation type Activity APP software activity interface Class APP software component name Time APP software user operation time Object APP software user operation object Content APP software user operation object content

[0057] As shown in Table 4, according to the general rules and development standards of APP software user interface design on the Android system, we initialized 7 content features related to parallel APP software user operations and executed Step 2.2;

[0058] Step 2.2, based on FeatureUI_operation, obtain the parallel APP software user operation content from operation_data and resp_data, and store it in raw_operation_content. Raw_operation_content also contains part of the content of operation_data and resp_data, and execute Step 2.3;

[0059] Step 2.3, based on the Android user interface framework, establish a parallel APP software user operation event response tree resp_tree, and initialize the node list node_list according to resp_tree. The response of the APP software to the user operation event is a tree-like structure. The event response process can be effectively analyzed through the tree structure, and then execute Step 2.4;

[0060] node_list=[node1, node2,...,noden];

[0061] Step 2.4, extract the APP software response node from node_list based on raw_operation_content, and store the node information in node_info. The node information contains the information of some interactive components in the APP software user interface, as shown in Table 5.

[0062] Table 5 node_info information

[0063] Node Information meaning Component APP software component name Text APP software component text content BoundsScreen APP software components can interact with the coordinate area ChildNode Subnodes of APP software components ParentNode Parent node of APP software component

[0064] As shown in Table 5, the node information includes but is not limited to the five elements in the table, from which information related to the user operation of the APP software is screened out to supplement and verify the raw_operation_content, and Step 2.5 is executed;

[0065] Step 2.5, for each layer of the parallel APP software user operation event response tree resp_tree, determine whether node_info is empty. If node_info is empty, execute Step 2.6, otherwise execute Step 2.8;

[0066] Step 2.6, based on resp_tree, determine whether there is a parent node. If so, execute Step 2.7, otherwise execute Step 2.10;

[0067] Step 2.7, store the parent node information into parent_node_info, and use parent_node_info to overwrite node_info, and execute Step 2.5;

[0068] node_info=parent_node_info;

[0069] Step 2.8, extract the APP software user operation interaction component information used to supplement the parallel APP software user operation content in node_info, recorded as Class_Info, as shown in Table 6.

[0070] Table 6 Supplementary information on APP software user operation interaction components

[0071] Supplementary interactive component information meaning Activity APP software function interface name Class APP software component name Text APP software component text content

[0072] As shown in Table 6, the supplementary information filtered from node_info contains three contents, namely, the name of the APP software function interface, the name of the APP software component, and the text content of the APP software component. These supplementary information will be used for cross-validation in the next step to obtain the final parallel APP software user operation content features, and execute Step 2.9;

[0073] Step 2.9, cross-validate raw_operation_content and Class_Info to obtain the parallel APP software user operation content features, recorded as operation_content.

[0074] operation_content=cross_validate(Class_Info, raw_operation_content, key=class) as [Package, EventType, Activity, Class, Time, Object, Content]

[0075] Step 2.10. If node_info is empty and node_info does not have a parent node, no cross-validation is performed. Only raw_operation_content is used as the user operation content feature of the parallel APP software. The information in raw_operation_content is extracted and stored in operation_content, and Step 2 is ended.

[0076] Step 3: Establish the contextual semantic relationship of the content features of the parallel APP software user operations;

[0077] Step 3.1. Structure the parallel APP software user operation content based on operation_content, recorded as structured data set D. Table 7 shows the information of a parallel APP software user operation data.

[0078] Table 7 Example data in structured dataset D

[0079] name data Package com.tencent.mm Activity android.SoftInputWindow Class android.widget.EditText Type TextSelection Content 'GoodMorning' Object query Time 2021-12-1710：13：27

[0080] The data in Table 7 means that the user used the Tencent APP software at 10:13:27 am on December 17, 2021, and operated the text component of the input function interface in the APP software, performed a text type operation, the operation content was 'GoodMorning', and executed Step 3.2;

[0081] Step 3.2, store the user operation objects in the structured data set D into object, and execute Step 3.3;

[0082] object = D['Object']

[0083] Step 3.3, store the user operation component content in the structured data set D into component, and execute Step 3.4;

[0084] component = D['Content']

[0085] Step 3.4, determine whether the object is empty, if it is empty, execute Step 3.5, otherwise execute Step 3.6;

[0086] Step 3.5, use component as the user operation object and execute Step 3.6;

[0087] object=component

[0088] Step 3.6, sort the structured data set D by time to obtain the parallel APP software user operation sequence and store it in operation_sequence;

[0089] operation_sequence = sorted(D), execute Step 3.7;

[0090] Step 3.7, remove duplicate words in operation_sequence, store the removed duplicate words into the dictionary matrix word_matrix, and execute Step 3.8;

[0091] word_matrix=dict(operation_sequence)

[0092] Get the dictionary matrix Where N v Indicates the number of words after deduplication, execute Step 3.9;

[0093] Step 3.8, using word_matrix as the dictionary, segment each parallel APP software user operation in operation_sequence, separate them with spaces, and store the data in sequence_list. Part of the sequence_list results are shown in Table 8 (where Time is not involved in the segmented data, and Object and Content are displayed together);

[0094] Table 8. Example of some data in sequence_list

[0095]

[0096] As shown in Table 8, the length of the data after word segmentation is not necessarily equal; it is displayed in array form. In the array structure, the number at the array subscript position 1 represents the type of APP software user operation event. The related events corresponding to the number are shown in Table 9 below. Execute Step 3.8;

[0097] Table 9 APP software user operation event types

[0098] serial number APP software user operation event type 1 TYPE_VIEW_CLICKED 2 TYPE_VIEW_TEXT_SELECTION_CHANGED 3 TYPE_WINDOWS_CHANGED 4 TYPE_VIEW_LONG_CLICKED 5 TYPE_VIEW_TEXT_CHANGED

[0099] Step 3.9, based on sequence_list, initialize the length of the parallel APP software user operation sequence to 6, use 0 to fill it if it is less than the length, truncate it if it exceeds the length and store the result in the APP user operation standardized sequence sequences, and execute Step 3.9; it should be noted that the application sets the length to 6, which can avoid the need for excessive padding due to too long a length, and can also avoid the deficiency of being too short a length and being unable to effectively retain semantic information;

[0100] sequences=Padding(operation_sequence, length=6, fill=0)

[0101] Step 3.10, initialize window windows = 6, training sample set vec_train = [];

[0102] Among them, the dimension of the initialized vec_train is R Nv×Nf , N v 、N f Represent the length of the dictionary and the output dimension of word embedding respectively. We set the output dimension of word embedding to 128 and execute 3.11;

[0103] Step 3.11, store the elements in sequences into the neural network training set vec_train, and execute Step 3.12;

[0104] vec_train = sequences;

[0105] Step 3.12, initialize the neural network, use vec_train to train the neural network, establish the contextual semantic relationship of the parallel APP software user operation content features, and store the results in the word_embedding matrix; among them, the neural network model selects Word2Vec;

[0106] vec_train=model(vec_train,N d );

[0107] word_embedding=model.wv

[0108] Among them, model.wv stores the trained word embedding vector representation. For example, Table 10 shows the word embedding vector representation for a word;

[0109] Table 10 Word embedding representation

[0110]

[0111] End Step 3.

[0112] Step 4: Extract the temporal context of the parallel APP software user operation sequence;

[0113] Step 4.1, initialize the global timing sequence global_sequence and the parallel APP timing sequence App_sequence, and execute Step 4.2;

[0114] Step 4.2, store the elements in the parallel APP user operation word segmentation sequence sequence_list into global_sequence in chronological order, mark g_label as the type of the parallel APP software user operation sequence, and execute Step 4.3; for example, the g_label of the first data is the type of the APP software user operation sequence of the second data, and so on;

[0115] Step 4.3, extract the user operation sequence of each parallel APP software in the parallel APP user operation word segmentation sequence sequence_list and store it in App_sequence, marking a_label as the type of one App_sequence apart; To better describe Step 4.4 and the following content, assume that the obtained global_sequence part of the data is shown in Table 11 and Table 12:

[0116] Table 11 Partial data of global_sequence

[0117]

[0118] Table 12 App_sequence partial data

[0119]

[0120] Tables 11 and 12 are the same batch of data from sequence_list. The data in Table 11 is sorted according to the global timing angle to generate global_sequence, and the data in Table 12 is sorted according to the timing of each parallel APP software user operation to generate App_sequence, and execute Step 4.4;

[0121] Step 4.4, Initialization step length input_length = 6, initial state h t0 = 0 LSTM-based global temporal context extractor:

[0122]

[0123] Among them, g represents the global timing; h t0 The timing context representing the initial moment of the global timing dimension, Respectively represent the temporal context extractor of the previous moment in the global temporal dimension, the bias parameter of the temporal context extractor of the previous moment in the global temporal dimension, the context extractor of the current moment in the global temporal dimension, the bias parameter of the context extractor of the current moment in the global temporal dimension, the temporal context extractor of the current moment in the global temporal dimension, the bias parameter of the temporal context extractor of the current moment in the global temporal dimension, the global temporal context extractor, and the bias parameter of the global temporal context extractor, and execute Step 4.5;

[0124] Step 4.5, input global_sequence, use the global time dimension last moment time context extractor to extract the last moment time context, and output G_memory as:

[0125]

[0126] Among them, h t-1 It is the time series context of the previous moment in the global time series dimension, indicating that the dimension is N f , N f X is the dimension of word_embedding word vector; gt It is the contextual semantic relationship of the content features of the parallel APP software user operations at the current moment in the global_sequence, and the dimension is N f ; The dimension is N f ;σ represents the activation function; execute Step4.6;

[0127] Step 4.6, use the global time series dimension current moment context extractor to extract the current moment context, and the output Gin_context is:

[0128]

[0129] Among them, Gin_context and Indicates dimension N f , execute Step 4.7;

[0130] Step 4.7, use the global timing dimension current time series context extractor to extract the current time series context, and output Gcell_context as:

[0131]

[0132] Among them, Gcell_context and The dimension is N f ; Indicates the time series context of the previous moment in the global time series dimension, indicating that the dimension is N f , execute Step 4.8;

[0133] Step 4.8. Use the global timing context extractor to extract the global timing context, and the output Global_context is:

[0134]

[0135] Among them, Global_context, The dimension is N f , execute Step 4.9;

[0136] Step 4.9, input Global_context into a fixed-size fully connected neural network layer, and then feed it back to the output layer, use the fully connected neural network output to predict the type of the next parallel global time series APP software user operation sequence and the g_label corresponding to the training batch batch to calculate the deviation, and according to the back propagation process, obtain the error of each layer parameter of the first LSTM-based neural network, complete the optimization of the network model, and execute Step 4.10; the LSTM-based neural network includes an input layer, an LSTM layer, a fully connected layer, and an output layer;

[0137] Step4.10. Continue to iterate Step4.9 until the model converges, extract the model LSTM layer output Global_context as the global temporal context and store it in global_temporal_context, and execute Step4.11.

[0138] Step 4.11, Initialization step length input_length = 6, initial state h t0 =0 LSTM-based parallel APP temporal context extractor:

[0139]

[0140] Among them, a represents the parallel APP timing; Respectively represent the timing context extractor of the previous moment of the parallel APP timing dimension, the bias parameter of the timing context extractor of the previous moment of the parallel APP timing dimension, the context extractor of the current moment of the parallel APP timing dimension, the bias parameter of the context extractor of the current moment of the parallel APP timing dimension, the timing context extractor of the current moment of the APP timing dimension, the bias parameter of the timing context extractor of the current moment of the parallel APP timing dimension, the parallel APP timing context extractor, and the bias parameter of the parallel APP timing context extractor, and execute Step4.12;

[0141] Step 4.12. Input App_sequence, use the previous moment temporal context extractor of the parallel APP temporal dimension to extract the temporal context of the previous moment, and output A_memory as:

[0142]

[0143] Among them, h t-1 It is the time series context of the previous moment in the parallel APP time series dimension, indicating that the dimension is N f , N f X is the dimension of word_embedding word vector; at It is the contextual semantic relationship of the content features of the parallel APP software user operations at the current moment in App_sequence, and the dimension is N f ; The representation dimension is N f ;Execute Step4.13;

[0144] Step 4.13. Use the current moment context extractor of the parallel APP time series dimension to extract the current moment context, and the output Ain_context is:

[0145]

[0146] Among them, Ain_context and Indicates dimension N f , execute Step 4.14;

[0147] Step 4.14. Use the current time series context extractor of the parallel APP time series dimension to extract the current time series context, and output Acell_context as:

[0148]

[0149] Among them, Acell_context and Indicates dimension N f ; Indicates the time series context of the previous moment in the parallel APP time series dimension, indicating that the dimension is N f , execute Step 4.15;

[0150] Step 4.15, use the parallel APP timing context extractor to extract the parallel APP timing context, and output App_contex t as;

[0151]

[0152] Among them, App_context, The representation dimension is N f , execute Step 4.16;

[0153] Step 4.16, input App_context into the fixed-size fully connected neural network layer, and then feed it back to the output layer, use the fully connected neural network output to predict the type of APP software user operation sequence for the next parallel APP time sequence and calculate the deviation with the a_label corresponding to the training batch batch, and according to the back propagation process, obtain the error of each layer parameter of the second LSTM-based neural network, complete the optimization of the network model, and execute Step 4.17;

[0154] Step 4.17. Continue to iterate Step 4.16 until the model converges, extract the model LSTM layer output App_context as the parallel APP temporal context and store it in app_temporal_context, and end Step 4.

[0155] Step 5. Complete parallel APP software user operations through a fully connected neural network;

[0156] Step 5.1. Concatenate the global temporal context global_temporal_context extracted from the first LSTM-based neural network in Step 4 and the parallel APP temporal context app_temporal_context extracted from the second LSTM-based neural network, and store the result in the temporal context N d is the number of user operations in the parallel APP software user operation sequence, N f is the dimension of the word vector in word_embedding;

[0157] To better describe Step 5.3 and subsequent steps, assume that the global_temporal_context and app_temporal_context obtained from Step 5.1 are as shown in Tables 13 and 14:

[0158] Table 13 global temporal context

[0159]

[0160] Table 14 app_temporal_context

[0161]

[0162] As shown in Tables 13 and 14, the dimensions of the obtained global_temporal_context and app_temporal_context are both N. f , after splicing, we get each Execute Step 5.2;

[0163] Step 5.2, based on the word embedding vector representation of Step 3.12, weight the average of the parallel APP software user operations in the parallel APP software user operation word sequence sequence_list, obtain the label of the APP software user operation, and store it in Label; the label of each APP software user operation is the weighted average representation of the next parallel APP software user operation; the Label dimension is N d ×N f , execute Step 5.3;

[0164] Step 5.3, initialize a temporal feature detector based on TextCNN:

[0165] Kernel_size,Stride,W convt , b convt

[0166] Among them, kernel_size represents the size of the convolution kernel, Stride represents the step size of the convolution, and W convt , b convt Respectively represent the timing feature detector and the timing feature detector bias parameter, and execute Step 5.4;

[0167] Step 5.4, use the temporal feature detector to slide in the global context of the parallel APP software user operation sequence represented by temporal_context and the parallel APP temporal context, and the output y_context is:

[0168] y_context = Relu(W convt temporal_contextx i +b convt )

[0169] Among them, y_context and temporal_contextx i The i-th group of timing contexts representing the timing contexts of the user operation sequences of the parallel APP software, Execute Step 5.5;

[0170] Step 5.5, perform maximum pooling on y_context, take the maximum value of the features of the two time series dimensions, and reduce the extracted features to a fixed size. After maximum pooling, connect y_context to y and execute Step 5.6;

[0171] Step 5.6, input y into a fixed-size fully connected neural network layer, and then feed it back to the output layer, use the temporal context features of the parallel APP software user operation sequence output by the fully connected neural network and the label corresponding to the training batch to calculate the deviation, and according to the back propagation process, obtain the error of each layer parameter of the neural network based on TextCNN, complete the optimization of the network model, and execute Step 5.7; wherein, the neural network based on TextCNN includes input layer, splicing layer, TextCNN layer, pooling layer, fully connected layer, and output layer;

[0172] Step5.7, continue to iterate step Step5.6 until the network converges, save the model as model, Step5.8;

[0173] Step 5.8. Use the model to complete the parallel APP software user operations and complete the complete parallel APP software user operation sequence at each time point. In order to more intuitively illustrate the completion effect, assume that Table 15 is a list of parallel APP software at different time processes within 1 minute:

[0174] Table 15 Parallel APP software list

[0175]

[0176] As shown in Table 15, assuming that there are 4 parallel APP software running in parallel within 1 minute, the user may interact with different APP software at different time points, then the APP software operations that need to be completed are shown in Table 16:

[0177] Table 16 APP software user operations that need to be completed

[0178] Time APP software where user operation occurred APP software user operations that need to be completed 1 Bilibili, Tencent QQ Sogou Pinyin, Youdao Dictionary 2 Bilibili Tencent QQ, Youdao Dictionary, Sogou Pinyin 3 Tencent QQ, Sogou Pinyin Youdao Dictionary, Bilibili 4 Youdao Dictionary, Sogou Pinyin Tencent QQ, Bilibili

[0179] The final completion effect is shown in Table 17:

[0180] Table 17 Completed user operation sequence for each parallel APP software

[0181]

[0182] According to the parallel APP software user operations that need to be completed in Table 16, we complete them through the trained neural network model, as shown in Table 17, the completed sequence of each parallel APP software user operation (the numbers in Table 17 represent the operation sequence numbers).

[0183] Embodiment 2: According to another aspect of the present invention, a parallel APP software user operation completion system is also provided, including: a capture module, used to capture parallel APP software user operation event content data and parallel APP software response data in real time; an extraction module, used to extract parallel APP software user operation content features; an establishment module, used to establish contextual semantic relationships of parallel APP software user operation content features; an extraction module, used to extract the timing context of the parallel APP software user operation sequence; wherein the timing context includes the global timing context and the parallel APP timing context; a completion module, used to complete the parallel APP software user operation.

[0184] It should be noted here that the above-mentioned capture module, extraction module, establishment module, extraction module and completion module correspond to steps Step 1 to Step 5 in Example 1. The examples and application scenarios implemented by the above-mentioned modules are the same as those of the corresponding steps, but are not limited to the contents disclosed in the above-mentioned Example 1.

[0185] According to another aspect of an embodiment of the present invention, a processor is provided, wherein the processor is used to run a program, wherein the program executes any one of the above-mentioned parallel APP software user operation completion methods when running.

[0186] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned parallel APP software user operation completion methods.

[0187] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A method for completing user operations of parallel APP software, characterized by: include: Step 1: Real-time capture of parallel APP software user operation event content data and parallel APP software response data; Step 2, extract the user operation content features of parallel APP software; Step 3, establish the contextual semantic relationship of the content features of parallel APP software user operations; Step 4, extract the timing context of the parallel APP software user operation sequence; wherein the timing context includes the global timing context and the parallel APP timing context; Step 5. Complete the parallel APP software user operations; The Step 3 specifically includes: According to the content characteristics of parallel APP software user operations, a structured data set D is obtained; Sort the structured data set D by time to obtain the parallel APP user operation sequence; Segment each parallel APP software user operation in the parallel APP user operation sequence to obtain the parallel APP user operation segmentation sequence; Construct a neural network training set based on the set sequence length of the parallel APP user operation word segmentation sequence; Use the neural network training set to train the neural network and establish the contextual semantic relationship of the content features of the parallel APP software user operations; The Step 4 specifically includes: Get the global timing sequence and parallel APP timing sequence; Input the global time series sequence into the first LSTM-based neural network model, and extract the model LSTM layer output as the global time series context; Input the parallel APP time series sequence into the second LSTM-based neural network model, and extract the model LSTM layer output as the parallel APP time series context; The Step 5 specifically includes: The global timing context and the parallel APP timing context are concatenated as the timing context; Input the temporal context into the TextCNN-based neural network model until the network converges to obtain an optimized TextCNN-based neural network model; The parallel APP software user operations are completed through the optimized TextCNN-based neural network model to obtain the complete APP software user operation sequence at each time point.

2. The method for completing user operations of parallel APP software according to claim 1, characterized in that: The Step 2 specifically includes: Based on the Android user interface framework, initialize the parallel APP software user operation content features and establish the parallel APP software user operation event response tree; Based on the parallel APP software user operation content characteristics, the parallel APP software user operation content is obtained from the parallel APP software user operation event content data and the parallel APP software response data; Initialize the node list based on the parallel APP software user operation event response tree; Extracting parallel APP software response node information from the node list based on the parallel APP software user operation content; The user operation content characteristics of the parallel APP software are determined based on the parallel APP software response node information.

3. The method for completing user operations of parallel APP software according to claim 2, characterized in that: Determining the parallel APP software user operation content characteristics based on the parallel APP software response node information includes: If the parallel APP software response node information is empty and there is no parent node, the parallel APP software user operation content characteristics are obtained based on the parallel APP software user operation content; otherwise, the parallel APP software user operation content and the APP software user operation interaction component information are cross-validated to obtain the parallel APP software user operation content characteristics; among which, the APP software user operation interaction component information used to supplement the parallel APP software user operation content is extracted from the parallel APP software response node information.

4. The method for completing user operations of parallel APP software according to claim 1, characterized in that: The obtaining of the global timing sequence and the parallel APP timing sequence includes: Store the elements in the parallel APP user operation word segmentation sequence into the global time sequence in chronological order; The user operation sequence of each parallel APP software in the parallel APP user operation word segmentation sequence is extracted and stored in the parallel APP time sequence.

5. The method for completing user operations of parallel APP software according to claim 1, characterized in that: The first LSTM-based neural network model and the second LSTM-based neural network model both include an input layer, an LSTM layer, a fully connected layer, and an output layer.

6. The method for completing user operations of parallel APP software according to claim 1, characterized in that: The TextCNN-based neural network includes an input layer, a concatenation layer, a TextCNN layer, a pooling layer, a fully connected layer, and an output layer.

7. A parallel APP software user operation completion system, implementing the parallel APP software user operation completion method of claim 1, characterized in that: include: Capture module, used to capture parallel APP software user operation event content data and parallel APP software response data in real time; Extraction module, used to extract the content features of user operations of parallel APP software; Establishing a module for establishing contextual semantic relationships of content features of parallel APP software user operations; An extraction module is used to extract the timing context of the parallel APP software user operation sequence; wherein the timing context includes the global timing context and the parallel APP timing context; The completion module is used to complete the user operations of the parallel APP software.

Images