Method and device for sequence processing using neural network

Abstract

The embodiments of this specification provide a method and a device for sequence processing using a neural network. In the sequence processing method, in the generation layer, the original vector sequence corresponding to the indicator sequence formed by the business indicator values ​​sorted by time is obtained, and multi-level vector aggregation is iteratively performed on the original vector sequence to obtain multi-level processing vector sequences corresponding to different time scales. The original vector sequence and each vector in each processing vector sequence form a multi-level tree relationship diagram. In the attention layer, according to the tree relationship diagram, the association vector of any input target vector is determined, and the target vector is subjected to weighted comprehensive processing based on the attention mechanism according to the association vector to obtain the corresponding update vector; the update vector is used to determine the feature representation of the indicator sequence.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of computer information processing, and more particularly to a method and device for sequence processing using a neural network. Background Art

[0002] An indicator sequence is a sequence of indicator values ​​of business indicators arranged in chronological order. In many application scenarios, it is necessary to predict the indicator sequence (usually to predict the indicator value at one or more moments in the future) so as to provide early warning and detect anomalies when the predicted value fluctuates greatly or the predicted value differs greatly from the actual value.

[0003] In conventional technology, machine learning is usually used to predict indicator sequences, which requires feature representation of the indicator sequence first. However, traditional feature representation methods are either less accurate or more complex. Therefore, it is necessary to provide a more effective sequence processing method that can balance the factors of accuracy and complexity. Summary of the invention

[0004] One or more embodiments of this specification describe a method and a device for sequence processing using a neural network, which can balance the factors of accuracy and time complexity.

[0005] In a first aspect, a method for sequence processing using a neural network is provided, comprising:

[0006] In the generation layer, an original vector sequence corresponding to an indicator sequence formed by business indicator values ​​sorted by time is obtained, and multi-level vector aggregation is iteratively performed on the original vector sequence to obtain multi-level processing vector sequences corresponding to different time scales. The original vector sequence and each vector in each processing vector sequence form a multi-level tree relationship diagram;

[0007] In the attention layer, the association vector of any input target vector is determined according to the tree relationship diagram, and the target vector is subjected to weighted comprehensive processing based on the attention mechanism according to the association vector to obtain a corresponding update vector; the update vector is used to determine the feature representation of the indicator sequence.

[0008] In a second aspect, a sequence processing device is provided, comprising:

[0009] A generating unit is used to obtain an original vector sequence corresponding to an indicator sequence formed by business indicator values ​​sorted by time, and iteratively perform multi-level vector aggregation on the original vector sequence to obtain a multi-level processing vector sequence corresponding to different time scales, wherein the original vector sequence and each vector in each processing vector sequence form a multi-level tree relationship diagram;

[0010] An acquisition unit is used to determine the association vector of any input target vector according to the tree relationship diagram, and perform weighted comprehensive processing based on the attention mechanism on the target vector according to the association vector to obtain a corresponding update vector; the update vector is used to determine the feature representation of the indicator sequence.

[0011] According to a third aspect, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method of the first aspect.

[0012] According to a fourth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method of the first aspect is implemented.

[0013] One or more embodiments of the present specification provide a method and device for sequence processing using a neural network. On the one hand, the target vector is updated only based on the associated vector of the target vector instead of all vectors, which can greatly reduce the number of calculations of the attention score, thereby improving the processing efficiency of the indicator sequence. On the other hand, since the tree relationship diagram is formed based on the processing vector sequence corresponding to different time scales, the associated vector of any target vector can be determined based on the tree relationship diagram, and multi-scale feature learning of the indicator sequence can be achieved, thereby improving the accuracy of the feature representation of the indicator sequence. In summary, this solution can balance the factors of accuracy and time complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0015] Figure 1 A schematic diagram showing the structure of a neural network in one embodiment is shown;

[0016] Figure 2 A schematic diagram showing the structure of a generation layer in one embodiment is shown;

[0017] Figure 3 A flow chart of a method for sequence processing using a neural network according to an embodiment is shown;

[0018] Figure 4 A schematic diagram showing a tree relationship diagram in one embodiment;

[0019] Figure 5A schematic diagram of a device for sequence processing according to an embodiment is shown. DETAILED DESCRIPTION

[0020] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0021] As mentioned earlier, when using machine learning to predict indicator sequences, it is usually necessary to represent the indicator sequences with features.

[0022] In one solution, LogTrans can be used to process the index sequence. Specifically, each element in the index sequence can be made to pay attention to the distance 2 before it. n elements to introduce long-range dependencies. This attention mechanism can be combined with a local attention mechanism similar to the one-dimensional CNN to simultaneously strengthen the locality of the attention mechanism. However, its time and space complexity is O(LlogL), and the author does not provide a code implementation that can utilize the sparsity of this attention mechanism. Where L is the length of the indicator sequence.

[0023] In another solution, Informer can be used to process indicator sequences. This solution notices that the attention score of the query to the key is often sparse. First, the sparsity of the attention score of each query is approximately estimated, and then the query with strong sparsity is made to pay attention to all keys. Its time and space complexity is also O(LlogL), and because the measurement of the sparsity of the attention mechanism is approximate, it is more random.

[0024] The embodiments of this specification provide a method for sequence processing using a neural network, which can be compatible with both accuracy and time complexity.

[0025] The structure of the neural network is first described below.

[0026] Figure 1 A schematic diagram of the structure of a neural network in one embodiment is shown. Figure 1 As shown, the neural network includes at least a generation layer and an attention layer (also called a pyramid attention layer). The generation layer is used to form a multi-level tree relationship diagram based on multi-level processing vector sequences and original vector sequences corresponding to different time scales. The attention layer is used to determine the association vector for any input target vector based on the tree relationship diagram, and perform weighted comprehensive processing on the target vector based on the attention mechanism according to the association vector to obtain the corresponding update vector.

[0027] Furthermore, the neural network may also include a summing and normalization layer and a feedforward layer, and the summing and normalization layer, the feedforward layer and the attention layer together constitute a transformer layer. In a specific example, the number of transformer layers may be N, and thus the number of the above-mentioned attention layers may be N, where N is a positive integer.

[0028] In addition, the neural network may further include an embedding layer for obtaining an indicator sequence formed by business indicator values ​​sorted in time, embedding each business indicator value in the indicator sequence to obtain a corresponding plurality of feature vectors, and forming an original vector sequence based on the plurality of feature vectors.

[0029] Among them, the embedding processing for any first indicator value may include: respectively obtaining the first indicator value, the covariate, and each embedding vector corresponding to the position, and superimposing each embedding vector to obtain the feature vector of the first indicator value. The covariate here can be understood as a parameter that assists in determining the feature vector of the first indicator value. For example, the above covariate can be the sequence number of the first indicator value corresponding to the moment, etc.

[0030] Finally, the neural network may further include a fully connected layer for predicting one or more business indicator values ​​at future moments (also called predicted indicator values). Alternatively, the neural network may further include two decoding layers and a fully connected layer for predicting business indicator values ​​at multiple future moments. The specific prediction process will be described later.

[0031] Figure 2 A schematic diagram of the structure of a generation layer in one embodiment is shown. Figure 2 In the example, the generation layer may include a plurality of convolutional layers stacked in sequence, wherein the processing result of the previous convolutional layer is input into the next convolutional layer.

[0032] In an example, the lengths of the convolution kernels of the convolution layers may be the same or different, which is specifically determined according to the number of child nodes of the nodes at the corresponding level in the tree relationship diagram.

[0033] In addition, the number of layers of the above-mentioned convolutional layers may be positively correlated with the number of levels of the tree relationship graph to be generated.

[0034] In another example, the generation layer may further include two fully connected layers. One of the fully connected layers is located before the multiple convolutional layers and is used to perform dimensionality reduction processing on each input vector in the input vector sequence. The other fully connected layer is located after the multiple convolutional layers and is used to perform dimensionality increase processing on each output vector in the output vector sequence.

[0035] The following uses Figure 1 The method of performing sequence processing using a neural network is described in detail.

[0036] Figure 3 A flow chart of a method for sequence processing using a neural network according to an embodiment is shown. The method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Figure 3 As shown, the method may at least include the following steps.

[0037] Step 302, at the generation layer, obtain the original vector sequence corresponding to the indicator sequence formed by the business indicator values ​​sorted by time, iteratively perform multi-level vector aggregation on the original vector sequence, and obtain multi-level processing vector sequences corresponding to different time scales. The original vector sequence and each vector in each processing vector sequence form a multi-level tree relationship diagram.

[0038] The business indicator value here may refer to data that changes over time, such as transaction volume, transaction amount, application visit volume, or user power consumption.

[0039] In one example, the above indicator sequence may include actual indicator values ​​at t past moments.

[0040] In another example, the above index sequence may include t actual index values ​​at past moments and M predetermined index values ​​at future moments, where M is a positive integer. In one example, the predetermined index value may be 0, for example.

[0041] In addition, the original vector sequence corresponding to the above indicator sequence can be formed based on multiple feature vectors obtained by embedding each business indicator value in the indicator sequence.

[0042] Among them, the embedding processing for any first indicator value may include: respectively obtaining the first indicator value, the covariate, and each embedding vector corresponding to the position, and superimposing each embedding vector to obtain the feature vector of the first indicator value. The covariate here can be understood as a parameter that assists in determining the feature vector of the first indicator value. For example, the above covariate can be the sequence number of the first indicator value corresponding to the moment, etc.

[0043] like Figure 2 As shown, the generation layer provided by the embodiment of the present specification may include multiple convolution layers stacked in sequence, so that the above-mentioned iterative execution of multi-level vector aggregation for the original vector sequence may include: performing multiple convolution processes in sequence in multiple convolution layers respectively, and obtaining multiple processing results corresponding to the multiple convolution processes. The multiple processing results are respectively used as multi-level processing vector sequences. In the process of the above-mentioned multiple convolution processes, the processing result of the previous convolution layer is input into the next convolution layer.

[0044] It should be noted that the processing vector sequences obtained by vector aggregation may have different lengths. The second length of any processing vector sequence is determined according to the first length of the original vector sequence and the convolution kernel length and level of the convolution layer where the corresponding convolution processing is located.

[0045] For example, assume that the first length of the original vector sequence is: L. In addition, assume that the number of convolutional layers is 3, then the second length of the processed vector sequence corresponding to the first convolutional layer is: L / C 1 . Where C is the length of the convolution kernel and 1 is the level of the convolution layer. Similarly, the second length of the processing vector sequence corresponding to the second convolution layer is: L / C 2 , the second length of the processing vector sequence corresponding to the 3rd convolutional layer is: L / C 3 That is, as the number of convolutional layers increases, the resulting processed vector sequence becomes shorter. Thus, the second length is negatively correlated with the number of convolutional layers.

[0046] It should also be noted that the above-mentioned processing vector sequences correspond to different time scales. For example, assuming that the time interval between two adjacent business indicator values ​​in the indicator sequence is 1 hour, or that the time interval between two adjacent feature vectors in the original vector sequence is 1 hour, then, after performing the first level of vector aggregation (for example, the above-mentioned convolution processing) on ​​the original vector sequence, the time interval between two adjacent vectors in the obtained processing vector sequence will be greater than 1 hour, for example, one day. Then, after performing the second level of vector aggregation on the processing vector sequence, the time interval between two adjacent vectors in the obtained processing vector sequence will be greater than 1 day, for example, one week. And so on, as the level increases, the time interval between two adjacent vectors in the obtained processing vector sequence becomes larger and larger. In addition, each processing vector sequence constitutes a hierarchical relationship, and the time scale corresponding to each processing vector sequence constituting the hierarchical relationship is from fine to coarse.

[0047] In addition, it should be understood that the time scale corresponding to each processed vector sequence is greater than the time scale corresponding to the original vector sequence.

[0048] The following describes the process of forming a multi-level tree relationship diagram.

[0049] In one example, the above-mentioned formation of a multi-level tree relationship diagram may include: taking the original vector sequence and each processed vector sequence as each initial vector sequence, and establishing a corresponding relationship between each initial vector sequence and each level of the tree relationship diagram. Wherein, from low level to high level corresponds to time scale from fine to coarse. Based on the corresponding relationship, a multi-level tree relationship diagram is formed.

[0050] For example, in the above example, the original vector sequence of length L can correspond to the first level of the tree relationship diagram. 1 The processing vector sequence can correspond to the second level of the tree relationship diagram. The length is L / C 2 The processing vector sequence can correspond to the third level of the tree relationship diagram. The length is L / C 3 The processing vector sequence can correspond to the 4th level of the tree relationship diagram.

[0051] The above-mentioned tree-like relationship diagram formed with multiple levels based on the corresponding relationship can specifically include: for any i-th level, construct each node corresponding to each vector in the i-th initial vector sequence (that is, the initial vector sequence corresponding to the i-th level); similarly, each node in each level can be constructed. For any first node in the i-th level, select several nodes of the same level (also called nodes of the same scale) from each node in the i-th level, select a parent node (also called a coarse-scale node) from each node in the i+1-th level, and select several child nodes (also called fine-scale nodes) from each node in the i-1-th level. The selected several nodes of the same level, a parent node, and several child nodes are all used as neighbor nodes of the first node, and the connection edges between the first node and the neighbor nodes are constructed. Similarly, the connection edges between each node in each level and the neighbor nodes can be constructed.

[0052] It should be understood that when the i-th level is the highest level, the first node does not have a parent node, so the step of selecting a parent node from each node in the i+1-th level may not be performed. In addition, when the i-th level is the lowest level, the first node does not have a child node, so the step of selecting a number of child nodes from each node in the i-1-th level may not be performed.

[0053] It should be noted that the above-mentioned same-layer node is called the same-scale node because the same-layer node is at the same level (i.e., the current level) as the first node, and one level corresponds to one time scale. The above-mentioned parent node is called the coarse-scale node because the parent node is at the previous level of the first node, and the time scale corresponding to the previous level is coarser than the current level. The above-mentioned child node is called the fine-scale node because the child node is at the next level of the first node, and the time scale corresponding to the next level is finer than the current level.

[0054] In one example, the difference between the node identifier of the above-mentioned node in the same layer and the first identifier of the first node is not greater than a first constant, and the first constant is positively correlated with the preset number of nodes in the same layer. The node identifier of the above-mentioned parent node is determined based on the quotient of the first identifier and the preset number of child nodes. The node identifier of the above-mentioned child node is determined based on the product of the first identifier and the preset number of child nodes.

[0055] In a specific example, the same-layer nodes, parent nodes, and child nodes of the first node are selected according to the following formula.

[0056]

[0057] Among them, l is the first identifier of the first node, j is the node identifier of the same-layer node, parent node or child node, and s is the level where the first node is located, also called the current level. is the set of nodes at the same level, is any node at the same level in the node set at the same level, A is the preset number of nodes at the same level, L is the first length of the original vector sequence, and C is the preset number of child nodes. is a collection of child nodes, Any child node in the child node collection. Is a parent node set, which contains only one parent node S is the number of levels in the tree relationship diagram.

[0058] Of course, in other examples, the above formula 1 may also be modified. For example, j in the above formula may be replaced by j%L, etc. This specification does not limit this.

[0059] Figure 4 A schematic diagram showing a tree relationship diagram in one embodiment. Figure 4 In the diagram, the tree relationship diagram includes 4 levels, wherein the first level contains 8 nodes, and the 8 nodes correspond to each vector in the original vector sequence. The node identifiers of the above 8 nodes are: 1, 2, 3, 4, 5, 6, 7 and 8 respectively. The second level contains 4 nodes, and the 4 nodes correspond to each vector in the processed vector sequence obtained after performing the first level of vector aggregation on the original vector sequence. The node identifiers of the 4 nodes are: 1, 2, 3 and 4 respectively. The third level contains two nodes, and the two nodes correspond to each vector in the processed vector sequence obtained after performing the second level of vector aggregation on the original vector sequence. The node identifiers of the two nodes are: 1 and 2 respectively. The third level contains 1 node, and the 1 node corresponds to the vector obtained after performing the third level of vector aggregation on the original vector sequence. The node identifier of the 1 node is: 1.

[0060] from Figure 4It can be seen that in each level, the first and last nodes each have a peer node, and each node except the first and last nodes has three peer nodes. The peer nodes here include the node itself. For example, the peer nodes of node 3 in the second level are: node 2, node 3 and node 4. Except for the nodes in the lowest level, that is, except for the leaf nodes, each node has two child nodes. For example, the child nodes of node 3 in the second level are: node 5 and node 6 in the first level. Except for the nodes in the highest level, that is, except for the root node, each node has a parent node. For example, the parent node of node 3 in the second level is: node 2 in the third level.

[0061] It should be understood that Figure 3 This is just an exemplary explanation. In actual applications, the number of child nodes in each level can also be 3 or more, and the number of child nodes of nodes in each level can also be different. For example, the number of child nodes of the nodes in the second level is 3, the number of child nodes of the nodes in the third level is 2, and so on. It can be specifically set by controlling the convolution level kernel length of the convolution layer.

[0062] Of course, in practical applications, the above steps of forming a tree relationship diagram can also be performed in the generation layer and the attention layer respectively. Specifically, in the generation layer, based on the results of multiple convolutions, an initial tree relationship diagram is constructed, which only includes the connection edges of each node in each level and the nodes in the two adjacent layers that have a convolution relationship. Then, in the attention layer, the connection edges between the nodes in the same level are constructed, that is, the connection edges between each node and the nodes in the same layer are constructed, so as to obtain the final tree relationship diagram (see Figure 4 ). The specific generation process is as described above, and this specification will not be repeated here.

[0063] Back to Figure 3 The method may further include the following steps.

[0064] Step 304, at the attention layer, determine the association vector of any input target vector according to the tree relationship diagram, perform weighted comprehensive processing on the target vector based on the attention mechanism according to the association vector, and obtain the corresponding update vector, which is used to determine the feature representation of the indicator sequence.

[0065] The determining of the association vector of any input target vector may specifically include: selecting neighbor nodes from each node in the tree relationship diagram that are reached along the connection edge from the node corresponding to the target vector, and determining the vector corresponding to the selected neighbor node as the association vector.

[0066] by Figure 4For example, assuming that the target vector corresponds to node 3 in the second level, its neighbor nodes may include: node 2, node 3 and node 4 in the second level, node 5 and node 6 in the first level, and node 2 in the third level.

[0067] That is to say, for node 3 in the second layer, the information of 6 nodes can be aggregated at the same time. In fact, node 2 in the second layer also aggregates nodes 3 and 4 in the first layer, node 1 in the second layer, and node 1 in the third layer. Therefore, this scheme can capture the information of distant nodes for any node, which can greatly improve the accuracy of sequence representation. And because this scheme aggregates node information layer by layer according to the level, it will not increase the time complexity. And it has been proved by experiments that the time complexity is only a constant multiple of the length L of the original vector sequence, so this scheme can balance the factors of accuracy and time complexity.

[0068] In step 340, performing weighted comprehensive processing based on the attention mechanism on the target vector according to the association vector may include:

[0069] The target vector and the associated vector are transformed by using the first transformation matrix Q, the second transformation matrix K and the third transformation matrix V respectively, and the first intermediate vector, the second intermediate vector and the third intermediate vector corresponding to the target vector are obtained, and the first intermediate vector, the second intermediate vector and the third intermediate vector corresponding to the associated vector are obtained. Based on the dot product operation of the first intermediate vector corresponding to the target vector and the second intermediate vector corresponding to the associated vector, the degree of association between the target vector and the associated vector is determined. With the degree of association between the target vector and the associated vector as a weight factor, the third intermediate vector corresponding to the associated vector is weightedly combined to obtain the first combination vector corresponding to the target vector. Based on the first combination vector, the update vector corresponding to the target vector is obtained. Similarly, the update vector of each input vector can be obtained.

[0070] It should be noted that the above process of determining the update vector of each input vector in the attention layer can also be understood as the update process of the tree relationship graph. The update here is to replace the vector corresponding to each node in the tree relationship graph with the corresponding update vector.

[0071] It should be understood that when the neural network includes N attention layers, the tree relationship diagram can be iteratively updated N times. After that, the feature representation of the indicator sequence can be determined based on each update vector obtained by the last attention layer in the N attention layers. In other words, the feature representation of the indicator sequence is determined based on the tree relationship diagram after N updates.

[0072] In one example, the above-mentioned determination of the characteristic representation of the indicator sequence may specifically include: for each level in the tree relationship diagram after N updates, integrating the update vectors corresponding to each node in the level to obtain each integrated vector corresponding to each level. From the low level to the high level, the integrated vectors of each level are integrated layer by layer to obtain the characteristic representation of the indicator sequence.

[0073] At this point, the processing of the indicator sequence is completed.

[0074] In addition, as mentioned above, the neural network may further include a fully connected layer (hereinafter referred to as the first fully connected layer) for predicting one or more business indicator values ​​at future moments (also referred to as predicted indicator values). Alternatively, the neural network may further include two decoding layers (including the first decoding layer and the second decoding layer) and a fully connected layer (hereinafter referred to as the second fully connected layer) for predicting business indicator values ​​at multiple future moments. The prediction process is described in detail below.

[0075] First, the case where the neural network includes the first fully connected layer and predicts a business indicator value at a future moment is described. In this case, the indicator sequence may include actual indicator values ​​at t past moments and a predetermined indicator value at one future moment.

[0076] Specifically, after the above-mentioned indicator sequence is processed corresponding to the above-mentioned generation layer and N attention layers, in the first fully connected layer, a target vector sequence formed by the update vector corresponding to the last node of each level in the tree relationship diagram after N updates can be obtained. Based on the target vector sequence, a predicted indicator value at one future moment is obtained. The N updates here correspond to N attention layers.

[0077] Similarly, the predicted index values ​​for multiple future moments can be obtained.

[0078] The following further describes the case where the neural network includes two decoding layers and one fully connected layer and predicts the business indicator values ​​at multiple future moments. In this case, the indicator sequence may include the actual indicator values ​​at t past moments and the predetermined indicator values ​​at M future moments. Here, M is a positive integer greater than 1.

[0079] Specifically, after the above-mentioned indicator sequence is processed corresponding to the above-mentioned generation layer and N attention layers, in the first decoding layer, a first target vector sequence formed by each update vector corresponding to each node in the tree relationship diagram after N updates is obtained. According to the first target vector sequence, each vector in the original vector sequence is subjected to weighted comprehensive processing based on the attention mechanism to obtain each intermediate vector. Here, the N updates correspond to the N attention layers.

[0080] At the second decoding layer, each intermediate vector is concatenated with the first target vector sequence to obtain a second target vector sequence. According to the second target vector sequence, each intermediate vector is subjected to weighted comprehensive processing based on the attention mechanism to obtain each final vector.

[0081] In the second fully connected layer, based on each final vector, the predicted index values ​​for M future moments are obtained.

[0082] The weighted comprehensive processing here can refer to the above-mentioned weighted comprehensive processing of the target vector based on the association vector based on the attention mechanism, and this manual will not repeat it here.

[0083] In summary, the method for sequence processing using a neural network provided in the embodiment of this specification, at the attention layer, updates the target vector only based on the associated vector of the target vector instead of based on all vectors, which can greatly reduce the number of calculations of the attention score, thereby improving the processing efficiency of the indicator sequence. In addition, since the tree relationship diagram is formed based on the processing vector sequence corresponding to different time scales, the associated vector of any target vector can be determined based on the tree relationship diagram, which can achieve multi-scale feature learning of the indicator sequence, thereby improving the accuracy of the feature representation of the indicator sequence. Finally, when the accuracy of the feature representation of the indicator sequence is improved, the prediction accuracy of the indicator sequence can be further improved.

[0084] Corresponding to the above-mentioned method for sequence processing using a neural network, an embodiment of the present specification also provides a sequence processing device, such as Figure 5 As shown, the device may include:

[0085] The generation unit 502 is used to obtain the original vector sequence corresponding to the indicator sequence formed by the business indicator values ​​sorted by time, iteratively perform multi-level vector aggregation on the original vector sequence, and obtain multi-level processing vector sequences corresponding to different time scales. The original vector sequence and each vector in each processing vector sequence form a multi-level tree relationship diagram.

[0086] The acquisition unit 504 is used to determine the association vector of any input target vector according to the tree relationship diagram, and perform weighted comprehensive processing on the target vector based on the attention mechanism according to the association vector to obtain a corresponding update vector, which is used to determine the feature representation of the indicator sequence.

[0087] Optionally, the generating unit 502 may include a plurality of convolution modules 5022 stacked in sequence, and the processing result obtained in the previous convolution module 5022 is input into the next convolution module 5022;

[0088] The generating unit 502 is specifically used for:

[0089] Performing multiple convolution processes in sequence in the multiple convolution modules 5022 respectively, to obtain multiple processing results corresponding to the multiple convolution processes;

[0090] The multiple processing results are respectively used as multi-level processing vector sequences.

[0091] The number of convolution modules 5022 is positively correlated with the number of levels of the tree relationship diagram.

[0092] Among them, each processing vector sequence has a different length, and the second length of any processing vector sequence is determined according to the first length of the original vector sequence and the convolution kernel length and level of the corresponding convolution module 5022, and the second length is negatively correlated with the level of the convolution module 5022.

[0093] The length of the convolution kernel is determined according to the number of child nodes of the non-leaf node in the tree relationship diagram.

[0094] Optionally, the generating unit 502 further includes:

[0095] The establishment module 5024 is used to use the original vector sequence and each processed vector sequence as each initial vector sequence, and establish a corresponding relationship between each initial vector sequence and each level of the tree relationship diagram, wherein the time scale from low level to high level corresponds to the time scale from fine to coarse.

[0096] The forming module 5026 is used to form a multi-level tree relationship diagram based on the above corresponding relationship.

[0097] The formation module 5026 is specifically used for:

[0098] For any i-th level, construct each node corresponding to each vector in the i-th initial vector sequence, thereby obtaining each node in each level;

[0099] For any first node in the i-th level, select the same-level nodes, parent nodes and child nodes from the nodes in the i-th level, i+1-th level and i-1-th level as the neighbor nodes of the first node. Construct the connection edges between the first node and the neighbor nodes, and thus construct the connection edges between each node in each level and the neighbor nodes.

[0100] The difference between the node identifier of the node in the same layer and the first identifier of the first node is not greater than a first constant, and the first constant is positively correlated with the preset number of nodes in the same layer. The node identifier of the parent node is determined based on the quotient of the first identifier and the preset number of child nodes. The node identifier of the child node is determined based on the product of the first identifier and the preset number of child nodes.

[0101] Optionally, the generating unit 502 further includes:

[0102] A transformation module (not shown in the figure), used to transform the target vector and the associated vector using a first transformation matrix Q, a second transformation matrix K and a third transformation matrix V respectively, to obtain a first intermediate vector, a second intermediate vector and a third intermediate vector corresponding to the target vector and the associated vector;

[0103] A dot product module (not shown in the figure), used to determine the degree of association between the target vector and the association vector based on a dot product operation of a first intermediate vector corresponding to the target vector and a second intermediate vector corresponding to the association vector;

[0104] A combination module (not shown in the figure) is used to perform weighted combination on the third intermediate vectors corresponding to the association vectors using the degree of association between the target vector and the association vector as a weight factor to obtain a first combination vector corresponding to the target vector, and obtain a corresponding update vector based on the first combination vector.

[0105] Optionally, the acquiring unit 504 may specifically include:

[0106] The selection module 5042 is used to select, from each node in the tree relationship diagram, a neighboring node that is reached along the connecting edge from the node corresponding to the target vector.

[0107] The determination module 5044 is used to determine the vector corresponding to the selected neighbor node as the association vector.

[0108] Optionally, the device may further include:

[0109] The embedding unit (not shown in the figure) is used to obtain the indicator sequence, and embed each business indicator value therein to obtain a corresponding plurality of feature vectors, and form an original vector sequence based on the plurality of feature vectors.

[0110] Optionally, the number of the acquisition units 504 is N, and N is greater than 1, and the device further includes:

[0111] The determination unit 506 is used to determine the characteristic representation of the indicator sequence based on each update vector obtained by the last acquisition unit 504 among the N acquisition units 504 .

[0112] Optionally, the indicator sequence includes t actual indicator values ​​at past moments and 1 predetermined indicator value at a future moment, and the device further includes:

[0113] The first prediction unit (not shown in the figure) is used to obtain a target vector sequence formed by an update vector corresponding to the last node of each level in the tree relationship diagram after N updates. Based on the target vector sequence, a prediction index value at a future moment is obtained; wherein the N updates correspond to the N acquisition units 504 respectively.

[0114] Optionally, the indicator sequence includes t actual indicator values ​​at past moments and M predetermined indicator values ​​at future moments, and the device further includes:

[0115] The first decoding unit (not shown in the figure) is used to obtain a first target vector sequence formed by each update vector corresponding to each node in the tree relationship diagram after N updates. According to the first target vector sequence, each vector in the original vector sequence is subjected to weighted comprehensive processing based on the attention mechanism to obtain each intermediate vector. The N updates correspond to the N acquisition units 504 respectively.

[0116] The second decoding unit (not shown in the figure) is used to concatenate each intermediate vector with the first target vector sequence to obtain a second target vector sequence. According to the second target vector sequence, each intermediate vector in each intermediate vector is subjected to weighted comprehensive processing based on the attention mechanism to obtain each final vector.

[0117] The second prediction unit (not shown in the figure) is used to obtain prediction index values ​​at M future moments based on each final vector.

[0118] The functions of the functional modules of the device in the above-mentioned embodiment of this specification can be implemented through the steps of the above-mentioned method embodiment. Therefore, the specific working process of the device provided by one embodiment of this specification will not be repeated here.

[0119] An embodiment of the present specification provides a sequence processing device that can balance the factors of accuracy and time complexity.

[0120] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0121] The steps of the method or algorithm described in conjunction with the disclosure of this specification can be implemented in hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, register, hard disk, mobile hard disk, CD-ROM or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a server. Of course, the processor and the storage medium can also be present in the server as discrete components.

[0122] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the present invention may be implemented in hardware, software, firmware, or any combination thereof. When implemented using software, the functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium may be any available medium that a general or special-purpose computer can access.

[0123] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0124] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of this specification. It should be understood that the above description is only the specific implementation method of this specification and is not intended to limit the protection scope of this specification. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of this specification should be included in the protection scope of this specification.

Claims

1. A method for processing visit sequence using a neural network, wherein the neural network includes a generation layer, N attention layers and a fully connected layer; the method includes: In the generation layer, an original vector sequence corresponding to a sequence of visits including t actual visit values ​​at t past moments and M scheduled visit values ​​at future moments is obtained, and multi-level vector aggregation is iteratively performed on the original vector sequence to obtain multi-level processing vector sequences corresponding to different time scales. The original vector sequence and each vector in each processing vector sequence form a multi-level tree relationship diagram; in each level of the tree relationship diagram, from low level to high level corresponds to time scales from fine to coarse; At any attention layer, according to the tree relationship diagram, determine the association vector of any input target vector, and perform weighted comprehensive processing based on the attention mechanism on the target vector according to the association vector to obtain a corresponding update vector; In the fully connected layer, a target vector sequence is obtained based on the update vector corresponding to the last node of each level in the tree relationship diagram after N updates; based on the target vector sequence, a predicted visit value at a future moment is obtained; wherein the N updates correspond to the N attention layers respectively.

2. The method according to claim 1, wherein: The generation layer includes a plurality of convolutional layers stacked in sequence, and the processing result obtained in the previous convolutional layer is input into the next convolutional layer; The iterations perform multiple levels of vector aggregation, including: Performing multiple convolution processes in sequence on the multiple convolution layers respectively to obtain multiple processing results corresponding to the multiple convolution processes; The multiple processing results are respectively used as the multi-level processing vector sequences.

3. The method according to claim 2, wherein: The processing vector sequences have different lengths, and the second length of any processing vector sequence is determined according to the first length of the original vector sequence and the convolution kernel length and level of the corresponding convolution layer; the second length is negatively correlated with the level.

4. The method according to claim 3, wherein: The convolution kernel length is determined according to the number of child nodes of non-leaf nodes in the tree relationship graph.

5. The method according to claim 2, wherein: The number of the convolutional layers is positively correlated with the number of levels of the tree relationship graph.

6. The method according to claim 1, wherein: The forming of the multi-level tree relationship diagram includes: The original vector sequence and each processed vector sequence are used together as each initial vector sequence, and a corresponding relationship between each initial vector sequence and each level of the tree relationship diagram is established; Based on the corresponding relationship, the multi-level tree relationship diagram is formed.

7. The method according to claim 6, wherein: Forming the multi-level tree relationship diagram includes: For any i-th level, construct nodes corresponding to each vector in the i-th initial vector sequence; thereby obtaining nodes in each level; For any first node in the i-th level, select the same-layer nodes, parent nodes and child nodes from the nodes in the i-th level, the i+1-th level and the i-1-th level as the neighbor nodes of the first node; construct the connecting edges between the first node and the neighbor nodes; in this way, construct the connecting edges between each node in each level and the neighbor nodes.

8. The method according to claim 7, wherein: The difference between the node identifier of the same-layer node and the first identifier of the first node is not greater than a first constant, and the first constant is positively correlated with the preset number of same-layer nodes; the node identifier of the parent node is determined based on the quotient of the first identifier and the preset number of child nodes; the node identifier of the child node is determined based on the product of the first identifier and the preset number of child nodes.

9. The method according to claim 1, wherein: The step of determining an associated vector of any input target vector comprises: From each node in the tree relationship diagram, select a neighbor node that is reached along the connecting edge from the node corresponding to the target vector; The vector corresponding to the selected neighbor node is determined as the association vector.

10. The method according to claim 1, wherein: The step of performing weighted comprehensive processing on the target vector based on the association vector and the attention mechanism includes: The target vector and the associated vector are transformed by using a first transformation matrix Q, a second transformation matrix K and a third transformation matrix V respectively, so as to obtain a first intermediate vector, a second intermediate vector and a third intermediate vector corresponding to the target vector and the associated vector; Determining the degree of association between the target vector and the association vector based on a dot product operation of a first intermediate vector corresponding to the target vector and a second intermediate vector corresponding to the association vector; Taking the degree of association between the target vector and the association vector as a weight factor, weighted combination is performed on the third intermediate vector corresponding to the association vector to obtain a first combination vector corresponding to the target vector; Based on the first combined vector, a corresponding update vector is obtained.

11. The method according to claim 1, wherein: The neural network also includes an embedding layer; In the embedding layer, the pageview sequence is obtained, and each actual pageview value therein is embedded to obtain a corresponding plurality of feature vectors; and the original vector sequence is formed based on the plurality of feature vectors.

12. The method according to claim 1, wherein: N is greater than 1; the method further comprises: Based on each update vector obtained from the last attention layer among the N attention layers, a feature representation of the access volume sequence is determined.

13. The method according to claim 12, wherein: M is a positive integer greater than 1; the neural network further includes a first decoding layer and a second decoding layer; the method further includes: At the first decoding layer, a first target vector sequence formed by each update vector corresponding to each node in the tree relationship diagram after N updates is obtained; and each vector in the original vector sequence is subjected to weighted comprehensive processing based on the attention mechanism according to the first target vector sequence to obtain each intermediate vector; wherein the N updates correspond to the N attention layers respectively; At the second decoding layer, the intermediate vectors are concatenated with the first target vector sequence to obtain a second target vector sequence; and weighted comprehensive processing based on the attention mechanism is performed on each of the intermediate vectors according to the second target vector sequence to obtain final vectors; In the fully connected layer, based on the final vectors, the predicted visit volume values ​​at the M future moments are obtained.

14. A sequence processing device, comprising: A generating unit is used to obtain an original vector sequence corresponding to a visit amount sequence including t actual visit amount values ​​at t past moments and M scheduled visit amount values ​​at future moments, and iteratively perform multi-level vector aggregation on the original vector sequence to obtain a multi-level processing vector sequence corresponding to different time scales, wherein each vector in the original vector sequence and each processing vector sequence forms a multi-level tree relationship diagram; in each level of the tree relationship diagram, from a low level to a high level corresponds to a time scale from fine to coarse; An acquisition unit, configured to determine, according to the tree relationship diagram, an associated vector of any input target vector, and perform weighted comprehensive processing based on an attention mechanism on the target vector according to the associated vector to obtain a corresponding update vector; A prediction unit is used to obtain a target vector sequence formed by an update vector corresponding to the last node of each level in the tree relationship diagram after N updates; based on the target vector sequence, a predicted access value at a future moment is obtained; wherein the N updates correspond to N acquisition units respectively.

15. The device according to claim 14, wherein: The generating unit includes a plurality of convolution modules stacked in sequence, and the processing result obtained in the previous convolution module is input into the next convolution module; The generating unit is specifically used for: Performing multiple convolution processes in the multiple convolution modules in sequence to obtain multiple processing results corresponding to the multiple convolution processes; The multiple processing results are respectively used as the multi-level processing vector sequences.

16. The device according to claim 15, wherein: The processing vector sequences have different lengths, and the second length of any processing vector sequence is determined based on the first length of the original vector sequence and the convolution kernel length and level of the corresponding convolution module; the second length is negatively correlated with the level.

17. The device according to claim 14, wherein: The generating unit further comprises: An establishment module is used to use the original vector sequence and each processed vector sequence as each initial vector sequence, and establish a corresponding relationship between each initial vector sequence and each level of the tree relationship diagram; wherein, from low level to high level corresponds to a time scale from fine to coarse; A forming module is used to form the multi-level tree relationship diagram based on the corresponding relationship.

18. The device according to claim 17, wherein: The forming module is specifically used for: For any i-th level, construct nodes corresponding to each vector in the i-th initial vector sequence; thereby obtaining nodes in each level; For any first node in the i-th level, select nodes in the same layer, parent nodes and child nodes from the nodes in the i-th level, the i+1-th level and the i-1-th level as neighbor nodes of the first node; and construct connecting edges between the first node and the neighbor nodes; In this way, the connection edges between each node and its neighboring nodes in each level are constructed.

19. The device according to claim 14, wherein: The acquisition unit comprises: A selection module, used to select, from each node in the tree relationship diagram, a neighbor node that is reached along a connection edge from the node corresponding to the target vector; The determination module is used to determine the vector corresponding to the selected neighbor node as the association vector.

20. The device according to claim 14, wherein: The number of the acquisition units is N, and N is greater than 1; the device further includes: The determination unit is used to determine the characteristic representation of the pageview sequence based on each update vector obtained by the last acquisition unit among the N acquisition units.

21. The device according to claim 20, wherein: M is a positive integer greater than 1; the device further comprises: A first decoding unit is used to obtain a first target vector sequence formed by each update vector corresponding to each node in the tree relationship diagram after N updates; and perform weighted comprehensive processing based on the attention mechanism on each vector in the original vector sequence according to the first target vector sequence to obtain each intermediate vector; wherein the N updates correspond to the N acquisition units respectively; A second decoding unit is used to concatenate the intermediate vectors with the first target vector sequence to obtain a second target vector sequence; and perform weighted comprehensive processing based on an attention mechanism on each of the intermediate vectors according to the second target vector sequence to obtain final vectors; The prediction unit is further used to obtain the predicted visit volume values ​​at the M future moments based on the final vectors.

22. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 13.

23. A computing device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 13 is implemented.

Images