Behavior graph analysis method and device

Abstract

The invention discloses a behavior graph analysis method and device, and the method comprises the steps: obtaining a text sequence feature of a behavior graph and an image sequence feature of the behavior graph, and enabling the behavior graph to comprise a logic relation between graph symbols with an incidence relation and a text, and between the graph symbols; the text sequence features and the image sequence features are fused to obtain fused sequence features, the fused sequence features are input into a trained analysis model to obtain a semantic text describing the behavior graph, and the semantic text is used for describing the incidence relation between graph symbols and texts and the logic relation between the graph symbols in a text mode. By adopting the method, the graph symbols in the behavior graph can be aligned with the text, and the logical relationship between the graph symbols can be identified.

Description

Technical Field

[0001] This application relates to the field of image analysis technology, and in particular to a behavioral graph analysis method and apparatus. Background Technology

[0002] In software development, product design, and many other fields, numerous flowcharts and sequence diagrams exist to represent product business logic. During the knowledge accumulation process, accurate parsing of flowcharts and sequence diagrams is crucial. However, existing technical solutions, which use Optical Character Recognition (OCR) models to parse flowcharts or sequence diagrams, can only extract the text within the diagram and cannot achieve complete knowledge extraction from the flowchart or sequence diagram. Summary of the Invention

[0003] This application provides a behavior graph parsing method and apparatus for aligning graph symbols and text in behavior graphs and identifying logical relationships between graph symbols.

[0004] In a first aspect, embodiments of this application provide a behavior graph parsing method, which can be executed by a behavior graph parsing device. The method includes: acquiring text sequence features and image sequence features of the behavior graph; the behavior graph includes graph symbols and text with association relationships, and logical relationships between graph symbols; fusing the text sequence features and the image sequence features to obtain fused sequence features; inputting the fused sequence features into a trained parsing model to obtain semantic text describing the behavior graph, wherein the semantic text is used to describe the association relationships between graph symbols and text, and the logical relationships between graph symbols in a textual manner.

[0005] Using the above method, the text sequence features and image sequence features extracted from the behavior graph are fused into a fused sequence feature. The parsing model analyzes this fused sequence feature to obtain semantic text representing the semantic information of the behavior graph. Semantic text can include text content, the relationships between text and graph symbols, and the logical relationships between graph symbols. Since there is a correspondence between graph symbols and text content, the logical relationships between texts can also be determined based on the logical relationships between graph symbols. In this way, complete knowledge of the behavior graph can be extracted, yielding the complete knowledge of the behavior graph.

[0006] In one possible implementation, the parsing model includes an encoder and a decoder; the encoder is used to learn the logical relationships between graph symbols through the fused sequence features; the decoder is used to learn the association between graph symbols and text through masked sequence features; the masked sequence features are obtained by masking the fused sequence features.

[0007] In one possible implementation, the analytical model is trained as follows: determining the sample fusion sequence features of the sample graphs; learning the predicted logical relationships between the symbols of each sample graph in the sample fusion sequence features through an encoder; determining a first loss value based on the predicted logical relationships and the true logical relationships between the symbols of each sample graph in the sample graphs; performing random partial masking on the sample fusion sequence features to obtain sample mask sequence features; learning the sample mask sequence features through a decoder to obtain sample prediction sequence features; determining a second loss value based on the sample prediction sequence features and the sample fusion sequence features; and performing the next round of training on the analytical model based on the first loss value and the second loss value.

[0008] In one possible implementation, the sample mask sequence features include text mask sequence features and image mask sequence features; the sample prediction sequence features include sample text prediction sequence features and sample image prediction sequence features; determining a second loss value based on the sample prediction sequence features and the sample fusion sequence features includes: determining a second loss value based on the loss value of the text sequence features in the sample text prediction sequence features and the loss value of the image sequence features in the sample fusion sequence features.

[0009] In one possible implementation, the sample fusion sequence features are randomly partially masked to obtain sample mask sequence features, including: randomly partially masking the text sequence features in the sample fusion sequence features and then fusing them with the image sequence features in the sample fusion sequence features to obtain text mask sequence features; and / or randomly partially masking the image sequence features in the sample fusion sequence features and then fusing them with the text sequence features in the sample fusion sequence features to obtain image mask sequence features.

[0010] In one possible implementation, the parsing model is a deep learning model for long sequence prediction; the behavior graph is a flowchart or a timing graph.

[0011] In one possible implementation, obtaining the text sequence features and image sequence features of the behavior graph includes: extracting text information from the behavior graph using an optical character recognition (OCR) model and converting the text information into the text sequence features; traversing the behavior graph using a depth-first search (DFS) algorithm to determine the topological sequence features representing the connection relationships between the graph symbols in the behavior graph; determining the layout sequence features representing the positional information of the graph symbols in the behavior graph based on their positions; obtaining the visual sequence features of the behavior graph, and combining the topological sequence features and the layout sequence features to obtain the image sequence features.

[0012] Secondly, embodiments of this application provide a behavior graph parsing apparatus, the apparatus comprising: an acquisition module, configured to acquire text sequence features and image sequence features of the behavior graph; the behavior graph includes graph symbols and text with association relationships, and logical relationships between graph symbols; a fusion module, configured to fuse the text sequence features and the image sequence features to obtain fused sequence features; and a processing module, configured to input the fused sequence features into a trained parsing model to obtain semantic text describing the behavior graph, the semantic text being used to describe the association relationships between graph symbols and text, and the logical relationships between graph symbols in a textual manner.

[0013] In one possible implementation, the parsing model in the behavioral graph parsing device includes an encoder and a decoder; the encoder is used to learn the logical relationships between graph symbols through the fused sequence features; the decoder is used to learn the association between graph symbols and text through mask sequence features; the mask sequence features are obtained by masking the fused sequence features.

[0014] In one possible implementation, the acquisition module is further configured to: determine the sample fusion sequence features of the sample image; the processing module is further configured to: learn the predicted logical relationship between each sample image symbol in the sample fusion sequence features through an encoder; determine a first loss value based on the predicted logical relationship and the true logical relationship between each sample image symbol in the sample image; perform random partial masking on the sample fusion sequence features to obtain sample mask sequence features; learn the sample mask sequence features through a decoder to obtain sample prediction sequence features; determine a second loss value based on the sample prediction sequence features and the sample fusion sequence features; and perform the next round of training on the analytical model based on the first loss value and the second loss value.

[0015] In one possible implementation, the sample mask sequence features include text mask sequence features and image mask sequence features; the sample prediction sequence features include sample text prediction sequence features and sample image prediction sequence features; the processing module is further configured to determine a second loss value based on the loss value of the text sequence features in the sample text prediction sequence features and the loss value of the image sequence features in the sample fusion sequence features, and the loss value of the image sequence features in the sample image prediction sequence features and the sample fusion sequence features.

[0016] In one possible implementation, the processing module is further configured to: perform random partial masking on the text sequence features in the sample fusion sequence features, and then fuse them with the image sequence features in the sample fusion sequence features to obtain text mask sequence features; and / or perform random partial masking on the image sequence features in the sample fusion sequence features, and then fuse them with the text sequence features in the sample fusion sequence features to obtain image mask sequence features.

[0017] In one possible implementation, the parsing model is a deep learning model for long sequence prediction; the behavior graph is a flowchart or a timing graph.

[0018] In one possible implementation, the acquisition module is specifically used to: extract text information from the behavior graph using an optical character recognition (OCR) model; convert the text information into the text sequence features; traverse the behavior graph using a depth-first search (DFS) algorithm to determine the topological sequence features representing the connection relationships between the graph symbols in the behavior graph; determine the layout sequence features representing the position information of the graph symbols in the behavior graph based on their positions; acquire the visual sequence features of the behavior graph; and combine the topological sequence features and the layout sequence features to obtain the image sequence features.

[0019] Thirdly, embodiments of this application provide a behavior graph parsing device, which includes a memory and a processor. The memory is used to store computer programs or instructions; the processor is used to call the computer programs or instructions stored in the memory to execute the method as described in any possible implementation of the first aspect.

[0020] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when read and executed by a computer, cause the computer to perform a method as described in any possible implementation of the first aspect.

[0021] Fifthly, embodiments of this application provide a computer program product storing instructions that, when read and executed by a computer, cause the computer to perform the method in any possible implementation of the first aspect described above. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating a behavior graph parsing method provided in an embodiment of this application. Figure 2 This is a schematic diagram of the internal structure of an analytical model provided in an embodiment of this application; Figure 3 This is a flowchart illustrating a method for training an analytical model provided in an embodiment of this application. Figure 4 This application provides an embodiment of a behavior graph parsing device with internal modules as shown in the schematic diagram of its internal modules. Figure 5 This is a schematic diagram of the structure of a behavior graph parsing device provided in an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] In the field of software development and product design, there are many flowcharts and sequence diagrams used to convey product business logic and code logic. During knowledge accumulation, existing methods rely on OCR models to recognize text within flowcharts or sequence diagrams and list the text in the recognition order. However, this method cannot identify the relationships between text and diagram symbols, or the logical relationships between the diagram symbols themselves.

[0026] For example, flowcharts include various symbols with special meanings in addition to text. For instance, diamond symbols represent judgments; rounded rectangle symbols represent the start or end of a process; right-angled rectangle symbols represent processing steps or task execution; and connecting arrows represent the flow direction of the process. Sequence diagrams also include various symbols with special meanings in addition to text. For example, a timeline represents the timeline throughout the interaction process; and message arrows represent synchronous, asynchronous, or return messages. Of course, in actual business scenarios, in addition to flowcharts and sequence diagrams, there are other types of execution process diagrams. This application uses behavioral diagrams to represent these types of diagrams.

[0027] Based on this, this application proposes a behavior graph parsing method to align graph symbols and text in a behavior graph and to identify the logical relationships between graph symbols.

[0028] Figure 1 This is a flowchart illustrating a behavior graph parsing method provided in an embodiment of this application. This process can be executed by a behavior graph parsing device, such as... Figure 1 As shown, the process includes the following steps: Step 101: Obtain the text sequence features and image sequence features of the behavior graph.

[0029] Specifically, behavioral diagrams include related diagram symbols and text, and the logical relationships between diagram symbols. Diagram symbols can be rounded rectangles, diamonds, or right-angled rectangles. The relationship between diagram symbols and text can be that the text is inside or outside the diagram symbol, or a correspondence between text and diagram symbols. The logical relationships between diagram symbols can be data flow relationships, decision branch relationships, or no relationship, etc. A data flow relationship indicates that data in diagram symbol A flows to data in diagram symbol B; a decision branch relationship indicates that diagram symbol B is one of the branches of diagram symbol A; no relationship indicates that there is no direct relationship between diagram symbol A and diagram symbol B. Behavioral diagrams can be flowcharts or sequence diagrams.

[0030] Obtaining text sequence features from behavior graphs involves extracting text information from the behavior graph using an OCR model and converting the text information into text sequence features. Specifically, the OCR model extracts the text corresponding to the graph symbols contained in the behavior graph, such as "approval approved" or "notification to finance." The text includes recognizable characters such as letters, numbers, and symbols. Further, the text information is sorted according to the logical relationships between the graph symbols, and the sorted text fragments are converted into text sequence features. For example, each token included in a text fragment is mapped to an index in the vocabulary; for example, "enterprise" can correspond to index 105. Then, a text embedding model is used to convert the token index into text sequence features.

[0031] The image sequence features of the behavior graph are obtained by traversing the behavior graph using the depth-first search (DFS) algorithm to determine the topological sequence features that represent the connection relationship between the graph symbols in the behavior graph; determining the layout sequence features that represent the position information of the graph symbols in the behavior graph based on the position of the graph symbols in the behavior graph; obtaining the visual sequence features of the behavior graph; and combining the topological sequence features and the layout sequence features to obtain the image sequence features.

[0032] Specifically, the Depth-First Search (DFS) algorithm is used to traverse or search trees and graphs. Its core idea is to start from the initial node and visit nodes as deeply as possible along a path until it cannot continue. Then, it backtracks to the previous node, tries another path, and repeats this process until all nodes have been visited. By traversing the behavioral graph using the DFS algorithm, the connections between all graph symbols are identified, such as the flow from symbol A to symbol B, and from symbol B to symbol C. These connections are then converted into vector form, yielding the topological sequence features.

[0033] The behavior graph is segmented, and the spatial coordinates of the graph symbols in each segmented subgraph are calculated. The spatial coordinates of the graph symbols can be the coordinates of the top-left corner of the graph symbol. The coordinates of the lower right corner of the symbol are The layout sequence features that represent the positional information of the graphic symbols are determined based on the spatial coordinates of each graphic symbol.

[0034] The behavior graph is segmented into subgraphs, each containing a single graph symbol, an arrow, etc. A visual feature extraction network converts each subgraph into fixed-dimensional visual sequence features. Segmenting the behavior graph into subgraphs before extracting visual sequence features allows the visual feature extraction model to focus more on the visual features of individual subgraphs, such as shape and boundary morphology. The visual sequence features, layout sequence features, and topological sequence features of the behavior graph are then fused to obtain an image sequence feature that can be understood by the parsing model.

[0035] Step 102: Fuse text sequence features and image sequence features to obtain fused sequence features.

[0036] Specifically, there are various ways to fuse text sequence features and image sequence features. For example, one can add the text sequence features to the image sequence features to obtain fused sequence features, or simply concatenate the text sequence features and image sequence features to obtain fused sequence features. The fusion method can be adjusted according to actual needs. For example, fused sequence features can be represented as follows: .

[0037] Step 103: Input the fused sequence features into the trained parsing model to obtain the semantic text describing the behavior graph.

[0038] The fused sequence features of the behavior graph are input into a trained parsing model, which is a deep learning model for long sequence prediction. The parsing model outputs semantic text describing the behavior graph. This semantic text describes the relationships between graph symbols and text, as well as the logical relationships between graph symbols. For example, the semantic text includes the type of graph symbol (e.g., decision symbol, processing symbol, start symbol); the logical relationships between graph symbols (e.g., decision-branch relationship, data flow relationship, no relationship, or other logical relationships); and the correspondence between text and graph symbols (e.g., text 1 corresponds to graph symbol 1, text 2 corresponds to graph symbol 2). The semantic text can be structured JavaScript key-value pair data (JavaScript Object Notation, JSON) formatted text.

[0039] Using the above method, the text sequence features and image sequence features extracted from the behavior graph are fused into a fused sequence feature. The parsing model analyzes this fused sequence feature to obtain semantic text representing the semantic information of the behavior graph. Semantic text can include text content, the relationships between text and graph symbols, and the logical relationships between graph symbols. Since there is a correspondence between graph symbols and text content, the logical relationships between texts can also be determined based on the logical relationships between graph symbols. In this way, complete knowledge of the behavior graph can be extracted, yielding the complete knowledge of the behavior graph.

[0040] In one possible implementation, the parsing model includes an encoder and a decoder, wherein the encoder is used to learn the logical relationships between graph symbols through fused sequence features, and the decoder is used to learn the association between graph symbols and text through masked sequence features, which are obtained by masking the fused sequence features.

[0041] Figure 2 This is a schematic diagram of the internal structure of an analytical model provided in an embodiment of this application. For example... Figure 2 As shown, the parsing model is a deep learning model designed for long sequence prediction. It can capture sequence dependencies in the data and performs particularly well in long sequence prediction tasks. The encoder includes a multi-head probabilistic sparse self-attention module, a self-attention distillation module, and a relation prediction head module. The decoder includes a multi-head attention module and a masked multi-head probabilistic and sparse self-attention module. The feature map output by the encoder represents some parameters passed from the encoder to the decoder.

[0042] Specifically, the analytical model is trained in the following manner. Figure 3 This is a flowchart illustrating a method for training an analytical model provided in an embodiment of this application. It includes the following steps: Step 301: Determine the sample fusion sequence features of the sample image.

[0043] Specifically, the sample graph is a behavioral graph, which can be a flowchart or a sequence diagram. The sample fusion sequence features of the sample graph are obtained by fusing the text sequence features and image sequence features of the sample graph. The methods for obtaining the text sequence features and image sequence features of the sample graph, as well as the methods for fusing the text sequence features and image sequence features, are the same as described above, and will not be elaborated further here.

[0044] Step 302: The encoder learns the predicted logical relationship between the symbols of each sample map in the sample fusion sequence features. Based on the predicted logical relationship and the real logical relationship between the symbols of each sample map in the sample map, the first loss value is determined.

[0045] Specifically, when calculating the attention, the multi-head probabilistic sparse self-attention module in the encoder only focuses on a few graph symbols with the highest semantic association degree with the current graph symbol. For example, when calculating the graph symbol corresponding to "Approval Passed", only the graph symbols of its parent node "Decision Making" and the direct sub-node "Notify Finance" are concerned. In this way, unnecessary calculations are greatly reduced.

[0046] Between the pooling layers of the encoder, the self-attention distillation (Flow-Distilling) module compresses and reduces the dimension of the fused sequence features, removing redundant image and text details while maintaining key information, ensuring that the high-level processes the macro-logic of the behavior graph.

[0047] The encoder also includes a relation prediction head module. The encoder learns the fused sequence features of the samples , and outputs the predicted sequence features . The relation prediction head outputs ( ) the probability distribution of satisfying a certain relation. For any feature in the fused sequence features of the samples , the parsing model is required to predict the type of relation between it and all historical features (j < i) in the sequence.

[0048] Based on the predicted logical relation and the true logical relation between each sample graph symbol in the sample graph, the first loss value is determined. The calculation formula of the first loss value is shown in Formula 1: Formula 1 Where, n represents the sequence length; K is the total number of relation types (such as data flow relation, decision branch relation, no relation, etc.); and are the predicted features of the elements [[ID=X]]and after being output by the encoder; is the true value label based on one-hot encoding, indicating whether there is the k-th relation between the features [[ID=X]]and <X ; is the relation probability predicted by the model based on [[ID=X]]and . This loss function can train the parsing model to learn how to extract the logical relations between graph symbols from the fused sequence features, solving the problem of extracting the logical relations between graph symbols.

[0049] Step 303, perform random partial masking on the fused sequence features of the samples to obtain the masked sequence features of the samples.

[0050] The sample mask sequence features include text mask sequence features and image mask sequence features. Specifically, after randomly masking a portion of the text sequence features in the sample fusion sequence features, it is fused with the image sequence features in the sample fusion sequence features to obtain the text mask sequence features. For example, the sample fusion sequence {7,9,11,13,15} is obtained by fusing the text sequence features {1,2,3,4,5} and the image sequence features {6,7,8,9,10}. Randomly selecting one of the text features for masking yields {1,2,0,4,5}, which is then fused with the image sequence features {6,7,8,9,10} to obtain {7,9,8,13,15}, which is the text mask sequence feature.

[0051] After randomly masking a portion of the image sequence features in the sample fusion sequence features, this masked portion is fused with the text sequence features in the sample fusion sequence features to obtain the image mask sequence features. Using the sample fusion sequence features from the example above, one image feature is randomly selected for masking to obtain {6,7,8,0,10}, and fused with the text sequence features {1,2,3,4,5} to obtain {7,9,11,4,15}, which is the image mask sequence features. Generally, 10% to 30% of the text sequence features and / or 10% to 30% of the image sequence features can be selected for masking.

[0052] Step 304: Learn the sample mask sequence features through the decoder to obtain the sample prediction sequence features. Based on the sample prediction sequence features and the sample fusion sequence features, determine the second loss value.

[0053] The text mask sequence features are input into the decoder of the parsing model. After passing through the decoder and the fully connected layer, the sample text prediction sequence features corresponding to the text mask sequence features are obtained. Based on the text sequence features in the sample text prediction sequence features and the sample fusion sequence features, the loss value of the text sequence features in the sample text prediction sequence features and the sample fusion sequence features is determined. The formula for calculating the loss value is shown in Formula 2. Formula 2 in, V represents the number of masked text tokens; V represents the size of the vocabulary; m represents the set. The m-th masked text token in the string; express The true One-Hot encoding indicates that its actual word is the v-th word in the vocabulary; The decoder represents the output vector h of the encoded data. m Predicted The probability of belonging to word v. By randomly masking the text sequence features in the fused sequence features of the samples, the parsing model uses the unmasked image sequence features to predict the masked text sequence features. The predictive ability of the training parsing model depends on the image that coincides with the spatial location of the text, ensuring the accuracy of the alignment of text and image symbols in complex behavior graphs.

[0054] The image mask sequence features are input into the decoder of the parsing model. After passing through the decoder and the fully connected layer, the sample image prediction sequence features corresponding to the image mask sequence features are obtained. Based on the image sequence features in the sample image prediction sequence features and the sample fusion sequence features, the loss value of the image sequence features in the sample image prediction sequence features and the sample fusion sequence features is determined. The formula for calculating this loss value is shown in Formula 3.

[0055] Among them, | | represents the number of image features that are masked; m represents the m-th image feature that is masked; Represents the true value of the masked image features; This represents the predicted value of the masked image features. By randomly masking the image sequence features in the fusion sequence features of the samples, the analytical model uses the unmasked text sequence features to predict the masked image sequence features. The predictive ability of the analytical model depends on the text sequence that coincides with the spatial location of the image, ensuring the accuracy of the alignment of text and image symbols in complex behavioral graphs.

[0056] The second loss value is calculated according to Formula 2 and Formula 3. The second loss value can be the sum of the loss values ​​calculated by Formula 2 and Formula 3.

[0057] Step 305: Based on the first loss value and the second loss value, perform the next round of training on the analytical model.

[0058] Calculate the sum of the first and second loss values ​​to determine the total loss value of the analytical model in this training round. Based on the magnitude of the total loss value and the stopping training condition, determine whether to proceed with the next round of training for the analytical model. For example, the stopping training condition could be that the total loss value is less than a first threshold, or that the number of training rounds is greater than a second threshold.

[0059] Figure 4 This is a schematic diagram of the internal modules of a behavior graph parsing device provided in an embodiment of this application. Figure 4 As shown, the device may include: an acquisition module 401, a fusion module 402, and a processing module 403. Optionally, it may also include a storage module for storing computer instructions or programs, and the processing module 403 may call the computer instructions or programs stored in the storage module.

[0060] The acquisition module 401 is used to acquire the text sequence features and image sequence features of the behavior graph; the behavior graph includes graph symbols and text with association relationships, and logical relationships between graph symbols; the fusion module 402 is used to fuse the text sequence features and the image sequence features to obtain fused sequence features; the processing module 403 is used to input the fused sequence features into a trained parsing model to obtain semantic text describing the behavior graph, the semantic text being used to describe the association relationships between graph symbols and text, and the logical relationships between graph symbols in a textual manner.

[0061] In one possible implementation, the parsing model in the behavioral graph parsing device includes an encoder and a decoder; the encoder is used to learn the logical relationships between graph symbols through the fused sequence features; the decoder is used to learn the association between graph symbols and text through mask sequence features; the mask sequence features are obtained by masking the fused sequence features.

[0062] In one possible implementation, the acquisition module 401 is further configured to determine the sample fusion sequence features of the sample image; the processing module 403 is further configured to learn the predicted logical relationship between each sample image symbol in the sample fusion sequence features through an encoder; determine a first loss value based on the predicted logical relationship and the true logical relationship between each sample image symbol in the sample image; perform random partial masking on the sample fusion sequence features to obtain sample mask sequence features; learn the sample mask sequence features through a decoder to obtain sample prediction sequence features; determine a second loss value based on the sample prediction sequence features and the sample fusion sequence features; and perform the next round of training on the analytical model based on the first loss value and the second loss value.

[0063] In one possible implementation, the sample mask sequence features include text mask sequence features and image mask sequence features; the sample prediction sequence features include sample text prediction sequence features and sample image prediction sequence features; the processing module 403 is further configured to determine a second loss value based on the loss value of the text sequence features in the sample text prediction sequence features and the loss value of the image sequence features in the sample fusion sequence features, and the loss value of the image sequence features in the sample image prediction sequence features and the sample fusion sequence features.

[0064] In one possible implementation, the processing module 403 is further configured to: perform random partial masking on the text sequence features in the sample fusion sequence features, and then fuse them with the image sequence features in the sample fusion sequence features to obtain text mask sequence features; and / or perform random partial masking on the image sequence features in the sample fusion sequence features, and then fuse them with the text sequence features in the sample fusion sequence features to obtain image mask sequence features.

[0065] In one possible implementation, the parsing model is a deep learning model for long sequence prediction; the behavior graph is a flowchart or a timing graph.

[0066] In one possible implementation, the acquisition module 401 is specifically used to: extract text information from the behavior graph using an optical character recognition (OCR) model; convert the text information into the text sequence features; traverse the behavior graph using a depth-first search (DFS) algorithm to determine the topological sequence features representing the connection relationship between the graph symbols in the behavior graph; determine the layout sequence features representing the position information of the graph symbols in the behavior graph based on the position of the graph symbols in the behavior graph; acquire the visual sequence features of the behavior graph; and combine the topological sequence features and the layout sequence features to obtain the image sequence features.

[0067] Figure 5 This is a schematic diagram of the structure of a behavior graph parsing device provided in an embodiment of this application, such as... Figure 5 As shown, it includes at least one processor 501 and a memory 502 connected to at least one processor 501. In this embodiment, the specific connection medium between the processor 501 and the memory 502 is not limited. Figure 5 Taking the connection between processor 501 and memory 502 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc.

[0068] In this embodiment of the application, the memory 502 stores instructions that can be executed by at least one processor 501. By executing the instructions stored in the memory 502, the at least one processor 501 can implement the steps of the behavior graph parsing method described above.

[0069] The processor 501 is the control center of the computer device, capable of connecting various parts of the computer device via various interfaces and lines. It performs resource configuration by running or executing instructions stored in the memory 502 and accessing data stored in the memory 502. Optionally, the processor 501 may include one or more processing units. The processor 501 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may not be integrated into the processor 501. In some embodiments, the processor 501 and the memory 502 may be implemented on the same chip; in other embodiments, they may be implemented on separate chips.

[0070] Processor 501 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0071] Memory 502, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 502 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory 502 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 502 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0072] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0073] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0074] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0075] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0076] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A behavioral graph resolution method, characterized by, The method comprises: obtaining text sequence features of a behavior graph and image sequence features of the behavior graph; the behavior graph comprises graph symbols and texts having a correlation relationship and a logical relationship between the graph symbols; fusing the text sequence features and the image sequence features to obtain fused sequence features; inputting the fused sequence features into a trained analysis model to obtain semantic texts describing the behavior graph, the semantic texts being used to describe the correlation relationship between the graph symbols and the texts and the logical relationship between the graph symbols in a text manner.

2. The method of claim 1, wherein, The analysis model comprises an encoder and a decoder; the encoder is used to learn the logical relationship between the graph symbols through the fused sequence features; the decoder is used to learn the correlation relationship between the graph symbols and the texts through a mask sequence feature; the mask sequence feature is obtained by masking the fused sequence features.

3. The method of claim 1, wherein, The analysis model is trained in the following manner: determining sample fused sequence features of sample graphs; learning, by the encoder, a predicted logical relationship between sample graph symbols in the sample fused sequence features; determining a first loss value based on the predicted logical relationship and a real logical relationship between the sample graph symbols in the sample graphs; randomly partially masking the sample fused sequence features to obtain sample mask sequence features; learning, by the decoder, the sample mask sequence features to obtain sample predicted sequence features; determining a second loss value based on the sample predicted sequence features and the sample fused sequence features; based on the first loss value and the second loss value, performing next-round training on the analysis model.

4. The method of claim 3, wherein: the sample mask sequence features comprise text mask sequence features and image mask sequence features; the sample predicted sequence features comprise sample text predicted sequence features and sample image predicted sequence features; determining a second loss value based on the sample predicted sequence features and the sample fused sequence features comprises: determining the second loss value based on a loss value of the sample text predicted sequence features and text sequence features in the sample fused sequence features and a loss value of the sample image predicted sequence features and image sequence features in the sample fused sequence features.

5. The method of claim 4, wherein, randomly partially masking the sample fused sequence features to obtain sample mask sequence features comprises: after randomly partially masking the text sequence features in the sample fused sequence features, fusing the text sequence features and image sequence features in the sample fused sequence features to obtain text mask sequence features; and / or after randomly partially masking the image sequence features in the sample fused sequence features, fusing the image sequence features and text sequence features in the sample fused sequence features to obtain image mask sequence features.

6. The method according to any one of claims 1 to 5, characterized in that, The analysis model is a deep learning model for long sequence prediction; and the behavior graph is a flowchart or a time sequence diagram.

7. The method according to any one of claims 1 to 5, characterized in that, Obtaining text sequence features of a behavior graph and image sequence features of the behavior graph comprises: extracting text information in the behavior graph by an optical character recognition (OCR) model and converting the text information into the text sequence features; The topological sequence feature representing the connection relationship of the graph symbols in the behavior graph is determined by traversing the behavior graph through a depth-first search (DFS) algorithm; The layout sequence feature representing the position information of the graph symbols in the behavior graph is determined according to the positions of the graph symbols in the behavior graph; The visual sequence feature of the behavior graph is obtained, and the image sequence feature is obtained in combination with the topological sequence feature and the layout sequence feature.

8. A behavior diagram resolution device, characterized by, Comprise: a memory for storing computer programs or instructions; a processor for calling the computer programs or instructions stored in the memory to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores instructions, and when the computer reads and executes the instructions, the computer executes the method according to any one of claims 1 to 7.

10. A computer program product, characterised in that, The computer program product stores instructions, and when the computer reads and executes the instructions, the computer executes the method according to any one of claims 1 to 7.

Images