Motion information processing method, device, terminal, electronic device and storage medium

Abstract

The present disclosure relates to a motion information processing method, apparatus, terminal, electronic device, and storage medium. The method includes obtaining a first target text; performing semantic parsing on the first target text to obtain at least one abstract semantic representation graph; performing syntactic dependency analysis on the first target text to obtain at least one dependency tree; extracting a corresponding semantic role information set from at least one abstract semantic representation graph for any target node indicating an action in the at least one abstract semantic representation graph; extracting a corresponding word element information set from at least one dependency tree; fusing the semantic role information set and the word element information set to obtain a corresponding target semantic information set; determining a target node sequence for indicating the order in which actions of each target node occur; and obtaining a first motion information representation result based on the target node sequence and the target semantic information set corresponding to each target node. The present disclosure improves the efficiency and accuracy of motion information extraction.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a motion information processing method, device, terminal, electronic device, and storage medium. Background Art

[0002] Related technologies can rely on manual extraction of motion information from text, but this method has problems such as low efficiency, prone to errors, and poor standardization. In addition, related technologies can also extract motion information based on preset rules, but fixed rules and patterns cannot adapt to the diversity and flexibility of text expression, resulting in easy errors. Moreover, different texts have large differences in language style and the presentation of motion elements. Therefore, this further reduces the accuracy of motion information extraction based on preset rules. Therefore, related technologies currently have difficulty in achieving accurate and automated extraction of motion information from text. Summary of the Invention

[0003] The present disclosure provides a motion information processing method, device, terminal, electronic device, and storage medium to solve the problems in the related art. The technical solution of the present disclosure is as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a motion information processing method is provided, the method comprising:

[0005] Get the first target text;

[0006] Performing semantic parsing on the first target text to obtain at least one abstract semantic representation graph, wherein the abstract semantic representation graph is used to indicate semantic roles and semantic role relationships in a corresponding sentence;

[0007] Performing syntactic dependency analysis on the first target text to obtain at least one dependency tree, wherein the dependency tree is used to indicate the part of speech of each word in the corresponding sentence and the dependency relationship between the word words;

[0008] For any target node indicating an action in the at least one abstract semantic representation graph, extracting a corresponding semantic role information set from the at least one abstract semantic representation graph; extracting a corresponding word element information set from the at least one dependency tree; and fusing the semantic role information set and the word element information set to obtain a corresponding target semantic information set;

[0009] Determining a target node sequence for indicating an order in which actions of each target node occur;

[0010] Based on the target node sequence and the target semantic information set corresponding to each target node, a first motion information representation result is obtained.

[0011] In an exemplary embodiment, determining a target node sequence for indicating the order in which actions of each target node occur includes:

[0012] traversing the at least one dependency tree to identify temporal conjunctions;

[0013] Determining the order in which actions of the relevant target nodes occur based on the temporal conjunctions;

[0014] Based on the at least one abstract semantic representation graph, identifying nodes representing time roles; based on the nodes representing time roles, determining an order in which actions of related target nodes occur;

[0015] When multiple target nodes belong to the same sentence and there is no temporal conjunction between the multiple target nodes, determining the order in which actions of the multiple target nodes occur based on the text order of the multiple target nodes in the sentence;

[0016] Based on the determined action occurrence sequence, generating a directed acyclic graph, wherein the directed acyclic graph is used to describe the action occurrence sequence of different target nodes;

[0017] Based on the directed acyclic graph, the target nodes are sorted to obtain the target node sequence.

[0018] In an exemplary embodiment, sorting the target nodes based on the directed acyclic graph to obtain the target node sequence includes:

[0019] Initialize the preset sequence;

[0020] Determining the in-degree of each target node in the directed acyclic graph;

[0021] Determine a current node, add the current node to the preset sequence, remove the current node from the directed acyclic graph, and the current node is a target node with an in-degree of a preset value;

[0022] If the directed acyclic graph is not empty, updating the in-degrees corresponding to the remaining target nodes, repeating the steps of determining the current node, adding the current node to the preset sequence, and removing the current node from the directed acyclic graph;

[0023] When the directed acyclic graph is empty, the preset sequence is determined as the target node sequence.

[0024] In an exemplary embodiment, the target semantic information set includes at least one target semantic item, and the fusion of the semantic role information set and the word element information set to obtain the corresponding target semantic information set includes:

[0025] For any of the target semantic items, determining a corresponding value based on the semantic role set;

[0026] In the case where the value is empty, determining the value corresponding to the target semantic item based on the word-unit information set;

[0027] Based on each of the target semantic items and its corresponding value, the target semantic information set is obtained.

[0028] In an exemplary embodiment, the semantic role information set includes information corresponding to at least one of the following semantic roles: action subject role, action receptor role, action target role, action direction role, execution mode role, and action location role;

[0029] The word-unit information set includes information corresponding to at least one of the following word-units: subject, object, verb-related adverb, verb-related prepositional phrase, and modifier of non-temporal conjunction;

[0030] The at least one target semantic item includes at least one of the following: action type, action execution subject, action action object, action modification dimension information, action direction, and action time dimension information.

[0031] In an exemplary embodiment, the method further comprises:

[0032] Traversing each node in the at least one abstract semantic representation graph to extract candidate nodes for indicating an action;

[0033] The target node is determined among the candidate nodes based on a predefined verb set.

[0034] In an exemplary embodiment, obtaining the first target text includes:

[0035] Get the first initial text;

[0036] Coreference resolution is performed on the first initial text to obtain the first target text.

[0037] In an exemplary embodiment, the first target text indicates a predicted video description generated by a video description generation model based on a sample video, and the method further includes:

[0038] Acquire a second target text, where the second target text indicates a standard video description corresponding to the sample video;

[0039] performing motion information processing on the second target text to obtain a second motion information representation result;

[0040] Based on the difference between the first motion information representation result and the second motion information representation result, the parameters of the video description generation model are adjusted.

[0041] According to a second aspect of an embodiment of the present disclosure, a motion information processing apparatus is provided, the apparatus comprising:

[0042] A text acquisition module is configured to acquire a first target text;

[0043] a semantic parsing module configured to perform semantic parsing on the first target text to obtain at least one abstract semantic representation graph, wherein the abstract semantic representation graph is used to indicate semantic roles and semantic role relationships in a corresponding sentence;

[0044] a syntactic analysis module configured to perform syntactic dependency analysis on the first target text to obtain at least one dependency tree, wherein the dependency tree is used to indicate the part of speech of each word in the corresponding sentence and the dependency relationship between the word words;

[0045] An information fusion module is configured to extract a corresponding semantic role information set from any target node indicating an action in the at least one abstract semantic representation graph; extract a corresponding word element information set from the at least one dependency tree; and fuse the semantic role information set and the word element information set to obtain a corresponding target semantic information set;

[0046] A sequence generation module is configured to determine a target node sequence for indicating an order in which actions of each target node occur;

[0047] The motion information generating module is configured to obtain a first motion information representation result based on the target node sequence and the target semantic information set corresponding to each target node.

[0048] In an exemplary embodiment, the sequence generation module is configured to execute:

[0049] traversing the at least one dependency tree to identify temporal conjunctions;

[0050] Determining the order in which actions of the relevant target nodes occur based on the temporal conjunctions;

[0051] Based on the at least one abstract semantic representation graph, identifying nodes representing time roles; based on the nodes representing time roles, determining an order in which actions of related target nodes occur;

[0052] When multiple target nodes belong to the same sentence and there is no temporal conjunction between the multiple target nodes, determining the order in which actions of the multiple target nodes occur based on the text order of the multiple target nodes in the sentence;

[0053] Based on the determined action occurrence sequence, generating a directed acyclic graph, wherein the directed acyclic graph is used to describe the action occurrence sequence of different target nodes;

[0054] Based on the directed acyclic graph, the target nodes are sorted to obtain the target node sequence.

[0055] In an exemplary embodiment, the sequence generation module is configured to execute:

[0056] Initialize the preset sequence;

[0057] Determining the in-degree of each target node in the directed acyclic graph;

[0058] Determine a current node, add the current node to the preset sequence, remove the current node from the directed acyclic graph, and the current node is a target node with an in-degree of a preset value;

[0059] If the directed acyclic graph is not empty, updating the in-degrees corresponding to the remaining target nodes, repeating the steps of determining the current node, adding the current node to the preset sequence, and removing the current node from the directed acyclic graph;

[0060] When the directed acyclic graph is empty, the preset sequence is determined as the target node sequence.

[0061] In an exemplary embodiment, the target semantic information set includes at least one target semantic item, and the information fusion module is configured to perform:

[0062] For any of the target semantic items, determining a corresponding value based on the semantic role set;

[0063] In the case where the value is empty, determining the value corresponding to the target semantic item based on the word-unit information set;

[0064] Based on each of the target semantic items and its corresponding value, the target semantic information set is obtained.

[0065] In an exemplary embodiment, the semantic role information set includes information corresponding to at least one of the following semantic roles: action subject role, action receptor role, action target role, action direction role, execution mode role, and action location role;

[0066] The word-unit information set includes information corresponding to at least one of the following word-units: subject, object, verb-related adverb, verb-related prepositional phrase, and modifier of non-temporal conjunction;

[0067] The at least one target semantic item includes at least one of the following: action type, action execution subject, action action object, action modification dimension information, action direction, and action time dimension information.

[0068] In an exemplary embodiment, the information fusion module is configured to perform:

[0069] Traversing each node in the at least one abstract semantic representation graph to extract candidate nodes for indicating an action;

[0070] The target node is determined among the candidate nodes based on a predefined verb set.

[0071] In an exemplary embodiment, the text acquisition module is configured to execute:

[0072] Get the first initial text;

[0073] Coreference resolution is performed on the first initial text to obtain the first target text.

[0074] In an exemplary embodiment, the first target text indicates a predicted video description generated by a video description generation model based on a sample video, and the motion information generation module is configured to perform:

[0075] Acquire a second target text, where the second target text indicates a standard video description corresponding to the sample video;

[0076] performing motion information processing on the second target text to obtain a second motion information representation result;

[0077] Based on the difference between the first motion information representation result and the second motion information representation result, the parameters of the video description generation model are adjusted.

[0078] According to a fifth aspect of the embodiments of the present disclosure, there is provided an electronic device, including:

[0079] processor;

[0080] a memory for storing instructions executable by the processor;

[0081] The processor is configured to execute the instructions to implement the motion information processing method as described in any of the above embodiments.

[0082] According to a sixth aspect of an embodiment of the present disclosure, a computer storage medium is provided. When instructions in the computer storage medium are executed by a processor of an electronic device, the electronic device executes the motion information processing method described in any of the above embodiments.

[0083] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, which includes a computer program, and when the computer program is executed by a processor, it implements the motion information processing method described in any of the above embodiments.

[0084] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

[0085] The present invention provides a motion information processing method that fully mines the information in the sentences in the text from the perspectives of semantic parsing and syntactic dependency analysis, obtains an abstract semantic representation graph for representing semantic roles and the relationship between semantic roles, and obtains a syntactic dependency graph for representing the syntactic dependency relationship between words in the sentence, thereby achieving a deep understanding and accurate extraction of text information. By fusing these two parts of information, it is possible to more comprehensively grasp the relevant information of the target node corresponding to the motion element conveyed by the text. The present invention integrates the information mining results of the two dimensions of semantics and grammar to improve the accuracy of extracting the relevant information of the target node. This method has been proven in practice to significantly improve the understanding of complex structured information in the text and automatically identify the relevant information of multi-action and multi-level target nodes. Moreover, the present invention further sorts these target nodes based on the order of action occurrence, which can more clearly sort out the sequential logic of the motion process. This makes the motion information representation result obtained based on the sorting results and the relevant information of the target node comprehensively cover the motion information in the text and has the significant advantage of high accuracy. The present invention can be fully automated, thereby significantly improving the efficiency of motion information extraction.

[0086] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] The accompanying drawings herein are incorporated into the specification and constitute a part of the present disclosure, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation to the present disclosure.

[0088] Figure 1 is a flowchart of a motion information processing method according to an exemplary embodiment;

[0089] Figure 2 is a flow chart of a method for determining a target node according to an exemplary embodiment;

[0090] Figure 3 is a flow chart of a method for determining a target node sequence according to an exemplary embodiment;

[0091] Figure 4 is a flow chart of a method for generating a target node sequence according to an exemplary embodiment;

[0092] Figure 5 A method for determining a target semantic information set according to an exemplary embodiment is shown;

[0093] Figure 6 A video description generation model optimization method is shown according to an exemplary embodiment;

[0094] Figure 7 is a block diagram of a motion information processing apparatus according to an exemplary embodiment;

[0095] Figure 8 A structural block diagram of a computer device according to an exemplary embodiment is shown. Figure 1 ;

[0096] Figure 9 A structural block diagram of a computer device according to an exemplary embodiment is shown. Figure 2 . DETAILED DESCRIPTION

[0097] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0098] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0099] Figure 1It is a flowchart of a motion information processing method according to an exemplary embodiment. The motion information processing method can be applied to an electronic device, which can be implemented by a server or a terminal alone, or can be implemented by a terminal and a server in collaboration. The terminal can be, but is not limited to, a physical device such as a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart wearable device, a digital assistant, an augmented reality device, a virtual reality device, etc., and can also include software such as an application running in the physical device. The server can be, but is not limited to, an independent server, or a server cluster or distributed system composed of multiple physical servers, or one or more cloud servers that provide cloud services, cloud databases, cloud computing, cloud, cloud storage, network services, cloud communications, intermediate services, domain name services, security services, and basic cloud computing services such as big data and artificial intelligence platforms. Reference Figure 1 As shown, the method includes the following steps.

[0100] In S110 , a first target text is acquired.

[0101] This disclosure does not limit the first target text. It can be a text describing the content conveyed in a video, such as a video summary or introduction, or a text summarizing the audio content, such as a thematic summary or abstract. Furthermore, the first target text can also be a textual representation of the information conveyed by an image, such as an annotation or description of the image.

[0102] In an exemplary embodiment, obtaining the first target text includes: obtaining a first initial text; and performing coreference resolution on the first initial text to obtain the first target text.

[0103] The substantive meaning of the content of the first initial text and the aforementioned first target text can be the same, and the first target text can be obtained by performing coreference resolution on the first initial text. Coreference resolution refers to the process of identifying and associating different expressions with the same referent in a text. In the context of motion information processing, coreference resolution helps to accurately understand and integrate relevant information. For example, in a text describing a sport, athletes may be mentioned many times, sometimes with their names and sometimes with their nicknames. Through coreference resolution, it can be made clear that these different expressions all point to the same person, thereby more accurately extracting and processing motion information, providing a solid foundation for subsequent analysis and application. The present disclosure can ensure that the references of all action subjects, objects, modifiers, etc. in the first target text are clear and consistent through coreference resolution, thereby reducing the difficulty of subsequent motion information extraction and improving the accuracy of motion information extraction.

[0104] This disclosure does not limit the implementation method of coreference resolution. For example, a rule-based approach can be used to identify and associate different expressions with the same referent using preset rules. The advantages of this approach are clear rules, easy to understand and implement, and can achieve good results in specific fields or scenarios. Machine learning-based methods can also be used, but this disclosure does not limit this and does not constitute an implementation obstacle.

[0105] In S120 , semantic parsing is performed on the first target text to obtain at least one abstract semantic representation graph, where the abstract semantic representation graph is used to indicate semantic roles and semantic role relationships in a corresponding sentence.

[0106] The present disclosure does not limit the implementation method of S120. An abstract semantic representation graph corresponding to each sentence in the first target text can be obtained based on the AMR (Abstract Meaning Representation) parsing model of the related technology, or a customized semantic parsing method can be used, which does not constitute an implementation obstacle.

[0107] An AMR parsing model is a tool used to convert natural language text into a structured semantic representation. By inputting a first target text into the AMR parsing model, it triggers an in-depth semantic analysis of the first target text, outputting the corresponding AMR graph (Abstract Semantic Representation Graph).

[0108] Nodes in the AMR graph correspond to semantic roles, which can be concrete objects like "apple" and "car," or abstract concepts like "happiness" and "time." The edges in the AMR graph represent the semantic role relationships between semantic roles. These relationships describe how they relate to each other. Common relationships include agent, patient, time, and location. For example, consider a sentence in the first target text: "Xiao Ming eats apples." After processing this sentence using the AMR parsing model, the output AMR graph contains three nodes corresponding to "Xiao Ming," "eat," and "apple," respectively. Regarding the edges, there is an agent relationship between "Xiao Ming" and "eat," as "Xiao Ming is the performer of the action "eat." There is also a patient relationship between "apple" and "eat," as "apple" is the recipient of the action "eat." With "eat" as the central node, an edge extends from "Xiao Ming" to "eat," labeled "agent." Another edge extends from "eat" to "apple," labeled "patient." Such an AMR graph clearly shows the semantic structure of "Xiao Ming eats apple".

[0109] In S130 , a syntactic dependency analysis is performed on the first target text to obtain at least one dependency tree, where the dependency tree is used to indicate the part of speech of each word in the corresponding sentence and the dependency relationship between the word words.

[0110] The present disclosure does not limit the implementation of S130. A dependency tree corresponding to each sentence in the first target text can be obtained using a syntactic analysis tool (such as SpaCy) in the relevant art, or a custom syntactic dependency analysis method can be used without any implementation obstacles. The dependency tree records each token and its corresponding part of speech (POS tags), thereby indicating the main grammatical structure of the sentence.

[0111] SpaCy, a natural language processing tool, is an open-source tool widely used in the field of natural language processing. It provides a rich set of language models and efficient processing capabilities, capable of processing multiple languages. SpaCy supports common natural language processing tasks such as part-of-speech tagging, syntactic analysis, and named entity recognition, and has high processing speed and accuracy. In an exemplary embodiment, the natural language processing tool SpaCy can perform operations such as word segmentation, part-of-speech tagging, and dependency parsing on input text. Word segmentation breaks the input text into individual tokens, part-of-speech tagging labels each token with its corresponding part of speech, and dependency parsing analyzes the dependencies between words and constructs a dependency tree. For example, the first target text contains the sentence "She likes reading." After processing using SpaCy, the tokens are "she," "like," and "read." In terms of part-of-speech tagging, "she" is a pronoun, "like" is a verb, and "read" can be considered a verb phrase. In the dependency tree, "like" is the central node, "she" is the agent of "like," and a dependency relationship exists between them. "Reading" is the object of the action "like," and also has a dependency relationship with "like." A simple graphical representation of the dependency tree can be used, with "like" as the center, an edge extending from "she" to "like," labeled "agent," and an edge extending from "like" to "reading," labeled "object." This clearly displays the text's lemmas, part-of-speech tags, and dependency relationships.

[0112] In S140, for any target node for indicating an action in the at least one abstract semantic representation graph, a corresponding semantic role information set is extracted from the at least one abstract semantic representation graph; a corresponding word element information set is extracted from the at least one dependency tree; and the semantic role information set and the word element information set are fused to obtain a corresponding target semantic information set.

[0113] The present disclosure does not limit the method of fusing the semantic role information set and the word element information set to obtain the corresponding target semantic information set. For example, a rule-based fusion method can be adopted, a machine learning algorithm can be used, and a knowledge graph can be used.

[0114] The purpose of the present disclosure is to extract motion information from the first target text. Therefore, the motion information is extracted around the target node for indicating the action. The present disclosure points out that there may be three types of actions in the text, namely static actions, state actions and dynamic actions, of which only dynamic actions are related to real motion information. Real motion information refers to the information generated by the movement of the moving object in the physical space. For example, "running" is a dynamic action, which represents the movement of a character in space and is closely related to motion information; "like" is a static action, which represents a psychological action, and "happy" is a state action, neither of which directly corresponds to real motion information. By distinguishing these three actions, information related to real motion can be filtered out from the text more accurately, thereby reducing the difficulty of motion information processing and improving the accuracy of motion information processing. In order to determine the target node, please refer to Figure 2 , which shows a flow chart of a target node determination method in an exemplary embodiment. The method includes:

[0115] S210. Traverse each node in the at least one abstract semantic representation graph and extract candidate nodes for indicating actions.

[0116] The abstract semantic representation graph may include three types of nodes indicating actions, namely nodes indicating dynamic actions, nodes indicating static actions, and nodes indicating state actions. Among them, the nodes indicating dynamic actions correspond to dynamic actions related to real motion information in the text. The description of actions such as "running" and "jumping" are represented by nodes indicating dynamic actions in the abstract semantic representation graph; the nodes indicating static actions correspond to static actions such as "like" and "hate" that represent psychological actions; and the nodes indicating state actions correspond to state actions such as "happy" and "sad". By identifying these three different types of nodes indicating actions, information related to real motion can be more accurately filtered out from the abstract semantic representation graph. In an exemplary embodiment, each word can be labeled based on the PropBank standard in the aforementioned coreference resolution stage. The PropBank standard is a system for annotating the argument structure of verbs. It establishes an argument structure framework for each verb and defines in detail the various semantic roles and argument relationships that may appear in the verb. If the labeling result is a dynamic predicate, it may be a node corresponding to a dynamic action. In an exemplary embodiment, S210 may extract nodes marked as dynamic predicates as candidate nodes, thereby excluding nodes corresponding to static actions or state actions. Of course, if static actions or state actions are not clearly distinguished, nodes corresponding to dynamic actions, static actions, or state actions may all be selected as candidate nodes.

[0117] S220. Determine the target node among the candidate nodes based on a predefined verb set.

[0118] The predefined verb set may be a non-dynamic action, such as a verb set consisting of static actions and state actions. The target node may be obtained by excluding the candidate nodes hit by the verb set.

[0119] The S140 combines information from dependency trees and AMR graphs. This is because dependency trees primarily focus on the grammatical dependencies between words in a sentence, concentrating on the syntactic structure of a sentence to clarify the roles and connections of each word within the sentence's grammatical framework. AMR graphs (Abstract Meaning Representation graphs), on the other hand, focus on the semantic level of a sentence, ignoring some grammatical details and focusing on the relationships between the core semantics and concepts expressed in the sentence. They highlight semantic relationships such as causality, purpose, and subject, rather than the grammatical structure emphasized by dependency trees. The information from dependency trees and AMR graphs, each with its own emphasis, can significantly enhance the richness of motion information for target nodes, enabling full mining and in-depth integration of motion information.

[0120] In S150 , a target node sequence is determined to indicate the order in which actions of the target nodes occur.

[0121] The present disclosure does not limit the specific method for determining the target node sequence for indicating the order in which the actions of each target node occur. Exemplarily, an event-driven method or a machine learning method can be used to determine the target node sequence for indicating the order in which the actions of each target node occur. In an exemplary embodiment, the target node sequence for indicating the order in which the actions of each target node occur can be determined based on explicit time tags in the first target text, verb dependencies parsed from an abstract semantic representation graph or a dependency tree, nodes of time roles in the abstract semantic representation graph, and the implicit order of the first target text.

[0122] In S160 , a first motion information representation result is obtained based on the target node sequence and the target semantic information set corresponding to each target node.

[0123] The first motion information representation result includes not only the target semantic information set corresponding to each target node, but also the action occurrence order information of each target node, thus comprehensively covering the motion information in the text. The present disclosure fully mines the information in the sentences in the text from the two perspectives of semantic parsing and syntactic dependency analysis, and obtains an abstract semantic representation graph for representing semantic roles and the relationship between semantic roles, as well as a syntactic dependency graph for representing the syntactic dependency relationship between words in the sentence, thereby achieving a deep understanding and precise extraction of text information. By fusing these two parts of information, it is possible to more comprehensively grasp the relevant information of the target nodes corresponding to the motion elements conveyed by the text. The present disclosure integrates the information mining results of the two dimensions of semantics and grammar to improve the extraction accuracy of the relevant information of the target nodes. This method has been proven in practice to significantly improve the understanding of complex structured information in the text and automatically identify the relevant information of multi-action and multi-level target nodes.

[0124] In an exemplary embodiment, please refer to Figure 3 , which shows a flow chart of a method for determining a target node sequence according to an exemplary embodiment. The method of determining a target node sequence for indicating the order in which actions of each target node occur includes:

[0125] S310. Traverse the at least one dependency tree and identify time conjunctions; based on the time conjunctions, determine the order in which actions of related target nodes occur.

[0126] In one exemplary embodiment, the relationship between each verb can be scanned sentence by sentence to identify explicit temporal conjunctions (e.g., after, before, then, while, etc.). For dependent clauses, if the dependent clause verbs have explicit temporal conjunctions, the temporal order they represent is recorded. For parallel clauses, if there are conjunctions or adverbs that explicitly represent temporal sequence (e.g., then), the temporal order they represent is recorded, thereby determining the order in which the actions of the multiple target nodes occur.

[0127] For example, when the time conjunctions "first", "secondly", "then", etc. are identified, the order of the corresponding target node actions can be clarified according to their sequence. For example, if the word "first" in the dependency tree modifies the word "action A", expressing "first perform the operation of node A, and then carry out the activity of node B", it can be determined that the action of node A occurs before node B. The word "secondly" modifies the word "action B", and the word "then" modifies the word "action C", then it can be determined that action A occurs first, followed by action B, and finally action C. Through this recognition and analysis of time conjunctions, the order in which the actions of the target node occur can be effectively sorted out.

[0128] S320. Based on the at least one abstract semantic representation graph, identify nodes representing time roles; based on the nodes representing time roles, determine the order in which actions of related target nodes occur.

[0129] In the AMR graph, for each action node, its related time role nodes (such as after, before, then) can be analyzed to determine the order in which the actions of the related target nodes occur.

[0130] For example, when there are nodes representing time roles in the abstract semantic representation graph, such as "morning", "afternoon", "evening", "before", "after", etc., the order of the target node's actions can be clarified based on these time role nodes. For example, if the time role node "morning" is associated with "action D" and the time role node "afternoon" is associated with "action E" in the abstract semantic representation graph, then it can be determined that action D occurs before action E. For another example, if there is a representation that "action F" comes before "action G", then it can be determined that the order of action F occurs before action G. By identifying and judging these nodes representing time roles, the order in which the actions of the target node occur can be accurately determined.

[0131] S330. When multiple target nodes belong to the same sentence and there is no time conjunction among the multiple target nodes, determine the order in which the actions of the multiple target nodes occur based on the text order of the multiple target nodes in the sentence.

[0132] Furthermore, in the case where no clear time conjunction is identified, the action verbs that appear adjacently in the same sentence in the text can be further scanned, and the implicit temporal relationship is inferred by default according to the text order, and the action represented by the previous verb is set to occur before the next action, thereby determining the order in which the actions of multiple target nodes occur.

[0133] For example, in a sentence, there are two target nodes, "pick up the cup" and "drink water," without a temporal conjunction between them. According to the text order, "pick up the cup" appears first, followed by "drink water." Therefore, it can be determined that the action of "pick up the cup" occurred first, followed by the action of "drink water." In this way, even without an obvious temporal conjunction, the order of the target node actions can be accurately determined based on the text order.

[0134] S340. Based on the determined action occurrence sequence, generate a directed acyclic graph, where the directed acyclic graph is used to describe the action occurrence sequence of different target nodes.

[0135] In an exemplary embodiment, the directed acyclic graph has the following requirements:

[0136] (1) The target node is a node in the graph;

[0137] (2) The direction of the edge indicates the order of actions of the connected target nodes.

[0138] For example, suppose there are three target nodes: "open the door," "enter the room," and "turn on the light." The order of actions that have been determined is "open the door" first, then "enter the room," and finally "turn on the light." A directed acyclic graph is constructed based on these three target nodes. Each node in the graph represents "open the door," "enter the room," and "turn on the light," respectively. A directed edge is drawn from the "open the door" node to the "enter the room" node, and then a directed edge is drawn from the "enter the room" node to the "turn on the light" node. In this way, this directed acyclic graph can clearly and intuitively describe the order in which the actions of these three different target nodes occur. Due to its directed acyclic nature, there will be no circular or contradictory sequence relationships, which can provide a basis for generating a target node sequence.

[0139] S350. Based on the directed acyclic graph, sort the target nodes to obtain the target node sequence.

[0140] The present disclosure does not limit the method for sorting the target nodes based on the directed acyclic graph to obtain the target node sequence. The present disclosure is based on S310-S350, which can effectively sort and accurately sort the target node action sequence. By constructing a directed acyclic graph, the action sequence relationship between the target nodes is clearly and intuitively presented, avoiding the occurrence of circular and contradictory sequences. The target node sequence obtained on this basis is faster and more accurate.

[0141] For example, please refer to Figure 4 , which shows a flow chart of a method for generating a target node sequence in an exemplary embodiment. The method of sorting the target nodes based on the directed acyclic graph to obtain the target node sequence includes:

[0142] S410. Initialize the preset sequence.

[0143] The preset sequence is initialized as an empty sequence.

[0144] S420. Determine the in-degree of each target node in the directed acyclic graph.

[0145] The present disclosure does not limit the method for determining the in-degree of each target node in the directed acyclic graph. Exemplarily, the method for determining the in-degree of each target node in the directed acyclic graph is to traverse each directed edge in the directed acyclic graph, and for each directed edge, the in-degree of the target node it points to is increased by 1. Starting from the starting node of the graph, the edges are processed in sequence according to their directions until all edges are traversed, so that the in-degree of each target node can be counted. For example, if there is a directed edge from node A to node B, then the in-degree of node B is increased by 1. In this way, the in-degree of each target node in the directed acyclic graph can be accurately determined.

[0146] S430. Determine a current node, add the current node to the preset sequence, remove the current node from the directed acyclic graph, and the current node is a target node with an in-degree of a preset value.

[0147] Exemplarily, the preset value may be 0. A point with an in-degree of 0 may be used as the current node.

[0148] S440. When the directed acyclic graph is not empty, update the in-degree corresponding to the remaining target nodes, repeat the steps of determining the current node, adding the current node to the preset sequence, and removing the current node from the directed acyclic graph.

[0149] Of course, the method for updating the in-degrees corresponding to the remaining target nodes may be the same as that in S420 .

[0150] S450. When the directed acyclic graph is empty, determine the preset sequence as the target node sequence.

[0151] Through the target node sequence determination method proposed in the present disclosure, the nodes in the directed acyclic graph can be effectively sorted, and the target node sequence used to indicate the order in which the actions of each target node occur can be quickly and accurately obtained.

[0152] In an exemplary embodiment, the target semantic information set includes at least one target semantic item. Figure 5 , which illustrates a method for determining a target semantic information set in an exemplary embodiment. The fusion of the semantic role information set and the word element information set to obtain the corresponding target semantic information set includes:

[0153] S510. For any of the target semantic items, determine a corresponding value based on the semantic role set.

[0154] The embodiments of the present disclosure do not limit the semantic roles involved in the semantic role set and their corresponding information. The semantic role information set may include information corresponding to at least one of the following semantic roles: action subject role, action receptor role, action target role, action direction role, execution mode role, and action location role.

[0155] For example, taking the output of the semantic abstract representation graph through the AMR parsing model as an example, then:

[0156] The action subject role is the ARG0 role: usually represents the subject of the action;

[0157] The action receptor role is ARG1 role: it represents the object or recipient of the action;

[0158] The action target role is the ARG2 role, which represents the target of the action;

[0159] The execution direction role is the direction role: it indicates the direction of movement of the action;

[0160] The execution mode role is the manner role: it indicates the way the action is performed;

[0161] The location role is the location role: it represents the spatial location where the action occurs;

[0162] In addition, the aforementioned time role may also be included, which is a time role: indicating the temporal sequence relationship between actions.

[0163] Among them, ARG0 role, ARG1 role, ARG2 role, direction role, manner role, location role, and time role can be understood as information in the output result of the AMR parsing model.

[0164] The present disclosure also does not limit the target semantic item, which may include at least one of the following: action type, action execution subject, action object, action modification dimension information, action direction, and action time dimension information: among which, the action modification dimension information may be information used to record the action amplitude and other information to describe or modify the action, and the action time dimension information may be information recording the time or sequence of the action.

[0165] For any target semantic item, information corresponding to the target semantic item may be first extracted from the semantic role set as the value corresponding to the target semantic item. If the information cannot be extracted, S520 is executed.

[0166] S520. When the value is empty, determine the value corresponding to the target semantic item based on the word-unit information set.

[0167] The embodiments of this disclosure do not limit the word-grams involved in the word-gram information set and their corresponding information. The word-gram information set includes information corresponding to at least one of the following word-grams: subject, object, verb-related adverb, verb-related prepositional phrase, and modifiers of non-temporal conjunctions. Modifiers of non-temporal conjunctions are other verb-modifying components excluding temporal conjunctions.

[0168] S530. Based on each of the target semantic items and its corresponding value, obtain the target semantic information set.

[0169] Based on S510 - S530 , the semantic role set and the word element information set can be combined to flexibly determine the value corresponding to each target semantic item, thereby ensuring the integrity and accuracy of the obtained target semantic information set.

[0170] An important application scenario for the motion information processing method proposed in this disclosure is related to video description generation models. Video description (also known as video captioning) is a classic task in the fields of computer vision and multimodality. This task aims to automatically describe video content using a trained video description generation model, covering key elements such as the subject, behavior, major events, and scene. However, video description generation models can produce hallucinations, generating description information that does not match the actual video content. This hallucination phenomenon can seriously affect the accuracy and reliability of video descriptions, reducing the effectiveness of video description generation models in practical applications.

[0171] However, the related art lacks a method for quantifying hallucinations. Using an AI model to directly score hallucinations on the output of a video description generation model can introduce hallucination biases inherent in the AI ​​model, leading to inaccurate scoring results. Manual hallucination scoring also presents high costs and unstable accuracy. The lack of an effective approach to hallucination quantification in the related art makes it difficult to optimize the video description generation model to reduce hallucinations, making it difficult to reduce hallucinations in the video description generation model. Applying the motion information processing method proposed in this disclosure to scenarios related to video description generation models can address these issues.

[0172] In an exemplary embodiment, the first target text indicates a predicted video description generated by the video description generation model based on a sample video. Figure 6 , which illustrates a method for optimizing a video description generation model in an exemplary embodiment. The method includes:

[0173] S610. Obtain a second target text, where the second target text indicates a standard video description corresponding to the sample video.

[0174] Unlike the first target text, the second target text corresponds to the ground truth (standard video description) of the predicted video description generated by the video description generation model based on the sample video. For example, the standard video description can be coreference resolved to obtain the second target text.

[0175] S620. Perform motion information processing on the second target text to obtain a second motion information representation result.

[0176] The process of performing motion information processing on the second target text to obtain the second motion information representation result is based on the same inventive concept as the method of performing motion information processing on the first target text to obtain the first motion information representation result in the previous article, and will not be repeated here.

[0177] S630. Adjust parameters of the video description generation model based on the difference between the first motion information representation result and the second motion information representation result.

[0178] The difference between the first motion information characterization result and the second motion information characterization result indicates the hallucination generated by the video description generation model in the process of generating a predicted video description based on a sample video. Therefore, by determining the difference between the first motion information characterization result and the second motion information characterization result, the hallucination generated by the video description generation model can be quantified. The present disclosure does not limit the specific calculation method of the difference between the first motion information characterization result and the second motion information characterization result. Exemplarily, the difference can be calculated from two perspectives: the difference in sequence order and the difference in the target semantic item of the target node itself, and the two differences are weighted to obtain a quantified result of the difference between the first motion information characterization result and the second motion information characterization result. Of course, the present disclosure does not limit the weights. The present disclosure does not limit the quantification methods of the difference in sequence order and the difference in the target semantic item of the target node itself. Exemplarily, for the difference in sequence order, the edit distance algorithm can be used for quantification. For the difference in the target semantic item of the target node itself, the semantic similarity algorithm can be used for quantification.

[0179] The present disclosure does not limit the parameter adjustment method and the parameter adjustment stopping condition. For example, the parameter adjustment method can use the gradient descent method. The parameter adjustment stopping condition can be set to reach a preset maximum number of iterations, or the model performance indicator no longer improves after several consecutive iterations.

[0180] Based on S610-S630, the present disclosure can optimize the video description generation model, thereby reducing the hallucinations of the video description generation model, thereby achieving the purpose of automatically identifying and optimizing common hallucinations such as "increased action", "reversed timing", and "wrong direction", and supporting full-process, human-free quality control.

[0181] Furthermore, the motion information processing disclosed in this disclosure can also be applied to scenarios involving fine-grained video action annotation and high-quality video-text data quality assessment. It can parse fine-grained video descriptions to accurately identify complex and subtle actions, assigning precise and detailed labels to each action, making the annotation results more accurate and professional. In terms of high-quality video-text data quality assessment, action information processing can be performed on text descriptions to obtain corresponding motion information representation results. This analysis can be used to assess the quality of the text data, thereby screening for high-quality video-text data.

[0182] Figure 7 This is a block diagram of a motion information processing device according to an exemplary embodiment. The device includes:

[0183] The text acquisition module 710 is configured to acquire a first target text;

[0184] A semantic parsing module 720 is configured to perform semantic parsing on the first target text to obtain at least one abstract semantic representation graph, wherein the abstract semantic representation graph is used to indicate semantic roles and semantic role relationships in a corresponding sentence;

[0185] The syntactic analysis module 730 is configured to perform syntactic dependency analysis on the first target text to obtain at least one dependency tree, wherein the dependency tree is used to indicate the part of speech of each word in the corresponding sentence and the dependency relationship between the word words;

[0186] The information fusion module 740 is configured to extract, for any target node indicating an action in the at least one abstract semantic representation graph, a corresponding semantic role information set from the at least one abstract semantic representation graph; extract a corresponding word-meta information set from the at least one dependency tree; and fuse the semantic role information set with the word-meta information set to obtain a corresponding target semantic information set.

[0187] The sequence generation module 750 is configured to determine a target node sequence for indicating the order in which actions of each target node occur;

[0188] The motion information generating module 760 is configured to obtain a first motion information representation result based on the target node sequence and the target semantic information set corresponding to each target node.

[0189] In an exemplary embodiment, the sequence generation module 750 is configured to execute:

[0190] traversing the at least one dependency tree to identify temporal conjunctions;

[0191] Determining the order in which actions of the relevant target nodes occur based on the temporal conjunctions;

[0192] Based on the at least one abstract semantic representation graph, identifying nodes representing time roles; based on the nodes representing time roles, determining an order in which actions of related target nodes occur;

[0193] When multiple target nodes belong to the same sentence and there is no temporal conjunction between the multiple target nodes, determining the order in which actions of the multiple target nodes occur based on the text order of the multiple target nodes in the sentence;

[0194] Based on the determined action occurrence sequence, generating a directed acyclic graph, wherein the directed acyclic graph is used to describe the action occurrence sequence of different target nodes;

[0195] Based on the directed acyclic graph, the target nodes are sorted to obtain the target node sequence.

[0196] In an exemplary embodiment, the sequence generation module 750 is configured to execute:

[0197] Initialize the preset sequence;

[0198] Determining the in-degree of each target node in the directed acyclic graph;

[0199] Determine a current node, add the current node to the preset sequence, remove the current node from the directed acyclic graph, and the current node is a target node with an in-degree of a preset value;

[0200] If the directed acyclic graph is not empty, updating the in-degrees corresponding to the remaining target nodes, repeating the steps of determining the current node, adding the current node to the preset sequence, and removing the current node from the directed acyclic graph;

[0201] When the directed acyclic graph is empty, the preset sequence is determined as the target node sequence.

[0202] In an exemplary embodiment, the target semantic information set includes at least one target semantic item, and the information fusion module 740 is configured to perform:

[0203] For any of the target semantic items, determining a corresponding value based on the semantic role set;

[0204] In the case where the value is empty, determining the value corresponding to the target semantic item based on the word-unit information set;

[0205] Based on each of the target semantic items and its corresponding value, the target semantic information set is obtained.

[0206] In an exemplary embodiment, the semantic role information set includes information corresponding to at least one of the following semantic roles: action subject role, action receptor role, action target role, action direction role, execution mode role, and action location role;

[0207] The word-unit information set includes information corresponding to at least one of the following word-units: subject, object, verb-related adverb, verb-related prepositional phrase, and modifier of non-temporal conjunction;

[0208] The at least one target semantic item includes at least one of the following: action type, action execution subject, action action object, action modification dimension information, action direction, and action time dimension information.

[0209] In an exemplary embodiment, the party information fusion module 740 is configured to perform:

[0210] Traversing each node in the at least one abstract semantic representation graph to extract candidate nodes for indicating an action;

[0211] The target node is determined among the candidate nodes based on a predefined verb set.

[0212] In an exemplary embodiment, the text acquisition module 710 is configured to execute:

[0213] Get the first initial text;

[0214] Coreference resolution is performed on the first initial text to obtain the first target text.

[0215] In an exemplary embodiment, the first target text indicates a predicted video description generated by a video description generation model based on a sample video, and the motion information generation module 760 is configured to execute:

[0216] Acquire a second target text, where the second target text indicates a standard video description corresponding to the sample video;

[0217] performing motion information processing on the second target text to obtain a second motion information representation result;

[0218] Based on the difference between the first motion information representation result and the second motion information representation result, the parameters of the video description generation model are adjusted.

[0219] Regarding the device in the above embodiment, the specific manner of each step has been described in detail in the embodiment of the aforementioned method and will not be elaborated here.

[0220] Please refer to Figure 8 , which shows a structural block diagram of a computer device provided by an exemplary embodiment of the present disclosure Figure 1 The computer device may be a terminal. The computer device is used to implement the motion information processing method provided in the above embodiment. Specifically:

[0221] Typically, the computer device 800 includes a processor 801 and a memory 802 .

[0222] Processor 801 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 801 may be implemented in hardware using at least one of a DSP (Digital Signal Processing), an FPGA (Field Programmable Gate Array), and a PLA (Programmable Logic Array). Processor 801 may also include a main processor and a coprocessor. The main processor is used to process data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state. In one exemplary embodiment, processor 801 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing content required for display. In one exemplary embodiment, processor 801 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0223] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include high-speed random access memory and non-volatile memory, such as one or more magnetic disk storage devices or flash memory storage devices. In an exemplary embodiment, the non-transitory computer-readable storage medium in the memory 802 is used to store at least one instruction, at least one program, code set, or instruction set, and is configured to be executed by one or more processors to implement the above-mentioned motion information processing method.

[0224] In an exemplary embodiment, computer device 800 may optionally include a peripheral device interface 803 and at least one peripheral device. Processor 801, memory 802, and peripheral device interface 803 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 803 via a bus, signal lines, or circuit boards. Specifically, the peripheral device includes at least one of a radio frequency circuit 804, a touchscreen display 805, a camera assembly 806, an audio circuit 807, a positioning assembly 808, and a power supply 809.

[0225] Those skilled in the art will understand that Figure 8The structure shown in the figure does not constitute a limitation on the computer device 800, and the computer device 800 may include more or fewer components than shown in the figure, or combine some components, or adopt a different arrangement of components.

[0226] Please refer to Figure 9 It shows a structural block diagram of a computer device provided by another exemplary embodiment of the present disclosure. Figure 2 The computer device may be a server for executing the above-mentioned motion information processing method. Specifically:

[0227] Computer device 900 includes a central processing unit (CPU) 901, a system memory 904 including random access memory (RAM) 902 and read-only memory (ROM) 903, and a system bus 905 connecting system memory 904 and CPU 901. Computer device 900 also includes a basic input / output system (I / O) 906 that facilitates information transfer between various components within the computer, and a mass storage device 907 for storing an operating system 913, application programs 914, and other program modules 911.

[0228] The basic input / output system 906 includes a display 908 for displaying information and an input device 909, such as a mouse and keyboard, for user input. Both the display 908 and the input device 909 are connected to the central processing unit 901 via an input / output controller 910 connected to the system bus 905. The basic input / output system 906 may also include an input / output controller 910 for receiving and processing input from a variety of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 910 also provides output to a display screen, printer, or other types of output devices.

[0229] Mass storage device 907 is connected to central processing unit 901 via a mass storage controller (not shown) connected to system bus 905. Mass storage device 907 and its associated computer-readable media provide non-volatile storage for computer device 900. In other words, mass storage device 907 may include computer-readable media (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0230] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, magnetic cassettes, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media is not limited to the aforementioned types. The aforementioned system memory 904 and mass storage device 907 may be collectively referred to as memory.

[0231] According to various embodiments of the present disclosure, the computer device 900 may also be connected to a remote computer on a network such as the Internet for operation. That is, the computer device 900 may be connected to a network 912 via a network interface unit 911 connected to the system bus 905. Alternatively, the network interface unit 911 may be used to connect to other types of networks or remote computer systems (not shown).

[0232] The memory further includes a computer program, which is stored in the memory and configured to be executed by one or more processors to implement the motion information processing method.

[0233] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one instruction, at least one program, a code set or an instruction set is stored. When the at least one instruction, the at least one program, the code set or the instruction set is executed by a processor, the motion information processing method is implemented.

[0234] Optionally, the computer-readable storage medium may include: ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drives), or an optical disk. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0235] In an exemplary embodiment, a computer-readable storage medium including program code is also provided, such as a memory including the program code. The program code can be executed by a processor to perform the motion information processing method. Alternatively, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0236] In an exemplary embodiment, a computer program product is further provided, including a computer program, which implements the above-mentioned motion information processing method when executed by a processor.

[0237] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0238] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A motion information processing method, characterized in that: The method comprises: Get the first target text; Performing semantic parsing on the first target text to obtain at least one abstract semantic representation graph, wherein the abstract semantic representation graph is used to indicate semantic roles and semantic role relationships in a corresponding sentence; Performing syntactic dependency analysis on the first target text to obtain at least one dependency tree, wherein the dependency tree is used to indicate the part of speech of each word in the corresponding sentence and the dependency relationship between the word words; For any target node indicating an action in the at least one abstract semantic representation graph, extracting a corresponding semantic role information set from the at least one abstract semantic representation graph; extracting a corresponding word element information set from the at least one dependency tree; and fusing the semantic role information set and the word element information set to obtain a corresponding target semantic information set; Traversing the at least one dependency tree to identify a temporal conjunction; and determining an order in which actions of the relevant target nodes occur based on the temporal conjunction; Based on the at least one abstract semantic representation graph, identifying nodes representing time roles; based on the nodes representing time roles, determining an order in which actions of related target nodes occur; When multiple target nodes belong to the same sentence and there is no temporal conjunction between the multiple target nodes, determining the order in which actions of the multiple target nodes occur based on the text order of the multiple target nodes in the sentence; Based on the determined action occurrence sequence, a directed acyclic graph is generated, wherein the directed acyclic graph is used to describe the action occurrence sequence of different target nodes; based on the directed acyclic graph, each of the target nodes is sorted to obtain a target node sequence; Based on the target node sequence and the target semantic information set corresponding to each target node, a first motion information representation result is obtained.

2. The method according to claim 1, characterized in that The step of sorting the target nodes based on the directed acyclic graph to obtain the target node sequence includes: Initialize the preset sequence; Determining the in-degree of each target node in the directed acyclic graph; Determine a current node, add the current node to the preset sequence, remove the current node from the directed acyclic graph, and the current node is a target node with an in-degree of a preset value; If the directed acyclic graph is not empty, updating the in-degrees corresponding to the remaining target nodes, repeating the steps of determining the current node, adding the current node to the preset sequence, and removing the current node from the directed acyclic graph; When the directed acyclic graph is empty, the preset sequence is determined as the target node sequence.

3. The method according to claim 1 or 2, characterized in that The target semantic information set includes at least one target semantic item, and the fusion of the semantic role information set and the word element information set to obtain the corresponding target semantic information set includes: For any of the target semantic items, determining a corresponding value based on the semantic role set; In the case where the value is empty, determining the value corresponding to the target semantic item based on the word-unit information set; Based on each of the target semantic items and its corresponding value, the target semantic information set is obtained.

4. The method according to claim 3, characterized in that The semantic role information set includes information corresponding to at least one of the following semantic roles: action subject role, action receptor role, action target role, action direction role, execution mode role, and action location role; The word-unit information set includes information corresponding to at least one of the following word-units: subject, object, verb-related adverb, verb-related prepositional phrase, and modifier of non-temporal conjunction; The at least one target semantic item includes at least one of the following: action type, action execution subject, action action object, action modification dimension information, action direction, and action time dimension information.

5. The method according to claim 1, wherein The method further comprises: Traversing each node in the at least one abstract semantic representation graph to extract candidate nodes for indicating an action; The target node is determined among the candidate nodes based on a predefined verb set.

6. The method according to claim 1, characterized in that The obtaining of the first target text includes: Get the first initial text; Coreference resolution is performed on the first initial text to obtain the first target text.

7. The method according to claim 1, characterized in that The first target text indicates a predicted video description generated by a video description generation model based on a sample video, and the method further includes: Acquire a second target text, where the second target text indicates a standard video description corresponding to the sample video; performing motion information processing on the second target text to obtain a second motion information representation result; Based on the difference between the first motion information representation result and the second motion information representation result, the parameters of the video description generation model are adjusted.

8. A motion information processing device, characterized in that: The device comprises: A text acquisition module is configured to acquire a first target text; a semantic parsing module configured to perform semantic parsing on the first target text to obtain at least one abstract semantic representation graph, wherein the abstract semantic representation graph is used to indicate semantic roles and semantic role relationships in a corresponding sentence; a syntactic analysis module configured to perform syntactic dependency analysis on the first target text to obtain at least one dependency tree, wherein the dependency tree is used to indicate the part of speech of each word in the corresponding sentence and the dependency relationship between the word words; An information fusion module is configured to extract a corresponding semantic role information set from any target node indicating an action in the at least one abstract semantic representation graph; extract a corresponding word element information set from the at least one dependency tree; and fuse the semantic role information set and the word element information set to obtain a corresponding target semantic information set; The sequence generation module is configured to traverse the at least one dependency tree and identify time conjunctions; determine the order of action occurrence of related target nodes based on the time conjunctions; identify nodes representing time roles based on the at least one abstract semantic representation graph; determine the order of action occurrence of related target nodes based on the nodes representing time roles; when multiple target nodes belong to the same sentence and there is no time conjunction among the multiple target nodes, determine the order of action occurrence of the multiple target nodes based on the text order of the multiple target nodes in the sentence; generate a directed acyclic graph based on the determined order of action occurrence, the directed acyclic graph being used to describe the order of action occurrence of different target nodes; and sort the target nodes based on the directed acyclic graph to obtain a target node sequence; The motion information generating module is configured to obtain a first motion information representation result based on the target node sequence and the target semantic information set corresponding to each target node.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the motion information processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device executes the motion information processing method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The computer program product includes a computer program stored in a readable storage medium. At least one processor of a computer device reads and executes the computer program from the readable storage medium, so that the device performs the motion information processing method according to any one of claims 1 to 7.

Images