Video behavior prediction method and system based on world knowledge model

The video behavior prediction system addresses limitations of existing methods by integrating expert and sampled trajectories to construct a world knowledge model, enhancing adaptability and accuracy in recognizing and predicting behaviors across various scenarios.

CN119068401AInactive Publication Date: 2024-12-03BI SHENGYUN (WUHAN) INFORMATION TECH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411564721.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-05
Publication Date
2024-12-03
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Current image and video recognition methods, including deep learning, traditional machine learning, and traditional image processing, struggle with large data requirements, limited generalization, poor interpretability, vulnerability to adversarial attacks, and difficulty in handling complex scenes and contextual information, necessitating extensive training for specific tasks.

Method used

A video behavior prediction method and system based on a world knowledge model that integrates expert and sampled trajectories to construct a world knowledge model (WKM), using action-state-action triples to synthesize task and state knowledge, enabling adaptive planning without fixed scenes or specific training.

Benefits of technology

Enhances the model's generalization ability, robustness, and efficiency by allowing it to understand and predict behaviors in new scenarios without prior training, reducing hallucinations and improving planning accuracy and flexibility.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068401A_ABST
    Figure CN119068401A_ABST
Patent Text Reader

Abstract

The invention discloses a video behavior prediction method based on a world knowledge model, and the method comprises the steps: detecting and tracking an object in a video through employing a target tracking algorithm, and drawing a target sampling track; converting the target sampling track into a preliminary text description, and generating a coherent time sequence text description in combination with the time sequence information of the video frame; the time sequence text description is input into a proxy model, and the proxy model obtains the prediction probability of the next action from the WKM according to the current state as query; and the agent model combines the prediction probability of the agent model and the prediction probability given by the WKM to carry out weighted fusion to determine the next action. According to the method, the WKM is constructed, and global task knowledge and local state knowledge are combined, so that the model can adapt to different tasks and scenes, the generalization ability of the model is improved, invalid behaviors and illusion behaviors of the intelligent agent are reduced, and the task execution efficiency and accuracy are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a video behavior prediction method and system based on a world knowledge model. Background Art

[0002] In the field of image and video recognition, there are currently a variety of methods and algorithms, mainly including deep learning methods, traditional machine learning methods, and traditional image processing methods. These methods have their own advantages and disadvantages, but overall there are still some obvious limitations.

[0003] Deep learning methods, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs) and their variants, generative adversarial networks (GANs), attention mechanisms, and transformers, can automatically extract image features and process time series data, but they have disadvantages such as large data requirements, limited model generalization, poor interpretability, and vulnerability to adversarial attacks. These methods usually require a lot of training on specific datasets and have limited generalization capabilities for new datasets or unknown tasks. At the same time, the internal working principles of deep learning models are difficult to explain, and adversarial samples can easily deceive the model, affecting its robustness.

[0004] Traditional machine learning methods, such as support vector machines (SVM), random forests, and gradient boosted decision trees (GBDT), do not require as much data as deep learning methods, but feature engineering is complex and requires manual feature design, which usually requires a lot of domain knowledge and experience. In addition, traditional machine learning methods are less efficient when processing large-scale image data, and their performance ceiling is also limited by manually designed features.

[0005] Traditional image processing methods, such as SIFT and HOG features combined with SVM classifiers, can handle image recognition problems to a certain extent, but they are not robust enough and have poor adaptability to scenes with illumination changes, occlusion, scale changes, etc. At the same time, the feature extraction and matching process is computationally intensive and not suitable for real-time applications. In addition, the hierarchical expression capability is limited, making it difficult to capture complex structures and contextual information in images.

[0006] In summary, each of the currently known methods is based on mathematics and machine learning, and can only capture and analyze current or past video behaviors, and requires training, learning, and adaptation for different scenarios and specific tasks. These methods have limited judgment and recognition capabilities for new scenarios and unknown tasks. Summary of the invention

[0007] The purpose of the present invention is to address the problems existing in the prior art and to provide a video behavior prediction method and system based on a world knowledge model, which can understand and predict related image and video behaviors without fixed scenes and without the need for advance training for specific tasks, thereby improving the accuracy and efficiency of recognition and enhancing the generalization ability and robustness of the model.

[0008] To achieve the above object, the technical solution adopted by the present invention is: A method for constructing a world knowledge model comprises the following steps: P1. Collect expert trajectories and sampled trajectories. Expert trajectories are action records from domain experts and are considered to be the optimal steps to complete a task. Sampled trajectories are generated by random sampling methods and may contain errors or invalid steps for comparison and training. P2. Compare and analyze expert trajectories and sample trajectories, identify key behavior patterns and rules required to complete the task, extract task knowledge; integrate all task knowledge and build a task knowledge base; P3, extract state information and behavior information from expert trajectories to form state knowledge, and express it in the form of action-state-action triples; integrate all state knowledge and build a state knowledge base; P4. Based on the task knowledge base and the state knowledge base, a world knowledge model WKM is constructed. The task knowledge and state knowledge extracted from the video are integrated with the expert trajectory and used as training data to retrain the WKM.

[0009] In step P3, state information and behavior information are extracted from the expert trajectory to form state knowledge, which is organized into the form of action-state-action triples, including: Extract state information including object position, velocity, acceleration, and environmental conditions; Extracting behavioral data includes action sequences and interactive behaviors; Extract state knowledge from state and behavior data, including the transition relationship between states, the correspondence between states and behaviors, and the impact of states on task execution; The extracted state knowledge is represented in the form of state-action-state triples.

[0010] In step P3, all state knowledge is integrated to build a state knowledge base, which specifically includes the following steps: P3.1. Traverse all expert trajectories and extract a series of action-state-action triplets from the expert trajectories, where each triple contains: State s represents the state of the world at a certain time t; The previous action a_pre is the action executed before the state s; The next action a_next is the action to be executed after state s; P3.2. Combine these triplets into a state knowledge base B, expressed as B = {(s, a_pre, a_next)}_i^|B|, where i is the index of the triple and |B| is the size of the state knowledge base.

[0011] Step P3 also includes, based on the current state and historical behavior, combining the expert's action trajectory, self-summarizing new state knowledge, and adding it to the state knowledge base B, specifically including the following steps: P3.3, predict the next action a_next_pred by comparing the current state s with similar states in the expert trajectory and analyzing the matching degree between the historical behavior sequence and the expert behavior pattern; P3.4, combine the newly identified state s_new, the previous action a_pre, and the predicted next action a_next_pred into a new action-state-action triple; P3.5. Add this new triple to the state knowledge base B and update the state knowledge base B.

[0012] A video behavior prediction method based on a world knowledge model comprises the following steps: S1, use the target tracking algorithm to detect and track objects in the video and draw the target sampling trajectory; S2, convert the target sampling trajectory in the video into a preliminary text description, and then combine the timing information of the video frame to generate a coherent temporal text description; S3, input the temporal text description into the agent model, and the agent model uses the current state s as a query to obtain the predicted probability of the next action from the world knowledge model WKM; S4. The proxy model combines its own prediction probability with the prediction probability of the next action given by WKM, performs weighted fusion, and decides the next action.

[0013] Step S1 includes the following steps: S1.1, decompose the video into individual frames; S1.2, preprocessing the video frames, including scaling, cropping, and normalization; S1.3, extract features of each frame using the pre-trained CNN model; S1.4, use the target detection algorithm to detect objects in the video and track them; S1.5. Draw the target sampling trajectory through coordinate recording.

[0014] Step S2 includes the following steps: S2.1. Generate text description of the video using Show, Attend and Tell or Transformer-based models to convert the video content into natural language descriptions. S2.2. Use RNN or Transformer to process the timing information of video frames and generate a coherent temporal text description.

[0015] Step S3 specifically includes: WKM uses the expert trajectory as the selected trajectory and the sampled trajectory as the deviated trajectory, synthesizes the task knowledge, and then queries the state knowledge base to obtain the predicted probability of the next action.

[0016] The specific process of task knowledge synthesis includes: S3.1. Input the temporal text description into the LLM model to judge the behavior type and generate preliminary task objectives by combining it with the task knowledge base.

[0017] S3.2. Call the expert trajectory corresponding to the task target, compare the target sampling trajectory with the expert trajectory called above, if they match, confirm the behavior type, if not, continue to generate new task targets in the LLM model until the task target that best matches the current sampling trajectory is found.

[0018] A video behavior prediction system based on a world knowledge model, comprising: The video processing module is used to detect and track objects in the video using a target tracking algorithm and draw target sampling trajectories; The text generation module is used to convert the target sampling trajectory in the video into a preliminary text description, and then combine the timing information of the video frames to generate a coherent temporal text description; The world knowledge model module provides the agent model with the predicted probability of the next action; The agent planning module combines its own prediction probability with the prediction probability of the next action given by WKM, performs weighted fusion, and decides the next action.

[0019] Compared with the prior art, the present invention has the following beneficial effects: 1. By building a world knowledge model (WKM) to assist agent planning, the global task knowledge and local state knowledge are combined, so that the model can adapt to different tasks and scenarios without the need for separate training for each specific scenario, thus significantly improving the generalization ability of the model; 2. The WKM model can automatically synthesize task knowledge and state knowledge from experts and sampled trajectories, and use this knowledge to guide the global and local planning of the agent, effectively reducing the invalid and hallucinatory behaviors of the agent, and improving the efficiency and accuracy of task execution; 3. By updating the state knowledge base in real time, the WKM model can reflect the information of the current world state in real time during the planning process, enabling the intelligent agent to better adapt to environmental changes and improve the flexibility and adaptability of planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 This is a flow chart of a method for constructing a world knowledge model in one embodiment of the present application; Figure 2 A schematic diagram of a flow chart of a video behavior prediction method in an embodiment of the present application; Figure 3 Schematic diagram of the process of task knowledge synthesis method in one embodiment of the present application. DETAILED DESCRIPTION

[0022] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] In the first aspect of this application, Figure 1 As shown, a method for constructing a world knowledge model is provided, comprising the following steps: P1. Collect expert trajectories and sampling trajectories. Expert trajectories are action records from domain experts and are considered to be the optimal steps to complete a task. Sampling trajectories are generated through random sampling methods and may contain errors or invalid steps for comparison and training.

[0024] For example, regarding the video sampling of falling into water, if the task is "go swimming in Donghu Lake", the expert's trajectory includes the following steps: a. Change into swimsuit; b. Wear swimming goggles; c. Adapt to the water temperature and prepare for swimming; d. Choose an area suitable for swimming and enter the water; e. Start swimming.

[0025] The sampled action trajectories include the wrong steps: a. Attempting to jump into water without changing clothes; b. Incorrect diving action; c. Wrong swimming area. The area sampled had a “No Swimming” sign.

[0026] By comparing these two trajectories, it is possible to identify which steps are necessary to successfully complete the task and which steps may lead to task failure or inefficiency. As prior knowledge, the expert trajectory can guide global planning and prevent it from falling into blind trial and error; P2. Compare and analyze expert trajectories and sampled trajectories, identify the key behavior patterns and rules required to complete the task, extract task knowledge, integrate all task knowledge, and build a task knowledge base.

[0027] Specifically, task knowledge includes the general steps and processes that need to be followed to complete the task. Through the successful steps and strategies in the expert trajectory, the key knowledge and path to complete the task are extracted. Through the wrong steps and invalid behaviors in the sampling trajectory, learn how to avoid these errors and improve the efficiency of task execution. Integrate all the extracted and organized task knowledge into a knowledge base to form a structured and queryable knowledge base system; P3, extract state information and behavior information from expert trajectories to form state knowledge, and express it in the form of action-state-action triples, integrate all task knowledge, and build a state knowledge base; P4. Based on the task knowledge base and the state knowledge base, a world knowledge model WKM is constructed. The task knowledge and state knowledge extracted from the video are integrated with the expert trajectory and used as training data to retrain the WKM.

[0028] Through the above steps, the world knowledge model WKM is constructed. As a model used to assist agent planning, it combines global task knowledge and local state knowledge. It uses global task knowledge to guide the agent's global planning, and uses local state knowledge to assist local planning to prevent hallucinations and blind trial and error.

[0029] The WKM model can automatically synthesize task knowledge and state knowledge from experts and sampled trajectories, rather than relying entirely on preset rules or training data. This ability to self-synthesize knowledge enables the WKM model to adapt to different tasks and scenarios and improve the generalization ability of the model.

[0030] In some embodiments, in step P3, state information and behavior information are extracted from the expert trajectory to form state knowledge, and represented in the form of action-state-action triples, which specifically includes the following steps: Extracting state information including object position, velocity, acceleration, environmental conditions, and other factors that may affect task execution; The extracted behavior data includes action sequences, interactive behaviors, and the results or impacts corresponding to these behaviors.

[0031] State knowledge is extracted from state and behavior data, including the conversion relationship between states, the correspondence between states and behaviors, and the impact of states on task execution.

[0032] The extracted state knowledge is represented in the form of state-action-state triples.

[0033] In some embodiments, step P3 further includes integrating all task knowledge and constructing a state knowledge base B, which specifically includes the following steps: P3.1. Traverse all expert trajectories and extract a series of action-state-action triplets from the expert action trajectories, where each triple contains: State s represents the state of the world at a certain time t; The previous action a_pre is the action executed before the state s; The next action a_next is the action to be executed after state s; P3.2. Combine these triplets into a state knowledge base B, expressed as B = {(s, a_pre, a_next)}_i^|B|, where i is the index of the triple and |B| is the size of the state knowledge base.

[0034] In some embodiments, step P3 further includes, based on the current state and historical behavior, combining the expert trajectory, self-summarizing new state knowledge, and adding it to the state knowledge base B, specifically including the following steps: P3.3, predict the next action a_next_pred by comparing the current state s with similar states in the expert trajectory and analyzing the matching degree between the historical behavior sequence and the expert behavior pattern; P3.4, combine the newly identified state s_new, the previous action a_pre, and the predicted next action a_next_pred into a new action-state-action triple; P3.5. Add this new triple to the state knowledge base B and update the state knowledge base B.

[0035] Through the above steps, the local dynamic state knowledge base is maintained, and the WKM model can update the information of the current world state in real time during the planning process, thereby effectively reducing the invalid behavior and hallucination behavior of the intelligent agent.

[0036] The second aspect of this application is as follows Figure 2 As shown, a video behavior prediction method based on a world knowledge model is provided, comprising the following steps: S1. Use the target tracking algorithm to detect and track objects in the video and draw the target sampling trajectory.

[0037] S2: Convert the target sampling trajectory in the video into a preliminary text description, and then combine it with the timing information of the video frames to generate a coherent temporal text description.

[0038] S3. Taking the temporal text description as input, the agent model obtains the predicted probability of the next action from the world knowledge model WKM based on the current state s and historical actions as queries.

[0039] S4. The proxy model combines its own prediction probability with the prediction probability of the next action given by WKM, performs weighted fusion, and decides the next action.

[0040] Traditional CV recognition technology can only understand specific scenarios and behaviors through pre-training, and lacks the ability to understand and judge complex situations in the real world. Through the above steps, the WKM model is introduced to combine global task knowledge with local state knowledge, so that the intelligent agent can think and judge like humans, improving the accuracy and efficiency of planning tasks.

[0041] In some embodiments, step S1 includes the following steps: S1.1, decompose the video into individual frames; S1.2, preprocessing the video frames, including scaling, cropping, and normalization; S1.3, extract features of each frame using the pre-trained CNN model; S1.4, use the target detection algorithm to detect objects in the video and track them; S1.5. Draw the target sampling trajectory through coordinate recording.

[0042] In some embodiments, step S2 includes the following steps: S2.1. Generate text description of the video using Show, Attend and Tell or Transformer-based models to convert the video content into natural language descriptions. S2.2. Use RNN or Transformer to process the timing information of video frames and generate a coherent temporal text description.

[0043] In some embodiments, step S3 specifically includes: WKM uses the expert trajectory as the selected trajectory and the sampled trajectory as the deviated trajectory, synthesizes the task knowledge, and then queries the state knowledge base to obtain the predicted probability of the next action.

[0044] Specifically, the examples provided below describe in detail how WKM can understand and predict human behavior through task knowledge synthesis and state knowledge summarization.

[0045] (1) The first set of video samples: Scene: By the river, there is a sign prohibiting swimming.

[0046] Behavior sequence: stop by the river, take off shirt, take off shoes.

[0047] Task knowledge synthesis: Swimming is prohibited here, and someone is swimming illegally.

[0048] Triplet judgment: ready to swim.

[0049] (2) The second set of video samples: Scene: Near the river.

[0050] Behavior sequence: Bending over and pulling repeatedly on the water surface.

[0051] Task knowledge synthesis: New task, someone is picking aquatic plants.

[0052] Triple determination: Picking water plants (based on behavior sequence).

[0053] (3) The third group of video samples: Scene: Riverside, with bamboo baskets.

[0054] Behavior sequence: Pull the bamboo basket out of the water, reach into the basket, and take out a fish.

[0055] Task knowledge synthesis: New task, someone is fishing.

[0056] Triplet determination: Ready to fish (based on behavior sequence).

[0057] After sampling each set of videos, WKM will think based on the observed behavior sequence and scene information, predict the target's behavioral intention, and synthesize the corresponding task knowledge.

[0058] WKM is also able to dynamically adjust its predictions and judgments based on newly learned knowledge and experience. For example, after sampling the third set of videos, WKM learned that "there are fish in this area, and someone has also caught fish", so when it observes a behavior sequence similar to the first set in this area in the future, it will prioritize generating fishing tasks.

[0059] This dynamic adjustment is not only based on the currently observed behavioral sequence, but also takes into account historical knowledge and experience (such as the information that "there are fish in this area").

[0060] In this process, WKM acts as an auxiliary tool to help the AI ​​agent model better understand task requirements, avoid ineffective trials and erroneous actions, and improve planning efficiency and accuracy.

[0061] In some embodiments, the specific process of task knowledge synthesis includes: S3.1. Input the time series text description into the LLM model to judge the behavior type, and combine it with the task knowledge base to generate preliminary task objectives. The LLM model is a general large language model.

[0062] S3.2. Call the expert trajectory corresponding to the task target, compare the target sampling trajectory with the expert trajectory called above, if they match, confirm the behavior type, if not, continue to generate new task targets in the LLM model until the task target that best matches the current sampling trajectory is found.

[0063] Specifically, Figure 3 As shown in Figure 1, we assume that K is a task knowledge base, where κ∈K is the task knowledge, ρtask represents the prompt indicating the task knowledge extraction, and τw, τl are the selected trajectory and the deviation trajectory, respectively. It should be noted that given the same task u, τw and τl always satisfy r (u, τw) = 1 ≥ r(u, τl). Even when r (u, τw) = r(u, τl), that is, the evaluation criteria scores of the two trajectories are the same, the sampled trajectory is still regarded as a deviation trajectory. This is because the expert's trajectory usually has a shorter step length, allowing WKM to learn more effective planning knowledge for prediction.

[0064] That is to say, when a new task u is received, the corresponding task knowledge k is first extracted from K according to the ρ task prompt. Then, the target sampling trajectory extracted from the video (i.e., τl) is compared with the expert trajectory extracted from K (i.e., τw). If the two match, i.e., r(u, τw)>r(u, τl), then the current behavior type is confirmed to be consistent with k. If the two do not match, even if r(u, τw) = r(u, τl), since the expert's trajectory has a shorter step length, τl is still regarded as a deviation trajectory, then new task knowledge is continued to be searched and extracted in K until the task target that best matches the current sampling trajectory is found.

[0065] by Figure 2 For example, assume that the current video target detection trajectory is initially judged as "swimming behavior" through LLM, the system will generate a ρ task (i.e., swimming task), which will generate an expert trajectory τl of swimming behavior. This trajectory (K) includes the action trajectory of normal behavior, such as waving arms, breathing, and position movement. The target sampling trajectory τw in the video, the trajectory (κ) includes waving arms and breathing, that is, k∈K, is judged as swimming. If the trajectory (k) is without waving arms, without breathing, and the target suddenly disappears, , continue to match τw in LLM to generate a new ρ task (diving task). If the task continues to be judged as inconsistent, continue to match it in LLM to generate a new ρ task (drowning task)... until the most consistent task is matched.

[0066] A third aspect of the present application provides a video behavior prediction system based on a world knowledge model, comprising: The video processing module is used to detect and track objects in the video using a target tracking algorithm and draw target sampling trajectories; The text generation module is used to convert the target sampling trajectory in the video into a preliminary text description, and then combine the timing information of the video frames to generate a coherent temporal text description; The world knowledge model module provides the agent model with the predicted probability of the next action; The agent planning module combines its own prediction probability with the prediction probability of the next action given by WKM, performs weighted fusion, and decides the next action.

[0067] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0068] In the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish descriptions and cannot be understood as indicating or suggesting relative importance. It should also be understood that although the terms "first", "second", etc. are used to describe various elements in some embodiments of the present application in the text, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first table can be named as the second table, and similarly, the second table can be named as the first table without departing from the scope of the various described embodiments. The first table and the second table are both tables, but they are not the same table.

[0069] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0070] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for constructing a world knowledge model, characterized in that: The following steps are involved: P1. Collect expert trajectories and sampled trajectories. Expert trajectories are action records from domain experts and are considered to be the optimal steps to complete a task. Sampled trajectories are generated by random sampling methods and may contain errors or invalid steps for comparison and training. P2. Compare and analyze expert trajectories and sample trajectories, identify key behavior patterns and rules required to complete the task, extract task knowledge; integrate all task knowledge and build a task knowledge base; P3, extract state information and behavior information from expert trajectories to form state knowledge, and express it in the form of action-state-action triples; integrate all state knowledge and build a state knowledge base; P4. Based on the task knowledge base and the state knowledge base, a world knowledge model WKM is constructed. The task knowledge and state knowledge extracted from the video are integrated with the expert trajectory and used as training data to retrain the WKM.

2. A method for constructing a world knowledge model according to claim 1, characterized in that: In step P3, state information and behavior information are extracted from the expert trajectory to form state knowledge, which is organized into the form of action-state-action triples, including the following steps: Extract state information including object position, velocity, acceleration, and environmental conditions; Extracting behavioral data includes action sequences and interactive behaviors; Extract state knowledge from state and behavior data, including the transition relationship between states, the correspondence between states and behaviors, and the impact of states on task execution; The extracted state knowledge is represented in the form of state-action-state triples.

3. The method for constructing a world knowledge model according to claim 1, characterized in that: In step P3, all state knowledge is integrated to build a state knowledge base, including the following steps: P3.

1. Traverse all expert trajectories and extract a series of action-state-action triplets from the expert trajectories, where each triple contains: State s represents the state of the world at a certain time t; The previous action a_pre is the action executed before the state s; The next action a_next is the action to be executed after state s; P3.

2. Combine these triplets into a state knowledge base B, expressed as B = {(s, a_pre, a_next)}_i^|B|, where i is the index of the triple and |B| is the size of the state knowledge base.

4. The method for constructing a world knowledge model according to claim 3, characterized in that: Step P3 also includes, based on the current state and historical behavior, combining the expert's action trajectory, self-summarizing new state knowledge, and adding it to the state knowledge base B, specifically including the following steps: P3.3, predict the next action a_next_pred by comparing the current state s with similar states in the expert trajectory and analyzing the matching degree between the historical behavior sequence and the expert behavior pattern; P3.4, combine the newly identified state s_new, the previous action a_pre, and the predicted next action a_next_pred into a new action-state-action triple; P3.

5. Add this new triple to the state knowledge base B and update the state knowledge base B.

5. A video behavior prediction method based on a world knowledge model, characterized in that: The following steps are involved: S1, use the target tracking algorithm to detect and track objects in the video and draw the target sampling trajectory; S2, convert the target sampling trajectory in the video into a preliminary text description, and then combine the timing information of the video frame to generate a coherent temporal text description; S3, input the temporal text description into the agent model, and the agent model uses the current state s as a query to obtain the predicted probability of the next action from the world knowledge model WKM; S4. The proxy model combines its own prediction probability with the prediction probability of the next action given by WKM, performs weighted fusion, and decides the next action.

6. The video behavior prediction method based on the world knowledge model according to claim 5 is characterized in that: Step S1 includes the following steps: S1.1, decompose the video into individual frames; S1.2, preprocessing the video frames, including scaling, cropping, and normalization; S1.3, extract features of each frame using the pre-trained CNN model; S1.4, use the target detection algorithm to detect objects in the video and track them; S1.

5. Draw the target sampling trajectory through coordinate recording.

7. The video behavior prediction method based on the world knowledge model according to claim 5 is characterized in that: Step S2 includes the following steps: S2.

1. Generate text description of the video using Show, Attend and Tell or Transformer-based models. S2.

2. Use RNN or Transformer to process the timing information of video frames and generate a coherent temporal text description.

8. The video behavior prediction method based on the world knowledge model according to claim 5, characterized in that: Step S3 specifically includes: WKM uses the expert trajectory as the selected trajectory and the sampled trajectory as the deviated trajectory, synthesizes the task knowledge, and then queries the state knowledge base to obtain the predicted probability of the next action.

9. The video behavior prediction method based on the world knowledge model according to claim 8, characterized in that: The specific process of task knowledge synthesis includes: S3.1, input the time series text description into the LLM model to judge the behavior type, and generate preliminary task objectives by combining with the task knowledge base; S3.

2. Call the expert trajectory corresponding to the task target, compare the target sampling trajectory with the expert trajectory called above, if they match, confirm the behavior type, if not, continue to generate new task targets in the LLM model until the task target that best matches the current sampling trajectory is found.

10. A video behavior prediction system based on a world knowledge model, characterized in that: include: The video processing module is used to detect and track objects in the video using a target tracking algorithm and draw target sampling trajectories; The text generation module is used to convert the target sampling trajectory in the video into a preliminary text description, and then combine the timing information of the video frames to generate a coherent temporal text description; The world knowledge model module provides the agent model with the predicted probability of the next action; The agent planning module combines its own prediction probability with the prediction probability of the next action given by WKM, performs weighted fusion, and decides the next action.

Citation Information

Patent Citations

  • Maneuvering decision training method based on online / offline hybrid reinforcement learning

    CN118605156A

  • Sender-receiver interface for artificial intelligence communication assistance for augmenting communications

    US10594757B1

  • Network surveillance and security system

    US20030051026A1