Target intention reasoning method and system based on context awareness
Through dynamic graph neural network and video query language combined with GPT's common sense context, the problem of integrating common sense knowledge and context perception in the remote sensing field target intention reasoning is solved, which improves the accuracy and robustness of intention reasoning, and enhances the complex scenarios Understanding ability.
Patent Information
- Application Number
- CN202411231032.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-03
- Publication Date
- 2024-11-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing target intention reasoning algorithms in the field of remote sensing are difficult to effectively integrate common sense knowledge and context perception, resulting in inaccurate interpretation of actions and intentions, especially in understanding complex latent variables and spatial relationships.
Dynamic graph neural network is used for visual encoding and language encoding, and video query language (VQL) method is introduced to obtain the situational context in the video, and the common sense context is obtained by combining GPT to predict the final result, thereby improving the accuracy of intention reasoning and Lu Great.
It significantly improves the accuracy and robustness of visual target intention reasoning in the remote sensing field, can better understand the action intentions in different scenarios, enhances attention to the video context, provides detailed text descriptions and analysis capabilities in different scenarios .
Smart Images

Figure CN118917420A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a target intention reasoning method and system based on context perception. Background Art
[0002] Remote sensing imaging refers to the technology of obtaining information about the earth's surface and its environment in a non-contact manner. This technology uses electromagnetic radiation or other types of information collected from remote sensors to generate images or data of the ground and atmosphere. Remote sensing imaging is widely used in urban planning, disaster prevention and mitigation, weather forecasting, military reconnaissance and other fields.
[0003] Target intention reasoning is a very important research task in the field of artificial intelligence and computer vision. By analyzing the object behavior and scene information in visual data (such as videos or images), it aims to understand and predict the intention and purpose of the target object. For example, in the security monitoring system of public places, target intention reasoning can detect whether pedestrians have abnormal behavior in advance and prevent potential security threats.
[0004] In the recent boom in cross-modal visual language understanding research, context-aware target intent reasoning is one of the most prominent research, which has the ability to understand and communicate dynamic visual scenes through natural language. Target intent reasoning technology is of great benefit to several applications such as autonomous driving, security monitoring, and robotics, so it has become one of the hot research directions in the field of computer vision.
[0005] In recent years, researchers have proposed many neural network-based algorithms for the problem of target intent reasoning in the field of remote sensing, and the performance of these algorithms is very good. However, visual target intent reasoning is still a challenging task because it requires the model to fully understand the video to correctly answer questions, which include not only factual questions but also reasoning questions. Reasoning questions require logical reasoning about latent variables beyond the observed visual facts (such as the spatial relationship between entities, the environment they are in, etc.). Therefore, context-aware target intent reasoning algorithms for remote sensing, which enable the algorithms to better understand complex latent variables, are a challenging and urgent research direction.
[0006] Although existing methods have excellent target intention reasoning capabilities, there are still three difficulties and challenges: (1) Effective integration of common sense knowledge is crucial for accurate intention reasoning. Current models often have difficulty in effectively integrating this knowledge, resulting in inaccurate interpretation of actions and intentions; (2) The biggest challenge in video target intention reasoning is context, because intention understanding is very sensitive to context. Humans can interpret different intentions behind the same action "placing a package" based on different video backgrounds and common sense knowledge. If the scenario is "station or airport", the intention is more likely to be "maybe a bomb, there is a security threat"; if the scenario is "shopping mall entrance", the intention is more likely to be "delivering a package"; if the scenario is "trash can", the intention is more likely to be "discarding trash". Uncertainty does not come from the placed package, but from the environment, which is the key to context-aware target intention reasoning; (3) Intention reasoning involves understanding and inferring latent variables, such as the target environment and the spatial relationship between objects, which is more complex and challenging for models.
[0007] Therefore, proposing a context-aware target intent reasoning method and system is an urgent problem that technical personnel in this field need to solve. Summary of the invention
[0008] The present invention proposes a method and system for reasoning target intent based on context perception, which uses a dynamic graph neural network for visual encoding and a language encoder for language encoding to obtain visual features and language features; introduces a video query language (VQL) method for visual-language interaction to obtain the situational context related to the question from the video; uses GPT to obtain common sense context, and combines the model prediction distribution based on situational context and contrast context to obtain the final result, so as to improve the accuracy and robustness of visual target intent reasoning in the field of remote sensing. The method provided by the invention can effectively solve the problems raised in the above background technology.
[0009] In order to achieve the above object, the present invention adopts the following technical solution:
[0010] A context-aware target intention reasoning method includes the following steps:
[0011] S1, collect original remote sensing video;
[0012] S2, use dynamic graph neural network to visually encode the remote sensing video in S1 to obtain visual features; perform language encoding on the question and answer set to obtain language features;
[0013] S3, based on the visual features and language features obtained in S2, a video query language method is used to perform visual-language interaction to obtain the situational context related to the question from the remote sensing video;
[0014] S4. Select positive and negative examples based on the similarity between the actions and answers of the anchor sample and other samples, extract features respectively, perform feature alignment, calculate the triplet loss, and obtain a comparative context model.
[0015] S5. Input training data, calculate the loss value, and optimize the comparative context model according to the loss value to obtain an optimized comparative context model;
[0016] S6. Use GPT to obtain common sense context based on the question and answer set, and combine the obtained common sense context with the optimized comparative context model to obtain the final intention reasoning prediction result.
[0017] In the above method, optionally, the visual feature acquisition in S2 includes:
[0018] Define V f is a continuous frame of a remote sensing video, V r For the objects in each frame, given the remote sensing video frame feature V f and remote sensing video regional features V r , using N dynamic graph Transformers to transform the remote sensing video region features V f Modeling is performed to form a remote sensing video area map G r ; Then with the remote sensing video frame feature V f Connect to get the visual features, that is, the frame area graph G f,r :
[0019] G f,r =Concat(V f ,G r )
[0020] Among them, Concat(·) is the concatenation function.
[0021] In the above method, optionally, the language feature acquisition in S2 includes:
[0022] First, concatenate the given question q and the answer set A, and use Bert to extract the language feature F q,A :
[0023] F q,A =Bert(Concat(q,A))
[0024] Among them, Bert(·) is a language feature extraction module.
[0025] In the above method, optionally, the specific contents in S3 include:
[0026] S301, calculating remote sensing video area map G r and language feature F q,AThe similarity matrix S r|q,A :
[0027] S r|q,A =G r (F q,A ) T
[0028] in,(·) T is the transpose of the matrix;
[0029] S302, multiply the similarity matrix by the language feature, transform the language feature into the visual feature space; then multiply it with the remote sensing video area map G r Fusion, to obtain the situational context related to the problem, that is, the cross-modal graph G r|q,A :
[0030] G r|q,A =G r +S r|q,A F q,A .
[0031] S303, use the multi-head self-attention function to convert the frame area graph G f,r and the cross-modal graph G r|q,A Perform feature fusion to obtain the synthetic feature G f,r|q,A :
[0032] G f,r|q,A =MHSA(G r +S r|q,A F q,A )
[0033] Where MHSA(·) is the multi-head self-attention function.
[0034] In the above method, optionally, the specific steps in S4 include:
[0035] S401. Compare the similarity of the answers of two samples through WUPS score:
[0036]
[0037] When two question-answer samples X and Y are in the correct answer When the WUPS score between them is greater than or equal to the threshold t1, X and Y are positive samples of each other; if the WUPS score is lower than the threshold t2, X and Y are negative samples of each other;
[0038] S402. Collect positive samples and negative samples for each question-answering data to allow the anchor sample to randomly select a positive sample and a negative sample to form a triplet as input.
[0039] S403, according to the similarity matrix S r|q,A , from the cross-modal graph Gr|q,A Select the top k nodes most relevant to the question and answer set to extract the cross-modal feature F r|q,A :
[0040] F r|q,A =top-k(G r|q,A )
[0041] Among them, top-k(·) represents the selection of the first k maximum values;
[0042] Repeat the operation for the three samples in the triplet to obtain
[0043] in, They are the cross-modal features of anchor samples, positive samples, and negative samples. a, p, and n represent anchor samples, positive samples, and negative samples, respectively;
[0044] Align the features of negative samples separately And the characteristics of the positive sample To the anchor sample:
[0045]
[0046] in, The features of the negative samples are aligned to the features of the anchor samples. Align the features of the positive sample to the features of the anchor sample;
[0047] The distances d(a,p) and d(a,n) between the anchor sample and the positive / negative samples are calculated as:
[0048]
[0049] The triplet loss is as follows:
[0050] L triplet =max(d(a,p)-d(a,n)+margin,0)
[0051] Among them, max(·,·) represents the maximum value between the two; margin is a hyperparameter set artificially;
[0052] For each sample in a triplet, the cross entropy loss is:
[0053]
[0054] Among them, ∑(·) is the summation operation, y i is the true matching score of the i-th sample, S i Predict the matching score for the model of the i-th sample;
[0055] The formula for calculating the matching score S is as follows:
[0056] S=G f,r|q,A (F q,A ) T .
[0057] In the above method, optionally, in S5, the loss function mentioned in S4 is used to apply the triplet loss L to the anchor sample, the positive sample and the negative sample. triplet and the cross entropy loss L ce Two loss functions together constitute the complete loss function:
[0058]
[0059] in, is the cross loss entropy function of the benchmark sample, is the cross loss entropy function of the positive sample, is the cross loss entropy function of negative samples.
[0060] In the above method, optionally, in S6, the question and answer set A are used as the input of GPT, and the most likely answer is selected from the answer set according to the question to obtain the confidence distribution S of the answer set of the question. gpt , and then use a coefficient λ to combine the two confidence distributions to obtain the joint distribution S of the matching scores of the answer set A joint :
[0061] S joint =S+λS gpt
[0062] Finally, the candidate answer with the highest confidence is returned as the final predicted answer α * :
[0063]
[0064] in, An operation to select the maximum value in a collection.
[0065] A context-aware target intention reasoning system, executing any of the context-aware target intention reasoning methods described above,
[0066] It includes a feature encoder, a video query language module, a comparative context module, and a common sense context module connected in sequence;
[0067] Feature encoder, used to provide high-dimensional features of original remote sensing video and text, and realize digital encoding of original data;
[0068] Video query language module, used to obtain video context information related to the question;
[0069] The contrastive context module is used to preliminarily infer the situational context information;
[0070] The common sense context module is used for common sense reasoning and combines the preliminary reasoning results with the common sense reasoning to obtain the final reasoning results.
[0071] It can be seen from the above technical solutions that, compared with the prior art, the present invention provides a method and system for reasoning target intention based on context perception, which has the following beneficial effects:
[0072] Compared with the existing video intent reasoning technology, the context-aware video intent reasoning provided by the present invention: (1) adopts the mentioned video query language module to obtain the visual context related to the question from the video, which effectively improves the attention to the correct context in the video, thereby significantly increasing the correctness of visual question answering; (2) uses the mentioned common sense context module, through its powerful corpus and text analysis and reasoning capabilities, and after combining with the basic module, makes the answer text description very detailed; (3) introduces a comparative learning module, through the detailed analysis of different scenes, to ensure different answers to the same action in different scenes. BRIEF DESCRIPTION OF THE DRAWINGS
[0073] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0074] Figure 1 A schematic diagram of a context-aware target intention reasoning method disclosed in the present invention;
[0075] Figure 2 This is a structural diagram of the context-aware target intent reasoning model disclosed in the present invention. DETAILED DESCRIPTION
[0076] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0077] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0078] Reference Figure 1-2 As shown, a context-aware target intention reasoning method includes the following steps:
[0079] S1, collect original remote sensing video;
[0080] S2, use dynamic graph neural network to visually encode the remote sensing video in S1 to obtain visual features; perform language encoding on the question and answer set to obtain language features;
[0081] S3, based on the visual features and language features obtained in S2, a video query language method is used to perform visual-language interaction to obtain the situational context related to the question from the remote sensing video;
[0082] S4. Select positive and negative examples based on the similarity between the actions and answers of the anchor sample and other samples, extract features respectively, perform feature alignment, calculate the triplet loss, and obtain a comparative context model.
[0083] S5. Input training data, calculate the loss value, and optimize the comparative context model according to the loss value to obtain an optimized comparative context model;
[0084] S6. Use GPT to obtain common sense context based on the question and answer set, and combine the obtained common sense context with the optimized comparative context model to obtain the final intention reasoning prediction result.
[0085] Furthermore, the visual feature acquisition in S2 includes:
[0086] Define V f is a continuous frame of a remote sensing video, V r For the objects in each frame, given the remote sensing video frame feature V f and remote sensing video regional features V r , using N dynamic graph Transformers to transform the remote sensing video region features V fModeling is performed to form a remote sensing video area map G r ; Then with the remote sensing video frame feature V f Connect to get the visual features, that is, the frame area graph G f,r :
[0087] G f,r =Concat(V f ,G r )
[0088] Among them, Concat(·) is the concatenation function.
[0089] Furthermore, the language feature acquisition in S2 includes:
[0090] First, concatenate the given question q and the answer set A, and use Bert to extract the language feature F q,A :
[0091] F q,A =Bert(Concat(q,A))
[0092] Among them, Bert(·) is a language feature extraction module.
[0093] Furthermore, the specific contents of S3 include:
[0094] S301, calculating remote sensing video area map G r and language feature F q,A The similarity matrix S r|q,A :
[0095] S r|q,A =G r (F q,A ) T
[0096] in,(·) T is the transpose of the matrix;
[0097] S302, multiply the similarity matrix by the language feature, transform the language feature into the visual feature space; then multiply it with the remote sensing video area map G r Fusion, to obtain the situational context related to the problem, that is, the cross-modal graph G r|q,A :
[0098] G r|q,A =G r +S r|q,A F q,A .
[0099] S303, use the multi-head self-attention function to convert the frame area graph G f,r and the cross-modal graph G r|q,A Perform feature fusion to obtain the synthetic feature Gf,r|q,A :
[0100] G f,r|q,A =MHSA(G r +S r|q,A F q,A )
[0101] Where MHSA(·) is the multi-head self-attention function.
[0102] Furthermore, the specific steps in S4 include:
[0103] S401. Compare the similarity of the answers of two samples through WUPS score:
[0104]
[0105] When two question-answer samples X and Y are in the correct answer When the WUPS score between them is greater than or equal to the threshold t1, X and Y are positive samples of each other; if the WUPS score is lower than the threshold t2, X and Y are negative samples of each other;
[0106] S402. Collect positive samples and negative samples for each question-answering data to allow the anchor sample to randomly select a positive sample and a negative sample to form a triplet as input.
[0107] S403, according to the similarity matrix S r|q,A , from the cross-modal graph G r|q,A Select the top k nodes most relevant to the question and answer set to extract the cross-modal feature F r|q,A :
[0108] F r|q,A =top-k(G r|q,A )
[0109] Among them, top-k(·) represents the selection of the first k maximum values;
[0110] Repeat the operation for the three samples in the triplet to obtain
[0111] in, They are the cross-modal features of anchor samples, positive samples, and negative samples. a, p, and n represent anchor samples, positive samples, and negative samples, respectively;
[0112] Align the features of negative samples separately And the characteristics of the positive sample To the anchor sample:
[0113]
[0114] in, The features of the negative samples are aligned to the features of the anchor samples. Align the features of the positive sample to the features of the anchor sample;
[0115] The distances d(a,p) and d(a,n) between the anchor sample and the positive / negative samples are calculated as:
[0116]
[0117] The triplet loss is as follows:
[0118] L triplet =max(d(a,p)-d(a,n)+margin,0)
[0119] Among them, max(·,·) represents the maximum value between the two; margin is a hyperparameter set artificially;
[0120] For each sample in a triplet, the cross entropy loss is:
[0121]
[0122] Among them, ∑(·) is the summation operation, y i is the true matching score of the i-th sample, S i Predict the matching score for the model of the i-th sample;
[0123] The formula for calculating the matching score S is as follows:
[0124] S=G f,r|q,A (F q,A ) T .
[0125] Furthermore, in S5, the loss function mentioned in S4 is used to apply the triplet loss L to the anchor sample, positive sample and negative sample. triplet and the cross entropy loss L ce Two loss functions together constitute the complete loss function:
[0126]
[0127] in, is the cross loss entropy function of the benchmark sample, is the cross loss entropy function of the positive sample, is the cross loss entropy function of negative samples.
[0128] Furthermore, in S6, the question and answer set A are used as the input of GPT, and the most likely answer is selected from the answer set according to the question, and the confidence distribution S of the answer set of the question is obtained. gpt, and then use a coefficient λ to combine the two confidence distributions to obtain the joint distribution S of the matching scores of the answer set A joint :
[0129] S joint =S+λS gpt
[0130] Finally, the candidate answer with the highest confidence is returned as the final predicted answer α * :
[0131]
[0132] in, An operation to select the maximum value in a collection.
[0133] and Figure 1 Correspondingly, this embodiment further discloses a context-aware target intention reasoning system, which executes any of the context-aware target intention reasoning methods described above.
[0134] It includes a feature encoder, a video query language module, a comparative context module, and a common sense context module connected in sequence;
[0135] Feature encoder, used to provide high-dimensional features of original remote sensing video and text, and realize digital encoding of original data;
[0136] Video query language module, used to obtain video context information related to the question;
[0137] The contrastive context module is used to preliminarily infer the situational context information;
[0138] The common sense context module is used for common sense reasoning and combines the preliminary reasoning results with the common sense reasoning to obtain the final reasoning results.
[0139] Specifically, the provided context-aware target intent reasoning method was trained using the PyTorch programming framework, and the training data used the IntentQA dataset. The training process was carried out on a server equipped with 4 NVIDIA RTX 3090 GPU graphics cards. The inventors trained the proposed method using the AdamW optimizer, with the learning rate set to 5e-4 and the batch size set to 8. The WUPS score positive sample boundary t1 and negative sample boundary t2 in the contrastive learning were set to 0.912 and 0.478, respectively, and the Top-k of the scenario context was set to 3. After 20,000 iterations, the model finally completed the training.
[0140] The inventors compared the provided context-aware target intent reasoning method with the benchmark models of various typical VideoQA methods that have been published. The specific experimental results are as follows:
[0141] Table 1. Comparative experimental results
[0142]
[0143] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0144] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A context-aware target intention reasoning method, characterized by: The following steps are involved: S1, collect original remote sensing video; S2, use dynamic graph neural network to visually encode the remote sensing video in S1 to obtain visual features; Perform language encoding on the question and answer sets to obtain language features; S3, based on the visual features and language features obtained in S2, the video query language method is used to perform visual-language interaction, obtain the situation context related to the question from the remote sensing video, and perform feature fusion to obtain synthetic features; S4. Select positive and negative examples based on the similarity between the actions and answers of the anchor sample and other samples, extract features respectively, perform feature alignment, calculate the triplet loss, and obtain a comparative context model. S5. Input training data, calculate the loss value, and optimize the comparative context model according to the loss value to obtain an optimized comparative context model; S6. Use GPT to obtain common sense context based on the question and answer set, and combine the obtained common sense context with the optimized comparative context model to obtain the final intention reasoning prediction result.
2. The context-aware target intention reasoning method according to claim 1, characterized in that: Visual feature acquisition in S2 includes: Define V f is a continuous frame of a remote sensing video, V r For the objects in each frame, given the remote sensing video frame feature V f and remote sensing video regional features V r , using N dynamic graph Transformers to transform the remote sensing video region features V f Modeling is performed to form a remote sensing video area map G r ; Then with the remote sensing video frame feature V f Connect to get the visual features, that is, the frame area graph G f,r : G f,r =Concat(V f ,G r ) Among them, Concat(·) is the concatenation function.
3. The context-aware target intention reasoning method according to claim 2 is characterized in that: The language feature acquisition in S2 includes: First, concatenate the given question q and the answer set A, and use Bert to extract the language feature F q,A : F q,A =Bert(Concat(q,A)) Among them, Bert(·) is a language feature extraction module.
4. The context-aware target intention reasoning method according to claim 1, characterized in that: The specific contents of S3 include: S301, calculating remote sensing video area map G r and language feature F q,A The similarity matrix S r|q,A : S r|q,A =G r (F q,A ) T in,(·) T is the transpose of the matrix; S302, multiply the similarity matrix by the language feature, transform the language feature into the visual feature space; then multiply it with the remote sensing video area map G r Fusion, to obtain the situational context related to the problem, that is, the cross-modal graph G r|q,A : G r|q,A =G r +S r|q,A F q,A S303, use the multi-head self-attention function to convert the frame area graph G f,r and the cross-modal graph G r|q,A Perform feature fusion to obtain the synthetic feature G f,r|q,A : G f,r|q,A =MHSA(G r +S r|q,A F q,A ) Where MHSA(·) is the multi-head self-attention function.
5. The context-aware target intention reasoning method according to claim 1, characterized in that: The specific steps in S4 include: S401. Compare the similarity of the answers of two samples through WUPS score: When two question-answer samples X and Y are in the correct answer When the WUPS score between them is greater than or equal to the threshold t1, X and Y are positive samples of each other; if the WUPS score is lower than the threshold t2, X and Y are negative samples of each other; S402, collecting positive samples and negative samples for each question-answering data, so as to allow the anchor sample to randomly select a positive sample and a negative sample, thereby forming a triplet as input; S403, according to the similarity matrix S r|q,A , from the cross-modal graph G r|q,A Select the top k nodes most relevant to the question and answer set to extract the cross-modal feature F r|q,A : F r|q,A =top-k(G r|q,A ) Among them, top-k(·) represents the selection of the first k maximum values; Repeat the operation for the three samples in the triplet to obtain in, They are the cross-modal features of anchor samples, positive samples, and negative samples respectively; a, p, and n represent anchor samples, positive samples, and negative samples respectively; Align the features of negative samples separately And the characteristics of the positive sample To the anchor sample: in, The features of the negative samples are aligned to the features of the anchor samples. Align the features of the positive sample to the features of the anchor sample; The distances d(a,p) and d(a,n) between the anchor sample and the positive / negative samples are calculated as: The triplet loss is as follows: L triplet =max(d(a,p)-d(a,n)+margin,0 Among them, max(·,·) represents the maximum value between the two; margin is a hyperparameter set artificially; For each sample in a triplet, the cross entropy loss is: Among them, ∑(·) is the summation operation, y i is the true matching score of the i-th sample, S i Predict the matching score for the model of the i-th sample; The calculation formula of the matching score S is as follows: S=G f,r|q,A (F q,A ) T 。 6. The context-aware target intention reasoning method according to claim 1, characterized in that: In S5, the loss function mentioned in S4 is used to apply the triplet loss L to the anchor sample, positive sample and negative sample. triplet and the cross entropy loss L ce Two loss functions together constitute the complete loss function: in, is the cross loss entropy function of the benchmark sample, is the cross loss entropy function of the positive sample, is the cross loss entropy function of negative samples.
7. The context-aware target intention reasoning method according to claim 1, characterized in that: In S6, the question and answer set A are used as the input of GPT, and the most likely answer is selected from the answer set according to the question, and the confidence distribution S of the question answer set is obtained. gpt , and then use a coefficient λ to combine the two confidence distributions to obtain the joint distribution S of the matching scores of the answer set A joint : S joint =S+λS gpt Finally, the candidate answer with the highest confidence is returned as the final predicted answer α * : in, An operation to select the maximum value in a collection.
8. A context-aware target intention reasoning system, executing a context-aware target intention reasoning method according to any one of claims 1 to 7, characterized in that: It includes a feature encoder, a video query language module, a comparative context module, and a common sense context module connected in sequence; Feature encoder, used to provide high-dimensional features of original remote sensing video and text, and realize digital encoding of original data; Video query language module, used to obtain video context information related to the question; The contrastive context module is used to preliminarily infer the situational context information; The common sense context module is used for common sense reasoning and combines the preliminary reasoning results with the common sense reasoning to obtain the final reasoning results.
Citation Information
Patent Citations
Visual common sense reasoning method and system based on knowledge perception attention network
CN115759262A
Video question and answer processing method and device, electronic equipment and storage medium
CN117112751A
Cited By
Video-based traffic accident analysis method, device, equipment and medium
CN119206593A
