Satellite video analysis method and system fusing low-rank expression and large language model

Through the low-rank representation, the moving objects and static backgrounds in satellite videos are separated and analyzed in combination with large language models, the problem of high-level understanding of satellite videos in the existing technology is solved, and the accurate understanding and in-depth description of satellite videos is achieved. Suitable for remote sensing.

CN119071408APending Publication Date: 2024-12-03WUHAN UNIV +1
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411171923.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-26
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing satellite video analysis methods are difficult to understand dynamic and static information, lack of mining of the interactive relationship between motion goals and scenes, resulting in the inability to complete high-level understanding tasks, and ordinary video understanding methods cannot be applied to the remote sensing field.

Method used

Low-rank representation (LRR) is used to separate moving objects from static backgrounds and input them into the Large Language Model (LLM) for detailed description and analysis to form insightful reports.

Benefits of technology

It realizes accurate inter-frame registration and motion target extraction of satellite videos, can provide in-depth and high-level understanding, bridge the gap between research and practical applications, and provides a comprehensive and systematic description of the scene background and motion targets.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071408A_ABST
    Figure CN119071408A_ABST
Patent Text Reader

Abstract

The invention provides a satellite video analysis method and system fusing low-rank expression and a large language model, and the method comprises the following steps: 1, obtaining an original satellite video, separating a moving object and a static background in the satellite video based on the low-rank expression LRR, and solving to obtain a low-rank background L and a sparse motion S; 2, respectively inputting the low-rank background L and the sparse motion S obtained in the step 1 into an LLM large language model to generate formatted language description; and 3, respectively inputting formatted language descriptions generated by the low-rank background L and the sparse motion S into the LLM large language model for conjoint analysis to form a final report with an insight. According to the invention, the extraction of the moving target and the separation of the moving target from the background can be realized more accurately, especially for a tiny moving target.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of video analysis technology, and in particular to a satellite video analysis method and system integrating low-rank expression and a large language model. Background Art

[0002] Video satellites are like flying robots in space, observing the earth with high spatiotemporal resolution, bringing various new demands for short video live broadcasts. Video satellites have gradually become a solid pillar of earth observation in the field of remote sensing, along with classic static images. Generally speaking, compared with static images, moving targets are the most unique and attractive elements because they can only be obtained from satellite videos. Therefore, many researchers have devoted themselves to tracking moving targets and proposed many algorithms, from traditional algorithms [1-3] to learning algorithms [4-6], which have performed remarkable in recent years. However, actual satellite video applications usually fail to achieve the expected fruitful results. It is believed that the core problem causing this dilemma is the insufficient exploration of dynamic and static information in satellite videos, resulting in a lack of comprehensive understanding. More specifically, there are two main problems with existing methods: First, in the field of remote sensing, most of the existing satellite video interpretation methods focus on extracting motion information of single or multiple targets, such as trajectories. There is a lack of understanding of higher-level information, a lack of cognition of scene background, and a lack of mining of the interactive relationship between moving targets and scenes. As a result, it is difficult for existing methods to truly have a significant impact on practical applications and lack valuable understanding. On the other hand, in the field of daily video understanding, although with the emergence of large models, various video understanding methods based on large models have shown good performance on daily videos, especially for video plot analysis. However, due to the characteristics of satellite videos such as small moving targets and few textures, ordinary video understanding methods cannot be directly used in the remote sensing field. This inadaptability further hinders the high-level understanding of satellite videos. Therefore, existing methods cannot complete the task of high-level understanding of satellite videos.

[0003] Therefore, this problem is further addressed from the fields of remote sensing and artificial intelligence. Specifically, the separation of moving objects and static background is first achieved using LRR (low-order representation) because it has few assumptions about the data. Then, a set of examples for hierarchical satellite video understanding is proposed for contextual learning with a pre-trained LLM large language model. The separated background and moving objects are fed to the LLM for detailed descriptions. Finally, these descriptions are taken to the LLM again for further analysis and output reports. Extensive experiments have demonstrated the effectiveness of the approach, bridging the gap between current research and the pressing needs in practical applications. Summary of the invention

[0004] The present invention provides a satellite video parsing method integrating low-rank expression and large language model.

[0005] To achieve the above object, the technical solution of the method of the present invention is:

[0006] A satellite video parsing method integrating low-rank expression and large language model comprises the following steps:

[0007] Step 1: Obtain the original satellite video, separate the moving object and the static background in the satellite video based on the low-rank representation LRR, and solve to obtain the low-rank background L and the sparse motion S;

[0008] Step 2: Input the low-rank background L and sparse motion S obtained in step 1 into the LLM large language model to generate a formatted language description;

[0009] Step 3: Input the formatted language descriptions generated by the low-rank background L and the sparse motion S into the LLM large language model for joint analysis to form an insightful final report.

[0010] Furthermore, the step 1 includes the following sub-steps:

[0011] Step 1.1. Transform the dimension of each frame in the original satellite video and stack all columns to generate a one-dimensional vector, which is represented as vec(), and broadcast the vec() operation on all n frames to obtain a hyperdimensional matrix D;

[0012] Step 1.2. Introduce inter-frame geometric transformation into the hyper-dimensional matrix D to compensate for inter-frame pseudo-motion and obtain the hyper-dimensional matrix after geometric registration;

[0013] Step 1.3. Use the hyperdimensional matrix after geometric registration as a constraint condition, and construct the optimization objective function based on the matrix rank solution rank() operation and L0 norm;

[0014] Step 1.4. Replace the rank() operation and L0 norm in the optimization objective function with the nuclear norm L* and L1 norm for simplification;

[0015] Step 1.5. Linearize the constraints to obtain a convex optimization problem and solve it iteratively to obtain the low-rank background L and sparse motion S.

[0016] Furthermore, the hyperdimensional matrix in step 1.1 is:

[0017] D=[vec(F1)|vec(F2)|…|vec(F n )]

[0018] Among them, F1, F2, F n They are the 1st frame, 2nd frame, and nth frame in the input video respectively.

[0019] Furthermore, the hyperdimensional matrix after transformation in step 1.2 is:

[0020]

[0021] Where τ={τ1,τ2,…,τ n}, where τ1,τ2,,τ n They are the inter-frame geometric transformations of the 1st frame, the 2nd frame, and the nth frame, respectively. Indicates multiplication.

[0022] Furthermore, the optimization objective function in step 1.3 is:

[0023]

[0024] Among them, γ is the weight corresponding to the L0 norm of the sparse motion S in the optimization objective function, st is the constraint condition, ║║0 represents the L0 norm, and min represents the minimum value.

[0025] Furthermore, the optimization objective function in step 1.4 is simplified to:

[0026]

[0027] Among them, λ is the weight corresponding to the L1 norm of the optimization objective function sparse motion S, and ║║1 represents the L1 norm.

[0028] Furthermore, the convex optimization problem obtained by linearizing the constraint conditions in step 1.5 is:

[0029]

[0030] where Δτ i represents the change in geometric transformation between frames of the i-th frame, J i represents the Jacobian matrix,

[0031] ε i Indicates the inter-frame geometric transformation corresponding to the i-th frame in R n The basis of the transformation in space, i∈[1,n]; T represents the transpose, Δτ i represents the change in geometric transformation between frames of the i-th frame, R n represents the geometric transformation space of dimension n, τ i is the geometric transformation between frames of the i-th frame, F i is the i-th frame in the input video, and ζ is the iterative variable for derivation.

[0032] Furthermore, the step 2 includes the following sub-steps:

[0033] Step 2.1. Input remote sensing expertise into the large language model LLM in the form of prompt words;

[0034] Step 2.2. Input the low-rank background L sequence image obtained by low-rank decomposition into the LLM large language model. The LLM large language model analyzes the low-rank background L sequence image according to remote sensing expertise and outputs content analysis about the static background.

[0035] Step 2.3. Input the sparse motion S sequence images obtained by low-rank decomposition into the LLM large language model. The large model analyzes the remote sensing expertise and sparse motion S sequence images and outputs content analysis about dynamic targets.

[0036] Furthermore, the step 3 comprises:

[0037] Step 3.1. Design a comprehensive understanding paradigm for satellite video background and moving targets, and embed the understanding paradigm into the LLM large language model;

[0038] Step 3.2. Integrate the content analysis of the static background and the content analysis of the dynamic target, and input the integrated content into the large language model LLM;

[0039] Step 3.3. The large prediction model LLM analyzes the background and motion comprehensively and outputs a standard formatted analysis report.

[0040] On the other hand, the present invention provides a satellite video parsing system integrating low-rank expression and large language model, comprising:

[0041] Module 1: It is used to obtain the original satellite video, separate the moving object and the static background in the satellite video based on the low-rank representation LRR, and solve to obtain the low-rank background L and the sparse motion S;

[0042] Module 2: It is used to input the low-rank background L and sparse motion S obtained in module 1 into the LLM large language model to generate a formatted language description;

[0043] Module 3: It is used to input the formatted language descriptions generated by the low-rank background L and the sparse motion S into the LLM large language model for joint analysis to form an insightful final report.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. The geometric correction specially designed for inter-frame geometric inconsistency can achieve more accurate and reliable inter-frame registration compared with other methods, providing a good foundation for subsequent moving target extraction;

[0046] 2. Based on the robust low-rank expression theory, the present invention can more accurately extract moving targets and separate them from the background, especially for tiny moving targets;

[0047] 3. In addition to conventional motion extraction, the present invention can also achieve a deeper and higher-level understanding of satellite video;

[0048] 4. The systematic satellite video understanding paradigm designed by this method provides a more comprehensive and systematic description of satellite videos and is more valuable for actual production applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 It is a schematic diagram of the overall framework of an embodiment of the present invention.

[0051] Figure 2 Schematic diagram of satellite video understanding paradigm according to an embodiment of the present invention.

[0052] Figure 3 The output comparison and display diagram is a schematic diagram of the embodiment of the present invention.

[0053] Figure 4 The figure is a schematic diagram of user interaction output according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] Example 1

[0056] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0057] like Figure 1 As shown, a satellite video parsing method integrating low-rank expression and large language model provided by an embodiment of the present invention includes:

[0058] Step 1: Obtain the original satellite video, separate the moving object and the static background in the satellite video based on the low-rank representation LRR, and solve to obtain the low-rank background L and the sparse motion S;

[0059] Step 2: Input the low-rank background L and sparse motion S obtained in step 1 into the LLM large language model to generate a formatted language description;

[0060] Step 3: Input the formatted language descriptions generated by the low-rank background L and the sparse motion S into the LLM large language model for joint analysis to form an insightful final report.

[0061] In order to effectively help LLM understand satellite videos and bridge the domain gap between satellite videos and natural videos, in step 1 of this embodiment, the video is divided into background and motion parts, such as Figure 1 Specifically, LRR is adopted in this task (first stage) because it effectively extracts motion and reconstructs static background while theoretically converging. For a certain frame F in n frames of satellite video i (pixel resolution: w×h), first transform its dimension and stack all columns to generate a one-dimensional vector vi of length w×h, which is denoted as vec(). Then broadcast the vec() operation on all n frames and obtain a hyperdimensional matrix D. Due to the nature of satellite videos, the background is usually static, while moving objects occupy smaller areas. Therefore, satellite videos are regarded as a combination of low-rank background L and sparse motion S, as follows:

[0062] D=[vec(F1)|vec(F2)|…|vec(F n )]

[0063] In order to further eliminate the pseudo motion caused by imperfect video stabilization or ground height difference, a frame-by-frame transformation is introduced to compensate for this inconsistency as follows:

[0064]

[0065] Where τ={τ1,τ2,…,τ n}, where τ1,τ2,,τ n They are the inter-frame geometric transformations of the 1st frame, the 2nd frame, and the nth frame, respectively. Indicates multiplication.

[0066] Write an optimization objective function in Lagrangian form as follows:

[0067]

[0068] Among them, γ is the weight corresponding to the L0 norm of the sparse motion S in the optimization objective function, st is the constraint condition, ║║0 represents the L0 norm, and min represents the minimum value.

[0069] Furthermore, the rank() operation and L0 norm are replaced by the nuclear norm L* and L1 norm respectively for simplification, as shown in the following formula:

[0070]

[0071] Among them, λ is the weight corresponding to the L1 norm of the optimization objective function sparse motion S, and ║║1 represents the L1 norm. Among them, λ is also a weighting coefficient. Generally speaking, the frame-by-frame differences in satellite videos are small, but they may accumulate over a period of time. Therefore, the constraints are linearized as follows:

[0072]

[0073] where Δτ i represents the change in geometric transformation between frames of the i-th frame, J i represents the Jacobian matrix,

[0074] ε i Indicates the inter-frame geometric transformation corresponding to the i-th frame in R n The basis of the transformation in space, i∈[1,n]; T represents the transpose, Δτ i represents the change in geometric transformation between frames of the i-th frame, R n represents the geometric transformation space of dimension n, τ i is the geometric transformation between frames of the i-th frame, F i is the i-th frame in the input video, and ζ is the iterative variable for derivation.

[0075] The above formula is a convex optimization problem that can be solved efficiently through iteration. Through LRR, motion separation and background reconstruction can be easily achieved.

[0076] In step 2, in order to give full play to the power of LLM, a two-branch and loop structure is proposed to enable LLM to focus on motion and background details respectively, and then jointly analyze and output an insightful report. LLM receives the separated background or moving objects from LRR (stage 1) and acts as an "analyzer" to generate output. Since LLM was originally pre-trained on a language dataset and does not have the ability of multimodal input, it converts visual data and text data, following similar ideas of Visual ChatGPT and LLaMa-VID to use visual encoders and text decoders for visual embedding and text queries respectively. Then, a linear projector is used to process the visual embedding and text queries to generate basic tags for LLM. Finally, the LLM large language model accepts these tags and analyzes the response. Alternatively, users can provide expertise to help LLM better understand the data. Specifically, the pre-trained QFormer is used as the text decoder and EVA-G is used as the visual encoder. For LLM, the pre-trained Vicuna is used with weights of 7B and 13B. The specific steps are:

[0077] Step 2.1. Input remote sensing expertise into the large language model LLM in the form of prompt words;

[0078] Step 2.2. Input the low-rank background L sequence image obtained by low-rank decomposition into the LLM large language model. The LLM large language model analyzes the low-rank background L sequence image according to remote sensing expertise and outputs content analysis about the static background.

[0079] Step 2.3. Input the sparse motion S sequence images obtained by low-rank decomposition into the LLM large language model. The large model analyzes the remote sensing expertise and sparse motion S sequence images and outputs content analysis about dynamic targets.

[0080] In step 3, in order to better contextualize the LLM large language model in satellite videos, a hierarchical understanding paradigm is designed and used as cues to eliminate hallucinations, such as Figure 2 With the powerful LLM, multiple tasks (e.g., scene classification and object detection) can be achieved simultaneously, and these tasks can be further improved by leveraging satellite video knowledge.

[0081] For the static background in satellite videos, the scene background reconstructed by LRR is clean and static. For accurate and valuable description, a hierarchical description framework is designed, such as Figure 2 . It includes scene level, target level and feature level. In the scene level description, it is further divided into man-made scenes and natural scenes. In the man-made scenes, there are four categories: urban areas, suburbs, residential areas and industrial areas; in the natural scenes, there are four categories: vegetation, water bodies, snow and bare land. In the target level description, it is further divided into traffic targets, movable targets and static targets. Traffic targets include airports, railway stations and ports. Movable targets include cars, ships, airplanes and trains. Static targets include parking lots, buildings and trees. In the feature level description, it is further divided into lighting, texture and sensors.

[0082] Moving objects, compared to static backgrounds, intrinsically distinguish satellite videos from other remote sensing images and provide rich potential information under motion. Motion analysis is also implemented in a hierarchical manner, where the motion emphasized by LRR is analyzed from spatial aspects and temporal aspects at the top level, such as Figure 2 . In terms of space, two parallel aspects are set up, considering all moving objects as a group with a general trend and focusing on the trajectory of a single object. In terms of time, the last frame in the video is used as the time reference, and selected states such as speed are used to describe the past, present, and future. In dynamic targets, a spatiotemporal description framework is further designed. In terms of space, it contains two contents: overall motion trend and independent motion trajectory. In terms of space, it contains three contents: past motion state, current motion state, and future motion estimation.

[0083] Combined video. LLM is used again to fully understand the results of the static background and moving object branches. Finally, a video understanding report is output.

[0084] The method of this embodiment is tested using Nvidia RTX 4090 and Tesla V100 GPUs. LRR is mainly implemented using OpenCV and Eigen libraries in C++ language, while LLM is mainly built using PyTorch library. Python scripts are developed for the integration of LRR and LLM. 40 videos of Jilin-1 and Luojia-3 satellites are collected and manually annotated. There is little work in the field of remote sensing, so general video understanding methods are selected, including Video-LLaMA, VideoChat, Video-ChatGPT, and LLaMA-VID. Finally, accuracy is focused on by measuring the percentage of correct keywords.

[0085] The analysis results of separated background video (V-BKG), separated motion video (V-MOV), joint video (V-JNT), and original video (V-ORI) from the LRR module are obtained. The calculated accuracies are summarized in Table 1.

[0086] Table 1 Accuracy comparison analysis

[0087]

[0088] In general, the proposed framework achieves the best performance on satellite videos, scoring 85.52% (7B) and 87.36% (13B). First, due to the proposed two-branch attention with LRR, the accuracy of the unique background and motion parts (V-BKG and V-MOV) is generally better than the original video (V-ORI). Second, due to the synergistic effect of background and motion, the joint analysis result of the two parts (V-JNT) is also better than the original video (V-ORI). These comparisons show that the proposed mechanism not only benefits our method, but also brings improvements to other methods, demonstrating its effectiveness and generalization. In addition, our method has a more refined description of satellite videos, and the designed understanding paradigm further helps the model to perform a more unified and professional analysis. Some descriptions such as Figure 3 , where green, blue, and red represent true, correct, and false keywords, respectively.

[0089] The performance of frameworks with different weights is also compared, as shown in Table 1. In general, the framework with 13B weight performs the same in accuracy (87.36%) as the 7B framework (85.52%). This is mainly due to the designed two-branch structure. The complex understanding task is intentionally split into two simpler tasks so that the LLM large language model can focus more on each aspect, and then analyze the results together. For the 13B framework, it is found that it generally has higher accuracy and less hallucinations in some very complex videos, but the GPU consumption is more (13B is about 30GB, 7B is about 24GB). Therefore, a balance should be struck between understanding accuracy and hardware cost for a specific application. Figure 4 Some visualizations in (a) are visualized in a conversational manner. In addition, to demonstrate the consistency of different languages, the framework is also tested in Chinese and Japanese, such as Figure 4 (b). The proposed approach provides a consistent description for the video. In fact, due to its good generalization, the proposed framework can be easily applied to Unmanned Aerial Vehicle (UAV) videos.

[0090] Example 2

[0091] This embodiment provides a satellite video parsing system integrating low-rank expression and large language model, including:

[0092] Module 1: It is used to obtain the original satellite video, separate the moving object and the static background in the satellite video based on the low-rank representation LRR, and solve to obtain the low-rank background L and the sparse motion S;

[0093] Module 2: It is used to input the low-rank background L and sparse motion S obtained in module 1 into the LLM large language model to generate a formatted language description;

[0094] Module 3: It is used to input the formatted language descriptions generated by the low-rank background L and the sparse motion S into the LLM large language model for joint analysis to form an insightful final report.

[0095] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present invention.

[0096] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

[0097] Other parts not described in detail are all prior art.

Claims

1. A satellite video parsing method integrating low-rank expression and large language model, characterized in that: The steps include: Step 1: Obtain the original satellite video, separate the moving object and the static background in the satellite video based on the low-rank representation LRR, and solve to obtain the low-rank background L and the sparse motion S; Step 2: Input the low-rank background L and sparse motion S obtained in step 1 into the LLM large language model to generate a formatted language description; Step 3: Input the formatted language descriptions generated by the low-rank background L and the sparse motion S into the LLM large language model for joint analysis to form an insightful final report.

2. A satellite video parsing method integrating low-rank expression and large language model as claimed in claim 1, characterized in that: The step 1 includes the following sub-steps: Step 1.

1. Transform the dimension of each frame in the original satellite video and stack all columns to generate a one-dimensional vector, which is represented as vec(), and broadcast the vec() operation on all n frames to obtain a hyperdimensional matrix D; Step 1.

2. Introduce inter-frame geometric transformation into the hyper-dimensional matrix D to compensate for inter-frame pseudo-motion and obtain the hyper-dimensional matrix after geometric registration; Step 1.

3. Use the hyperdimensional matrix after geometric registration as a constraint condition, and construct the optimization objective function based on the matrix rank solution rank() operation and L0 norm; Step 1.

4. Replace the rank() operation and L0 norm in the optimization objective function with the nuclear norm L* and L1 norm for simplification; Step 1.

5. Linearize the constraints to obtain a convex optimization problem and solve it iteratively to obtain the low-rank background L and sparse motion S.

3. A satellite video parsing method integrating low-rank expression and large language model as claimed in claim 2, characterized in that: The hyperdimensional matrix in step 1.1 is: D=[vec(F1)|vec(F2)|…|vec(F n )] Among them, F1, F2, F n They are the 1st frame, 2nd frame, and nth frame in the input video respectively.

4. The satellite video parsing method integrating low-rank expression and large language model as claimed in claim 3, characterized in that: The hyperdimensional matrix after transformation in step 1.2 is: Where τ={τ1,τ2,…,τ n } represents the set of inter-frame geometric transformations, where τ1, τ2,, τ n They are the inter-frame geometric transformations of the 1st frame, the 2nd frame, and the nth frame, respectively. Indicates multiplication.

5. A satellite video parsing method integrating low-rank expression and large language model as claimed in claim 4, characterized in that: The optimization objective function in step 1.3 is: Among them, γ is the weight corresponding to the L0 norm of the sparse motion S in the optimization objective function, st is the constraint condition, ║ ║0 represents the L0 norm, and min represents the minimum value.

6. A satellite video parsing method integrating low-rank expression and large language model as claimed in claim 5, characterized in that: The optimization objective function in step 1.4 is simplified to: Where λ is the weight corresponding to the L1 norm of the sparse motion S of the optimization objective function, ‖‖ * represents the nuclear norm L*, and ‖ ‖1 represents the L1 norm.

7. A satellite video parsing method integrating low-rank expression and large language model as claimed in claim 6, characterized in that: The convex optimization problem obtained by linearizing the constraint conditions in step 1.5 is: where Δτ i represents the change in geometric transformation between frames of the i-th frame, J i represents the Jacobian matrix, ε i Indicates the inter-frame geometric transformation corresponding to the i-th frame in R n The basis of the transformation in space, i∈[1,n]; T represents the transpose, Δτ i represents the change in geometric transformation between frames of the i-th frame, R n represents the geometric transformation space of dimension n, τ i is the geometric transformation between frames of the i-th frame, F i is the i-th frame in the input video, and ζ is the iterative variable for derivation.

8. The satellite video parsing method integrating low-rank expression and large language model as claimed in claim 1, characterized in that: The step 2 includes the following sub-steps: Step 2.

1. Input remote sensing expertise into the large language model LLM in the form of prompt words; Step 2.

2. Input the low-rank background L sequence image obtained by low-rank decomposition into the LLM large language model. The LLM large language model analyzes the low-rank background L sequence image according to remote sensing expertise and outputs content analysis about the static background. Step 2.

3. Input the sparse motion S sequence images obtained by low-rank decomposition into the LLM large language model. The large model analyzes the remote sensing expertise and sparse motion S sequence images and outputs content analysis about dynamic targets.

9. A satellite video parsing method integrating low-rank expression and large language model as claimed in claim 8, characterized in that: The step 3 comprises: Step 3.

1. Design a comprehensive understanding paradigm for satellite video background and moving targets, and embed the understanding paradigm into the LLM large language model; Step 3.

2. Integrate the content analysis of the static background and the content analysis of the dynamic target, and input the integrated content into the large language model LLM; Step 3.

3. The large prediction model LLM analyzes the background and motion comprehensively and outputs a standard formatted analysis report.

10. A satellite video parsing system integrating low-rank representation and large language model, characterized in that: include: Module 1: It is used to obtain the original satellite video, separate the moving object and the static background in the satellite video based on the low-rank representation LRR, and solve to obtain the low-rank background L and the sparse motion S; Module 2: It is used to input the low-rank background L and sparse motion S obtained in module 1 into the LLM large language model to generate a formatted language description; Module 3: It is used to input the formatted language descriptions generated by the low-rank background L and the sparse motion S into the LLM large language model for joint analysis to form a final report with insights; The satellite video parsing system integrating low-rank expression and large language model is used to execute the steps in the satellite video parsing method integrating low-rank expression and large language model described in any one of claims 1-9.

Citation Information

Patent Citations

  • Video action recognition method based on sparse low-rank coding

    CN106056082A

  • On-orbit real-time image stabilizing method and system for video satellite image

    CN108076341A

  • Video foreground and background separation method based on improved low-rank sparse decomposition

    CN110490894A

  • Visual big language model construction method for multi-source ship remote sensing image interpretation

    CN117994677A

  • Apparatus for Processing High Dynamic Range Imaging

    KR101740647B1