Multi-modal large model inference method with timing sparsity and dynamic feedback

Abstract

The application discloses a kind of multi-modal large model inference methods of time sequence sparse and dynamic feedback, through sparse sampling strategy and dynamic feedback inference mechanism, improve model analysis precision and processing efficiency, effectively solve the key challenge in deep-sea geological video analysis.The method comprises: constructing structured graph-text data set by integrating multi-source marine data, constructing three-level annotation system;Based on multi-modal large model architecture, the parameters of the core module of Transformer are optimized using LoRA technology;Design time sequence sparse sampling and dynamic feedback inference mechanism, realize context frame management through sliding window and bidirectional queue, reduce the computational complexity by combining sparse sampling strategy and enhance the consistency of time series inference using historical prediction results.The application realizes the synchronous improvement of deep-sea geological analysis accuracy and efficiency through the collaborative innovation of multi-modal data processing, efficient model optimization and intelligent inference mechanism, and provides a reliable intelligent solution for marine geological research.

Description

Technical Field

[0001] The present invention relates to the technical fields of artificial intelligence and multimodal image processing, and in particular to a multimodal large model reasoning method with temporal sparseness and dynamic feedback. Background Art

[0002] With the rapid development of deep-sea exploration technology, video-based deep-sea geological analysis has become an important tool in marine science research. Deep-sea video data, with its high resolution, long duration, and complex geological features, presents new opportunities and challenges for automated geological type identification. Traditional methods typically rely on manual interpretation or single-frame image classification models, which face numerous challenges when applied to long-duration, high-resolution deep-sea video.

[0003] Computational efficiency bottleneck: Deep-sea videos usually have a resolution of 4K or even higher and last for several hours. Manual interpretation and frame-by-frame processing will generate huge computational overhead, making it difficult to achieve real-time or quasi-real-time analysis. Lack of spatiotemporal continuity: Seabed geological features often show a gradual distribution (such as the transition of sulfide deposits around hydrothermal vents). Traditional analysis methods cannot effectively utilize the spatiotemporal continuity characteristics of geological changes, resulting in inconsistent classification results. Insufficient use of contextual information: The evolution of geological types has clear regularities (such as the transition from sediments → hydrothermal sulfides → chimney structures), but existing methods lack effective modeling of inter-frame correlations. Difficulty in integrating domain knowledge: Deep-sea geology has special distribution patterns (such as hydrothermal vents are usually distributed along fault zones), and traditional computer vision methods find it difficult to effectively integrate this prior knowledge.

[0004] To address these challenges, deep-sea geological classification analysis urgently requires an efficient and intelligent solution that can significantly improve processing efficiency while maintaining analytical accuracy. Specifically, a method is needed that can significantly reduce the number of frames to be processed through intelligent sampling strategies, leverage multi-frame contextual information to improve classification consistency, effectively integrate geological prior knowledge, and maintain the ability to process high-resolution video.

[0005] This study proposes a multimodal large-scale inference method based on temporal sparseness and dynamic feedback, effectively addressing these issues by integrating an interval sampling strategy with prior knowledge of prediction results. The method's core innovations lie in: designing a scientific interval sampling scheme that significantly reduces computational effort while ensuring coverage; constructing a dynamic frame queue input mechanism that leverages information from previous and subsequent frames to enhance the reliability of current frame analysis; fine-tuning the multimodal large-scale model to enable it to simultaneously process visual features and textual prompts; and incorporating geological prior knowledge to ensure that classification results conform to the laws of geological evolution.

[0006] This innovative analysis method provides an efficient and intelligent means for deep-sea geological research. It is not only suitable for offline large-scale video data analysis, but also lays the foundation for future real-time ROV operation monitoring. The method's demonstrated efficiency and accuracy give it broad application prospects in fields such as marine resource exploration and ecological and environmental monitoring. Summary of the Invention

[0007] The purpose of the present invention is to provide a multimodal large model reasoning method with temporal sparseness and dynamic feedback, which solves the problems of low computational efficiency, lack of spatiotemporal continuity and insufficient fusion of domain knowledge in the prior art.

[0008] The technical solution adopted by the present invention is as follows: a multi-modal large model reasoning method with temporal sparseness and dynamic feedback, comprising:

[0009] Constructing and annotating a graphic and text dataset by integrating multi-source data, including video, image, and text data; and fine-tuning a multimodal large model using the annotated graphic and text dataset.

[0010] Extract target frames from the video data at preset intervals and generate a target frame index list T. Construct a sliding window with a window size of 2k+1 and use a bidirectional queue structure Q from the target frame index list T to store the current target frame and its preceding and following k frames. Sparse sampling is used to sample the bidirectional queue structure Q to obtain a frame sequence S.

[0011] The current frame sequence and the prediction result of the previous frame are used as prompts to input the fine-tuned multimodal large model, effectively integrating prior knowledge to predict the result of the current frame. The prediction result of the current frame is used as the inference for the next frame; an iterative processing mechanism is adopted until the prediction task of all frames in the target frame index list T is completed.

[0012] Furthermore, we construct and annotate image and text datasets through multi-source data integration, including:

[0013] The multi-source data is geological data; the graphic and text data sets are annotated, including: building a three-level annotation system including basic geological type labels, fine-grained feature descriptions, and spatiotemporal distribution annotations.

[0014] Furthermore, it also includes parsing and post-processing the results output by the model;

[0015] Parse the output results, including:

[0016] Regular expressions are used to extract basic geological type information from model outputs, parse fine-grained geological features, and automatically associate spatiotemporal metadata;

[0017] Post-processing, including:

[0018] Construct a time series database to calculate the changing patterns of geological phenomena, the temporal and spatial distribution trajectory of geological phenomena, and predict future evolution trends, and use GIS tools for visualization.

[0019] Furthermore, we use labeled image and text datasets to fine-tune the multimodal large model, including:

[0020] Based on the Qwen2.5-VL-7B model architecture, LoRA technology is used to optimize the model by injecting a trainable low-rank matrix while keeping the pre-trained model parameters unchanged.

[0021] Furthermore, when processing the target frame, first pre-read the target frame index list T The first and last boundary frames in the ,are used for out-of-bounds filling processing;

[0022] For the first frame, when the left side of the sliding window exceeds the video frame range, the content of the first frame is used for repeated filling; for the last frame, when the right side of the sliding window exceeds the video frame range, the content of the last frame is used for repeated filling.

[0023] Furthermore, it also includes:

[0024] Maintain a prediction result list R to store the prediction results of the previous frame; for the processing of the first frame position, only the frame sequence of the first frame is used as the input of the fine-tuned multimodal large model.

[0025] The present invention also provides a multi-modal large model reasoning device with temporal sparseness and dynamic feedback, comprising:

[0026] The first module is used to construct and annotate a graphic and text dataset by integrating multi-source data; the multi-source data includes video, image, and text data; and fine-tune the multimodal large model using the annotated graphic and text dataset;

[0027] The second module is used to extract target frames from the video data at preset intervals and generate a target frame index list T. It constructs a sliding window with a window size of 2k+1 and uses a bidirectional queue structure Q to store the current target frame and the k frames before and after it from the target frame index list T. It samples the bidirectional queue structure Q using a sparse sampling strategy to obtain a frame sequence S.

[0028] The third module is used to input the current frame sequence and the prediction result of the previous frame as prompts into the fine-tuned multimodal large model, effectively integrating prior knowledge to predict the result of the current frame. The prediction result of the current frame is used for inference of the next frame;

[0029] An iterative processing mechanism is adopted until the prediction tasks of all frames in the target frame index list T are completed.

[0030] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned multi-modal large model inference method with temporal sparseness and dynamic feedback.

[0031] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned multi-modal large model reasoning method with temporal sparsity and dynamic feedback.

[0032] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned multi-modal large model reasoning method with temporal sparseness and dynamic feedback.

[0033] The beneficial effects of the embodiments of the present invention are: proposing a multimodal large model inference method with temporal sparsity and dynamic feedback, significantly improving the model analysis accuracy and processing efficiency through intelligent sampling strategies and multimodal large model optimization, and effectively solving the key challenges in deep-sea geological video analysis.

[0034] By adopting scientific sampling strategies (such as 250-frame intervals) and dynamic frame queue processing mechanisms, it greatly reduces redundant frame calculations while ensuring video coverage, significantly lowers computing overhead, and maintains efficient processing capabilities for high-resolution videos.

[0035] In addition, contextual temporal sparse and dynamic reasoning are proposed, and a dynamic feedback mechanism is designed to input the historical frame prediction results as text prompts into the current frame analysis, forming a multi-modal input of "sparse context image sequence + text prior", which effectively improves the coherence of continuous frame classification. The temporal sparse sampling of this application is implemented by maintaining a dynamic queue in operation. Because a frame of image will be used multiple times, in order to avoid repeated reading, it is implemented through a sliding window. Combined with sparse sampling strategies (such as even frame interval selection), the accuracy of temporal reasoning is maintained while reducing the amount of calculation. This method has achieved breakthroughs in computational efficiency, temporal consistency, prior knowledge fusion and multimodal collaborative analysis, providing a high-precision, low-redundancy and scalable intelligent analysis framework for deep-sea geological research. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1A flowchart of a multimodal large model inference method with temporal sparseness and dynamic feedback is provided in one embodiment of the present invention.

[0038] Figure 2 A schematic diagram of a multi-modal large model inference device module with temporal sparseness and dynamic feedback provided by one embodiment of the present invention.

[0039] Figure 3 The present invention provides a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0041] It should be noted that, unless there is any conflict, the features in the following embodiments and implementations may be combined with each other.

[0042] Figure 1 A flowchart of a multi-modal large model inference method with temporal sparseness and dynamic feedback is provided in one embodiment of the present invention. Figure 1 As shown, the multimodal large model reasoning method with temporal sparseness and dynamic feedback in the embodiment of the present invention may include the following steps:

[0043] Step S1: Construct a graphic and text dataset by integrating multi-source data. 512×512 standardized images are used for pre-annotation. A three-level annotation system is constructed, including basic geological type labels, fine-grained feature descriptions, and spatiotemporal distribution annotations. Double-blind annotation and expert review are used to ensure quality. Multi-dimensional structured data is stored in a standardized format to provide data support for subsequent model training and evaluation.

[0044] Among them, according to the specific field problems, multi-source data of the corresponding field is obtained. For example, the multi-source data can be geological data. Furthermore, the multi-source data can also be deep-sea geological data.

[0045] In one embodiment, taking deep-sea geological data as an example, a deep-sea geological multimodal dataset is constructed to provide data support for model training and evaluation, specifically including the following sub-steps:

[0046] Step S11: By integrating international ocean data platforms (such as the NOAA Ocean Database), shared databases of scientific research institutions, and original images, videos, and text materials from scientific expeditions, a multi-source, heterogeneous deep-sea geological data acquisition system is constructed. Specifically for video data, to avoid collecting many similar images, a default sampling rate of 1 fps is used when extracting frames from videos. This ensures data integrity while effectively reducing data redundancy. The collected images are scaled to a uniform size of 512×512, then described using the Qwen2.5-VL-72B model. The description of each image is recorded for subsequent manual review and organization.

[0047] Step S12: Build a multi-level image-text annotation system, covering basic geological type annotation (sediment, bedrock, hydrothermal vents, etc.) and fine-grained geological feature descriptions (chimney structures, sulfides, basalt, gray-white sediments, etc.). Manual extraction and modification of these geological descriptions are required based on the image descriptions from step S11. For expedition images, time and longitude and latitude information are also recorded. To ensure annotation quality, two professionally trained annotators independently complete the back-to-back annotation process. In the event of disagreements, deep-sea geology experts make the final decision to ensure the scientific and reliable annotation results. All annotated data is stored in a standardized format to facilitate subsequent model training and testing.

[0048] Step S2: First, load the Qwen2.5-VL-7B pre-trained model, configure the image processor parameters and initialize the training environment; use LoRA technology to fine-tune the parameters of the model's seven key modules ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], and set technical parameters such as rank r=16 and scaling factor α=32; construct training data containing 512×512 standardized images and multimodal instructions (i.e., question-answer pairs, each image corresponds to a question-answer pair), adopt a mixed precision training strategy, configure hyperparameters such as batch size 4 and learning rate 1e-4, set 10 training cycles, save checkpoints every 1000 steps, and finally obtain the optimized fine-tuned model.

[0049] To further illustrate the question-answer pair, an example question-answer pair is shown:

[0050] Question: You are a seasoned deep-sea geologist with extensive knowledge of deep-sea geology. The input images are three spaced-apart images. The geological type of the first frame is hydrothermal vent. Based on the preceding and following frames, determine whether the middle frame is rock, hydrothermal vent, sediment, or other. First, extract the time from the image and format it as 'YYYY / MM / DD HH:MM:SS'. Then, provide a professional description of the middle deep-sea geological image. Finally, confirm the geological type.

[0051] Output: Based on the image, a large number of small organisms can be seen clustered around some yellow and black vertical structures. These structures resemble the chimney-like structures surrounding hydrothermal vents on the seafloor. They are typically formed by mineral deposits and are habitats for many deep-sea organisms. Given that these features are typical of hydrothermal vents, we can conclude that the middle frame belongs to the "hydrothermal vent" geological type. In summary, the middle frame is a hydrothermal vent.

[0052] Step S3: The temporal sparse sampling and dynamic feedback reasoning process is as follows: First, record the target frame index list T , and construct a sliding window with a length of 2 k +1 for the double queue Q Store the target frame and its preceding and following frames k Next, convert the frames in the queue to PIL.Image format and adjust their resolution to 480×270. At the same time, apply sparse sampling strategy to generate a new frame sequence. S At the same time, maintain a list of prediction results R During the inference process, multiple image sequences are S The prediction result before the current frame is used as a prompt to input the fine-tuned model to predict the result of the current frame and store the prediction result in R In order to be used for the next frame reasoning;

[0053] In one embodiment, step S3 specifically includes the following sub-steps:

[0054] Step S31: When the video is initialized, in order to avoid redundancy, the interval of extracting frames is set to 250 frames, and the extracted frame indexes are stored in a list in sequence according to the interval. T , which is the index list of the target frame. Build a dynamic frame queue processing mechanism based on the target frame. Build a sliding window and set the window size to 2 k +1 (where k Indicates the context radius. k =3). Using a bidirectional queue structure QTo store the current target frame and the 3 frames before and after it in list T.

[0055] Step S32: When processing the target frame, first pre-read the list T The first and last boundary frames in the video are used for out-of-bounds filling. For the first frame, when the left side of the sliding window exceeds the video frame range, the content of the first frame is used for repeated filling; for the last frame, when the right side of the sliding window exceeds the video frame range, the content of the last frame is used for repeated filling. Specifically, for the target frame index T [N] (where index N refers to the list T The index in the queue is not the actual index of the video frame), Q The actual stored frame sequence is indexed as [ T [N-3], T [N-2], ..., T [N], ..., T [N+2], T [N+3]]. Frame positioning is achieved through the cv2.CAP_PROP_POS_FRAMES property, which is used to position the video to the specific position of the target frame in the video. T [N], and then it is easy to obtain the image content of the corresponding frame.

[0056] Step S33: Convert all frames in the queue to PIL.Image format and uniformly adjust the resolution to 480×270 to reduce computational overhead. At the same time, formulate a sparse sampling strategy to Q Perform uniform sampling. In this example, the reserved index is [ T [N-2], T [N], T [N+2]] frames form a new frame sequence S , S The index is T [N-2], T [N], T [N+2] three-frame PIL.Image format image sequence, the sequence S As the image input to the fine-tuned model in step S2.

[0057] Step S34: Maintain a prediction result list R For the processing of the first frame position, since there is no advance prediction result as text prompt input, only the image sequence is used. S As input, a single image modality is used for inference, and the inference results are stored in R Specifically, for the target frame index T [N], previous frame (current frame T[N] In frame sequence S The index of the previous frame in T [N−2], then the corresponding prediction result should be R [N−2]. Therefore, when T [N] frames of inference, R [N−2] is used as the prompt text input for the fine-tuned model in step S2.

[0058] In one embodiment, the previous frame may also be the previous frame of the current frame in the list T, that is, T [N−1], the corresponding prediction result should be R [N−1], in the T [N] frames of inference, R [N−2] is used as the prompt text input for the fine-tuned model in step S2. Preferably, the previous frame is the current frame T [N] In frame sequence S The previous frame in , the corresponding prediction result should be R [N−2] is used as the prompt text input for the fine-tuned model in step S2. After multiple experiments, it is verified that the model output has a higher accuracy.

[0059] Step S35: The image sequence obtained in step S33 S Combined with the prediction result of the previous frame obtained in step S34, the model’s graphic data is formed. These data contain multi-frame image content and historical frame prediction information, which serve as the context for temporal reasoning. The model analyzes the image sequence S and context information (i.e., hints), predicting images S [N] geological reasoning results and store the results in R [N]. This result will be used in the inference of the next frame as part of the prompt input.

[0060] Step S36: adopting an iterative processing mechanism, cyclically executing the frame processing flow from step S32 to step S35 until the prediction tasks of all frames in the target frame index list T are completed.

[0061] This application uses the previous frame's prediction result as a prompt because each geological type in the video should appear in many consecutive frames, preventing sudden changes such as a hydrothermal vent in the current frame, sediment in the next frame, and then a hydrothermal vent again in the next frame. This prompt can largely avoid sudden changes in type, improve stability and accuracy, and effectively integrate geological prior knowledge. Furthermore, the prompt tells the model "You are a deep-sea geologist with extensive knowledge of deep-sea geology," further integrating geological prior knowledge.

[0062] In one embodiment, the model output is further parsed and post-processed: Regular expressions are used to extract basic geological information, such as sediments, bedrock, and hydrothermal vents, from the model output to generate structured geological data. Fine-grained geological feature descriptions, such as chimney structures, sulfides, basalt, grayish-white sediments, and fault orientations, are parsed. Spatiotemporal metadata, such as time and longitude and latitude information, are automatically associated. A time series database is constructed to calculate the changing patterns of geological phenomena, map the spatiotemporal distribution of geological phenomena, and predict future evolution trends, which are then visualized using GIS tools.

[0063] The present invention also provides a multi-modal large model reasoning device with temporal sparseness and dynamic feedback, such as Figure 2 Shown, including:

[0064] The first module is used to construct and annotate a graphic and text dataset by integrating multi-source data; the multi-source data includes video, image, and text data; and fine-tune the multimodal large model using the annotated graphic and text dataset;

[0065] The second module is used to extract target frames from the video data at preset intervals and generate a target frame index list T. It constructs a sliding window with a window size of 2k+1 and uses a bidirectional queue structure Q to store the current target frame and the k frames before and after it from the target frame index list T. It samples the bidirectional queue structure Q using a sparse sampling strategy to obtain a frame sequence S.

[0066] The third module is used to input the current frame sequence and the prediction result of the previous frame as prompts into the fine-tuned multimodal large model, effectively integrating prior knowledge to predict the result of the current frame. The prediction result of the current frame is used for inference of the next frame;

[0067] An iterative processing mechanism is adopted until the prediction tasks of all frames in the target frame index list T are completed.

[0068] It should be noted that the device embodiment shown in this embodiment matches the content of the above method embodiment. You can refer to the content of the above method embodiment and will not repeat it here.

[0069] Figure 3 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 3 The electronic device provided in this embodiment includes: a memory and a processor, wherein the memory is used to store information including program instructions, and the processor is used to control the execution of program instructions. When the program instructions are loaded and executed by the processor, a multi-modal large model inference method with temporal sparseness and dynamic feedback of the present invention is implemented.

[0070] It should be noted that, in addition to Figure 3In addition to the memory and processor shown, the electronic device may also include other hardware according to its actual functions, which will not be described in detail.

[0071] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned multi-modal large model reasoning method with temporal sparsity and dynamic feedback.

[0072] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned multi-modal large model reasoning method with temporal sparseness and dynamic feedback.

[0073] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0074] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0075] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0076] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0077] The above embodiments are intended only to illustrate the design concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. The scope of protection of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design concepts disclosed in the present invention are within the scope of protection of the present invention.

Claims

1. A multimodal large model inference method with temporal sparseness and dynamic feedback, characterized by: include: Constructing and annotating a graphic and text dataset by integrating multi-source data, including video, image, and text data; Fine-tune large multimodal models using labeled image and text datasets; Extract target frames from the video data at preset intervals and generate a target frame index list T; construct a sliding window with a window size of 2k+1, and use a bidirectional queue structure Q from the target frame index list T to store the current target frame and its previous and subsequent k frames; Through the sparse sampling strategy, the bidirectional queue structure Q is sampled to obtain the frame sequence S; The current frame sequence and the predicted result of the previous frame are used as prompts to input into the fine-tuned multimodal large model, which effectively integrates prior knowledge to predict the result of the current frame. The predicted result of the current frame is used as the inference for the next frame; an iterative processing mechanism is adopted until the prediction task of all frames in the target frame index list T is completed; the previous frame is the previous frame of the current frame in the list T or the previous frame in the frame sequence S.

2. The method according to claim 1, characterized in that By integrating multi-source data, we build and annotate graphic and text datasets, including: The multi-source data is geological data; the graphic and text data sets are annotated, including: building a three-level annotation system including basic geological type labels, fine-grained feature descriptions, and spatiotemporal distribution annotations.

3. The method according to claim 2, characterized in that It also includes parsing and post-processing the model output results; Parse the output results, including: Regular expressions are used to extract basic geological type information from model outputs, parse fine-grained geological features, and automatically associate spatiotemporal metadata; Post-processing, including: Construct a time series database to calculate the changing patterns of geological phenomena, the temporal and spatial distribution trajectory of geological phenomena, and predict future evolution trends, and use GIS tools for visualization.

4. The method according to claim 1, wherein Fine-tune large multimodal models using labeled image and text datasets, including: Based on the Qwen2.5-VL-7B model architecture, LoRA technology is used to optimize the model by injecting a trainable low-rank matrix while keeping the pre-trained model parameters unchanged.

5. The method according to claim 1, wherein When processing the target frame, first pre-read the target frame index list T The first and last boundary frames in the ,are used for out-of-bounds filling processing; For the first frame, when the left side of the sliding window exceeds the video frame range, the content of the first frame is used for repeated filling; for the last frame, when the right side of the sliding window exceeds the video frame range, the content of the last frame is used for repeated filling.

6. The method according to claim 1, characterized in that Also includes: Maintain a prediction result list R to store the prediction results of the previous frame; for the processing of the first frame position, only the frame sequence of the first frame is used as the input of the fine-tuned multimodal large model.

7. A multimodal large model inference device with temporal sparseness and dynamic feedback, characterized by: include: The first module is used to construct and annotate a graphic and text dataset by integrating multi-source data; the multi-source data includes video, image and text data; Fine-tune large multimodal models using labeled image and text datasets; The second module is used to extract target frames from the video data at preset intervals and generate a target frame index list T; construct a sliding window with a window size of 2k+1, and use a bidirectional queue structure Q from the target frame index list T to store the current target frame and its previous and subsequent k frames; Through the sparse sampling strategy, the bidirectional queue structure Q is sampled to obtain the frame sequence S; The third module is used to input the current frame sequence and the prediction result of the previous frame as prompts into the fine-tuned multimodal large model, effectively integrating prior knowledge to predict the result of the current frame. The prediction result of the current frame is used for inference of the next frame; An iterative processing mechanism is adopted until the prediction tasks of all frames in the target frame index list T are completed.

8. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein, the memory is used to store program data, and the processor is used to execute the program data to implement a multi-modal large model reasoning method with temporal sparseness and dynamic feedback as described in any one of claims 1-6 above.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a multi-modal large model reasoning method with temporal sparseness and dynamic feedback is implemented as described in any one of claims 1-6.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, it implements a multi-modal large model reasoning method with temporal sparseness and dynamic feedback as described in any one of claims 1-6.

Images