Inference decision method and device based on large language model, equipment and storage medium

Abstract

This application provides a reasoning and decision-making method, apparatus, device, and storage medium based on a large language model. The method includes: acquiring a decision instruction input for a target scene and a video frame sequence corresponding to the target scene; when the decision instruction corresponds to a logical reasoning task, identifying at least one object to be analyzed, and using a first large language model to perform dynamic trajectory analysis and scene association analysis on the object to be analyzed based on the video frame sequence to obtain corresponding scene description information; combining the decision instruction and scene description information, performing semantic retrieval in a preset domain rule base to obtain at least one target rule information; combining the decision instruction, video frame sequence, scene description information, and at least one target rule information to generate corresponding text prompt information; and using a second large language model to perform logical reasoning based on the text prompt information to obtain the corresponding decision result. This improves the accuracy of decision results for dynamic environments.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a reasoning and decision-making method, apparatus, device, and storage medium based on a large language model. Background Technology

[0002] With the development of artificial intelligence technology, various autonomous intelligent systems (such as unmanned ships, self-driving cars, low-altitude drones, and special industrial robots) have made significant progress in basic visual perception, enabling them to identify obstacles, target objects, and their basic attributes in the environment. However, the real-world operating environments faced by these systems are generally highly dynamic, such as complex water flows and tides, variable traffic flow and weather at intersections, dense obstacles in urban pipe networks, and strict operating procedures in hazardous chemical workshops.

[0003] In related technologies, deep learning models are generally used for object detection, semantic segmentation, or multi-object tracking to achieve environmental perception and state recognition. For example, convolutional neural networks or visual Transformers are used to extract features and match patterns from single-frame image data to detect obstacles on water surfaces, vehicles on roads, or aerial targets, and then output decision results based on the detection results. However, these technical solutions can only complete static pattern matching of visual features, and their query output is only a surface description of visual content. They cannot effectively capture the movement trends of targets and the evolution of events, and it is difficult to predict the movement trends of obstacles and collision risks, resulting in inaccurate decision results for dynamic environments. Summary of the Invention

[0004] This application proposes a reasoning and decision-making method, apparatus, device, and storage medium based on a large language model, which can improve the accuracy of decision results in dynamic environments.

[0005] In some implementations, to achieve the above objectives, a first aspect of this application proposes a reasoning and decision-making method based on a large language model, the method comprising: Obtain the decision instructions input for the target scene and the video frame sequence corresponding to the target scene; When the task type indicated by the decision instruction is a logical reasoning task, at least one object to be analyzed is determined based on the decision instruction, and dynamic trajectory analysis and scene association analysis are performed on the at least one object to be analyzed based on the video frame sequence using the first language model to obtain the corresponding scene description information. By combining the decision instruction with the scene description information, semantic retrieval is performed in a preset domain rule base to obtain at least one target rule information; By combining the decision instruction, the video frame sequence, the scene description information, and the at least one target rule information, corresponding text prompt information is generated; By using the second language model, logical reasoning is performed based on the text prompt information to obtain the corresponding decision result, wherein the parameter scale of the second language model is larger than that of the first language model.

[0006] Accordingly, a second aspect of the embodiments of this application proposes a reasoning and decision-making apparatus based on a large language model, the apparatus comprising: The acquisition module is used to acquire the decision instructions input for the target scene and the video frame sequence corresponding to the target scene; The analysis module is used to determine at least one object to be analyzed based on the decision instruction when the task type indicated by the decision instruction is a logical reasoning task, and to perform dynamic trajectory analysis and scene association analysis on the at least one object to be analyzed based on the video frame sequence using the first large language model to obtain corresponding scene description information. The retrieval module is used to combine the decision instruction with the scene description information to perform semantic retrieval in a preset domain rule base to obtain at least one target rule information; The generation module is used to combine the decision instruction, the video frame sequence, the scene description information, and the at least one target rule information to generate corresponding text prompt information; The reasoning module is used to perform logical reasoning based on the text prompt information using a second language model to obtain the corresponding decision result, wherein the parameter scale of the second language model is larger than that of the first language model.

[0007] In some embodiments, the reasoning and decision-making device based on a large language model further includes a classification module, used for: The first video frame is determined from the video frame sequence; Using the first language model, the decision instruction and the first video frame are combined to perform task classification, and the task classification result corresponding to the decision instruction is obtained. The type of the task classification result includes at least one of visual perception task, knowledge retrieval task and logical reasoning task. When the task classification result indicates that the decision instruction corresponds to a visual perception task, the corresponding decision result is generated based on the decision instruction and the video frame sequence through the first large language model. When the task classification result indicates that the decision instruction corresponds to a knowledge retrieval task, a semantic search is performed in a preset domain rule base based on the decision instruction to obtain the corresponding decision result.

[0008] In some embodiments, the reasoning and decision-making device based on a large language model further includes a determining module, used for: Using the first major language model, visual consistency analysis is performed on the decision results based on the video frame sequence to obtain the first analysis result; Using the first large language model, a logical compliance analysis is performed on the decision result based on the at least one target rule information to obtain a second analysis result; Using the first large language model, and based on the first analysis result and the second analysis result, determine the inference confidence corresponding to the decision result; When the inference confidence is less than a preset confidence threshold, an intermediate decision result is generated based on the decision instruction, the video frame sequence, and the scene description information, and the inference confidence of the intermediate decision result is determined. This process continues until the inference confidence corresponding to the intermediate decision result is greater than or equal to the preset confidence threshold, at which point the corresponding intermediate decision result is determined as the updated decision result corresponding to the decision instruction.

[0009] In some embodiments, the reasoning and decision-making device based on a large language model further includes a determining module, used for: Combining the decision instruction with the scenario description information, a semantic search is performed in a preset domain rule base to obtain multiple corresponding update target rule information, wherein the number of the multiple update target rule information is greater than the number of the at least one target rule information; By combining the decision instruction, the video frame sequence, the scene description information, and the multiple update target rule information, a corresponding update text prompt is generated; Using the second language model, logical reasoning is performed based on the updated text prompt information to generate intermediate decision results.

[0010] In some embodiments, the reasoning and decision-making device based on a large language model further includes an anomaly constraint information generation module, used for: Based on the first analysis result, the visual anomaly information contained in the decision result is determined; Based on the second analysis result, the missing rule information contained in the decision result is determined; Based on the visual anomaly information and the rule missing information, anomaly constraint information is generated; The step of combining the decision instruction, the video frame sequence, the scene description information, and the multiple update target rule information to generate corresponding update text prompt information includes: By combining the decision instruction, the video frame sequence, the scene description information, the multiple update target rule information, and the anomaly constraint information, a corresponding update text prompt is generated.

[0011] In some implementations, the reasoning and decision-making device based on a large language model further includes an uploading module for: Obtain the geographic location coordinates of the edge computing node; The decision instruction and the scene description information are used to generate a retrieval vector, and the retrieval vector and the geographic location coordinates are uploaded to the cloud node, so that the cloud node can extract at least one real-time rule information based on the announcement rule information associated with the geographic location coordinates within the target time period. Receive at least one real-time rule information returned by the cloud node; By combining the at least one target rule information and the at least one real-time rule information, at least one initial fused rule information is obtained; For each initial fusion rule information, a corresponding rule weight description information is generated, wherein the rule weight description information is used to describe that the real-time rule information has a higher priority in the inference process of the second large language model than the corresponding target rule information; Each initial fusion rule information is concatenated with the corresponding rule weight description information to obtain the corresponding fusion rule information; The step of combining the decision instruction, the video frame sequence, the scene description information, and the at least one target rule information to generate corresponding text prompt information includes: By combining the decision instruction, the video frame sequence, the scene description information, and each fusion rule information, a corresponding text prompt message is generated.

[0012] In some implementations, the acquisition is further used for: Obtain the decision instructions for the target scene and the original video stream corresponding to the target scene, and determine the target duration corresponding to the original video stream; When the target duration is less than a preset perception threshold, the corresponding number of copies is determined based on the ratio between the perception threshold and the target duration. Determine the original video frame sequence corresponding to the original video stream, and copy the original video frame sequence according to the number of copies to obtain multiple intermediate video frame sequences; The original video frame sequence is sequentially concatenated with the plurality of intermediate video frame sequences to obtain the target video frame sequence; The target video frame sequence is re-encoded according to a preset frame rate to obtain the video frame sequence corresponding to the target scene.

[0013] Accordingly, a third aspect of the embodiments of this application proposes a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the reasoning and decision-making method based on a large language model according to any one of the embodiments of the first aspect of this application.

[0014] Accordingly, a fourth aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the reasoning and decision-making method based on a large language model according to any one of the embodiments of the first aspect of this application.

[0015] This application embodiment acquires decision instructions input for a target scene and a corresponding video frame sequence. When the decision instructions correspond to a logical reasoning task, at least one object to be analyzed is determined based on the decision instructions. A first large language model is used to perform dynamic trajectory analysis and scene association analysis on the at least one object to be analyzed based on the video frame sequence, obtaining corresponding scene description information. Combining the decision instructions and scene description information, semantic retrieval is performed in a preset domain rule base to obtain at least one target rule information. Combining the decision instructions, video frame sequence, scene description information, and at least one target rule information, corresponding text prompt information is generated. A second large language model is used to perform logical reasoning based on the text prompt information to obtain the corresponding decision result. The parameter scale of the second large language model is larger than that of the first large language model. Thus, through a two-stage reasoning architecture driven by video frame sequence-based dynamic trajectory analysis and scene association analysis, semantic retrieval and target rule information injection under the constraints of the domain rule base, and collaboration between large and small models, a deep understanding of the movement trend of targets in a dynamic environment, compliant integration of professional domain rules, and accurate execution of complex logical reasoning can be achieved.

[0016] Specifically, compared to the limitations of related technologies that only perform static pattern matching on single-frame images and cannot capture temporal dynamic information, this application first utilizes a first-large language model to perform dynamic trajectory analysis and scene association analysis on video frame sequences, transforming discrete visual frames into structured scene description information containing entity motion attributes and spatiotemporal evolution relationships, thereby effectively modeling target motion trends and event evolution processes. Simultaneously, by performing semantic retrieval of decision instructions and scene description information in a domain rule base, target rule information is injected into the reasoning process in the form of decision constraint information, avoiding the problem of pure visual perception output being disconnected from professional rules. Finally, a second-larger language model with a larger parameter scale performs logical reasoning based on the multimodal fusion of video frame sequences, scene description information, and decision constraint information, ensuring that the decision results conform to both visual facts and domain rule constraints. In summary, this application can improve the accuracy of decision results for dynamic environments. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the architecture of a reasoning and decision-making system based on a large language model provided in an embodiment of this application; Figure 2 This is a flowchart of the reasoning and decision-making method based on a large language model provided in an embodiment of this application; Figure 3 This is an example diagram of the overall modules corresponding to the reasoning and decision-making method based on a large language model provided in the embodiments of this application; Figure 4 This is a schematic diagram of the functional modules of the reasoning and decision-making device based on a large language model provided in the embodiments of this application; Figure 5 This is a schematic diagram of the hardware structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0021] With the development of artificial intelligence technology, various autonomous intelligent systems (such as unmanned ships, self-driving cars, low-altitude drones, and special industrial robots) have made significant progress in basic visual perception, enabling them to identify obstacles, target objects, and their basic attributes in the environment. However, the real-world operating environments faced by these systems are generally highly dynamic, such as complex water flows and tides, variable traffic flow and weather at intersections, dense obstacles in urban pipe networks, and strict operating procedures in hazardous chemical workshops.

[0022] In related technologies, deep learning models are generally used for object detection, semantic segmentation, or multi-object tracking to achieve environmental perception and state recognition. For example, convolutional neural networks or visual Transformers are used to extract features and match patterns from single-frame image data to detect obstacles on water surfaces, vehicles on roads, or aerial targets, and then output decision results based on the detection results. However, these technical solutions can only complete static pattern matching of visual features, and their query output is only a surface description of visual content. They cannot effectively capture the movement trends of targets and the evolution of events, and it is difficult to predict the movement trends of obstacles and collision risks, resulting in inaccurate decision results for dynamic environments.

[0023] Based on this, embodiments of this application provide a reasoning and decision-making method, apparatus, device, and storage medium based on a large language model, which can improve the accuracy of decision results for dynamic environments.

[0024] The reasoning and decision-making method, apparatus, device, and storage medium based on a large language model provided in this application are specifically described through the following embodiments. First, the reasoning and decision-making system based on a large language model in this application is described.

[0025] Please refer to Figure 1 In some implementations, embodiments of this application provide a reasoning and decision-making system based on a large language model, including a terminal 11 and a server 12.

[0026] In some implementations, terminal 11 can be used to acquire video frame sequences of the target scene in real time, receive decision instructions input by the user, and execute complete perception, reasoning, and decision-making tasks.

[0027] For example, terminal 11 can be a hardware device such as an unmanned shipborne edge computing device, an autonomous vehicle-mounted computing unit, or an industrial robot field controller; terminal 11 can perform dynamic trajectory analysis and scene association analysis on the video frame sequence through the first large language model (lightweight visual language model) configured on it, generate scene description information, and perform semantic retrieval in the locally preset domain rule base based on decision instructions and scene description information to obtain target rule information; terminal 11 can also use the second large language model (with a larger parameter scale than the first large language model) configured on it to combine decision instructions, video frame sequence, scene description information and target rule information to generate text prompt information and perform logical reasoning to obtain the final decision result, thereby realizing full-link local autonomous reasoning and real-time response in environments without network or with weak network.

[0028] In some implementations, server 12 can be used to provide supplementary retrieval services for wide-area dynamic knowledge or the latest domain rules that are not available in the terminal's local knowledge base. For example, it can be a cloud computing cluster, an edge cloud server, or a remote server equipped with a wide-area graph database. Server 12 can receive query vectors and context information sent by terminal 11 through a cloud knowledge retrieval module (such as a vector database or graph database that integrates real-time hydrological and meteorological information, the latest navigation notices, or local regulations), perform multi-hop retrieval, and return supplementary temporary rules or special constraint clauses to enhance the terminal's knowledge coverage and decision-making compliance in special scenarios.

[0029] Furthermore, terminal 11 and server 12 can interact via wireless communication networks (such as 4G / 5G, satellite communication, or dedicated local area networks): When terminal 11 determines that the current logical reasoning task involves temporary, regional, or dynamically updated domain knowledge not covered by the local rule base, it uploads the query vector and geographical location information, which integrates decision instructions and scene description information, to server 12; after completing the retrieval, server 12 sends the supplementary rule clauses back to terminal 11; terminal 11's second language model uses the original video frame sequence, scene description information, local target rule information, and cloud-supplemented rule information as the reasoning context to regenerate or correct the decision results, thus forming a collaborative mechanism in which terminal 11's autonomous local reasoning is the main component and cloud-based dynamic knowledge enhancement is the auxiliary component. This ensures low-latency autonomous decision-making capabilities in normal scenarios and meets the compliance requirements for real-time dynamic knowledge in special scenarios.

[0030] The reasoning and decision-making method based on a large language model in this application can be illustrated through the following embodiments.

[0031] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user will be obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent will the necessary user-related data for the normal operation of the embodiments of this application be obtained.

[0032] In this application embodiment, the description will focus on a reasoning and decision-making device based on a large language model, which can be integrated into a computer device. See also Figure 2 , Figure 2The flowchart illustrates the steps of the reasoning and decision-making method based on a large language model provided in this embodiment. Taking the integration of the large language model-based reasoning and decision-making device into a terminal or server as an example, the specific process when the processor on the terminal or server executes the program instructions corresponding to the large language model-based reasoning and decision-making method is as follows: Step 101: Obtain the decision instructions input for the target scene and the video frame sequence corresponding to the target scene.

[0033] In some implementations, the basic input required for subsequent multimodal processing can be constructed by simultaneously collecting the user's natural language queries (decision instructions) and continuous video data (video frame sequences) captured by the shipborne camera at the system input, thereby laying the data foundation for subsequent calculations.

[0034] The target scenario can be the working environment and corresponding dynamic situation of the transport equipment that needs to perform intelligent perception and cognitive reasoning under rule constraints. For example, the water navigation environment of unmanned ships (rivers, lakes, canals, ports, oceans, etc.), or the road traffic scenario of autonomous vehicles, the urban pipeline inspection environment of low-altitude drones, the hazardous chemical workshop operation scenario of special industrial robots, or a dynamic environment containing elements such as obstacles, other moving entities, and boundary markers. It can be used to provide the system with the context background that needs to be perceived, understood and reasoned, and is the objective object on which subsequent multimodal processing and rule retrieval depend.

[0035] Among them, decision instructions can be queries or task requests that users input into the unmanned vessel assisted driving system in the form of natural language, such as "What is the relative motion trend of the red vessel to the right front, and is there a risk of collision?" or "What color is the buoy on the left side of the screen?" These instructions can be used to indicate the type of cognitive task (including visual perception tasks, knowledge retrieval tasks, or complex logical reasoning tasks) that the system needs to perform.

[0036] Among them, the video frame sequence can be a set of image frames extracted from a continuous video stream captured by a visual sensor (such as a bow camera) mounted on the front of the unmanned vessel and arranged in time sequence, such as a continuous image sequence with a frame rate of 25 frames per second and a duration of no less than 2 seconds, or a standardized video tensor generated after time-series completion processing, which can be used to provide dynamic visual observation data for the target scene.

[0037] Specifically, the system can receive text or voice commands input by the user through a human-computer interaction interface (such as a shipboard touchscreen, voice acquisition device, data link of a shore-based remote control station, or vehicle-mounted microphone). If the input is voice, the voice recognition module is invoked to convert it into a text command. For example, the decision command could be "What is the relative motion trend of the red vessel to the right, and is there a collision risk?", "What color is the buoy on the left side of the screen?", or "What is the obligation of the vessel that gives way in a cross-traffic situation according to maritime collision avoidance rules?" The system can temporarily store the received decision commands as strings in the memory buffer of the edge computing node for subsequent task classification and inference modules to call.

[0038] It should be noted that this application uses unmanned vessel assisted driving as an example for illustration, but the proposed multimodal multi-agent visual question answering system and method are also applicable to other intelligent transportation equipment that require visual perception and cognitive reasoning under rule constraints, such as autonomous vehicles, low-altitude drones, and special industrial robots. Correspondingly, the target scenario can be replaced with road traffic scenario, urban inspection environment, or hazardous chemical workshop operation scenario.

[0039] Furthermore, the system can simultaneously acquire video frame sequences corresponding to the target scene. These video frame sequences can be captured in real-time by visual sensors deployed at the front end of the platform (such as the bow visible light camera of an unmanned vessel, the forward-facing camera of an autonomous vehicle, or the pod camera of a drone) at a preset sampling rate (e.g., 25 frames per second). The system can read the raw video stream through the video acquisition interface and decode and normalize the size of each frame.

[0040] In some implementations, to ensure effective modeling of temporal dynamic information by subsequent multimodal large models, the acquired video stream can be duration-verified: First, the target duration corresponding to the original video stream is calculated. If this duration is less than a preset perception threshold (e.g., 2.0 seconds or 3.0 seconds), a temporal completion mechanism is triggered. Specifically, a lossless loop replay strategy can be adopted. The number of copies is determined based on the ratio of the perception threshold to the target duration. The original real frame sequence is cyclically copied and sequentially spliced, and then re-encoded at a standard frame rate to generate a video frame sequence that meets the model input requirements. For example, when an unmanned surface vessel encounters high sea states, resulting in packet loss in image transmission and only capturing a 0.4-second (10-frame) video fragment, the system detects that the duration is shorter than the 3.0-second threshold, calculates the number of copies to be 8, copies the 10-frame sequence 8 times to obtain 80 frames, and encodes it into a 3.2-second standardized video tensor at 25 frames / second. If the original video stream duration meets the standard, it is directly formatted into a standard video tensor. Ultimately, the system uses the decision command and the video frame sequence as a pair of input data and sends them to the subsequent task classification and processing chain.

[0041] For example, in addition to acquiring video frame sequences in real time via shipborne visual sensors, a more robust scene representation can be obtained through multi-source perception fusion. Specifically, the system can simultaneously access data from millimeter-wave radar, lidar, or infrared thermal imagers, aligning the timestamps of sensor data from different modalities and performing feature-level fusion to generate enhanced video frame sequences (e.g., projecting radar detection boxes onto video frames to form multi-modal overlay frames). This can compensate for the perception deficiencies of pure vision in low visibility or high-light environments, avoid the problem of data acquisition being hindered by a single camera's field of view being limited due to obstruction or inclement weather, and improve the completeness and reliability of the system's situational awareness in complex water or road scenarios.

[0042] In some implementations, the temporal padding processing for video frame sequences can adaptively select a padding strategy based on the scene's dynamic range. For example, the system first evaluates the amplitude of non-rigid motion in the original video frames (such as the intensity of water ripples or the rolling of waves); if the motion is gentle and computing power is sufficient, an intelligent frame interpolation algorithm based on an optical flow prediction model is activated to generate smooth composite frames to improve motion continuity; if the water surface is turbulent or computing power is limited, it automatically reverts to the lossless loop playback mode to avoid artifacts caused by frame interpolation. This balances inference fluency and visual realism, further optimizing the real-time performance and reliability at the edge.

[0043] By using the above methods, we can simultaneously acquire users' natural language queries and corresponding water scene video data, thus constructing the input pairs required for subsequent multimodal cognitive reasoning.

[0044] In some implementations, to overcome the problems of insufficient video clip duration acquired by existing unmanned surface vessel (USV) assisted driving systems in adverse communication or high sea state environments, failing to meet the minimum input requirements of large language models for temporal receptive fields, and the generation of visual artifacts and hallucinations in non-rigid dynamic scenes such as water ripples and wakes if interpolation algorithms are used, the actual duration of the original video stream can be detected and compared with a preset perception threshold. When the duration is insufficient, the number of copies is determined based on the ratio between the two. By cyclically copying and sequentially splicing the original real frame sequence, and then re-encoding at a standard frame rate, a standardized video tensor that meets the model's input format requirements is generated. This forcibly extends the exposure time of key actions in the model's attention mechanism without introducing synthetic artifacts, thereby ensuring that the downstream multimodal inference module can obtain temporally complete and visually realistic input data. For example, step 101 may include: (101.1) Obtain the decision instructions for the target scene and the original video stream corresponding to the target scene, and determine the target duration corresponding to the original video stream; (101.2) When the target duration is less than the preset perception threshold, the corresponding number of replications is determined based on the ratio between the perception threshold and the target duration; (101.3) Determine the original video frame sequence corresponding to the original video stream, and copy the original video frame sequence according to the number of copies to obtain multiple intermediate video frame sequences; (101.4) The original video frame sequence is sequentially concatenated with multiple intermediate video frame sequences to obtain the target video frame sequence; (101.5) The target video frame sequence is re-encoded according to the preset frame rate to obtain the video frame sequence corresponding to the target scene.

[0045] The raw video stream can be a continuous image data stream that is not processed by timing completion and is collected in real time by the front-end camera of the unmanned vessel or transmitted and received. For example, it can be the raw video clip output by the bow camera or a short video with frame loss after compression and transmission.

[0046] The target duration can be the actual time length experienced by the original video stream from the start frame to the end frame. It can be compared with the perception threshold to determine whether the current video segment meets the minimum receptive field requirements of the downstream multimodal model for temporal dynamic information.

[0047] The perception threshold can be a pre-set time length threshold used to determine whether the input video segment meets the model's temporal input requirements, such as 2.0 seconds or 3.0 seconds.

[0048] The number of copies can be calculated based on the ratio between the perception threshold and the target duration, used to determine the number of times the original video frame sequence should be completely copied. For example, when the perception threshold is 3.0 seconds and the target duration is 0.4 seconds, the ratio is 7.5, and the number of copies is determined to be 8.

[0049] The intermediate video frame sequence can be any independent copy of the original video frame sequence obtained by making a complete copy according to the number of copies made.

[0050] The target video frame sequence can be an extended frame sequence obtained by sequentially splicing the original video frame sequence and all intermediate video frame sequences in chronological order, with the total number of frames meeting the perception threshold requirement.

[0051] The preset frame rate can be a pre-set standard sampling frequency used to re-encode the target video frame sequence, such as 25 frames per second, which can be used to convert the expanded frame sequence into a standardized video tensor that conforms to the input specifications of the downstream multimodal model.

[0052] In some implementations, the system reads the raw video stream through the video acquisition interface and calculates the time from the first frame to the last frame as the target duration. For example, if an unmanned surface vessel captures only a 0.4-second video segment due to packet loss in the image transmission link during navigation, the system determines the target duration to be 0.4 seconds.

[0053] For example, the system has a preset perception threshold, which is used to determine whether the input video segment meets the minimum receptive field requirements of the downstream multimodal large model for temporal dynamic information. The perception threshold can be set according to the temporal input specifications of the large language model used, such as 2.0 seconds or 3.0 seconds, or it can be dynamically adjusted according to the security level of the specific application scenario.

[0054] Furthermore, when the target duration is less than a preset perception threshold, the corresponding number of copies can be determined based on the ratio between the perception threshold and the target duration. Specifically, the number of copies is obtained by rounding up the result of dividing the perception threshold by the target duration. For example, if the perception threshold is 3.0 seconds, the target duration is 0.4 seconds, and the ratio is 7.5, then the number of copies is determined to be 8. This ensures that the total number of frames after copying and splicing can meet the input length requirements of the downstream model.

[0055] Furthermore, the original video frame sequence corresponding to the original video stream can be determined, that is, each frame image arranged in chronological order. Then, according to the calculated number of copies, the original video frame sequence is completely copied to obtain multiple intermediate video frame sequences. Each intermediate video frame sequence is completely identical to the original video frame sequence, consisting of genuinely captured original frames without interpolation or generative synthesis. For example, if the original video frame sequence contains 10 images and the number of copies is 8, the system will sequentially generate intermediate video frame sequences 1 through 8, each consisting of the same 10 frames. This lossless loop playback mechanism avoids the distortion artifacts that may be generated by interpolation algorithms in non-rigid dynamic scenes such as water ripples and wakes.

[0056] Then, the original video frame sequence and all intermediate video frame sequences can be sequentially concatenated in chronological order to obtain the target video frame sequence. Specifically, the concatenation order is: original video frame sequence, first intermediate video frame sequence, second intermediate video frame sequence, ... until the last intermediate video frame sequence. The concatenated target video frame sequence contains a continuous set of the original frames and all copied frames, with the total number of frames equal to the number of original frames multiplied by (number of copies + 1). Continuing with the above example, concatenating the original 10 frames with 8 intermediate sequences of 10 frames each yields a target video frame sequence containing 90 frames (in the actual example, if copied 8 times, the total number of frames is 10 × (8 + 1) = 90 frames).

[0057] Alternatively, the copied multi-frame intermediate video sequence can be spliced ​​together without splicing it with the original video frame sequence. It is sufficient that the duration of the target video frame sequence obtained after splicing the multi-frame intermediate video sequence is greater than or equal to the preset perception threshold.

[0058] In this way, the motion features of key targets (such as the relative position of ships and the flashing of navigation lights) are realistically repeated in the time dimension, which not only satisfies the downstream model's mandatory verification of input length, but also enhances the model's attention mechanism's ability to capture key features through repeated exposure.

[0059] Furthermore, the target video frame sequence can be re-encoded according to a preset frame rate to generate a video frame sequence corresponding to the target scene that can be directly used by downstream multimodal large models. The preset frame rate is consistent with the sampling rate of the original video stream (e.g., 25 frames / second) to ensure temporal continuity. The re-encoding process can package each frame in the target video frame sequence into a standard video container format (such as MP4 or RAW tensor) at the preset frame rate. For example, the system encodes a 90-frame sequence at 25 frames / second to generate a standardized video tensor with a duration of 3.6 seconds (90 / 25=3.6 seconds), which exceeds the perception threshold of 3.0 seconds and meets the model input requirements.

[0060] It should be noted that if the target duration of the original video stream is already greater than or equal to the perception threshold, there is no need to trigger the padding process; the original video stream will be directly encoded and output at the preset frame rate.

[0061] For example, in addition to using a fixed perception threshold, the perception threshold can be dynamically adjusted according to the semantic type of the decision instruction. Specifically, after acquiring the decision instruction, the system first uses a lightweight text classifier to determine whether the instruction involves high-speed moving target prediction (such as "collision risk" or "relative speed") or only involves static attribute recognition (such as "color" or "quantity"). For instructions involving high-speed dynamics, the system automatically increases the perception threshold to 4.0 seconds to capture longer motion trajectories; for static attribute instructions, the perception threshold is reduced to 1.0 seconds, or even a single frame image is used directly, thereby avoiding unnecessary temporal padding calculations and further optimizing the computing efficiency at the edge.

[0062] In some implementations, besides rounding up the ratio, a smooth transition mechanism can be introduced to determine the number of copies. For example, when the target duration of the original video stream is extremely short (e.g., less than 0.1 seconds), simple copying may lead to periodic artifacts (e.g., repeated flickering). In this case, the system can first perform slight time jitter resampling on the original frame sequence (e.g., fine-tune the display duration of each frame) before performing copying and splicing to break the strict periodic repetition, making the generated target video frame sequence visually closer to natural continuous motion, thereby reducing the risk of overfitting the downstream model to the repetitive pattern.

[0063] By using the above methods, when the video input duration is insufficient, the timing can be supplemented by looping and replaying real frames instead of interpolation synthesis. This avoids distortion artifacts and visual illusions in non-rigid dynamic scenes such as water ripples and wakes, thereby providing a visually realistic and time-complete standardized input for downstream multimodal reasoning modules, thus ensuring the accuracy and reliability of subsequent cognitive reasoning.

[0064] Step 102: When the task type indicated by the decision instruction is a logical reasoning task, at least one object to be analyzed is determined based on the decision instruction, and dynamic trajectory analysis and scene association analysis are performed on at least one object to be analyzed based on the video frame sequence using the first large language model to obtain the corresponding scene description information.

[0065] In some implementations, in order to overcome the problem of the lack of structured factual basis for reasoning in unmanned vessel assisted driving systems, after determining that the decision command belongs to a logical reasoning task, the command can be parsed first to determine the objects to be analyzed that need to be focused on, and then a lightweight first language model can be called to generate structured text describing the entity attributes, dynamic behaviors and contextual relationships in the current scene, thereby transforming the original visual time series data into a semantic representation that can be directly used by the knowledge retrieval and rule reasoning modules.

[0066] Among them, logical reasoning tasks can be query-type tasks that require combining video temporal dynamic information, object movement trends, and domain rule knowledge to perform causal analysis, compliance judgment, or decision generation, such as "What is the relative movement trend of the red ship to the right and is there a risk of collision?" or questions involving the application of collision avoidance rules.

[0067] The objects to be analyzed can be specific entities or sets of entities extracted from decision instructions that require key attention and tracking, such as "the red ship to the right" or "the buoy to the left". This can be used to guide the video analysis module to focus on the target related to the instructions and reduce the interference of irrelevant visual information.

[0068] Among them, the first major language model can be a visual language model with a small number of parameters that can run efficiently at the edge, such as a lightweight multimodal large language model with fewer than 3 billion parameters, which can be used to perform tasks with low computational overhead, such as task classification, scene description generation, decision result verification, and summary extraction.

[0069] Dynamic trajectory analysis can be a process of extracting and judging the motion parameters such as position, azimuth, distance, and speed of the object to be analyzed in time based on video frame sequences. For example, analyzing whether the azimuth of a ship is constant or whether the distance is decreasing can be used to judge whether there is a collision risk or to predict the motion trend.

[0070] Scene association analysis can be the process of resolving the spatial, relative motion, and semantic relationships between the object to be analyzed and other entities (such as other ships, buoys, channel boundaries, and obstacles) and environmental elements (such as weather, visibility, and water type) in the video frame sequence.

[0071] The scene description information can be text output in structured natural language form, generated after dynamic trajectory analysis and scene association analysis, such as "The weather is clear and the visibility is good. There is a red cargo ship at a constant bearing of 45 degrees to the starboard side."

[0072] Specifically, the primary language model (i.e., a lightweight visual language model) can be invoked to extract entities and perform semantic parsing on decision-making instructions, identifying specific entities that require focused attention and tracking. Parsing methods include: performing dependency parsing on the instruction text to extract noun phrases and pronoun references, and combining this with target detection results from the first frame of the video for entity alignment to eliminate ambiguity. For example, for the decision-making instruction "What is the relative motion trend of the red vessel to the right, and is there a collision risk?", the system extracts "red vessel" as the object to be analyzed and specifies its orientation attribute as "to the right." For instructions containing multiple entities, such as "Will there be a collision risk between the white yacht on the port side and the cargo ship on the starboard side?", the system identifies "white yacht" and "cargo ship" as two independent objects to be analyzed. If the instruction does not explicitly specify an object (e.g., "Which ship in the frame has the highest collision risk?"), the system defaults to using all dynamic targets detected in the video frame sequence as the objects to be analyzed.

[0073] Furthermore, the identified objects to be analyzed, along with the video frame sequence, can generate corresponding prompt information, which is then input into the first language model. This model performs dynamic trajectory analysis and scene association analysis on each object to be analyzed.

[0074] For example, prompts can be generated by constructing a spatiotemporal anchor point prompt template. Specifically, spatiotemporal trajectory anchor points of the object to be analyzed can be extracted from a video frame sequence (e.g., the initial bounding box coordinates in the first frame, the terminating bounding box coordinates in the last frame, and the discrete positions in intermediate keyframes), and these coordinate data can be mapped to "relative relationships of reference points" in natural language descriptions, such as "the target is located at 45 degrees to starboard and 200 meters away in the first frame, and at 42 degrees to starboard and 150 meters away in the last frame"; at the same time, the time granularity requirements (such as "within the last 3 seconds" or "from appearance to present") can be extracted from the decision instructions, and the time granularity can be embedded into the prompt template. Then, after the first language model receives the prompt information, it does not need to calculate the low-level optical flow or coordinate difference itself, but directly infers the motion trend (e.g., "the azimuth angle is basically constant, the distance is decreasing, and there is a tendency to approach") based on the "anchor point difference". This significantly reduces the inference complexity of large language models when performing dynamic trajectory analysis, while maintaining the accurate transmission of temporal information and avoiding trend misjudgment caused by the model's insensitivity to small movements between consecutive frames. It is especially suitable for low-latency analysis of long video sequences by lightweight models at the edge.

[0075] For example, dynamic trajectory analysis may include: extracting information such as the position coordinates, bounding box size, azimuth (relative to the ship's or image coordinate system), and distance change rate of the object to be analyzed in each frame of the video frame sequence, and calculating its temporal characteristics, such as direction of motion, velocity vector, whether the azimuth change is constant, and whether the distance is continuously decreasing. Scene association analysis may include: resolving the spatial relationships, relative motion relationships, and semantic relationships between the object to be analyzed and other entities (such as other ships, buoys, channel boundaries, and obstacles) and environmental elements (such as weather, visibility, and water type) in the video frame sequence. Specific analysis elements can be set according to actual conditions.

[0076] For example, for the "red vessel," the model calculates its bearing at 45 degrees to the starboard side of the main vessel and maintains this position through continuous frame optical flow tracking and depth estimation, with the distance gradually decreasing from 200 meters to 150 meters. Simultaneously, it identifies no other obstructions around the vessel, and the main vessel's course forms an angle of approximately 60 degrees with the red vessel's, constituting a typical encounter situation. The first major language model can convert the structured results of the above dynamic trajectory analysis and scene association analysis into natural language descriptions, generating scene description information.

[0077] For example, scene description information can be output in the form of structured natural language text, including an environmental overview, entity attributes, dynamic behaviors, and contextual relationships. For the navigation scenario described above, the generated scene description information would be: "Clear weather, good visibility. A red cargo ship is located at a 45-degree bearing to the starboard side of this ship. This ship is approaching this ship at a speed of approximately 8 knots and a constant bearing angle, with the distance decreasing from 200 meters to 150 meters. The relative heading angle between this ship and the red cargo ship is approximately 60 degrees, constituting a cross-encounter situation. There are no other obstacles to the starboard side of the red cargo ship, and the channel is open." By generating such structured scene description information, the system can transform raw visual time-series data into semantic facts that can be directly used by downstream rule retrieval and logical reasoning modules.

[0078] By using the above methods, raw, unstructured visual time-series data can be transformed into structured scene semantic descriptions that can be directly used by rule retrieval and logical reasoning modules, thereby providing a clear factual basis for subsequent multimodal reasoning and improving the system's cognitive reasoning ability and output interpretability in complex navigation scenarios.

[0079] In some implementations, to overcome the problem that existing unmanned surface vessel (USV) assisted driving systems use a unified, complex inference chain for all query commands, resulting in high computational resource consumption even for simple visual question answering and knowledge retrieval tasks, making it difficult to meet the real-time and low-power deployment requirements of edge devices, tasks can be divided into three categories—visual perception, knowledge retrieval, and logical reasoning—using a lightweight primary language model. For the first two categories, the lightweight model or rule base can be directly invoked to complete the response, bypassing the high-energy-consuming complex inference module. This achieves adaptive computational power scheduling based on query semantic type, thereby significantly reducing the overall system inference latency and power consumption. For example, before step 102, i.e., before "when the decision command corresponds to a logical reasoning task, determine at least one object to be analyzed based on the decision command," the following may also be included: (A.1) Determine the first video frame from the video frame sequence; (A.2) Using the first language model, the task is classified by combining the decision command and the first video frame to obtain the task classification result corresponding to the decision command. The type of the task classification result includes at least one of the following: visual perception task, knowledge retrieval task and logical reasoning task. (A.3) When the task classification result represents a decision instruction that corresponds to a visual perception task, the corresponding decision result is generated based on the decision instruction and video frame sequence through the first language model. (A.4) When the task classification result represents a decision instruction that corresponds to a knowledge retrieval task, semantic retrieval is performed in the preset domain rule base based on the decision instruction to obtain the corresponding decision result.

[0080] The first video frame can be the first frame in the time sequence of a video frame sequence, such as the image captured by the camera of an unmanned vessel at the beginning of the moment.

[0081] The task classification result can be a category label output by the first language model based on the semantic features of the decision instructions and the first video frame, such as a visual perception task, a knowledge retrieval task, or a logical reasoning task.

[0082] Among them, visual perception tasks can be query types that only require extracting static visual attributes or simple spatiotemporal features from video frames, such as "What color is the buoy on the left side of the screen?" or "How many ships are in the screen?"

[0083] Among them, knowledge retrieval tasks can be query types that require searching for and returning symbolic knowledge or rule interpretations from a domain rule base, such as what is the obligation of a vessel to give way in a cross-encounter situation under maritime collision avoidance rules.

[0084] Among them, the domain rule base can be a pre-built high-dimensional semantic vector database containing professional maritime rule knowledge.

[0085] Specifically, the index of the video frame sequence can be read by a video decoder, the frame data with index 0 can be extracted, and it can be scaled to a preset input size (e.g., 224×224 pixels). This first video frame serves as the key visual input for subsequent task classification, and is used to quickly extract the static semantic features of the scene without loading all video frames, thereby reducing the computational overhead of the classification stage.

[0086] Furthermore, the decision instruction and the first video frame can be input into the first large language model (i.e., a lightweight visual language model) to perform task classification. This allows the semantic type of the decision instruction to be determined at a very low computational cost before entering the complex inference chain, thereby achieving adaptive scheduling of computing resources and avoiding all queries from entering the high-energy-consuming logical inference module.

[0087] For example, after receiving the text command and the first frame image, the first language model performs multimodal feature fusion and a classification head (e.g., a linear layer plus Softmax) to output three probability distributions, and takes the category with the highest probability as the task classification result. The task classification result includes at least the following three types: visual perception tasks, which only require extracting static visual attributes (such as color, shape, quantity) or simple spatiotemporal features (such as location, existence) from the video to complete the query; knowledge retrieval tasks, which do not require video temporal analysis, but only need to search for and return symbolic knowledge or rule interpretations from the domain rule base; and logical reasoning tasks, which require complex queries that combine video temporal dynamics, target trajectory prediction, and rule constraints to perform causal analysis and compliance decisions.

[0088] For example, the system can determine the color of the buoy to the left of the instruction as a visual perception task; the obligation of the vessel to give way in a cross-encounter situation as a knowledge retrieval task; and whether there is a collision risk with the red vessel to the right of the instruction as a logical reasoning task.

[0089] In some implementations, when the task classification result represents a decision instruction corresponding to a visual perception task, the corresponding decision result can be directly generated through the first language model based on the decision instruction and the complete video frame sequence (or the first frame if only the first frame is needed for an answer). Specifically, the system formats the decision instruction and the video frame sequence (or the first frame) according to a visual question-and-answer template and inputs them into the visual question-and-answer interface of the first language model. The model directly outputs a concise answer without calling the knowledge retrieval module and complex reasoning module. For example, for "What color is the buoy on the left?", the first language model analyzes the left area in the first frame image and outputs "red"; for "How many ships are in the picture?", the model counts the targets in the video frame sequence and outputs "3 ships". The processing latency of this path is usually less than 1 second, significantly lower than that of the logical reasoning link, making it suitable for high-frequency, fragmented environmental queries.

[0090] In some implementations, when the task classification result represents a decision instruction corresponding to a knowledge retrieval task, semantic retrieval can be performed directly in a pre-defined domain rule base based on the decision instruction to obtain the corresponding decision result. Specifically, the system embeds the decision instruction into text, generates a query vector, calculates the cosine similarity in the rule base, recalls the Top-K rule clauses with the highest similarity (e.g., K=3), and concatenates the text of these clauses as the decision result, outputting them directly without going through a visual model. For example, for the instruction "What is the obligation of a vessel to give way in a cross-encounter situation?", the system retrieves: "When two motorized vessels cross-encounter and there is a risk of collision, the vessel with another vessel on its starboard side should give way to the other vessel." The system outputs the original text of this clause as the decision result. This path bypasses the visual processing module, achieving rule query with extremely low latency.

[0091] In some implementations, the task classification results may also include a fourth category: emergency avoidance tasks. When the instruction contains emergency keywords such as "immediately," "collision," or "danger," and a high-confidence collision threat is detected in the first video frame, the system directly marks the task as the highest priority, calling only the core inference module to generate emergency avoidance instructions (such as "immediately hard right steering") with minimal delay, and simultaneously triggering a warning signal. This significantly improves the system's response speed in sudden dangerous scenarios, meeting the real-time requirements of safety-critical situations.

[0092] By adopting the above methods, the waste of computing resources caused by using a unified re-inference link for all queries can be avoided, thus laying the foundation for efficient computing power scheduling and low-latency response of edge unmanned ship assisted driving systems.

[0093] Step 103: Combining the decision instructions and scenario description information, perform semantic retrieval in the preset domain rule base to obtain at least one target rule information.

[0094] In some implementations, decision instructions can be semantically concatenated with scene descriptions that describe real-time visual dynamics to form a joint query vector rich in contextual information. Then, a semantic similarity-based retrieval can be performed in a pre-defined domain rule base to recall the rule clauses most relevant to the current scene. This significantly improves the scene-specificity and matching accuracy of rule retrieval, thereby providing compliant and accurate rule basis for subsequent logical reasoning.

[0095] Among them, the target rule information can be one or more professional rule clauses retrieved from the preset domain rule base that are related to the semantics of the decision command and the current navigation scenario. These can be used as rule constraint inputs in the logical reasoning process to guide the compliance generation of decision results.

[0096] In some implementations, the generated scene description information can be semantically concatenated with the original decision instructions to form a context-rich joint query vector. Then, semantic retrieval is performed in a pre-defined domain rule base to recall at least one target rule information most relevant to the current navigation scenario. Specifically, a text embedding model (e.g., an encoder based on BERT or Sentence-Transformer) can be invoked to map the concatenated text "Decision Instruction: [User Instruction]; Current Scene: [Scene Description Information]" to a high-dimensional semantic space to generate the query vector. This pre-defined domain rule base is an offline vector database that stores text blocks of maritime rule clauses and their corresponding vector representations. The system can calculate the cosine similarity between the query vector and each rule vector in the database, sort them by similarity from high to low, and extract the Top-K rule texts as target rule information (e.g., K=3).

[0097] For example, the decision instruction is "Is there a collision risk with the red vessel to the right front?", and the scenario description is "Clear weather, good visibility. There is a red cargo ship at a 45-degree bearing to the right of this vessel, approaching at a constant bearing, decreasing in distance from 200 meters to 150 meters." The target rule information retrieved after combining these descriptions includes: "Rule 15 (Intersecting Encounters): When two motorized vessels intersect and there is a risk of collision, the vessel with another vessel on its starboard side shall give way to the other vessel" and "Rule 16 (Acts of the Giving Way Vessel): The vessel that must give way to another vessel shall, as early as possible, take large-scale action to allow sufficient clearance for the other vessel." By incorporating the dynamic scenario description into the search query, the system can accurately match rule clauses related to the current situation, avoiding the problem of rule and scenario disconnect that may occur when searching solely based on the original instruction.

[0098] For example, a visual anchor-weighted retrieval enhancement strategy can be introduced. Specifically, the system extracts key visual anchor terms (such as 45 degrees to starboard, constant azimuth, and distance reduction) from the scene description information and assigns these anchor terms higher retrieval weights than ordinary words (e.g., adjusting the dimensional weights of the query vector through TF-IDF or attention mechanisms). This makes the retrieval results more likely to include rule clauses describing motion trends and spatial relationships. In this way, the semantic alignment accuracy between the retrieval results and the dynamic navigation scene can be further improved, especially suitable for complex scenarios involving collision avoidance rules such as encounters and overtaking that depend on relative motion relationships.

[0099] By combining static user commands with dynamic visual scene descriptions in the above manner, joint semantic retrieval can be performed. This allows the retrieved rule information to accurately match the actual needs of the current navigation situation, avoiding the problem of rules being out of touch with the scene caused by relying solely on text command retrieval. This provides a rule basis for subsequent multimodal logical reasoning that is both compliant with domain norms and fits the real-time scene, thereby improving the rule compliance and scene adaptability of the decision results.

[0100] Step 104: Combine the decision instructions, video frame sequence, scene description information, and at least one target rule information to generate corresponding text prompt information.

[0101] In some implementations, to overcome the problem that existing general multimodal models lack structured guidance when directly inputting raw video and text instructions, making it difficult for the reasoning process to focus on rule constraints and scene facts, a text prompt message containing a fact observation-rule matching-logical deduction thought chain paradigm can be constructed by multi-source fusion and formatted organization of decision instructions, video frame sequences, scene description information, and retrieved target rule information. This prompt message serves as the input of the second language model, thereby guiding the model to perform compliant reasoning according to the expected cognitive path, and thus improving the interpretability and rule alignment of the output decision.

[0102] The text prompts can be formatted text inputs that integrate user query commands, visual time-series data, structured scene semantic descriptions, and relevant domain rules. For example, a prompt text organized according to a thought chain template such as "Fact observation: The red ship's position on the starboard side remains unchanged and the distance is decreasing; Rule matching: According to the rules of cross-encounter situations, this ship should give way to starboard; Please give a decision" can be used to input into the second language model to guide it to perform logical reasoning and decision generation under the constraints of the rules according to the expected cognitive path.

[0103] In some implementations, decision instructions, video frame sequences, scene description information, and at least one retrieved target rule information can be fused together to generate structured text prompts according to a preset thought chain template. Specifically, the video frame sequence can first be sparsely sampled (e.g., 4-8 frames are uniformly extracted) and encoded into the text description "Video Frame Sequence: [Frame 1 timestamp, Frame N timestamp]" to reduce text length while retaining the temporal range indication; then, the decision instructions, scene description information, and target rule information are concatenated according to a three-part template of fact observation-rule matching-logical deduction.

[0104] For example, the decision instruction is "Is there a collision risk with the red vessel to the right?", the scenario description is "A red cargo ship at 45 degrees to the starboard side, with a constant bearing, and the distance is decreasing from 200 meters to 150 meters", and the target rule is "Rule 15: In a cross-traffic situation, a vessel with another vessel on its starboard side should give way to that vessel." Therefore, the generated text prompt could be: "[Fact Observation] Based on the video frame sequence (3.2 seconds, 25 frames / second), a red cargo ship at 45 degrees to the starboard side is observed with a constant bearing, and the distance is decreasing from 200 meters to 150 meters. [Rule Matching] According to Rule 15 of the Maritime Collision Avoidance Regulations, in a cross-traffic situation with a collision risk, if another vessel is on the starboard side of this vessel, this vessel is the give-way vessel. [Logical Deduction] Based on the above facts and rules, please determine whether there is a collision risk and provide the avoidance decision this vessel should make." This text prompt can be encapsulated as a natural language text, serving as input to the subsequent second-largest language model, guiding it to perform compliant reasoning according to the expected cognitive path.

[0105] By using the above methods, multimodal heterogeneous information (natural language instructions, video frames, scene descriptions, and rule clauses) can be integrated into unified, structured text prompts. This provides a clear factual basis, rule constraints, and reasoning path guidance for the second language model, enabling the subsequent decision generation process to strictly follow the cognitive chain of observation-matching-inference, thereby significantly improving the interpretability of reasoning results and compliance with navigation rules.

[0106] In some implementations, to overcome the problem that relying solely on a local, fixed rule base cannot adapt to dynamically changing aquatic environments (such as temporary no-navigation zones, exercise areas, or sudden hydrological and meteorological changes), leading to decisions that contradict the latest navigation notices, the geographical coordinates of the edge computing node can be obtained. The retrieval vector generated by fusing decision instructions with scene description information, along with the coordinates, can be uploaded to the cloud node. The cloud then extracts real-time rules based on the notice rules associated with those coordinates within the target timeframe. The target rules retrieved locally are then merged with the cloud's real-time rules. Weighted description information with higher priority than local rules is generated for the real-time rules and concatenated into fused rule information. This assigns higher decision weight to dynamic rules when generating text prompts, ensuring that the unmanned vessel's decisions in complex and ever-changing waters always comply with the latest control requirements. For example, the reasoning decision-making method based on a large language model can be applied to edge computing nodes. Before step 104, it may further include: (B.1) Obtain the geographic coordinates of the edge computing nodes; (B.2) Obtain the retrieval vector generated by the decision instruction and scene description information, and upload the retrieval vector and geographic location coordinates to the cloud node so that the cloud node can extract at least one real-time rule information based on the notification rule information associated with the geographic location coordinates within the target time period; (B.3) Receive at least one real-time rule information returned by the cloud node; (B.4) Combine at least one target rule information and at least one real-time rule information to obtain at least one initial fusion rule information; (B.5) Generate corresponding rule weight description information for each initial fusion rule information, wherein the rule weight description information is used to describe that the real-time rule information has a higher priority in the reasoning process of the second language model than the corresponding target rule information; (B.6) Concatenate each initial fusion rule information with the corresponding rule weight description information to obtain the corresponding fusion rule information; Then, by combining the decision-making instructions, video frame sequence, scene description information, and at least one target rule information, corresponding text prompts are generated, including: By combining decision instructions, video frame sequences, scene description information, and information from each fusion rule, corresponding text prompts are generated.

[0107] Among them, edge computing nodes can be computing units deployed on the end of unmanned vessels, such as embedded AI acceleration chips equipped with lightweight visual language models and local rule vector libraries.

[0108] Among them, the geographic location coordinates can be the latitude and longitude information of the current location of the edge computing node, such as the coordinates (114.2° east longitude, 22.3° north latitude) obtained by the global positioning system carried by the unmanned ship. These coordinates can be used by cloud nodes to retrieve temporary navigation notices or regional control rules associated with that location.

[0109] The retrieval vector can be a numerical representation obtained by concatenating decision instructions and scene description information into text and mapping it to a high-dimensional space through a semantic coding model. For example, it can be a 1280-dimensional floating-point vector generated based on a Transformer encoder.

[0110] Among them, cloud nodes can be computing resources deployed on shore-based or cloud servers, such as remote servers with wide-area map databases and real-time navigation notice parsing capabilities. They can be used to receive retrieval vectors and geographic coordinates uploaded by edge computing nodes, extract the latest real-time rule information of that location, and return it to the edge computing nodes.

[0111] The target timeframe can be the time range condition that cloud nodes use when retrieving notification rule information, such as being valid for the current day or from the date of publication to within the next 72 hours. It can be used to filter out temporary rules related to the current sailing time window and avoid using expired notification information.

[0112] Among them, the information in the notices and rules can be temporary navigation regulations with timeliness and regionality issued by the maritime regulatory authorities, such as the clause in the "Notice to Navigators" of a port on a certain day regarding a speed limit of 5 knots in the construction area, or a notice of navigation prohibition in the exercise area, which can be used to supplement dynamic information not included in the local basic rule base.

[0113] Among them, real-time rule information can be one or more announcement rules extracted from cloud nodes that are related to the geographical coordinates of edge computing nodes and the timeliness of targets. For example, if the waters are temporarily closed to navigation due to an exercise or a section of the waterway has a speed limit of 8 knots, it can be used to integrate with local target rule information to reflect the latest regulatory requirements.

[0114] The initial fusion rule information can be a set of rules obtained by merging the target rule information retrieved locally with the real-time rule information returned by the cloud. For example, the local intersection yield rule and the cloud temporary no-fly zone notice can be placed in the same set, which can be used to add weight descriptions and generate the final fusion rule information.

[0115] The rule weight description information can be a natural language instruction or metadata, used to clearly indicate the priority preference of the second language model when different rules conflict. For example, it can prioritize the temporary notification rules returned by the cloud, and when cloud rules conflict with local basic rules, cloud rules shall prevail. This can be used to guide the inference model to give higher compliance weights to real-time rules in decision generation.

[0116] Among them, the fusion rule information can be a rule text with priority indication obtained by concatenating the initial fusion rule information with the corresponding rule weight description information. For example, "Temporary no-fly zone notice" and "Priority: This rule is higher than the local basic rule" are concatenated into a complete string.

[0117] Specifically, the system can use a global positioning module (such as GPS or BeiDou receiver) integrated on the edge computing node to read the current latitude and longitude information in real time at a preset frequency (such as 1Hz) and temporarily store it in memory. For example, when the unmanned vessel navigates to the vicinity of a port, the system obtains the geographical coordinates (114.2°E, 22.3°N). These coordinates are used to subsequently retrieve temporary navigation notices or regional regulatory rules related to that waterway from the cloud node.

[0118] Furthermore, the system can obtain a retrieval vector generated jointly by decision instructions and scene description information, and package this retrieval vector with geographic coordinates and upload it to the cloud node. Specifically, a text embedding model (such as a BERT-based encoder) can be invoked to semantically encode the concatenated decision instructions and scene description information to generate a high-dimensional floating-point vector as the retrieval vector.

[0119] Furthermore, the retrieval vector and geographic coordinates can be sent to the cloud node in JSON format via 4G / 5G or satellite communication links. Upon receiving the request, the cloud node, based on the target timeframe (e.g., "valid today" or "within 72 hours of publication") and the geographic coordinates, retrieves the notification rule information associated with those coordinates from the cloud-based wide-area map database, extracting at least one real-time rule. For example, the cloud node, based on the coordinates (114.2°E, 22.3°N), retrieves the Notice to Navigation issued by the maritime authorities that day: a speed limit of 5 knots in the port construction area and a no-navigation zone in the exercise area, and returns this information as real-time rule information.

[0120] In some implementations, the edge computing node receives at least one real-time rule information returned by the cloud node and merges it with at least one target rule information (i.e., local basic rule) obtained by retrieval to obtain at least one initial fusion rule information. The initial fusion rule information includes real-time rule information and target rule information given for the same type of navigation behavior or the same scene element.

[0121] For example, the target rule information is "In open waters, the maximum speed of vessels shall not exceed 10 knots" (basic rule), while the real-time rule information returned from the cloud is "Within the current construction area, the maximum speed of vessels shall not exceed 5 knots" (temporary notice). Both address the behavior of speed limits, but the values ​​conflict. The system can merge these two rules into an initial fusion rule information containing two rule texts. At this point, the system identifies the source of each rule: if it comes from the cloud (real-time rule information), it generates corresponding rule weight description information, such as "Priority: This rule is a temporary notice rule, and its validity is higher than the local basic rule. It should be followed first during inference." For example, for the construction area speed limit rule, the additional description is "This rule is a temporary notice valid for the day, and its priority is higher than the normal speed clause in the local basic collision avoidance rules." In this way, the system only explicitly embeds the rule weight description information as a prompt when rules conflict, guiding the large language model to correctly choose when a conflict occurs; if the real-time rule and the target rule do not conflict (e.g., they constrain different behaviors respectively), there is no need to generate a weight description, and the rule texts can be directly merged.

[0122] In some implementations, when there is no conflict between real-time rule information and target rule information—that is, when their requirements for the same navigation behavior are consistent or compatible and do not generate contradictory instructions—the system does not need to generate additional conflict resolution description information. In this case, the rule weight description information can be simplified to "This rule is a valid navigation basis; please follow it normally," or the weight description can be omitted entirely, and the merged rule list can be concatenated in its original order as the fused rule information. For example, if the local target rule information is "The vessel should maintain a proper lookout when visibility is good," and the cloud-based real-time rule information is "A temporary warning buoy is located 500 meters ahead of the construction area; be careful to avoid it," and there is no conflict, the system directly concatenates the two rules into "Maintain a proper lookout; be careful to avoid the temporary warning buoy 500 meters ahead of the construction area," and generates a text prompt message accordingly. This reduces the redundancy of the prompt information, avoids unnecessary priority labeling interfering with model inference, and maintains the complete transmission of all available rule information.

[0123] In some implementations, a timeliness weight value can be attached to each real-time rule (e.g., linearly decaying from 1.0 to 0.2 over time). When the rule's publication time exceeds half of the target timeliness, its priority description strength is automatically reduced, for example, changing "must be followed first" to "recommended for reference". This avoids the problem of outdated rules being assigned excessively high priority due to network latency or untimely rule updates, further improving the timeliness and security of decision-making.

[0124] By adopting the above methods, the decision-making process can automatically follow the latest navigation notices and regional regulatory requirements, solving the problem that the local fixed rule base cannot adapt to the dynamic changes in waters. This provides a reliable rule basis for the subsequent generation of compliant decisions that conform to the current actual regulatory environment, and significantly improves the system's safety and adaptability in complex and ever-changing waters.

[0125] Step 105: Using the second language model, logical reasoning is performed based on the text prompts to obtain the corresponding decision results. The parameter scale of the second language model is larger than that of the first language model.

[0126] In some implementations, textual prompts that integrate decision instructions, video frame sequences, scene descriptions, and rule clauses can be input into a second, larger language model. This model can then leverage its enhanced logical deduction and thought chain generation capabilities to perform deep reasoning, thereby generating decision results that conform to navigation rules and the actual scene, thus achieving a high-quality response to complex queries.

[0127] The second largest language model can be a visual language model with a large number of parameters and powerful logical reasoning and complex thought chain generation capabilities, such as a large-scale multimodal language model with more than 8 billion parameters. It can be deployed on edge computing nodes and share GPU memory with the first largest language model.

[0128] The decision results can be text outputs generated based on logical reasoning from text prompts, including intermediate reasoning steps and the final navigation decision. For example, "Observing that the red boat on the starboard side remains unchanged in bearing and the distance is decreasing, a cross-encounter situation is formed. According to the rules, this vessel should turn right to avoid crossing in front of the other vessel," or machine-readable instructions after being summarized and refined, such as "There is a risk of collision. This vessel must turn right and pass through the stern of the red boat." These can be directly output to the user interface or the unmanned vessel motion control system to achieve assisted driving decision-making.

[0129] In some implementations, the second language model can receive formatted text prompts and then perform step-by-step reasoning according to a pre-defined thought chain template (such as "fact observation - rule matching - logical deduction") within the prompts, ultimately outputting a text result containing intermediate reasoning steps and the final navigation decision. For example, the text prompt might be: "[Fact Observation] According to the video frame sequence, the red cargo ship's bearing at 45 degrees to starboard remains constant, and the distance decreases from 200 meters to 150 meters. [Rule Matching] According to Rule 15 of the International Regulations for Preventing Collisions at Sea (COCR), when a cross-ship encounter poses a collision risk, this vessel is the give-way vessel. [Logical Deduction] Please determine if there is a collision risk and provide a avoidance decision." The second language model can then output the decision result: "Reasoning: The red cargo ship's bearing is constant and the distance is continuously decreasing, indicating a collision risk. According to Rule 15, this vessel should act as the give-way vessel and make a sharp right turn to avoid crossing in front of the other vessel. Decision: Immediately turn 20 degrees to starboard and pass behind the red ship." The decision result includes both an interpretable thought process and directly executable motion instructions. By employing a model with a larger number of parameters, the system can handle causal inference and rule trade-offs in complex scenarios, compensating for the shortcomings of lightweight models in deep reasoning capabilities.

[0130] For example, besides directly using the second language model to output the decision results, a self-verification mechanism can be introduced. The decision results are only output using the second language model after successful verification. Specifically, when generating the decision results, the second language model requires that its output format explicitly include three fields: [Observation Basis], [Applicable Rules], and [Action Decision]. The system then extracts these three fields using regular expressions and checks whether the [Action Decision] contains an action that matches the keyword in the [Applicable Rules] (e.g., if the rule contains "turn right," does the decision contain "turn right"?). If there is no match, regeneration is automatically triggered. This further improves the semantic alignment between the decision results and the rule clauses, reducing rule execution deviations caused by the model's arbitrary generation.

[0131] In some implementations, after obtaining the decision result, the system can further refine and extract the decision result to generate simplified instructions that can be directly used by the unmanned vessel's underlying motion control system or low-bandwidth communication link. Specifically, the system calls the first major language model (i.e., the lightweight model) as the terminal extraction unit. The lengthy decision result generated by the second major language model (containing a detailed thought chain reasoning process, such as "Reasoning: The red cargo ship's azimuth is constant and the distance is continuously decreasing, posing a collision risk. According to Rule 15, this vessel should act as the yielding vessel and take a sharp right turn to avoid crossing in front of the other vessel. Decision: Immediately turn 20 degrees to the right and pass behind the red ship") is input into this extraction unit, along with an extraction instruction template, such as "Please ignore the above reasoning process and only output the final machine-readable decision instruction, in the format 'Action: Direction + Amplitude', not exceeding 20 Chinese characters." The first major language model then outputs the simplified result, such as "Action: Turn 20 degrees to the right" or "Instruction: Turn to the right and pass behind the red ship." This reduces the bandwidth requirements for information transmission, making it easier to transmit decision results back over long distances or in low-quality communication channels (such as satellite links). On the other hand, the standardized output instructions can be directly parsed and executed by the programmable logic controller of the unmanned vessel without manual intervention or secondary conversion, thereby significantly improving the interface compatibility and response speed between the system and the underlying motion control system.

[0132] This application embodiment acquires decision instructions input for a target scene and a corresponding video frame sequence. When the decision instructions correspond to a logical reasoning task, at least one object to be analyzed is determined based on the decision instructions. A first large language model is used to perform dynamic trajectory analysis and scene association analysis on the at least one object to be analyzed based on the video frame sequence, obtaining corresponding scene description information. Combining the decision instructions and scene description information, semantic retrieval is performed in a preset domain rule base to obtain at least one target rule information. Combining the decision instructions, video frame sequence, scene description information, and at least one target rule information, corresponding text prompt information is generated. A second large language model is used to perform logical reasoning based on the text prompt information to obtain the corresponding decision result. The parameter scale of the second large language model is larger than that of the first large language model. Thus, through a two-stage reasoning architecture driven by video frame sequence-based dynamic trajectory analysis and scene association analysis, semantic retrieval and target rule information injection under the constraints of the domain rule base, and collaboration between large and small models, a deep understanding of the movement trend of targets in a dynamic environment, compliant integration of professional domain rules, and accurate execution of complex logical reasoning can be achieved. Specifically, compared to the limitations of related technologies that only perform static pattern matching on single-frame images and cannot capture temporal dynamic information, this application first utilizes a first-large language model to perform dynamic trajectory analysis and scene association analysis on video frame sequences, transforming discrete visual frames into structured scene description information containing entity motion attributes and spatiotemporal evolution relationships, thereby effectively modeling target motion trends and event evolution processes. Simultaneously, by performing semantic retrieval of decision instructions and scene description information in a domain rule base, target rule information is injected into the reasoning process in the form of decision constraint information, avoiding the problem of pure visual perception output being disconnected from professional rules. Finally, a second-larger language model with a larger parameter scale performs logical reasoning based on the multimodal fusion of video frame sequences, scene description information, and decision constraint information, ensuring that the decision results conform to both visual facts and domain rule constraints. In summary, this application can improve the accuracy of decision results for dynamic environments.

[0133] In some implementations, to overcome the problem that large language models may produce illusory outputs that are inconsistent with visual facts or violate maritime rules in safety-critical navigation scenarios, and lack the ability to assess the reliability of their own outputs and autonomously correct them, a first large language model can be used to perform visual consistency analysis of the decision results with the original video frame sequence and logical compliance analysis with the target rule information, respectively. The results of these two analyses are then fused to generate a quantified reasoning confidence level. When the confidence level is lower than a preset safety threshold, iterative re-reasoning based on the original input information is automatically triggered until a corrected decision with acceptable confidence is generated. This constructs a closed-loop self-reflective mechanism of generation-verification-correction, effectively suppressing illusory phenomena and ensuring the visual authenticity and rule compliance of the output results. For example, after step 105, the following may also be included: (C.1) Using the first major language model, visual consistency analysis of the decision results is performed based on the video frame sequence to obtain the first analysis result; (C.2) Using the first major language model, a logical compliance analysis is performed on the decision results based on at least one target rule information to obtain the second analysis result; (C.3) Using the first language model, based on the results of the first and second analyses, determine the inference confidence level corresponding to the decision result; (C.4) When the inference confidence is less than the preset confidence threshold, an intermediate decision result is generated based on the decision instruction, video frame sequence and scene description information, and the inference confidence of the intermediate decision result is determined until the inference confidence corresponding to the intermediate decision result is greater than or equal to the preset confidence threshold. Then, the corresponding intermediate decision result is determined as the updated decision result corresponding to the decision instruction.

[0134] The first analysis result can be the evaluation information output by the first language model after performing visual consistency verification on the decision result based on the video frame sequence. For example, it can determine whether the "red ship" described in the decision result actually exists in the video frame sequence, or whether the "unchanged orientation" movement trend is consistent with the visual observation. This can be used to quantify the degree of matching between the decision result and the original visual input.

[0135] The second analysis result can be the evaluation information output by the first language model after performing a logical compliance check on the decision result based on at least one target rule information. For example, it can check whether the decision result "this vessel should give way to the right" is consistent with the retrieved cross-encounter situation rules, or whether the key constraint "avoid crossing in front of other vessels" has been omitted. It can be used to quantify the degree of alignment between the decision result and the maritime rules.

[0136] The reasoning confidence score can be a comprehensive score generated by integrating the results of the first and second analyses, such as 0.45 or 0.80, which is used to characterize the overall reliability of the decision results in terms of both visual authenticity and rule compliance.

[0137] The preset confidence threshold can be a pre-set minimum confidence score used to determine whether the decision result meets the safe release conditions, such as 0.6. When the inference confidence is lower than this threshold, the system intercepts the current decision result and initiates a retry and correction mechanism.

[0138] Intermediate decision results can be temporary decision outputs generated by the system based on the same decision instructions, video frame sequences, and scene description information after the initial decision result fails the confidence verification. These outputs can be used to replace the original decision result and undergo confidence evaluation again until a preset threshold is reached, at which point they become the final updated decision result.

[0139] In some implementations, a first language model can be invoked to perform visual consistency analysis on the generated decision results based on the original video frame sequence, yielding a first analysis result. Specifically, the first language model receives the video frame sequence (or keyframe sampling) and the decision result text, and checks item by item whether the scene elements, object attributes, and dynamic behaviors described in the decision match the visual facts in the video frame sequence. Analysis dimensions include: whether the target exists (e.g., whether the "red ship" mentioned in the decision appears in the video), whether attributes match (e.g., color, shape, position), and whether the movement trend is consistent (e.g., whether "constant azimuth" and "distance decreasing" match the observed values). For example, if the decision result is "the red cargo ship is located at 45 degrees to starboard, and the distance is decreasing," the verification model compares the ship's trajectory data in the video frame sequence, confirming that the azimuth remains within the range of 43-47 degrees and the distance decreases from 200 meters to 150 meters, giving the first analysis result as "visual consistency passed, deviation within tolerance range"; if the decision incorrectly describes it as "port side," then the first analysis result is "visual anomaly: the target is actually located on the starboard side, the decision misjudged it as port side."

[0140] Furthermore, using the same primary language model, a logical compliance analysis can be performed on the decision result based on at least one target rule information retrieved, yielding a second analysis result. Specifically, the verification model can compare the decision result with the target rule information line by line, checking whether the rules referenced in the decision are accurate, whether key actions are omitted, and whether there are any instructions that contradict the rules. For example, the target rule information includes "In a cross-encounter situation, this vessel should turn right" and "Avoid crossing in front of other vessels." The decision result is "This vessel should turn right." After analysis, the verification model concludes that the decision includes a right turn action but omits the constraint "avoid crossing," giving a second analysis result of "Rule compliance partially passed, key constraint 'avoid crossing' missing." Another example: if the decision result is "This vessel should turn left," then the second analysis result is "Rule violation: The decision contradicts the right turn direction required by the rule."

[0141] In some implementations, the reasoning confidence level corresponding to the decision outcome can be determined using a first language model, based on the first and second analysis results. Specifically, weights can be assigned to the two analysis results (e.g., visual consistency weight 0.4, rule compliance weight 0.6), and the confidence level can be obtained by weighted summation. Alternatively, a natural language guidance model can be used to directly output a score, for example, prompting "Please provide a comprehensive confidence level based on visual consistency (maximum score 0.4) and rule compliance (maximum score 0.6)."

[0142] For example, if the first analysis result is a complete pass (0.4) and the second analysis result is a partial pass (0.3, due to missing constraints), then the first language model might output a comprehensive confidence score of 0.7. If the first analysis result indicates a visual anomaly (0.1) and the second analysis result indicates a rule violation (0), then the first language model might output a confidence score of 0.1. The system presets a confidence threshold, such as 0.6, to determine whether the decision result is reliable.

[0143] In some implementations, the first language model can be used to simultaneously perform visual consistency analysis and logical compliance analysis on the decision results, and directly output the corresponding reasoning confidence.

[0144] In some implementations, when the inference confidence level is less than a preset confidence threshold, the system triggers a correction process, generating an intermediate decision result based on the decision instructions, video frame sequences, and scene description information. Specifically, a second large language model (with a larger number of parameters, responsible for complex inference) can be invoked, combining the original decision instructions, video frame sequences, and scene description information, and expanding the rule retrieval scope and / or adding abnormal constraint information to re-perform logical inference and generate a new intermediate decision result. For example, if the initial decision has a confidence level of 0.55 (<0.6) due to the omission of "avoid crossing in front of other ships," the system will forcibly increase the number of retrievals from K=3 to K=5 during the second inference, and add the constraint "must include the action of avoiding crossing" in the prompt, generating an intermediate decision result of "this ship turns 30 degrees to the right, passes behind the red ship, and avoids crossing in front of it."

[0145] In some implementations, after generating intermediate decision results, the primary language model can be invoked again to re-verify the intermediate decision results in the manner described above, determining the inference confidence corresponding to the intermediate decision results. If it is still less than the threshold, the iteration continues (e.g., further expanding the search scope or adjusting the weight description) until the inference confidence of a certain version of the intermediate decision result is greater than or equal to the preset confidence threshold. At this point, the intermediate decision result is determined as the final updated decision result. In this way, the system can effectively suppress hallucination output, ensuring that the final decision meets safety standards in terms of both visual realism and rule compliance.

[0146] In some implementations, iterative correction can also employ a differentiated seed strategy. That is, during each correction, some non-core descriptive words can be randomly discarded or replaced in the scene description information (e.g., changing "sunny" to a synonym for "good visibility"), generating multiple fine-tuned versions of text prompts. These are then input into the second-largest language model to obtain multiple candidate intermediate decision results, and the one with the highest confidence is selected. This diversity sampling helps avoid local error reasoning paths and improves correction efficiency.

[0147] By using the above methods, a separate lightweight model can be used to perform visual and rule-based dual verification of the decision results generated by the main model. This enables the quantitative identification and automatic interception of unreliable outputs, and triggers a closed-loop correction process when the confidence level is insufficient. This reduces the fatal error rate from more than 40% in the current technology to about 3%, thereby providing a reliable quality assurance for the final decision to be output to the unmanned vessel motion control system in a safe and compliant manner.

[0148] In some implementations, to avoid insufficient decision confidence due to a narrow rule retrieval scope or omission of key constraints during the initial reasoning, and since simply repeating the same input cannot fundamentally solve the problem of missing information, the recall of semantic retrieval can be increased (e.g., from Top-3 to Top-5) when the correction process is triggered. More relevant rule clauses can be injected into the text prompt information, and the second language model can be called again for deep reasoning to generate a more complete and compliant intermediate decision result. This achieves effective closed-loop self-correction through dynamically expanding the knowledge context, thereby significantly improving the rule completeness and security reliability of the final output. For example, "generating intermediate decision results based on decision instructions, video frame sequences, and scene description information" in (C.4) may include: (C.4.1) Combining decision instructions and scenario description information, semantic retrieval is performed in a preset domain rule base to obtain multiple corresponding update target rule information, wherein the number of multiple update target rule information is greater than the number of at least one target rule information; (C.4.2) Combine decision instructions, video frame sequences, scene description information, and multiple update target rule information to generate corresponding update text prompts; (C.4.3) Using the second language model, logical reasoning is performed based on updated text prompts to generate intermediate decision results.

[0149] Among them, updating the target rule information can be a set of rule clauses that is more numerous than the target rule information retrieved in the first retrieval, obtained by expanding the semantic retrieval scope after triggering the correction process. For example, missing clauses such as "avoid crossing in front of other ships" can be obtained by increasing the recall quantity from Top-3 to Top-5. This can be used to provide a more comprehensive rule context for re-reasoning, so as to make up for the omission of key constraints caused by insufficient retrieval scope in the first decision.

[0150] The updated text prompt can be a formatted text that integrates decision instructions, video frame sequences, scene description information, and multiple update target rule information. Specifically, it can be a thought chain template that is additionally spliced ​​with the newly added rule clauses on the basis of the initial text prompt.

[0151] In some implementations, when the confidence level of the initial inference decision falls below a preset threshold, triggering a correction process, the system again combines the original decision instructions and scenario description information to perform a semantic search in a preset domain rule base. However, this time the search scope is dynamically expanded to obtain a greater number of updated target rule information than the initial search. Specifically, the retrieval recall parameter K can be increased from the default value of the initial search (e.g., K=3) to an expanded value (e.g., K=5 or K=7), significantly increasing the number of rule clauses returned by the search. For example, the initial search only recalls three rules, such as "give way in cross-encounter situations" and "avoid crossing," while the corrected search expands the K value to 5, additionally recalling two supplementary rules, such as "ships should maintain a safe speed" and "lookout duty." This greater number of updated target rule information aims to compensate for key constraint clauses that may have been missed in the initial inference due to an overly narrow search scope, providing a more complete rule context for re-inference.

[0152] Furthermore, decision instructions, video frame sequences, scene description information, and the multiple update target rule information can be fused to generate corresponding update text prompts. Structurally, these prompts can contain more rule paragraphs and selectively include abnormal constraint information (such as directional correction instructions like "must include avoid crossing action"). For example, the update text prompts can be organized as follows: "[Fact Observation] According to the video frame sequence, the red cargo ship's azimuth angle at 45 degrees to starboard is constant, and the distance is decreasing. [Rule Matching] Based on the following rules: Rule 1 (give way at intersections), Rule 2 (avoid crossing), Rule 3 (maintain safe speed), Rule 4 (obligation to lookout). [Correction Constraint] Note: The initial decision omitted 'avoid crossing in front of other vessels'; this reasoning must include this action. [Logical Deduction] Please reassess the collision risk and provide a complete avoidance decision." By injecting more rule clauses and abnormal constraint information into the prompts, a more sufficient basis is provided for the secondary reasoning of the second language model.

[0153] Furthermore, the updated text prompt can be input into the second language model, which then performs logical reasoning based on the expanded rule context and corrected constraints to generate an intermediate decision. For example, after receiving the updated prompt, the second language model outputs the following intermediate decision: "Reasoning: The red cargo ship's bearing is constant and the distance is continuously decreasing, indicating a collision risk. According to the intersection yield rule, this ship is the yielding ship; according to the avoidance of crossing rule, this ship must not cross in front of the red ship. Decision: Immediately turn 30 degrees to the right, pass behind the red ship, and simultaneously reduce speed to a safe speed to avoid it." This intermediate decision then enters the verification stage of C.1-C.3. If the confidence level meets the standard, it becomes the final updated decision. By dynamically expanding the search scope and re-reasoning, the system can effectively compensate for rule deficiencies that may occur during the initial reasoning, achieving closed-loop self-correction.

[0154] By using the above methods, the recall scope of rule retrieval can be proactively expanded when the confidence of the initial inference is insufficient, thereby avoiding rule omissions during the inference process and achieving effective correction of low-confidence results.

[0155] In some implementations, specific visual anomalies that contradict the video facts can be extracted from visual consistency analysis results, and specific rule omissions or violations in the decision results can be extracted from logical compliance analysis results. These two types of defects can then be transformed into structured anomaly constraint information (e.g., "must include actions to avoid crossing in front of other vessels" or "ignore incorrect orientation judgments"). This constraint information is then injected as an additional condition into the updated text prompt information, thereby guiding the second language model to proactively avoid discovered error types during secondary reasoning, achieving precise and efficient targeted correction, thus avoiding computational waste caused by blindly expanding the search scope and improving the correction success rate. For example, before (C.4.2), that is, before "combining decision instructions, video frame sequences, scene description information, and multiple update target rule information to generate corresponding updated text prompt information," it may also include: (D.1) Based on the results of the first analysis, determine the visual anomaly information contained in the decision results; (D.2) Based on the results of the second analysis, determine the missing rule information contained in the decision results; (D.3) Generate anomaly constraint information based on visual anomaly information and rule missing information; Then, by combining the decision command, video frame sequence, scene description information, and multiple update target rule information, corresponding update text prompts are generated, including: By combining decision instructions, video frame sequences, scene description information, multiple update target rule information, and abnormal constraint information, corresponding update text prompts are generated.

[0156] Among them, visual anomaly information can be specific information that is inconsistent or contradictory between the content described in the decision result identified in the first analysis and the actual visual facts in the video frame sequence. For example, the decision result may have omitted the key action of "avoiding crossing in front of other ships" or incorrectly described the direction of movement of the target ship.

[0157] Among them, missing rule information may be specific information identified in the second analysis results that is missing or violated between the rule clauses cited in the decision results and the target rule information. For example, the decision results may not include key constraints in the retrieved collision avoidance rules (such as "giving way vessel shall not cross in front of another vessel"), or may misinterpret the applicable circumstances of the rules.

[0158] Among them, the abnormal constraint information can be a targeted correction instruction expressed in natural language, generated based on visual abnormal information and rule missing information. For example, it can be "Note: When making a correction decision, the action constraint of 'avoid crossing in front of other ships' must be included" or "Suppress the incorrect keyword 'turn left' in the first reasoning". It can be used to inject update text prompt information together with the update target rule information to guide the second language model to actively avoid the identified error types in the second reasoning.

[0159] In some implementations, the initial analysis results can be presented in natural language or structured fields, such as "Visual consistency analysis: Target vessel bearing description error (decision states port side, actual starboard side)", "Inconsistent movement trend (decision states distance increasing, actual decreasing)", or "Target does not exist (decision mentions red buoy, no red buoy in video)". The system extracts these specific discrepancies into a list of visual anomalies through keyword extraction or field parsing. For example, if the initial decision result is "Red vessel on port side is moving away", while the first analysis result is "Target is actually on starboard side and distance is decreasing", then the system determines the visual anomaly information as: "Bearing error: Actual starboard side, decision mistakenly port side; Movement trend error: Actual distance decreasing, decision mistakenly moving away". This information will be stored in a structured manner for subsequent generation of directional correction instructions.

[0160] For example, the second analysis result could be "Rule compliance: The decision mentions giving way to the right, but omits the mandatory constraint of 'avoiding crossing in front of other vessels'", or "The decision does not cite applicable safe speed rules". The system generates a list of missing rule information by parsing these descriptions. For instance, the target rule information might include "When meeting at an intersection, the vessel giving way should turn right and must not cross in front of other vessels," but the decision result only outputs "give way to the right," then the missing rule information would be: "Missing rule constraint: Avoid crossing in front of other vessels". If the decision result outputs an action contrary to the rule (such as "turn left"), then the missing rule information would be: "Rule violation: The decision requires a left turn, contrary to the rule requiring a right turn."

[0161] Furthermore, based on the aforementioned visual anomaly information and rule missing information, abnormal constraint information can be generated. This generation can be achieved through template concatenation, for example, "Note: Correct the following errors: Visual part: {Visual anomaly information}; Rule part: {Rule missing information}. During secondary reasoning, these errors must be corrected, and it must be ensured that similar problems do not recur in new decisions."

[0162] For example, for visual anomalies such as "azimuth misjudged as port" and missing rules such as "omission of avoidance of crossing," the generated anomaly constraint information is: "Corrected constraint: The actual target is on the starboard side, and 'port' must not appear in the decision; it must explicitly include the action of 'avoid crossing in front of other vessels.'" Then, when generating update text prompt information (corresponding to C.4.2), the system integrates the decision instruction, video frame sequence, scene description information, multiple update target rule information, and this anomaly constraint information to form a complete update text prompt. An example update prompt is: "[Fact Observation]...[Rule Matching]...[Corrected Constraint] The actual target is on the starboard side and must include the action of 'avoid crossing.' [Logical Deduction] Please regenerate the decision." In this way, the anomaly constraint information can accurately correct deviations and avoid repeating the same errors in secondary reasoning.

[0163] In some embodiments, temporal causal anomaly information can also be introduced as an additional source of anomaly constraints. Specifically, a temporal causal analysis module can be invoked (which can utilize a first large language model combined with timestamp encoding or a dedicated causal reasoning model) to perform event temporal extraction and temporal causal consistency verification on the video frame sequence. Based on the timestamp information of each frame in the video frame sequence and the environmental state changes related to the object to be analyzed, the temporal causal analysis module can detect the occurrence times of key events (such as "the ship starts to turn", "the other ship enters the safety distance of this ship", "course change", "whistle warning", etc.), so as to obtain objective and quantified objective event temporal facts (for example, "event X occurs at time t1, event Y occurs at time t2, and t1 < t2"). Then, the module analyzes the implicit causal order statement from the decision result (for example, the implicit causal order of "after this ship turns right, the other ship immediately turns left to avoid" is: this ship turns right first, and the other ship turns left later). Compare the implicit causal order in the decision result with the objective event temporal facts: if the decision result claims that "A causes B" and thus implies that A should occur before B, but the objective time sequence shows that B occurs before A, then a temporal causal anomaly is determined. Exemplarily, the decision result is described as "after this ship turns right, the other ship immediately turns left to avoid, proving that this ship's turn effectively avoids a collision", while the actual video observation shows that the start timestamp of the other ship turning left is earlier than the start timestamp of this ship turning right, that is, the objective time sequence is "the other ship turns left first and this ship turns right later", which is completely opposite to the causal order in the decision result. At this time, the system generates temporal causal anomaly constraint information: "Temporal causal correction: The objective video time sequence shows that the other ship turns left before this ship turns right, and the causal relationship order of 'this ship's turn causes the other ship to avoid' in the decision is incorrect. When performing secondary reasoning, the causal logic must be corrected, and it cannot be asserted that this ship's turn triggers the other ship's action." By injecting such temporal causal anomaly information into the updated text prompt information, it can guide the second large language model to correct the implicit incorrect temporal assumptions in the reasoning, especially applicable to navigation scenarios with strong temporal dependencies such as determination of avoidance timing, division of collision liability, and discrimination of active and passive behaviors.

[0164] In some implementations, the system can also introduce "attention hotspot anomaly information" as a source of anomaly constraints. Specifically, when the first language model performs visual consistency analysis on the decision results, the attention weight map of the model during the processing of the video frame sequence can be extracted simultaneously (e.g., by extracting the cross-attention matrix of the last layer of the Transformer model and mapping the average attention value to the two-dimensional space of the input frame). The system compares the highlighted areas in the attention weight map (i.e., the target areas that the model focuses on) with the target areas actually described in the decision results to identify key areas that were ignored by the model in the first inference but are visually significant. For example, the decision results only focus on the red cargo ship directly in front of the ship on the starboard side, while the attention heatmap shows a small fishing boat in the distance on the starboard side that was not effectively focused on by the model. The system then generates attention hotspot anomaly constraint information: "Attention blind spot correction: There is a small fishing boat in the distance on the starboard side of the video frame sequence that was not focused on in the first inference. In the second inference, the fishing boat must be included in the collision risk assessment, and it must be determined whether there is a potential conflict between it and the ship." By generating anomaly information based on attention visualization, the system can start from the model's own perception blind spots and force secondary reasoning to cover the missed key visual targets, thereby significantly improving the completeness of situational awareness in complex aquatic environments.

[0165] In some implementations, besides directly concatenating abnormal constraint information into the update text prompt, a "constraint-weighted sampling" mechanism can also be used. Specifically, the system assigns different enforcement weights to multiple abnormal constraint pieces (e.g., "must include avoid crossing" is high weight, "pay attention to starboard" is medium weight), and then, when generating the update text prompt, the high-weight constraints are repeated multiple times (e.g., repeated three times in the prompt), or placed at the beginning and end of the prompt (the beginning and end positions in the prompt have the greatest influence on the model). This constraint strengthening strategy can effectively improve the model's compliance probability with key correction instructions and reduce the risk of failure in the second iteration.

[0166] By using the above methods, the secondary reasoning process can proactively avoid identified defects, rather than blindly expanding the search scope or simply repeating the same input, thereby significantly improving the correction efficiency and reducing the number of iterations, and thus ensuring that compliant and reliable final decision results are generated within a single or limited number of corrections.

[0167] Please refer to Figure 3 In some implementations, combined with Figure 3 This paper introduces the overall process of reasoning and decision-making based on a large language model in this application.

[0168] This embodiment uses the scenario of an unmanned surface vessel (USV) encountering a cross-entry situation in complex waters as an example to fully demonstrate the system's multimodal collaboration and closed-loop self-reflection process from input to output. The system is deployed on the edge computing node of the USV and is equipped with an AI acceleration chip that supports a unified memory architecture. The first large language model (less than 3 billion parameters) and the second large language model (more than 8 billion parameters) are both loaded into the video memory using 4-bit quantization technology.

[0169] Furthermore, the system receives the decision command sent from the shore-based system: "What is the relative motion trend of the red vessel to the right front, and is there a risk of collision?" Simultaneously, it acquires a 3.5-second raw video stream captured by the bow camera. Since the target duration (3.5 seconds) exceeds the preset perception threshold (2.0 seconds), the preprocessing module directly formats it into a standard video frame sequence, eliminating the need to trigger a loop completion mechanism.

[0170] The system then inputs the decision command and the first frame of the video into the routing agent (implemented by the first large language model). The routing agent extracts semantic features and determines that the command contains the keywords "relative motion" and "collision risk," belonging to a "logical reasoning task" that requires temporal analysis and rule verification. Therefore, the data flow is directed to the main reasoning link. If it is a visual perception task (such as buoy color) or a knowledge retrieval task (such as rule interpretation), the first large language model responds directly or only retrieves the rule base and returns the result, thus bypassing the high-energy-consuming module.

[0171] Furthermore, the video frame sequence can be fed in parallel into the shipping scene description unit (implemented by the first large language model). This unit decomposes the video into structured text and outputs scene description information: "The weather is clear and visibility is good. There is a red cargo ship at a constant bearing of 45 degrees to the starboard side of this ship, approaching this ship at a constant bearing angle, with the distance decreasing from 200 meters to 150 meters."

[0172] In some implementations, the shipping knowledge enhancement retrieval module can semantically concatenate the decision instructions with the above-mentioned scenario description information to generate a rich context query vector, and perform cosine similarity retrieval in the locally pre-loaded high-dimensional feature vector library (with built-in vectorized blocks of the "Marine Collision Avoidance Rules") to recall the top-3 target rule information with the highest similarity, such as the cross-encounter situation rule, the yielding vessel obligation rule, and the rule to avoid crossing in front of other vessels.

[0173] Furthermore, the multimodal fusion unit can fuse video frame sequences, scene description information, target rule information, and original decision instructions, generating text prompts according to a thought chain template of fact observation-rule matching-logical deduction. The reasoning module (implemented by the second large language model) receives this prompt, performs deep logical reasoning, and outputs the initial decision result: "Reasoning: Observing that the red boat's bearing on the starboard side remains unchanged and the distance is decreasing, a cross-encounter situation is formed. According to the rules, this ship should give way to starboard."

[0174] Specifically, the scoring agent (implemented by the first major language model) can independently read video frame sequences, target rule information, and initial decision results. On one hand, it can perform visual consistency analysis to confirm that the existence, azimuth, and distance changes of the target in the decision are consistent with the video facts; on the other hand, it can perform logical compliance analysis, discovering that the decision result omits a key constraint to avoid crossing in front of other boats, and giving a second analysis result of rule missing. The scoring agent, combining visual and rule dimensions, outputs an inference confidence score of 0.45 (below the preset safety threshold of 0.6).

[0175] In some implementations, if the confidence level is insufficient, the system can trigger a correction process. Specifically, an expansion command can be sent to the shipping knowledge enhancement retrieval module to increase the retrieval quantity to the Top-5, supplementing missing rules such as avoiding crossing; simultaneously, visual anomalies and missing rule information are used to generate anomaly constraint information. The inference module re-infers based on the amplified rule context and anomaly constraint information, generating an intermediate decision result: "This vessel should turn 30 degrees to the right, pass behind the red vessel, and avoid crossing in front of it." The scoring agent verifies again, and if the confidence level increases to 0.80, passage is granted.

[0176] In some implementations, the summarizing agent (implemented by the first major language model) can receive verified intermediate decision results, omitting lengthy intermediate steps in the thought process and forcibly distilling them into a single machine-readable instruction: "There is a risk of collision; this vessel must turn right and pass behind the red vessel." This instruction is ultimately sent to the unmanned vessel's motion control system for low-level execution.

[0177] In some implementations, for simple visual perception tasks (such as "the color of the left buoy"), the routing agent can directly bypass the shipping scene description unit, the knowledge enhancement retrieval module, and the second large language model, with the first large language model outputting the answer within 1 second. For video fragmented scenes (such as capturing only 0.4 seconds of footage), the preprocessing module performs lossless loop replay: the 10-frame sequence is copied 8 times and recoded at 25 frames / second into a 3.2-second standard video frame sequence, meeting the model input requirements and avoiding interpolation artifacts. For scenarios requiring dynamic knowledge injection, the system can simultaneously search the local rule base and cloud real-time announcements (such as temporary rate limits), and assign higher priority to cloud rules when there are rule conflicts.

[0178] Please see Figure 4 This application also provides a reasoning and decision-making device based on a large language model, which can implement the above-mentioned reasoning and decision-making method based on a large language model. The reasoning and decision-making device based on a large language model includes: The acquisition module 41 is used to acquire the decision instructions input for the target scene and the video frame sequence corresponding to the target scene; Analysis module 42 is used to determine at least one object to be analyzed based on the decision instruction when the task type indicated by the decision instruction is a logical reasoning task, and to perform dynamic trajectory analysis and scene association analysis on at least one object to be analyzed based on the video frame sequence through the first language model to obtain the corresponding scene description information. The retrieval module 43 is used to combine decision instructions and scene description information to perform semantic retrieval in a preset domain rule base to obtain at least one target rule information. The generation module 44 is used to combine the decision command, video frame sequence, scene description information and at least one target rule information to generate corresponding text prompt information; The reasoning module 45 is used to perform logical reasoning based on text prompts using the second language model to obtain the corresponding decision results. The parameter scale of the second language model is larger than that of the first language model.

[0179] The specific implementation of this large language model-based reasoning and decision-making device is basically the same as the specific embodiment of the large language model-based reasoning and decision-making method described above, and will not be repeated here. Subject to meeting the requirements of the embodiments of this application, the large language model-based reasoning and decision-making device may also be equipped with other functional modules to implement the large language model-based reasoning and decision-making method in the above embodiments.

[0180] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described reasoning and decision-making method based on a large language model. This computer device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0181] Please see Figure 5 , Figure 5 The hardware structure of a computer device according to another embodiment is illustrated. The computer device includes: The processor 51 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 52 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 52 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 52 and is called and executed by the processor 51 to execute the reasoning and decision-making method based on a large language model according to the embodiments of this application. Input / output interface 53 is used to implement information input and output; The communication interface 54 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 55 transmits information between various components of the device (e.g., processor 51, memory 52, input / output interface 53, and communication interface 54); The processor 51, memory 52, input / output interface 53, and communication interface 54 are connected to each other within the device via bus 55.

[0182] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described reasoning and decision-making method based on a large language model.

[0183] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0184] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0185] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0186] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0187] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0188] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0189] It should be understood that in this application, "at least one" and "several" refer to one or more, and "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0190] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0191] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0192] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0193] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0194] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A reasoning and decision-making method based on a large language model, characterized in that, The method includes: Obtain the decision instructions input for the target scene and the video frame sequence corresponding to the target scene; When the task type indicated by the decision instruction is a logical reasoning task, at least one object to be analyzed is determined based on the decision instruction, and dynamic trajectory analysis and scene association analysis are performed on the at least one object to be analyzed based on the video frame sequence using the first language model to obtain the corresponding scene description information. By combining the decision instruction with the scene description information, semantic retrieval is performed in a preset domain rule base to obtain at least one target rule information; By combining the decision instruction, the video frame sequence, the scene description information, and the at least one target rule information, corresponding text prompt information is generated; By using the second language model, logical reasoning is performed based on the text prompt information to obtain the corresponding decision result, wherein the parameter scale of the second language model is larger than that of the first language model.

2. The reasoning and decision-making method based on a large language model according to claim 1, characterized in that, Before determining at least one object to be analyzed based on the decision instruction when the decision instruction corresponds to a logical reasoning task, the method further includes: The first video frame is determined from the video frame sequence; Using the first language model, the decision instruction and the first video frame are combined to perform task classification, and the task classification result corresponding to the decision instruction is obtained. The type of the task classification result includes at least one of visual perception task, knowledge retrieval task and logical reasoning task. When the task classification result indicates that the decision instruction corresponds to a visual perception task, the corresponding decision result is generated based on the decision instruction and the video frame sequence through the first large language model. When the task classification result indicates that the decision instruction corresponds to a knowledge retrieval task, a semantic search is performed in a preset domain rule base based on the decision instruction to obtain the corresponding decision result.

3. The reasoning and decision-making method based on a large language model according to claim 1, characterized in that, After obtaining the corresponding decision result through logical reasoning based on the text prompt information using the second language model, the process further includes: Using the first major language model, visual consistency analysis is performed on the decision results based on the video frame sequence to obtain the first analysis result; Using the first large language model, a logical compliance analysis is performed on the decision result based on the at least one target rule information to obtain a second analysis result; Using the first large language model, and based on the first analysis result and the second analysis result, determine the inference confidence corresponding to the decision result; When the inference confidence is less than a preset confidence threshold, an intermediate decision result is generated based on the decision instruction, the video frame sequence, and the scene description information, and the inference confidence of the intermediate decision result is determined. This process continues until the inference confidence corresponding to the intermediate decision result is greater than or equal to the preset confidence threshold, at which point the corresponding intermediate decision result is determined as the updated decision result corresponding to the decision instruction.

4. The reasoning and decision-making method based on a large language model according to claim 3, characterized in that, The step of generating intermediate decision results based on the decision instruction, the video frame sequence, and the scene description information includes: Combining the decision instruction with the scenario description information, a semantic search is performed in a preset domain rule base to obtain multiple corresponding update target rule information, wherein the number of the multiple update target rule information is greater than the number of the at least one target rule information; By combining the decision instruction, the video frame sequence, the scene description information, and the multiple update target rule information, a corresponding update text prompt is generated; Using the second language model, logical reasoning is performed based on the updated text prompt information to generate intermediate decision results.

5. The reasoning and decision-making method based on a large language model according to claim 4, characterized in that, Before generating the corresponding update text prompt information by combining the decision instruction, the video frame sequence, the scene description information, and the multiple update target rule information, the method further includes: Based on the first analysis result, the visual anomaly information contained in the decision result is determined; Based on the second analysis result, the missing rule information contained in the decision result is determined; Based on the visual anomaly information and the rule missing information, anomaly constraint information is generated; The step of combining the decision instruction, the video frame sequence, the scene description information, and the multiple update target rule information to generate corresponding update text prompt information includes: By combining the decision instruction, the video frame sequence, the scene description information, the multiple update target rule information, and the anomaly constraint information, a corresponding update text prompt is generated.

6. The reasoning and decision-making method based on a large language model according to claim 1, characterized in that, Before generating the corresponding text prompt information by combining the decision instruction, the video frame sequence, the scene description information, and the at least one target rule information, the method applied to edge computing nodes further includes: Obtain the geographic location coordinates of the edge computing node; The decision instruction and the scene description information are used to generate a retrieval vector, and the retrieval vector and the geographic location coordinates are uploaded to the cloud node, so that the cloud node can extract at least one real-time rule information based on the announcement rule information associated with the geographic location coordinates within the target time period. Receive at least one real-time rule information returned by the cloud node; By combining the at least one target rule information and the at least one real-time rule information, at least one initial fused rule information is obtained; For each initial fusion rule information, a corresponding rule weight description information is generated, wherein the rule weight description information is used to describe that the real-time rule information has a higher priority in the inference process of the second large language model than the corresponding target rule information; Each initial fusion rule information is concatenated with the corresponding rule weight description information to obtain the corresponding fusion rule information; The step of combining the decision instruction, the video frame sequence, the scene description information, and the at least one target rule information to generate corresponding text prompt information includes: By combining the decision instruction, the video frame sequence, the scene description information, and each fusion rule information, a corresponding text prompt message is generated.

7. The reasoning and decision-making method based on a large language model according to claim 1, characterized in that, The step of acquiring the decision command input for the target scene and the video frame sequence corresponding to the target scene includes: Obtain the decision instructions for the target scene and the original video stream corresponding to the target scene, and determine the target duration corresponding to the original video stream; When the target duration is less than a preset perception threshold, the corresponding number of copies is determined based on the ratio between the perception threshold and the target duration. Determine the original video frame sequence corresponding to the original video stream, and copy the original video frame sequence according to the number of copies to obtain multiple intermediate video frame sequences; The original video frame sequence is sequentially concatenated with the plurality of intermediate video frame sequences to obtain the target video frame sequence; The target video frame sequence is re-encoded according to a preset frame rate to obtain the video frame sequence corresponding to the target scene.

8. A reasoning and decision-making device based on a large language model, characterized in that, The device includes: The acquisition module is used to acquire the decision instructions input for the target scene and the video frame sequence corresponding to the target scene; The analysis module is used to determine at least one object to be analyzed based on the decision instruction when the task type indicated by the decision instruction is a logical reasoning task, and to perform dynamic trajectory analysis and scene association analysis on the at least one object to be analyzed based on the video frame sequence using the first large language model to obtain corresponding scene description information. The retrieval module is used to combine the decision instruction with the scene description information to perform semantic retrieval in a preset domain rule base to obtain at least one target rule information; The generation module is used to combine the decision instruction, the video frame sequence, the scene description information, and the at least one target rule information to generate corresponding text prompt information; The reasoning module is used to perform logical reasoning based on the text prompt information using a second language model to obtain the corresponding decision result, wherein the parameter scale of the second language model is larger than that of the first language model.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the reasoning and decision-making method based on a large language model as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the reasoning and decision-making method based on a large language model as described in any one of claims 1 to 7.

Images