Visual perception and memory decision system and method of intelligent camera terminal

Abstract

This invention discloses a visual perception and memory decision-making system and method for an intelligent camera terminal. The system includes modules for video stream acquisition, basic perception, candidate event triggering, event fragment construction, on-device memory storage and retrieval, multimodal enhanced reasoning, strategy execution, and memory update. The method acquires audio and video and outputs video frames, triggers candidate events through lightweight detection and tracking and window-level weighted score calculation, and constructs structured event fragments; generates query vectors based on joint encoding of vision, region, and time, retrieves local historical memory, and returns it in summary form; uses an on-device multimodal large model to complete reasoning, outputs structured results such as event semantics and risk level, executes alerts, and updates local memory. This invention processes everything locally on the terminal, without cloud dependency, effectively reducing computational power consumption and false alarm rate, improving event judgment accuracy and privacy security, and is suitable for resource-constrained intelligent camera hardware platforms.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, computer vision, edge computing, intelligent security and smart home technology, and in particular relates to a visual perception and memory decision-making system and method for an intelligent camera terminal. Background Technology

[0002] Currently, the detection and processing of surveillance video by monitoring equipment typically includes the following forms: I. Solutions Based on Traditional Visual Algorithms and Rule Engines. These solutions typically utilize computer vision algorithms such as object detection, object tracking, face recognition, pet recognition, area intrusion detection, and behavior recognition to identify targets and behaviors in the video stream, and then trigger alarms, recording, or message pushes based on preset rules. Their main shortcomings are: lack of event-level semantic understanding; difficulty in handling complex scenes; reliance on manually generated rules, resulting in poor generalization; and usually, judgments are based only on the current frame or a short time window.

[0003] II. Cloud-based Multimodal Large Model-Based Solutions. This type of solution uploads images, video clips, event summaries, or voice data collected by camera terminals to the cloud, where a cloud-based multimodal large model performs analysis and inference. Its main drawbacks are: high network bandwidth and cloud inference costs; response speed affected by network conditions; privacy risks in home and office scenarios; decreased intelligence capabilities in weak network or offline scenarios; and a lack of continuous closed-loop processing between cloud analysis and edge execution.

[0004] III. Solutions combining edge-based basic vision with cloud-based supplementary analysis. This type of solution typically uses a lightweight vision model locally on the terminal to perform object detection, tracking, and initial event screening, only invoking a cloud-based model for further analysis when a local event is triggered. Its main problems are: a large number of low-value frames in continuous video streams are processed indiscriminately; there is a lack of unified historical context management between local detection results and cloud analysis results; the judgment of the importance of the current event mainly relies on instantaneous images; and it is difficult to form an edge-based closed loop of "event understanding—decision-execution—memory update".

[0005] The closest existing technology to this invention is the third approach, which involves running a lightweight object detection and rule engine locally on the smart camera terminal and uploading some suspicious images or detection results to the cloud for supplementary semantic understanding. This approach still suffers from at least the following drawbacks: First, the multimodal large model cannot participate in event processing at a high frequency and stably on the device side; second, directly processing a large number of low-value frames from continuous video streams in the large model wastes computing power; third, the judgment of the importance of events usually lacks historical memory enhancement and relies solely on the current image content, resulting in insufficient accuracy; fourth, it cannot continuously optimize subsequent judgments based on local historical events and user feedback; and fifth, it is difficult to simultaneously meet the core terminal requirements of low computing power, low latency, low false alarms, and high privacy.

[0006] Therefore, a design solution is needed that can solve at least one of the above problems. Summary of the Invention

[0007] The purpose of this invention is to provide a visual perception and memory decision-making system and method for intelligent camera terminals to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, one technical solution adopted by the present invention is: a visual perception and memory decision-making system for an intelligent camera terminal, including a video stream acquisition module, which receives video and audio captured by a monitoring device and sends continuous video frames and optional audio frames into a basic perception module; The basic perception module is used to receive the content transmitted by the video stream acquisition module, perform lightweight target detection, target tracking and basic behavior analysis on the video frames, and output the detection results, tracking status and basic behavior features. The candidate event triggering module calculates the candidate event triggering score based on the detection results, tracking status, basic behavioral characteristics and user rules obtained from the basic perception module. When the triggering score meets the preset threshold, a candidate event is generated. The event fragment construction module is used to receive candidate events generated by the candidate event triggering module, extract keyframes and regions of interest from the video and audio received by the video stream acquisition module, and construct a structured event fragment containing a set of keyframes, regions of interest, and event metadata. The edge-side memory storage and retrieval module is used to maintain object memories, event memories, scene memories, and user preference memories. After receiving an event fragment, this module encodes the visual, regional, and temporal information of the current event into a vector. From the local vector index or key-value index, it finds the top K most similar memories in descending order of similarity, generates a memory summary, and notifies the user. If no historical records similar to the current event are found, a separate memory summary is generated and the user is notified.

[0009] Preferably, it also includes a multimodal enhanced reasoning module, which is used to receive the event fragment and the memory summary output by the end-side memory storage and retrieval module, take the event fragment, memory summary, region description, time period description, and user rules as input, call the end-side multimodal large model, and output structured results: event semantic description, identity judgment, anomaly judgment, risk level, event risk type, and recommended action.

[0010] Preferably, it also includes a strategy execution module, which issues warnings based on the event risk type in the structured results output by the multimodal enhanced reasoning module according to the user-preset warning actions, and generates an execution log at the same time.

[0011] Preferably, it also includes a memory update module, which updates the object memory, event memory, scene memory and user preference memory in the end-side memory storage and retrieval module based on the structured results output by the multimodal enhanced inference module, the execution log generated by the strategy execution module and user feedback.

[0012] This application also provides a visual perception and memory decision-making method for an intelligent camera terminal, applicable to the aforementioned system, specifically including the following steps: S1: Collects video and audio captured by monitoring equipment; the video stream acquisition module outputs video frames according to the sampling frame rate. S2: Perform lightweight target detection, target tracking, and basic behavior analysis on the video frame to obtain detection results, tracking status, and basic behavior features; S3: Calculate the candidate event trigger score based on the detection results, tracking status, basic behavioral characteristics and user rules. Generate a candidate event when the trigger score meets the preset threshold. S4: Extract keyframes and regions of interest from the video and audio based on the candidate events, and construct a structured event fragment containing a set of keyframes, regions of interest, and event metadata; S5: Based on the event fragment, the visual, regional, and temporal information of the current event is encoded into a vector. The top K most similar memories are retrieved from the local vector index or key-value index in descending order of similarity, and a memory summary is generated. If no similar historical records are found, the corresponding memory summary is generated. S6: Receive the event fragment and memory summary, take the event fragment, memory summary, region description, time period description and user rules as input, call the terminal side multimodal big model for reasoning, and output structured results. The structured results include event semantic description, identity judgment, anomaly judgment, risk level, event risk type and recommended action. S7: Based on the event risk type in the structured results, execute the corresponding warning operation according to the user's preset warning action, and generate an execution log; S8: Update the object memory, event memory, scene memory, and user preference memory based on the structured results, execution logs, and user feedback. Preferably, in step S2, the method for calculating the candidate event trigger score is as follows: within a preset time window, the detection results and trajectory features of the video collected at a preset sampling frame rate are aggregated to calculate the dwell score s_dwell, the area sensitivity score s_zone, and the recurrence score s_repeat, respectively. s_dwell, s_zone, and s_repeat are all normalized to the [0,1] interval; the formula for calculating the candidate event trigger score is: S_event = a1・s_dwell + a2・s_zone + a3・s_repeat; Where a1, a2, and a3 are non-negative weights, and a1+a2+a3=1; if S_event≥preset trigger threshold T_event (value range [0.5,1]), and the target dwell time≥preset dwell threshold T_dwell, then it is determined to be a valid candidate event and output.

[0013] Preferably, in step S5, the query vector q is obtained by concatenating the visual feature fea_q, the region embedding e_zone, and the time embedding e_time, where q = [fea; e_zone; e_time]. For each memory m_i stored locally, its feature vector fea_i is obtained. The similarity calculation formula is as follows: Sim(q, m_i) = a1·cos(fea_q, fea_i) + a2·I(zone = zone_i) + a3·exp(-|Δt| / τ_t); m_i represents a historical memory entry, fea_i represents the visual features of the historical memory, Δt represents the time difference between the current event and the historical memory, τ_t represents the time decay coefficient, and a1, a2, and a3 represent non-negative weights that satisfy a1+a2+a3=1, sorted from high to low similarity.

[0014] Preferably, the multimodal enhanced reasoning module is implemented in any of the following ways: Method 1: A unified edge multimodal large model is adopted, which simultaneously receives image input and text input, and completes multimodal feature fusion and reasoning within the model; Method 2: A combination of a visual encoder and a language model is adopted, in which the visual encoder extracts image features, and then the image features and text information are jointly input into the language model for joint reasoning to obtain the final structured result.

[0015] The beneficial effects of this invention are as follows: This overall solution achieves high-accuracy, low-latency, and low-false-alarm monitoring and decision-making locally on the intelligent camera terminal through lightweight front-end perception, window-level weighted event triggering, structured event fragment construction, edge-side memory retrieval, and multimodal enhanced inference. The entire process is executed on the edge, protecting privacy and not relying on the network; through historical memory enhancement and closed-loop iterative updates, the accuracy of judgment is continuously improved, while significantly reducing unnecessary computing power consumption, making it suitable for various resource-constrained intelligent camera hardware platforms.

[0016] The basic perception module only performs lightweight target detection, target tracking and basic behavior analysis. It can complete real-time preprocessing without complex semantic understanding. It maintains low power consumption and high real-time performance in normal monitoring conditions and is compatible with the hardware environment of intelligent camera terminals with limited resources. Among them, the window-level multi-feature weighted triggering mechanism significantly reduces invalid large model calls. The candidate event triggering module performs weighted fusion calculation on the dwell score, regional sensitivity score, and recurrence score within a preset time window. Combined with the dual threshold judgment of triggering score and dwell time, only high-value events are sent to subsequent multimodal inference, filtering out instantaneous interference and single-frame misjudgment from the source, and greatly reducing the waste of computing power and false triggering rate on the edge. Among them, the event fragment construction module only extracts keyframes and regions of interest (ROI), and carries event metadata and related target trajectories, avoiding sending the complete video stream into the large model, effectively shortening the model input length, improving inference speed and reducing memory usage; Among them, the edge memory storage and recall module maintains object memory, event memory, scene memory and user preference memory, supports retrieval based on visual, regional and time joint similarity, enables the reasoning process to have historical context support, and significantly improves the accuracy of anomaly judgment, risk level assessment and identity recognition. Among them, historical memory is returned in the form of text summary and structured tags, without returning the original video or image data. While retaining effective information, it minimizes the context length of the input large model, reduces inference latency, and improves the stability of edge operation. The multimodal enhanced inference module can be implemented using a unified edge-side multimodal large model or a combination of a visual encoder and a language model, offering flexible deployment and strong compatibility. The model is obtained through cloud distillation, fine-tuning, and quantization, ensuring inference performance while meeting edge-side storage, computing power, and latency requirements. Among them, video data, perception results, memory data and reasoning results are processed locally on the intelligent camera terminal without uploading to the cloud, thus protecting user privacy; at the same time, network latency is avoided, enabling stable intelligent decision-making in weak network and no network environments; The memory update module continuously iterates and updates the local memory based on reasoning results, execution logs, and user feedback, enabling the system to continuously learn scene patterns and user preferences. Over the long term, this can sustainably reduce false alarm rates and improve judgment accuracy, becoming increasingly intelligent with use. Attached Figure Description

[0017] Figure 1 This is an architecture diagram of the visual perception and memory decision-making system of the intelligent camera terminal in this invention. Detailed Implementation

[0018] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0019] Example: This application provides a visual perception and memory decision-making system for an intelligent camera terminal, including: The video stream acquisition module receives video and audio captured by the monitoring equipment and sends continuous video frames and optional audio frames to the basic perception module. It uses a circular buffer to store video content before and after an event, with a pre-buffer length W_pre of 3-10 seconds and a post-buffer length W_post of 3-15 seconds. To adapt to the computing power of the edge device, the sampling frame rate f_s can be dynamically set to 5-15 fps by the resource scheduling module; when a large motion change or the appearance of a target in a sensitive area is detected, it can be temporarily increased to 10-20 fps.

[0020] The basic perception module receives the content transmitted by the video stream acquisition module and performs lightweight target detection, target tracking, and basic behavior analysis on the video frames. Target detection includes identifying people, vehicles, pets, moving objects, etc. in the scene. Target tracking includes cross-frame association of the same target, maintaining trajectory information, and calculating target dwell time and speed. Basic behavior analysis includes outputting basic behavioral features such as dwelling, loitering, intrusion, and rapid movement based on trajectory and position changes. After processing, the module outputs detection results, tracking status, and basic behavioral features for subsequent event triggering. Specifically, the basic perception module performs low-computing-power real-time pre-analysis on the video frames, including a target detection submodule, a target tracking submodule, and a basic behavior analysis submodule. The target detection submodule outputs bounding boxes, categories, and detection confidence scores; the target tracking submodule associates targets based on the similarity of features in adjacent frames and motion consistency, outputting the trajectory track_id, dwell time dwell_time, and speed velocity; the basic behavior analysis submodule outputs preliminary behavior labels based on posture, speed changes, and region rules. To reduce computing power, the basic perception module performs detection every frame and complex behavior analysis every 2 to 3 frames.

[0021] The candidate event triggering module calculates the candidate event triggering score based on the detection results, tracking status, basic behavioral characteristics and user rules obtained from the basic perception module. When the triggering score meets the preset threshold, a candidate event is generated. The event fragment construction module is used to receive candidate events generated by the candidate event triggering module, extract keyframes and regions of interest from the video and audio received by the video stream acquisition module, and construct structured event fragments containing information such as timestamps, device numbers, target trajectories, keyframe sets, regions of interest, and event metadata. The edge-side memory storage and retrieval module is used to maintain object memories, event memories, scene memories, and user preference memories. Object memories include target features, frequency of occurrence, common areas, and common time periods; event memories include historical events, risk levels, and handling results; scene memories include sensitive areas and normal scene patterns; and user preference memories include alarm levels, silent periods, and linkage rules. Upon receiving an event fragment, this module concatenates the visual features, region embedding, and time embedding of the current event into a query vector q. It then finds the top K most similar memories (e.g., K is set to 5) from the local vector index or key-value index, ranked by similarity from highest to lowest, generates a memory summary, and notifies the user. If no similar historical records are found, a separate memory summary is generated and the user is notified (specifically, if no similar memories are found, an empty summary placeholder is returned to ensure a consistent workflow structure). The multimodal enhanced reasoning module is used to receive the event fragments and the memory digests output by the end-side memory storage and recall module. It takes the event fragments, memory digests, region descriptions, time period descriptions, and user rules as inputs, calls the end-side multimodal large model, and outputs structured results: event semantic description, identity judgment, anomaly judgment, risk level, event risk type, and recommended actions. The entire reasoning process is completed locally on the terminal, protecting privacy and reducing latency. The strategy execution module issues warnings based on the event risk type in the structured results output by the multimodal enhanced inference module and the user-preset warning actions, and corresponding actions, including but not limited to: local recording, message push, voice broadcast, and linkage of light / alarm devices; after execution, it generates an execution log, recording the action type, execution time, execution status and result.

[0022] The memory update module updates the object memory, event memory, scene memory, and user preference memory in the edge memory storage and retrieval module based on the structured results output by the multimodal enhanced inference module, the execution logs generated by the strategy execution module, and user feedback. The update methods include adding memory entries, updating memory frequency, correcting event tags, and decaying old memories, so that the system can continuously learn scene habits and user preferences, continuously reduce false alarm rate, and improve judgment accuracy.

[0023] This application also provides a visual perception and memory decision-making method for an intelligent camera terminal used in the aforementioned system, which includes the following steps: S1: Collects video and audio captured by monitoring equipment; the video stream acquisition module outputs video frames according to the sampling frame rate. S2: Perform lightweight target detection, target tracking, and basic behavior analysis on the video frames to obtain detection results, tracking status, and basic behavior features; wherein, the calculation method for the candidate event trigger score is as follows: within a preset time window, aggregate the detection results and trajectory features of the video collected at a preset sampling frame rate, and calculate the dwell score s_dwell, the area sensitivity score s_zone, and the recurrence score s_repeat respectively, wherein s_dwell, s_zone, and s_repeat are all normalized to the [0,1] interval; the formula for calculating the candidate event trigger score is: S_event = a1・s_dwell + a2・s_zone + a3・s_repeat; Where a1, a2, and a3 are non-negative weights, and a1+a2+a3=1; if S_event≥preset trigger threshold T_event (value range [0.5,1]), and the target dwell time≥preset dwell threshold T_dwell, then it is determined to be a valid candidate event and output.

[0024] S3: Calculate the candidate event trigger score based on the detection results, tracking status, basic behavioral characteristics, and user rules. Generate a candidate event when the trigger score meets a preset threshold. Specifically, increase S_event if a target enters a sensitive area, stays for longer than the threshold, exhibits abnormal behavior exceeding the threshold, the same object repeatedly appears within a set time window, or a user-configured key object / key time period is triggered. Generate a candidate event when S_event ≥ T_event and dwell_time ≥ T_dwell; otherwise, only write to the normal monitoring log. T_event is 0.55~0.75, and T_dwell is 2~10 seconds. S4: Based on the candidate event, extract the set of keyframes and the region of interest (ROI) within the window before and after the event from the circular buffer, and perform ROI cropping based on the corresponding region of related_tracks to generate an event fragment. (Related_tracks is the set of related target trajectories, which is the continuous tracking identifier and trajectory information of one or more targets directly associated with the current candidate event. It is used to uniquely identify the target object that triggers the event, and includes at least one or a combination of the following: unique target trajectory ID (track_id), continuous position coordinates of the target in the video frame, target dwell time, target motion path, and target category (person / vehicle / pet, etc.)). The keyframe extraction method is "first frame + key action frame (if any) + end frame + several uniformly sampled frames" to ensure that both temporal changes are preserved and the input length is controlled. If the event involves multiple objects, the ROIs are sorted according to trajectory importance and target size, and only the first N_obj objects are retained. N_obj=1~4 to construct a structured event fragment containing the set of keyframes, the region of interest, and event metadata. S5: Based on the event fragment, the visual, regional, and temporal information of the current event is encoded into a vector. The top K most similar memories (e.g., the top 5) are retrieved from the local vector index or key-value index in descending order of similarity and a memory summary is generated. If no similar historical records are found, a corresponding memory summary is generated. Specifically, the query vector q is obtained by concatenating the visual feature fea_q, the region embedding e_zone, and the time embedding e_time (which can be obtained using the CLIP algorithm), q = [fea; e_zone; e_time]. For each memory m_i stored locally, its feature vector fea_i is obtained. The similarity calculation formula is as follows: Sim(q, m_i) = a1·cos(fea_q, fea_i) + a2·I(zone = zone_i) + a3·exp(-|Δt| / τ_t); m_i represents a historical memory entry, fea_i represents the visual features of the historical memory, Δt represents the time difference between the current event and the historical memory, τ_t represents the time decay coefficient, and a1, a2, and a3 are non-negative weights satisfying a1+a2+a3=1. This method simultaneously considers visual similarity, regional consistency, and temporal proximity. Specifically, a1 takes values ​​of [0.3, 0.5], a2 takes values ​​of [0.1, 0.3], and a3 takes values ​​of [0.3, 0.5]. The results are sorted from highest to lowest similarity.

[0025] The recall results are returned in the form of text summaries and structured tags, instead of the complete original video, to reduce the length of the context on the device. The text summary is a brief description of the historical memory in natural language, such as: personnel staying in sensitive areas, targets repeatedly appearing in the entrance area, common personnel activities during the daytime, etc. The structured tags are standardized tags that can be directly read by machines, such as: event type (staying, intrusion, loitering), risk level (low, medium, high), target type (person, vehicle, pet), location of occurrence (entrance, wall, passage), and time of occurrence (daytime, nighttime).

[0026] S6: Receive the event fragment and memory summary, and take the event fragment, memory summary, region description, time period description, and user rules as input. Call the on-device multimodal large model for reasoning and output a structured result. The structured result includes event semantic description, identity judgment, anomaly judgment, risk level, event risk type, and recommended action. The on-device multimodal large model is obtained through cloud model distillation. The specific steps for obtaining the on-device multimodal large model are as follows: S61: Input the keyframe or target area image corresponding to the candidate event, the basic perception results, the historical memory summary, and contextual information such as time, region, and device status as input data into the cloud teacher model. The teacher model outputs the event semantic description, the object identity judgment result, the event risk level, and the recommended handling strategy. The cloud model has no less than 72 bytes of parameters. S62: If the amount of labeled data obtained in the previous step is greater than or equal to 100,000 records, manual verification and correction shall be performed by uniform sampling at 1%; S63: Provide the same input data to the end-side student model (the model parameter size can be 2B-7B), and use the output of the teacher model as a supervision signal to fine-tune the training of the student model; S64: After training, the student model is quantized (4-int or 8-int) to meet the storage, computing power and latency requirements of the edge device; S65: Deploy the adapted student model on the smart camera terminal; S7: Based on the event risk type in the structured results, execute the corresponding warning operation according to the user's preset warning action, and generate an execution log; S8: Update the object memory, event memory, scene memory, and user preference memory based on the structured results, execution logs, and user feedback.

[0027] It is important to note that the solution presented in this application uses a single, edge-side multimodal large model that simultaneously supports image and text input. The implementation steps involve inputting keyframe / ROI images from event segments into the model's visual branch; inputting memory summaries, region descriptions, time periods, and user rules into the model's text branch; the model internally performs multimodal fusion; and directly outputs structured results such as event semantics, identity determination, anomaly detection, risk level, and recommended actions. The key feature of this solution is that a single model completes all reasoning. Other implementation schemes include a combination of a visual encoder and a language model, where two models collaborate to complete multimodal reasoning. Specifically, the visual encoder (such as lightweight visual models like CLIP or ViT) inputs keyframe / ROI images and outputs fixed-dimensional visual features; the large language model (LLM) inputs visual features, memory summaries, regions, time periods, and user rules; the language model performs joint understanding of visual features and textual information and outputs structured reasoning results. This approach separates visual encoding and language reasoning, offering flexibility, lightweight design, and better suitability for edge deployment.

[0028] The following examples will be provided to help you better understand the entire solution: Taking a home doorway camera terminal as an example: the video stream acquisition module continuously acquires video of the doorway area; the basic perception module identifies a person entering a sensitive area and staying there for more than 8 seconds; the candidate event triggering module calculates S_event=0.81 based on the dwell time, nighttime period, and area sensitivity, which is higher than the threshold T_event=0.65, thus generating a "doorway dwelling event"; the event fragment construction module extracts 5 seconds of video from the 5-second buffer before and the 8-second buffer after the event. The keyframes and corresponding ROIs are displayed; the memory retrieval module does not find highly similar common visitor memories in the local index, and only retrieves two historical "stranger loitering at night" event summaries; the multimodal enhanced inference module outputs abnormal_desc="stranger loitering at night", identity_desc="stranger", risk_score=0.86, event_type="high-priority alarm event", and provides an event description of "unfamiliar person loitering at the door for a long time at night"; the policy execution module executes local video recording marking, pushes high-priority alarm messages to the user, and links the door lights; the memory update module writes the event and the handling result into the event memory for reference in subsequent similar events.

[0029] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A visual perception and memory decision system for an intelligent video terminal, characterized by: include: The video stream acquisition module receives video and audio captured by the monitoring equipment and sends continuous video frames and optional audio frames to the basic sensing module. The basic perception module is used to receive the content transmitted by the video stream acquisition module, perform lightweight target detection, target tracking and basic behavior analysis on the video frames, and output the detection results, tracking status and basic behavior features. The candidate event triggering module calculates the candidate event triggering score based on the detection results, tracking status, basic behavioral characteristics and user rules obtained from the basic perception module. When the triggering score meets the preset threshold, a candidate event is generated. The event fragment construction module is used to receive candidate events generated by the candidate event triggering module, extract keyframes and regions of interest from the video and audio received by the video stream acquisition module, and construct a structured event fragment containing a set of keyframes, regions of interest, and event metadata. The edge-side memory storage and retrieval module is used to maintain object memories, event memories, scene memories, and user preference memories. After receiving an event fragment, this module encodes the visual features, regional information, and time information of the current event into a vector. From the local vector index or key-value index, it finds the top K most similar memories in descending order of similarity, generates a memory summary, and notifies the user. If no historical records similar to the current event are found, a separate memory summary is generated and the user is notified.

2. The visual perception and memory decision system of claim 1, wherein: It also includes a multimodal enhanced reasoning module, which receives the event fragments and the memory summaries output by the edge-side memory storage and retrieval module. Taking the event fragments, memory summaries, region descriptions, time period descriptions, and user rules as inputs, it calls the edge-side multimodal large model and outputs structured results: event semantic description, identity judgment, anomaly judgment, risk level, event risk type, and recommended actions.

3. The visual perception and memory decision system of claim 2, wherein: It also includes a strategy execution module, which issues warnings based on the event risk type in the structured results output by the multimodal enhanced reasoning module and according to the user's preset warning actions, while generating an execution log.

4. The visual perception and memory decision system of claim 3, wherein: It also includes a memory update module, which updates the object memory, event memory, scene memory, and user preference memory in the edge memory storage and retrieval module based on the structured results output by the multimodal enhanced reasoning module, the execution logs generated by the strategy execution module, and user feedback.

5. A visual perception and memory decision method of intelligent video terminal, used in the system of any one of claims 1-4, characterized in that: Includes the following steps: S1: Collects video and audio captured by monitoring equipment; the video stream acquisition module outputs video frames according to the sampling frame rate. S2: Perform lightweight target detection, target tracking, and basic behavior analysis on the video frame to obtain detection results, tracking status, and basic behavior features; S3: Calculate the candidate event trigger score based on the detection results, tracking status, basic behavioral characteristics and user rules. Generate a candidate event when the trigger score meets the preset threshold. S4: Extract keyframes and regions of interest from the video and audio based on the candidate events, and construct a structured event fragment containing a set of keyframes, regions of interest, and event metadata; S5: Based on the event fragment, the visual, regional, and temporal information of the current event is encoded into a vector. The top K most similar memories are retrieved from the local vector index or key-value index in descending order of similarity, and a memory summary is generated. If no similar historical records are found, the corresponding memory summary is generated. S6: Receive the event fragment and memory summary, take the event fragment, memory summary, region description, time period description and user rules as input, call the terminal side multimodal big model for reasoning, and output structured results. The structured results include event semantic description, identity judgment, anomaly judgment, risk level, event risk type and recommended action. S7: Based on the event risk type in the structured results, execute the corresponding warning operation according to the user's preset warning action, and generate an execution log; S8: Update the object memory, event memory, scene memory, and user preference memory based on the structured results, execution logs, and user feedback.

6. The visual perception and memory decision method of an intelligent camera terminal according to claim 5, characterized in that: In step S2, the method for calculating the candidate event trigger score is as follows: within a preset time window, the detection results and trajectory features of the video collected at a preset sampling frame rate are aggregated to calculate the dwell score s_dwell, the area sensitivity score s_zone, and the recurrence score s_repeat, respectively. s_dwell, s_zone, and s_repeat are all normalized to the [0,1] interval. The formula for calculating the candidate event trigger score is: S_event = a1・s_dwell + a2・s_zone + a3・s_repeat; Where a1, a2, and a3 are non-negative weights, and a1+a2+a3=1; if S_event≥preset trigger threshold T_event, and the target dwell time≥preset dwell threshold T_dwell, then it is determined to be a valid candidate event and output.

7. The visual perception and memory decision-making method for an intelligent camera terminal according to claim 5, characterized in that: In step S5, the query vector q is obtained by concatenating the visual feature fea_q, the region embedding e_zone, and the time embedding e_time, where q = [fea; e_zone; e_time]. For each memory m_i stored locally, its feature vector fea_i is obtained. The similarity calculation formula is as follows: Sim(q, m_i) = a1·cos(fea_q, fea_i) + a2·I(zone = zone_i) + a3·exp(-|Δt| / τ_t); m_i represents a historical memory entry, fea_i represents the visual features of the historical memory, Δt represents the time difference between the current event and the historical memory, τ_t represents the time decay coefficient, and a1, a2, and a3 represent non-negative weights that satisfy a1+a2+a3=1, sorted from high to low similarity.

8. The visual perception and memory decision-making method for an intelligent camera terminal according to claim 5, characterized in that: The multimodal enhanced inference module in step S6 is implemented in any of the following ways: Method 1: Use a unified edge-side multimodal large model to simultaneously receive image and text inputs, and complete multimodal feature fusion and inference within the model; Method 2: A combination of visual encoder and language model is used. The visual encoder extracts image features, and then the image features and text information are input into the language model for joint reasoning to obtain the final structured result.

Images