A Multimodal Model-Based AI Video Inspection System and Method Based on Cameras

Abstract

This invention relates to the field of video inspection technology and discloses a multimodal model AI video inspection system and method based on cameras. The system includes: a video acquisition and processing module, an embedded vector calculation module, a difference judgment and triggering module, a multimodal analysis module, a work order management integration module, and a strategy dynamic adjustment module. During video inspection, by dynamically adjusting the screenshot acquisition interval, the detection frequency can be adaptively optimized according to the degree of scene change, avoiding resource waste caused by fixed intervals. This allows the system to reduce unnecessary analysis operations when there are no obvious changes in the image, thereby improving the efficiency of computing resources and the real-time performance of system operation, ensuring energy-saving and efficient inspection. During video inspection, by combining a multimodal large model with text prompting technology, visual features and scene context information are integrated for anomaly recognition, which can reduce the risk of misjudging subtle changes by a single model and improve the adaptability to complex scenes.

Description

Technical Field

[0001] This invention relates to the field of video inspection technology, specifically to a multimodal model AI video inspection system and method based on cameras. Background Technology

[0002] With the development of technology and the progress of society, video surveillance systems have become an indispensable part of public safety, industrial production, traffic management and other fields. With the continuous maturation of Internet of Things, edge computing and artificial intelligence technologies, the application of multimodal perception in the field of intelligent monitoring has shown a rapid development trend. Cameras have played an important role in industrial production and safety management and have been widely used in various scenarios for video monitoring and video inspection.

[0003] Currently, in camera-based video inspection systems, screenshot analysis is typically performed at fixed time intervals. This method cannot adaptively adjust the detection frequency according to changes in the scene. If the inspection scene remains unchanged for a long time, analyzing it at a fixed frequency will waste computing resources and reduce system efficiency.

[0004] Therefore, a multimodal model AI video inspection system and method based on cameras is proposed to solve the above problems. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a multimodal model AI video inspection system and method based on cameras, which solves the problems of wasted computing resources and reduced system operating efficiency mentioned in the background.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multimodal model AI video inspection system and method based on cameras, the method comprising the following steps: S1, collecting video stream data from multiple cameras deployed in the monitoring area, and generating a screenshot sequence at predetermined time intervals, the time intervals being dynamically adjustable and initially set to 10 minutes; S2, receiving the screenshot sequence, processing it to generate embedding vectors for adjacent screenshots using a pre-trained embedding model, and outputting the embedding vectors, wherein the embedding vectors are high-dimensional visual feature representations; S3, receiving the embedding vectors, processing and calculating the cosine distance between the vectors to quantify the screenshot differences, and outputting the cosine distance, the distance being calculated by dot product and modulus, with a value range of [0,2]. S4. Receive the cosine distance, process it, compare it with a preset threshold to generate difference judgment data, and output the data. The threshold is set based on historical anomaly data. S5. Receive the difference judgment data, process it so that analysis is not triggered when the difference is less than the threshold, and multimodal large model analysis is triggered when the difference is greater than the threshold, and output a trigger signal. S6. Receive the trigger signal and the current screenshot, process it, identify abnormal objects and events by combining multimodal large model with text prompts, and output anomaly identification results. S7. Receive the anomaly identification results, process it, generate a structured report when an anomaly is identified and report it to the work order system, and output an anomaly report. S8. Receive historical feedback on anomaly identification results, process it, dynamically adjust the screenshot interval, and output an adjustment strategy.

[0007] Preferably, generating the screenshot sequence in S1 includes the following steps: S11, capturing a video stream in real time through a camera hardware interface and compressing the data using video encoding technology; S12, generating screenshots from the video stream using a frame extraction algorithm at predetermined time intervals, wherein the frame extraction algorithm supports adaptive resolution adjustment to adapt to different network bandwidth conditions; S13, storing the generated screenshot sequence in a distributed database and cloud storage system, and adding metadata tags, including camera ID, timestamp, and geographic location information.

[0008] Preferably, generating the embedding vector in S2 includes the following steps: S21, loading a pre-trained embedding model, which is based on a convolutional neural network and a Transformer architecture and pre-trained on a large-scale image dataset; S22, performing preprocessing operations on each screenshot, including image normalization, resizing, and noise filtering; S23, inputting the preprocessed screenshot into the embedding model, outputting a fixed-dimensional embedding vector, and performing standardization processing to make the vector value range consistent.

[0009] Preferably, the calculation of cosine distance in S3 includes the following steps: S31, extracting the embedding vectors of adjacent screenshots, denoted as vector. sum vector S32, Calculate the vector sum vector The dot product of the vectors is calculated, and the vectors are calculated respectively. Magnitude and vector The modulus length; S33, the cosine similarity is obtained by dividing the dot product by the product of the modulus lengths, the formula is: ;

[0010] in and Represents an embedding vector. Representing vectors and dot product, Representing vectors The length of the mold, Representing vectors The length of the mold, This is the distance value.

[0011] Preferably, generating difference judgment data in S4 includes the following steps: S41, receiving cosine distance data within a historical period and statistically analyzing its distribution characteristics, including the mean μ and standard deviation σ; S42, calculating a dynamic threshold based on the distribution characteristics, using the formula: ;in For dynamic thresholds, The mean of the historical cosine distance. The standard deviation of the historical cosine distance, To adjust the coefficient, the value range is 1.5-2.5; S43, compare the current cosine distance with the dynamic threshold in real time. If the distance is less than the threshold, output a signal with no difference; if the distance is greater than the threshold, output a signal with difference; S44, record the difference judgment data to the log system, and update the calculated dynamic threshold to the threshold comparison unit.

[0012] Preferably, the triggering analysis in S5 includes the following steps: S51, when the difference judgment data outputs no difference signal, skip the multimodal large model analysis stage and directly enter the next screenshot cycle; S52, when the output has a difference signal, trigger the multimodal large model loading process, including initializing model parameters and loading industry professional database; S53, configure prompt word templates for the multimodal large model, the templates are customized based on scene type, including detecting intrusive objects and identifying equipment failures.

[0013] Preferably, the anomaly identification in S6 includes the following steps: S61, inputting the current screenshot into the visual encoder of the multimodal large model to extract high-level visual features; S62, generating a context-aware query statement by combining text prompts, and parsing the scene semantics through a natural language processing module; S63, comparing visual features and text prompts using multimodal fusion technology, outputting anomaly probability scores, and determining an anomaly when the score exceeds a preset confidence level; S64, classifying and locating the identified anomalies, and generating bounding boxes and semantic segmentation maps as auxiliary outputs.

[0014] Preferably, generating an anomaly report in S7 includes the following steps: S71, parsing the output of the multimodal large model to generate a standardized anomaly report, the report fields of which include anomaly ID, camera ID, timestamp, anomaly description and confidence level; S72, automatically pushing the anomaly report to the work order management system through the API interface to trigger the work order creation and allocation process; S73, monitoring the work order processing status in real time and displaying the anomaly processing progress on the system interface.

[0015] Preferably, the dynamic adjustment of the screenshot interval in S8 includes the following steps: S81, receiving anomaly identification result data within a historical period, and statistically analyzing the frequency distribution of anomalies and system resource utilization data; S82, analyzing the frequency distribution and resource utilization data, generating a shortening interval instruction when the anomaly frequency is higher than a preset threshold, and generating an extending interval instruction when the system resource utilization remains too high; S83, calculating the adjustment coefficient based on the shortening interval instruction and the extending interval instruction, using the following formula: ;

[0016] in Based on the screenshot interval, This represents the current abnormal frequency. For abnormal frequency threshold, As the current resource utilization rate, For resource utilization threshold, S84. Calculate the new screenshot interval based on the adjustment coefficients, and perform boundary constraint processing on the new interval to ensure that the interval is within the range of the minimum and maximum allowable values; S85. Update the adjusted screenshot interval to the video acquisition module and record the adjustment log.

[0017] Preferably, the system includes: a video acquisition and processing module, which receives video stream data acquired by the camera, generates a screenshot sequence at predetermined time intervals through a frame extraction unit, and outputs the screenshot sequence using a storage unit; an embedding vector calculation module, which receives the screenshot sequence, generates an embedding vector through an embedding model, and outputs the cosine distance using a cosine distance calculation unit; a difference judgment and triggering module, which receives the cosine distance, generates difference judgment data through a threshold comparison unit, and outputs the trigger signal using a trigger logic unit; a multimodal analysis module, which receives the trigger signal and the current screenshot, identifies abnormal objects and events through a multimodal large model, and outputs the abnormal identification result using a result generation unit; a work order management integration module, which receives the abnormal identification result, creates a structured abnormal report through a report generation unit, and reports it to the work order management system using an API interface unit; and a strategy dynamic adjustment module, which receives historical feedback of the abnormal identification result, dynamically adjusts the screenshot interval through a strategy optimization unit, and outputs optimization instructions to the video acquisition and processing module using a feedback loop unit.

[0018] Compared with the prior art, the present invention provides a multimodal model AI video inspection system and method based on cameras, which has the following beneficial effects: 1. In the present invention, when performing video inspection, by dynamically adjusting the screenshot acquisition interval, the detection frequency can be adaptively optimized according to the degree of scene change, avoiding the waste of resources caused by fixed intervals, so that the system reduces unnecessary analysis operations when there is no obvious change in the picture, thereby improving the efficiency of computing resource utilization and the real-time performance of system operation, and ensuring energy saving and high efficiency in the inspection process.

[0019] 2. In this invention, when performing video inspection, a multimodal large model combined with text prompting technology is used to integrate visual features and scene context information for anomaly identification. This reduces the risk of misjudging subtle changes by a single model, improves the adaptability to complex scenes, and enables the system to accurately identify anomalies even under conditions of changing light and object occlusion, thereby enhancing the robustness and reliability of detection.

[0020] 3. In this invention, during video inspection, a structured anomaly report is automatically generated and submitted to the work order management system. At the same time, the detection strategy is dynamically optimized based on historical feedback, realizing closed-loop management of the entire process from anomaly discovery to processing optimization. This enables the system to continuously improve itself, adapt to the needs of different monitoring scenarios, and improve the accuracy of inspections and the level of system intelligence. Attached Figure Description

[0021] Figure 1 This is a flowchart of the camera-based multimodal model AI video inspection method of the present invention; Figure 2 This is a diagram illustrating the architecture of the camera-based multimodal model AI video inspection system of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] For specific implementation examples, please refer to: Figure 1-2 A multimodal model-based AI video inspection system and method based on cameras. The method includes the following steps: S1. Video stream data collected by multiple cameras deployed in the monitoring area, and screenshot sequences generated at predetermined time intervals. The time interval can be dynamically adjusted, initially set to 10 minutes; S2. Receiving the screenshot sequence, processing it to generate embedding vectors of adjacent screenshots using a pre-trained embedding model, and outputting the embedding vectors, which are high-dimensional visual feature representations; S3. Receiving the embedding vectors, processing and calculating the cosine distance between the vectors to quantify the screenshot differences, and outputting the cosine distance. The distance is calculated by dot product and modulus, with a value range of [0,2]. Smaller values ​​indicate higher similarity; S4. Receiving the cosine distance... S5. Receive difference judgment data, process it by comparing it with a preset threshold to generate difference judgment data, and output the data. The threshold is set based on historical anomaly data. S6. Receive the difference judgment data, process it by not triggering analysis when the difference is less than the threshold, and triggering multimodal large model analysis when the difference is greater than the threshold, and output the trigger signal. S7. Receive the trigger signal and the current screenshot, process it by identifying abnormal objects and events through multimodal large model combined with text prompts, and output the anomaly identification result. S8. Receive the anomaly identification result, process it by generating a structured report and reporting it to the work order system when an anomaly is identified, and output the anomaly report. S9. Receive historical feedback on the anomaly identification result, process it by dynamically adjusting the screenshot interval, and output the adjustment strategy.

[0024] The screenshot sequence generation in S1 includes the following steps: S11, capturing video streams in real time through the camera hardware interface and compressing the data using video encoding technology; S12, generating screenshots from the video stream using a frame extraction algorithm at predetermined time intervals. The frame extraction algorithm supports adaptive resolution adjustment to adapt to different network bandwidth conditions, specifically including the following steps: S121, initializing the frame extraction controller, setting predetermined time interval parameters, and starting the timed capture mechanism; S122, capturing the current video frame data from the video stream buffer when each predetermined time interval arrives; S123, detecting the current network bandwidth conditions in real time and automatically selecting the corresponding resolution level based on the bandwidth threshold; S124, scaling the captured video frames according to the selected resolution level and outputting screenshots adapted to the network bandwidth; S125, storing the processed screenshots in the analysis queue and recording the capture timestamp and resolution parameters; S13, storing the generated screenshot sequence in a distributed database and cloud storage system, and adding metadata tags, including camera ID, timestamp, and geographic location information.

[0025] The generation of embedding vectors in S2 includes the following steps: S21, loading a pre-trained embedding model. The embedding model is based on a convolutional neural network and Transformer architecture and is pre-trained on a large-scale image dataset. The specific structure of the embedding model includes the following processing steps: S211, adopting a multi-layer convolutional neural network architecture, including an input layer, five convolutional layers, three pooling layers, and two fully connected layers; S212, each convolutional layer is followed by batch normalization and a ReLU activation function, with convolutional kernel sizes of 11×11, 5×5, 3×3, 3×3, and 3×3 respectively; S213, the fully connected layer outputs a 1024-dimensional feature vector, which is then normalized by L2 to obtain the final embedding vector; the training process of the embedding model includes the following steps: S214, training using a triplet loss function, the formula of which is: ;in The triplet loss value, Indicates anchor point sample, Indicates a positive sample. Indicates a negative sample. anchor point Compared with positive samples The distance between them anchor point With negative samples The distance between them S215. Use a stochastic gradient descent optimizer with a learning rate of 0.001 and a batch size of 128. S216. Apply an online hard example mining strategy during training, selecting the most difficult positive and negative samples for computation. S22. Perform preprocessing operations on each screenshot, including image normalization, resizing, and noise filtering, including the following steps: S221. Perform image normalization on the screenshot, linearly transforming the pixel values ​​to the [0,1] floating-point range, using the formula: ;

[0026] in This represents the original screenshot pixel matrix. and These are the minimum and maximum pixel values, respectively. The image is normalized; S222, the normalized screenshot size is adjusted to a preset uniform resolution, bilinear interpolation algorithm is used to maintain the image aspect ratio, and image data of fixed size is output. The pixel value calculation formula of bilinear interpolation algorithm is: ;in For the target pixel coordinates, , S223. Apply Gaussian filtering to the resized screenshot to remove high-frequency noise interference. The formula for generating the Gaussian filter convolution kernel is: The convolution kernel size is 3×3, and the standard deviation of the Gaussian filter is... =1.5; S23. Input the preprocessed screenshot into the embedding model, output a fixed-dimensional embedding vector, and perform standardization to ensure consistent vector value range. This includes the following steps: S231. Input the preprocessed screenshot into the embedding model and start the model's forward computation process; S232. Process the input screenshot through the multi-layer network structure of the embedding model to generate a fixed-dimensional initial embedding vector, with the vector dimension consistent with the model's output layer configuration; S233. Perform L2 normalization on the initial embedding vector, calculated using the following formula: ;in Represents the initial embedding vector. The L2 norm of a vector. S234. Output the standardized embedding vector, ensuring that the magnitude of all vectors is uniformly 1, thus achieving consistency in the value range.

[0027] The calculation of cosine distance in S3 includes the following steps: S31, extract the embedding vector of adjacent screenshots, denoted as vector. sum vector S32, Calculate the vector sum vector The dot product of the vectors is calculated, and the vectors are calculated respectively. Magnitude and vector The modulus length; S33, the cosine similarity is obtained by dividing the dot product by the product of the modulus lengths, the formula is: ;in and Represents an embedding vector. Representing vectors and dot product, Representing vectors The length of the mold, Representing vectors The length of the mold, This is the distance value.

[0028] The steps for generating difference judgment data in S4 are as follows: S41, Receive cosine distance data from historical periods and statistically analyze its distribution characteristics, including the mean μ and standard deviation σ; S42, Calculate the dynamic threshold based on the distribution characteristics, using the following formula: ;in For dynamic thresholds, The mean of the historical cosine distance. The standard deviation of the historical cosine distance, To adjust the coefficient, the value range is 1.5-2.5; S43, compare the current cosine distance with the dynamic threshold in real time. When the distance is less than the threshold, output a signal with no difference; when it is greater than the threshold, output a signal with difference; S44, record the difference judgment data to the log system, and update the calculated dynamic threshold to the threshold comparison unit.

[0029] The S5 trigger analysis includes the following steps: S51. When the difference judgment data outputs no difference signal, skip the multimodal large model analysis stage and directly enter the next screenshot cycle; S52. When a difference signal is output, trigger the multimodal large model loading process, including initializing model parameters and loading industry-specific databases. The multimodal large model loading process includes the following steps: S521. When a difference signal is received, start the multimodal large model loading controller and allocate dedicated computing resources; S522. Load the pre-trained multimodal large model parameter file from the model repository. The parameter file contains the weight matrices of the visual encoder and text encoder; S523. Initialize the model inference environment, including setting the computing device type, memory allocation strategy, and parallel computing parameters; S524. Connect... Connect to industry-specific databases and load professional knowledge bases related to the current monitoring scenario, including equipment specification databases and anomaly pattern feature databases; S525, verify the integrity of model parameters and database connections, and complete the multimodal large model loading process; S53, configure prompt word templates for the multimodal large model. The templates are customized based on scenario types, including detecting intrusive objects and identifying equipment faults. Specifically, this includes the following steps: S531, load predefined prompt word templates according to the scenario type database. Scenario types include intrusion detection, equipment fault identification, and environmental anomalies; S532, dynamically populate the prompt word templates with parameters, including timestamps, camera locations, and object categories; S533, verify the grammatical rationality of the prompt words through the natural language processing module and optimize the template structure to ensure query accuracy.

[0030] Anomaly identification in S6 includes the following steps: S61, inputting the current screenshot into the visual encoder of the multimodal large model to extract high-level visual features; S62, generating a context-aware query statement by combining text prompts, and parsing scene semantics through the natural language processing module, including the following steps: S621, loading a pre-configured text prompt template, the template is customized based on the scene type and associated with the anomaly identification task; S622, dynamically filling the variable parameters in the prompt template with the context information of the current screenshot, including timestamp, camera position and scene category, to generate a personalized query statement; S623, generating... The generated query statement is input into the natural language processing module for semantic parsing, extracting key intents and entity relationships from the query; S624, the parsed scene semantic representation is output for subsequent multimodal fusion and anomaly detection; S63, multimodal fusion technology is used to compare visual features and text prompts, outputting anomaly probability scores. Anomalies are identified when the score exceeds a preset confidence level. The multimodal fusion technology includes the following steps: S631, projecting the visual feature vector and text prompt word vectors into the same latent space, and calculating feature weights using an attention mechanism; S632, applying a feature weighted fusion algorithm, the formula of which is: ;in Represents visual feature vectors. Represents the text feature vector. These are the weighting coefficients of visual features in feature fusion. S633: Normalize the fused features and output anomaly probability scores; S64: Classify and locate the identified anomalies, and generate bounding boxes and semantic segmentation maps as auxiliary outputs.

[0031] The steps for generating an anomaly report in S7 are as follows: S71, parse the output of the multimodal large model to generate a standardized anomaly report. The report fields include anomaly ID, camera ID, timestamp, anomaly description, and confidence level; S72, automatically push the anomaly report to the work order management system through the API interface to trigger the work order creation and allocation process; S73, monitor the work order processing status in real time and display the anomaly processing progress on the system interface.

[0032] The dynamic adjustment of the screenshot interval in S8 includes the following steps: S81, receiving anomaly identification results data from the historical period, and statistically analyzing the frequency distribution of anomalies and system resource utilization data; S82, analyzing the frequency distribution and resource utilization data, generating a shortening interval instruction when the anomaly frequency exceeds a preset threshold, and generating an extending interval instruction when the system resource utilization remains excessively high; S83, calculating the adjustment coefficient based on the shortening and extending interval instructions, using the following formula:

[0033] ;in Based on the screenshot interval, This represents the current abnormal frequency. For abnormal frequency threshold, As the current resource utilization rate, For resource utilization threshold, β and β are weighting coefficients; S84, calculate the new screenshot interval based on the adjustment coefficient, and perform boundary constraint processing on the new interval to ensure that the interval is within the range of the minimum and maximum allowable values. The boundary constraint processing includes the following steps: S841, calculate the suggested value of the new interval based on the adjustment coefficient: ;in Suggested values ​​for new screenshot intervals. To adjust the coefficient, Based on the screenshot interval; S842, using boundary constraint functions to... The constraint processing is performed, and the formula is: The interval between the screenshots is the adjusted interval. The minimum allowable time is 1 minute. The maximum allowable value is 30 minutes; S843, the interval value after verifying the constraints. Is it within the allowable range of system resource load? S85, Update the adjusted screenshot interval to the video capture module and record the adjustment log.

[0034] The system includes: a video acquisition and processing module, which receives video stream data acquired by a camera, generates a screenshot sequence at predetermined time intervals through a frame extraction unit, and outputs the screenshot sequence using a storage unit; an embedding vector calculation module, which receives the screenshot sequence, generates embedding vectors through an embedding model, and outputs cosine distance using a cosine distance calculation unit; a difference judgment and triggering module, which receives the cosine distance, generates difference judgment data through a threshold comparison unit, and outputs a trigger signal using a trigger logic unit; a multimodal analysis module, which receives the trigger signal and the current screenshot, identifies abnormal objects and events through a multimodal large model, and outputs the abnormal identification results using a result generation unit; a work order management integration module, which receives the abnormal identification results, creates a structured abnormal report through a report generation unit, and reports it to the work order management system using an API interface unit; and a strategy dynamic adjustment module, which receives historical feedback on the abnormal identification results, dynamically adjusts the screenshot interval through a strategy optimization unit, and outputs optimization instructions to the video acquisition and processing module using a feedback loop unit.

[0035] The operation steps of the camera-based multimodal model AI video inspection system and method are as follows: Step 1: Video data acquisition and adaptive screenshot generation system. The system acquires video stream data in real time through multiple cameras deployed in the monitoring area. It uses a frame extraction algorithm to automatically generate screenshot sequences at predetermined time intervals. The time interval has dynamic adjustment capability. The initial setting is 10 minutes. The system can automatically optimize the acquisition frequency according to the needs of scene changes. During the screenshot generation process, the system will detect the network bandwidth conditions in real time and ensure the stability of data transmission under different network environments through adaptive resolution adjustment technology. The generated screenshot sequences will be tagged with metadata and stored in a distributed database and cloud storage system.

[0036] Step 2: Embedding Vector Generation and Similarity Calculation. For adjacent screenshots generated by each camera, the system uses a pre-trained embedding model to generate corresponding embedding vectors. This embedding model is based on a convolutional neural network and Transformer architecture and is pre-trained on a large-scale image dataset. It can extract the visual features of the screenshots. The system performs image normalization, size adjustment, and noise filtering preprocessing operations on the screenshots to ensure the standardization of the input data. Then, it calculates the cosine distance between the embedding vectors of adjacent screenshots and calculates the similarity index by vector dot product and magnitude to quantify the degree of visual difference between the screenshots.

[0037] Step 3: Dynamic Threshold Judgment and Anomaly Detection Trigger System. Based on the statistical distribution characteristics of historical cosine distance data, the system calculates the dynamic threshold, optimizes the threshold using an adjustment coefficient, compares the real-time calculated cosine distance with the dynamic threshold, and generates a difference judgment signal when the distance value exceeds the threshold, triggering the multimodal large model analysis process.

[0038] Step 4: Multimodal Fusion Analysis and Anomaly Recognition. After an anomaly detection signal is triggered, the system starts the multimodal large model loading process, initializes model parameters and connects to the industry-specific database. By inputting the current screenshot into the visual encoder, high-level visual features are extracted. At the same time, text prompts are combined to generate context-aware query statements. The natural language processing module is used to parse the scene semantics. Multimodal fusion technology is used to compare visual features and text prompts, and an anomaly probability score is output. When the score exceeds the preset confidence level, it is determined to be an anomaly event. The system will also classify and locate the identified anomalies, and generate bounding boxes and semantic segmentation maps as auxiliary outputs.

[0039] Step 5: Intelligent Report Generation and Work Order Management System. The system automatically parses the output of the multimodal large model and generates a standardized anomaly report. The report includes anomaly ID, camera ID, timestamp, anomaly description, and confidence level key fields. The report is automatically pushed to the work order management system through the API interface, triggering the work order creation and allocation process. The system monitors the work order processing status in real time and dynamically displays the anomaly processing progress on the system interface, forming a complete closed loop for anomaly event handling.

[0040] Step Six: Adaptive Strategy Optimization and System Iteration. Based on historical feedback data of anomaly identification results, the system statistically analyzes the frequency distribution of anomalies and resource utilization. By analyzing this data, the screenshot acquisition strategy is dynamically adjusted. When the anomaly frequency is high, the screenshot interval is automatically shortened. When the system resource utilization is consistently too high, the interval is appropriately extended. This ensures that resource utilization efficiency is optimized while maintaining real-time detection. This continuous adaptive optimization mechanism enables the system to continuously improve its inspection strategy and enhance long-term operational efficiency.

[0041] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0042] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A camera-based multimodal model AI video inspection method, characterized in that: The method includes the following steps: S1. Video stream data collected by multiple cameras deployed in the monitoring area, and screenshot sequences generated according to a predetermined time interval, wherein the time interval can be dynamically adjusted and is initially set to 10 minutes; S2. Receive the screenshot sequence, process it to generate embedding vectors for adjacent screenshots using a pre-trained embedding model, and output the embedding vectors, wherein the embedding vectors are high-dimensional visual feature representations; S3. Receive the embedded vector, process and calculate the cosine distance between the vectors to quantify the screenshot differences, and output the cosine distance. The distance is calculated by the dot product and the modulus, and the value range is [0,2]. The smaller the value, the higher the similarity. S4. Receive the cosine distance, process it and compare it with a preset threshold to generate difference judgment data, and output the data. The threshold is set based on historical abnormal data. S5. Receive the difference judgment data, process it so that analysis is not triggered when the difference is less than the threshold, and multimodal large model analysis is triggered when the difference is greater than the threshold, and output a trigger signal; S6. Receive the trigger signal and the current screenshot, process the identification of abnormal objects and events by combining a multimodal large model with text prompts, and output the abnormal identification result; S7. Receive the anomaly identification result, process the generated structured report when an anomaly is identified and report it to the work order system, and output the anomaly report; S8. Receive historical feedback on anomaly identification results, process and dynamically adjust screenshot intervals, and output adjustment strategies. The anomaly identification process in S6 includes the following steps: S61. Input the current screenshot into the visual encoder of the multimodal large model to extract high-level visual features; S62. Generate context-aware query statements by combining text prompts, and parse the scene semantics through the natural language processing module; S63. Use multimodal fusion technology to compare visual features and text prompts, and output anomaly probability scores. If the score exceeds the preset confidence level, it is judged as an anomaly. S64. Classify and locate the identified anomalies, and generate bounding boxes and semantic segmentation maps as auxiliary outputs. The dynamic adjustment of screenshot interval in S8 includes the following steps: S81. Receive anomaly identification results data within the historical period, and statistically analyze the frequency distribution of anomalies and system resource utilization data. S82. Analyze the frequency distribution and resource utilization data. When the abnormal frequency is higher than the preset threshold, generate a shortening interval instruction. When the system resource utilization is continuously too high, generate an extending interval instruction. S83. Calculate the adjustment coefficient based on the shortened interval command and the extended interval command, using the following formula: ； in Based on the screenshot interval, This represents the current abnormal frequency. For abnormal frequency threshold, As of the current resource utilization rate, For resource utilization threshold, β and β are weighting coefficients; S84. Calculate the new screenshot interval based on the adjustment coefficient, and perform boundary constraint processing on the new interval to ensure that the interval is within the range of the minimum and maximum allowable values; S85. Update the adjusted screenshot interval to the video capture module and record the adjustment log.

2. The camera-based multimodal model AI video inspection method according to claim 1, characterized in that: The step of generating the screenshot sequence in S1 includes the following steps: S11. Capture video streams in real time through the camera hardware interface and compress data using video encoding technology; S12. Use a frame extraction algorithm to generate screenshots from the video stream at the predetermined time interval. The frame extraction algorithm supports adaptive resolution adjustment to adapt to different network bandwidth conditions. S13. Store the generated screenshot sequence in a distributed database and cloud storage system, and add metadata tags, including camera ID, timestamp, and geolocation information.

3. The camera-based multimodal model AI video inspection method according to claim 1, characterized in that: The step of generating the embedding vector in S2 includes the following steps: S21. Load the pre-trained embedding model, which is based on a convolutional neural network and a Transformer architecture and is pre-trained on a large-scale image dataset. S22. Perform preprocessing operations on each screenshot, including image normalization, resizing, and noise filtering; S23. Input the preprocessed screenshot into the embedding model, output a fixed-dimensional embedding vector, and perform standardization to make the vector value range consistent.

4. The camera-based multimodal model AI video inspection method according to claim 1, characterized in that: The calculation of the cosine distance in S3 includes the following steps: S31. Extract the embedding vectors of adjacent screenshots, denoted as vector. sum vector ; S32, Calculate vectors sum vector The dot product of the vectors is calculated, and the vectors are calculated respectively. Magnitude and vector The modulus length; S33. The cosine similarity is obtained by dividing the dot product by the product of the moduli, and the formula is: ； in and Represents an embedding vector. Representing vectors and dot product, Representing vectors The length of the mold, Representing vectors The length of the mold, This is the distance value.

5. The camera-based multimodal model AI video inspection method according to claim 1, characterized in that: The process of generating difference judgment data in S4 includes the following steps: S41. Receive cosine distance data within the historical period and statistically analyze its distribution characteristics, including the mean μ and standard deviation σ. S42. Calculate the dynamic threshold based on the distribution characteristics, using the following formula: ； in For dynamic thresholds, The mean of the historical cosine distance. The standard deviation of the historical cosine distance, The adjustment factor has a range of 1.5-2.

5. S43. Compare the current cosine distance with the dynamic threshold in real time. If the distance is less than the threshold, output a signal with no difference; if the distance is greater than the threshold, output a signal with difference. S44. Record the difference judgment data to the log system, and update the calculated dynamic threshold to the threshold comparison unit.

6. The camera-based multimodal model AI video inspection method according to claim 1, characterized in that: The trigger analysis in S5 includes the following steps: S51. When the difference judgment data outputs no difference signal, skip the multimodal large model analysis stage and directly enter the next screenshot cycle. S52. When there is a difference signal in the output, the multimodal large model loading process is triggered, including initializing model parameters and loading industry professional databases. S53. Configure prompt word templates for multimodal large models. The templates are customized based on scene types, including detecting intrusive objects and identifying device malfunctions.

7. The camera-based multimodal model AI video inspection method according to claim 1, characterized in that: The process of generating an anomaly report in S7 includes the following steps: S71. Parse the output of the multimodal large model and generate a standardized anomaly report. The report fields include anomaly ID, camera ID, timestamp, anomaly description, and confidence level. S72. Automatically push exception reports to the work order management system via API interface to trigger work order creation and assignment process; S73. Monitor the work order processing status in real time and display the progress of exception handling on the system interface.

8. A camera-based multimodal model AI video inspection system, used to implement the camera-based multimodal model AI video inspection method as described in any one of claims 1-7, characterized in that the system... include: The video acquisition and processing module receives video stream data acquired by the camera, generates a screenshot sequence at predetermined time intervals through the frame extraction unit, and outputs the screenshot sequence using the storage unit. The embedding vector calculation module receives the screenshot sequence, generates embedding vectors through the embedding model, and outputs the cosine distance using the cosine distance calculation unit; The difference judgment and triggering module receives the cosine distance, generates difference judgment data through the threshold comparison unit, and outputs the trigger signal using the trigger logic unit; The multimodal analysis module receives the trigger signal and the current screenshot, identifies abnormal objects and events through a multimodal large model, and outputs the abnormal identification results using the result generation unit; The work order management integration module receives the anomaly identification results, creates a structured anomaly report through the report generation unit, and reports it to the work order management system through the API interface unit. The strategy dynamic adjustment module receives historical feedback of the anomaly identification results, dynamically adjusts the screenshot interval through the strategy optimization unit, and outputs optimization instructions to the video acquisition and processing module using the feedback loop unit.

Images