Safety supervision scene image processing and analysis method, system and device based on multi-modal large model, processor and storage medium thereof

Abstract

The invention relates to a safety supervision scene image processing and analyzing method based on a multi-modal large model. The method comprises the following steps: acquiring an image captured by a safety supervision scene camera and identifying a human body target; based on the snapshot image, screening out low-quality and non-human body target images, and performing sample set division; carrying out attribute classification on the obtained image by using a multi-modal large model to obtain a safety supervision scene image attribute tag, manually determining a high-confidence sample to obtain a training set, and generating a brand new business small model to carry out image classification test; and updating the current data set and carrying out model fine tuning and optimization processing on the business small model. The invention also relates to a corresponding system, device, processor and storage medium. By adopting the method, the system, the device, the processor and the storage medium of the invention, the advantages of the multi-mode large model and the service system small model are fully played, target detection, attribute identification and classification and real-time accurate early warning in the safety supervision field are realized, and the safety of a safety supervision scene is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to the effective processing of safety and surveillance scene images, and specifically refers to a safety and surveillance scene image processing and analysis method, system, device, processor and computer-readable storage medium thereof based on a multimodal large model. Background Art

[0002] In the current field of image detection technology for safety and surveillance scenes, traditional image processing methods have many limitations and are unable to meet the growing needs of safety supervision.

[0003] First, in practical applications, single models are limited in performance. Early image detection for safety and security scenarios relied primarily on single traditional machine learning models or simple deep learning models. These models are typically trained only for specific image features, resulting in poor generalization when faced with complex and changing safety and security scenarios. For example, when a scene experiences lighting changes, occlusion by people, or unusual camera angles, the model's detection accuracy drops significantly, making it unable to accurately identify the target object and its attributes. This results in many safety hazards being missed and prevented from being detected and warned of in a timely manner.

[0004] Secondly, safety and security scenarios are prone to insufficient data processing capabilities. The amount of image data generated by safety and security scenarios is large and complex, containing a large amount of redundant information and noisy data. Traditional methods lack effective screening and processing mechanisms when dealing with low-quality images (such as images with abnormal brightness or blur). These low-quality images not only affect the training efficiency of the model, but may also cause the model to learn incorrect features, thereby reducing detection accuracy. At the same time, the recognition of non-human samples is not accurate enough, and some non-human objects are easily misidentified as human bodies, increasing the workload and error rate of subsequent processing.

[0005] Furthermore, the dispersed application of safety and security models across numerous locations can lead to untimely model updates. As safety and security services continue to evolve and change, new security risks and scenarios continue to emerge, requiring image detection models to be updated promptly to meet these demands. However, traditional model training and updating methods often require extensive manual labeling, resulting in low efficiency. Furthermore, when encountering new attribute categories or rare scenarios, the models struggle to quickly learn and adapt, failing to provide accurate detection results and early warning information in a timely manner.

[0006] Finally, traditional image processing and analysis methods for safety and security scenarios lack multimodal fusion capabilities. Real-world safety and security scenarios contain a wealth of multimodal information, such as visual images, sound, and temperature. However, traditional image processing methods often focus solely on image information, ignoring the value of other modal information. This prevents the model from fully utilizing the comprehensive information in the scene to make judgments, limiting its ability to understand and analyze complex scenarios and making it difficult to achieve comprehensive and accurate safety monitoring and early warning.

[0007] In summary, the existing safety and supervision scene image detection technology has shortcomings in dealing with complex scenes, processing massive data, and quickly updating models. A new technical solution is urgently needed to solve these problems in order to improve the safety and management efficiency of safety and supervision scenes. Summary of the Invention

[0008] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a safety and surveillance scene image processing and analysis method, system, device, processor and computer-readable storage medium based on a multimodal large model.

[0009] To achieve the above objectives, the present invention provides a method, system, device, processor, and computer-readable storage medium for image processing and analysis of safety and security scenes based on a multimodal large model as follows:

[0010] The main feature of the safety and supervision scene image processing and analysis method based on the multimodal large model is that the method comprises the following steps:

[0011] (1) Obtain images captured by the safety and security camera and identify human targets from them;

[0012] (2) Based on the captured images, low-quality and non-human target images are screened out and the sample set is divided;

[0013] (3) Use the multimodal large model to classify the acquired images, obtain the attribute labels of the safety and supervision scene images, and manually determine the high-confidence samples to obtain the training set, thereby generating a new business small model for image classification testing;

[0014] (4) Update the current data set and perform model fine-tuning and optimization on the small business model.

[0015] Preferably, the step (1) specifically includes:

[0016] (1.1) Obtain the original image captured or warned by the safety and security camera from the required analysis scene;

[0017] (1.2) Use the Yolo or PaddleDetection visual detection model or framework to detect people, and set a lower screening threshold to filter out blurry, human-like images for subsequent supplementation of various types of sample data.

[0018] Preferably, the step (2) specifically includes:

[0019] (2.1) First, low-quality images are screened to remove abnormal brightness in the dataset, including images that are too bright or too dark, as well as blurred images. These images are then processed uniformly according to task requirements and placed in the undeterminable category or temporarily excluded from the training dataset.

[0020] (2.2) Perform human image screening to distinguish between human data with higher credibility and non-human data. Non-human data is added to the negative sample category, and human attribute data enters the next step of processing.

[0021] Preferably, the step (3) specifically includes:

[0022] (3.1) First, determine whether a basic business model already exists. If not, proceed to step (3.2). Otherwise, proceed to step (3.4).

[0023] (3.2) Using the multimodal large model to assist in generating a small business model, clearly classify attribute categories based on business needs and actual data distribution, and use the multimodal large model to perform attribute classification to obtain attribute labels for safety and security scene images;

[0024] (3.3) Manually determine high-confidence sample attributes and generate a small model training data set to generate a small business model;

[0025] (3.4) Attribute classification is performed on the existing business models, and corresponding result evaluation is performed. The obtained evaluation results are then processed in the next step.

[0026] Preferably, the step (4) specifically includes:

[0027] (4.1) Based on the model classification confidence, the currently acquired data is divided into low confidence and high confidence parts;

[0028] (4.2) For low-confidence data, use the multimodal large model to perform cause analysis and select reliable data to be added to the training set;

[0029] (4.3) For high confidence data, it is added as a new training set to the small business model for data update, thereby achieving model fine-tuning and optimization.

[0030] Preferably, the step (4.2) specifically includes:

[0031] (4.2.1) If there is low confidence data due to unqualified captured images, the multimodal large model can be used to adjust the camera angle and capture rules, or images containing only partial human bodies can be added to the unidentifiable category;

[0032] (4.2.2) For low-confidence data caused by new attributes or new examples that have not been covered before, the multimodal large model is used to output the category judgment results, the sample attribute labels are modified accordingly, and they are added to the corresponding training dataset under manual review;

[0033] (4.2.3) If there is small sample category data, use the multimodal large model to generate available sample supplementary data and add it to the training data set.

[0034] The above-described method is used to implement a safety and surveillance scene image processing and analysis system based on a multimodal large model. The main features of the system are as follows:

[0035] Image acquisition and person detection module, used to obtain the original image and identify whether the image contains human targets;

[0036] An image screening module, connected to the image acquisition and person detection module, for screening out images with abnormal brightness, blurred images, and non-human images;

[0037] A business model training and testing module, connected to the image screening module, is used to generate a small business model using a multimodal large model and perform classification tests based on model attributes;

[0038] The business model fine-tuning and optimization module is connected to the business model training and testing module, and is used to update the training set based on the classification results and perform corresponding model fine-tuning and optimization processing.

[0039] The main features of the safety and surveillance scene image processing and analysis device based on a multimodal large model are as follows:

[0040] a processor configured to execute computer-executable instructions;

[0041] The memory stores one or more computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned safety and supervision scene image processing and analysis method based on the multimodal large model are implemented.

[0042] The main feature of the safety and supervision scene image processing and analysis processor based on a multimodal large model is that the processor is configured to execute computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned safety and supervision scene image processing and analysis method based on a multimodal large model are implemented.

[0043] The main feature of this computer-readable storage medium is that a computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the above-mentioned safety and supervision scene image processing and analysis method based on a multimodal large model.

[0044] The safety and surveillance scene image processing and analysis method, system, device, processor, and computer-readable storage medium thereof based on the multimodal large model of the present invention have the following technical advantages over the existing technology:

[0045] Improve detection accuracy: By combining a large multimodal model with a small business model, and fully leveraging the powerful understanding and generation capabilities of the large multimodal model and the professionalism of the small business model for safety and supervision scenarios, it is possible to more accurately detect targets, identify attributes, and classify safety and supervision scene images, thereby effectively improving detection accuracy and reducing misjudgments and missed judgments.

[0046] Enhanced data processing capabilities: The image screening module can effectively remove low-quality images and non-human samples, thereby improving data quality and reducing the interference of noisy data on model training. At the same time, it uses a large multimodal model to process low-confidence data and supplement new data to make model training data richer and more comprehensive, thereby enhancing the model's generalization ability.

[0047] Realize real-time and accurate early warning: This technical solution can process safety and surveillance scene images in real time, quickly identify abnormal situations and issue early warnings, helping staff to take timely measures, reduce the possibility of safety accidents, and ensure the safety of people and property in safety and surveillance scenes.

[0048] Improve model adaptability: The business model fine-tuning and optimization module can continuously update the small business model based on new data, enabling it to quickly adapt to the development and changes of safety and supervision business as well as emerging security risks and scenarios, and maintain good detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a structural diagram of the safety and surveillance scene image processing and analysis system based on a multimodal large model of the present invention.

[0050] Figure 2 Schematic diagram of the workflow of the image acquisition and personnel detection module of the present invention.

[0051] Figure 3 Schematic diagram of the workflow of the image screening module of the present invention.

[0052] Figure 4 This is a workflow diagram of the business model training and testing module of the present invention.

[0053] Figure 5 This is a flow chart of the present invention for using a multimodal large model to assist in generating a small business model in the absence of a basic small business model.

[0054] Figure 6 Schematic diagram of the workflow of the business model fine-tuning optimization module of the present invention.

[0055] Figure 7 This is a schematic diagram of classifying the causes of low-confidence results and proposing improvement measures according to the present invention. DETAILED DESCRIPTION

[0056] In order to more clearly describe the technical content of the present invention, further description is given below in conjunction with specific embodiments.

[0057] Before describing in detail embodiments according to the present invention, it should be noted that, hereinafter, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, whereby a process, method, article, or apparatus comprising a list of elements includes not only those elements, but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0058] The safety and supervision scene image processing and analysis method based on the multimodal large model includes the following steps:

[0059] (1) Obtain images captured by the safety and security camera and identify human targets from them;

[0060] (2) Based on the captured images, low-quality and non-human target images are screened out and the sample set is divided;

[0061] (3) Use the multimodal large model to classify the acquired images, obtain the attribute labels of the safety and supervision scene images, and manually determine the high-confidence samples to obtain the training set, thereby generating a new business small model for image classification testing;

[0062] (4) Update the current data set and perform model fine-tuning and optimization on the small business model.

[0063] As a preferred embodiment of the present invention, the step (1) specifically includes:

[0064] (1.1) Obtain the original image captured or warned by the safety and security camera from the required analysis scene;

[0065] (1.2) Use the Yolo or PaddleDetection visual detection model or framework to detect people, and set a lower screening threshold to filter out blurry, human-like images for subsequent supplementation of various types of sample data.

[0066] As a preferred embodiment of the present invention, the step (2) specifically includes:

[0067] (2.1) First, low-quality images are screened to remove abnormal brightness in the dataset, including images that are too bright or too dark, as well as blurred images. These images are then processed uniformly according to task requirements and placed in the undeterminable category or temporarily excluded from the training dataset.

[0068] (2.2) Perform human image screening to distinguish between human data with higher credibility and non-human data. Non-human data is added to the negative sample category, and human attribute data enters the next step of processing.

[0069] As a preferred embodiment of the present invention, the step (3) specifically includes:

[0070] (3.1) First, determine whether a basic business model already exists. If not, proceed to step (3.2). Otherwise, proceed to step (3.4).

[0071] (3.2) Using the multimodal large model to assist in generating a small business model, clearly classify attribute categories based on business needs and actual data distribution, and use the multimodal large model to perform attribute classification to obtain attribute labels for safety and security scene images;

[0072] (3.3) Manually determine high-confidence sample attributes and generate a small model training data set to generate a small business model;

[0073] (3.4) Attribute classification is performed on the existing business models, and corresponding result evaluation is performed. The obtained evaluation results are then processed in the next step.

[0074] As a preferred embodiment of the present invention, the step (4) specifically includes:

[0075] (4.1) Based on the model classification confidence, the currently acquired data is divided into low confidence and high confidence parts;

[0076] (4.2) For low-confidence data, use the multimodal large model to perform cause analysis and select reliable data to be added to the training set;

[0077] (4.3) For high confidence data, it is added as a new training set to the small business model for data update, thereby achieving model fine-tuning and optimization.

[0078] As a preferred embodiment of the present invention, the step (4.2) specifically includes:

[0079] (4.2.1) If there is low confidence data due to unqualified captured images, the multimodal large model can be used to adjust the camera angle and capture rules, or images containing only partial human bodies can be added to the unidentifiable category;

[0080] (4.2.2) For low-confidence data caused by new attributes or new examples that have not been covered before, the multimodal large model is used to output the category judgment results, the sample attribute labels are modified accordingly, and they are added to the corresponding training dataset under manual review;

[0081] (4.2.3) If there is small sample category data, use the multimodal large model to generate available sample supplementary data and add it to the training data set.

[0082] The above-described method is used to implement a safety and supervision scene image processing and analysis system based on a multimodal large model, wherein the system includes:

[0083] Image acquisition and personnel detection module: ① Image acquisition: Use the timed task scheduler to trigger the camera capture operation at a pre-set time interval (such as every 5 minutes) to obtain the original image of the timed capture. For early warning capture, by integrating anomaly detection sensors (such as infrared human intrusion detection sensors) on the camera device side, when the sensor detects an abnormality, it immediately sends a capture instruction to the camera to obtain the original warning image. ② Personnel detection: Taking the yolo model as an example, when performing personnel detection, the input image is first sent to a series of convolution layers. These convolution layers are equipped with convolution kernels of different sizes, such as 3×3, 5×5, 7×7 and other sizes. Convolution kernels of different sizes have different advantages in extracting image features: smaller convolution kernels, such as 3×3 convolution kernels, are good at capturing detailed information in the image, such as the edges and textures of objects; while larger convolution kernels, such as 7×7 convolution kernels, are more helpful in obtaining larger range features in the image and the approximate shape information of the target object. After multiple layers of convolution operations, a feature map with rich semantic information is obtained. The specific process is: the input size is H in ×W in ×C in image X, where H in is the height of the input image, W in is the width of the input image, C in is the number of channels, for which size K is used h ×K w ×C in The convolution operation is performed with the convolution kernel K, where K h ×K w is the spatial dimension feature, C inThe convolution kernel is the channel dimension. Each convolution kernel is multiplied by the local image region elements, summed, and biased to obtain the intermediate feature map Y. After processing with the ReLU activation function Z = max(0, Y), the output feature map Z of that layer is obtained. If there are L convolutional layers, the subsequent layers use the activated output of the previous layer as input and repeat the above convolution and activation steps. Finally, the feature maps obtained from different layers are fused through methods such as splicing. If the fused feature maps have different spatial sizes, they are first upsampled or downsampled and then spliced ​​according to the channel dimension to obtain the final feature map with rich semantic information. The feature map is then processed using the spatial pyramid pooling (SPP) structure. The feature map is first divided into sub-regions of different sizes (such as 1×1, 2×2, 4×4, etc.). Then, a pooling operation (commonly maximum pooling or average pooling) is performed within each sub-region. The pooled results of each sub-region are sequentially spliced ​​together to form a fixed-length feature vector. This achieves the fusion of features of different scales, allowing the model to effectively detect people of different sizes. During the detection process, the screening threshold is lowered (for example, from the default 0.5 to 0.2). At this time, the model will output more detection frames that are likely to be human bodies, even if the image areas corresponding to these detection frames are blurred or similar to human bodies. These can be used to supplement sample data later.

[0084] Image screening module: ① Low-quality image screening: In terms of brightness anomaly detection, a method based on histogram statistics is adopted. For the input image, its brightness histogram is calculated, and the number of pixels with different brightness values ​​is statistically distributed. Set an over-brightness threshold (such as 200, ranging from 0 to 255) and an over-darkness threshold (such as 30). If the proportion of pixels in the image with brightness values ​​higher than the over-brightness threshold or lower than the over-darkness threshold exceeds a certain proportion (such as 10%), it is determined to be a brightness abnormality image. In terms of blurred image detection, the Laplace operator is used to evaluate the image clarity. Apply the Laplace operator convolution kernel to the image to calculate the Laplace response value, and convert the Laplace operator convolution kernel (such as the common ) is multiplied element-by-element with the corresponding pixel area in the image and then summed. Each pixel of the image is traversed (the boundary pixels are calculated according to the specific boundary processing method) to obtain the response value of each pixel position. The matrix composed of these response values ​​is the Laplace response value result of the image. Its numerical value reflects the richness of the edge and detail information of the corresponding image area. Set a blur threshold (such as 0.1, ranging from 0 to 1). If the Laplace response value is lower than the blur threshold, it is judged to be a blurred image. After screening out the abnormal brightness and blurred images, they can be marked as specific undeterminable categories according to the task requirements and stored in a special low-quality image database. They are not included in the training data set for the time being. ② Human image screening: A human classification model based on a convolutional neural network is used, such as the ResNet architecture. The model inputs a preliminarily screened image. The image first enters the convolutional layer, where different convolution kernels are slid across the image to perform convolution operations, extracting local features and generating a feature map. The feature map then enters the pooling layer, where downsampling is performed, typically using max pooling or average pooling, to reduce the amount of data while retaining key features. Finally, the pooled feature map is flattened and fed into the fully connected layer. This layer multiplies the input features by a weight matrix and adds a bias, mapping the extracted features to different class spaces. After processing with activation functions and softmax functions, the fully connected layer outputs the probability that the image belongs to each class, completing the image's feature extraction and classification. The model is trained using a large amount of labeled human and non-human image data, enabling it to learn the characteristic differences between human and non-human features. The trained model predicts the input image and outputs the probability of the image belonging to a human or non-human. If the probability is greater than a set threshold (e.g., 0.5), the image is considered human and its human attribute data is passed to the next module. If the probability is less than the threshold, the image is considered non-human and placed in the negative sample dataset.

[0085] Business model training and testing module: ① Business small model generation (when there is no basic business small model): clearly divide the attribute categories according to business needs and actual data distribution. Taking the multimodal large model as an example, the screened images and corresponding attribute category texts are input into the multimodal large model. The model uses its internal image feature extraction network to perform step-by-step feature extraction on the input image through a series of convolutional layers and pooling layer operations, and converts the image data into representative feature vectors; at the same time, the text encoder processes the attribute category text, first splitting the text into words or characters, and then mapping each word or character into a low-dimensional vector through the word embedding layer, and then using the Transformer architecture to sequentially model these vectors to capture the semantics and contextual information in the text, and finally output the encoded representation of the attribute category text. Then, feature fusion is performed through a cross-modal interaction mechanism. For example, an attention mechanism is used to first calculate the correlation score between image features and text features. By performing operations such as the inner product on the text feature vector and the image feature vector, the score is converted into a probability distribution through the Softmax function. The text features are weighted and summed according to this probability distribution to obtain a text feature representation related to the image features. This is then concatenated or element-by-element added to the image features to obtain the fused features. Finally, the fused features are input into the fully connected layer, which linearly transforms the fused features using a weight matrix and bias. Nonlinearity is then introduced through an activation function (such as ReLU). Finally, the output is converted into the probability of each attribute category through the Softmax function. The label corresponding to the category with the highest probability is the safety and security scene image attribute label for the image. These attribute labels are manually reviewed, and samples with high confidence (such as samples with a model prediction probability greater than 0.9) are selected and organized into a small model training dataset. Input the training dataset into the deep learning framework and use an appropriate loss function (such as the cross-entropy loss function) and optimizer (such as the Adam optimizer) to train the small business model (such as a classification model based on a convolutional neural network) to generate the small business model. ② Attribute classification testing and evaluation (when a basic small business model already exists): Input the newly acquired image into the existing small business model, and the model will perform attribute classification predictions on the image. When evaluating the results, use metrics such as accuracy, recall, and F1 value. Accuracy = number of correctly predicted samples / number of predicted samples, recall = number of correctly predicted samples / actual number of samples, and F1 value = 2 × (accuracy × recall) / (accuracy + recall). By calculating these metrics, the performance of the small business model is evaluated, providing a basis for subsequent model optimization.

[0086] Business Model Fine-tuning and Optimization Module: ① Low-Confidence Data Processing: For low-confidence data that presents challenges for model recognition, such as irregular capture angles, partially captured subjects, or significant occlusion, the image understanding capabilities of the multimodal large model are leveraged. Low-confidence images and associated scene descriptions (such as capture location and time) are fed into the multimodal large model. The model analyzes the relationship between the subject and the surrounding environment and outputs a judgment regarding the capture angle issue. For example, if the model detects that a key body part (such as the head) is at the edge of the image and incomplete, combined with the scene description, it may be due to a camera perspective issue and outputs adjustment suggestions, such as shifting the camera 10 degrees to the left or rotating it 5 degrees upward. Images containing only partial subjects are directly marked as unclassifiable. For low-confidence data containing new attributes or new examples, the image and associated descriptions are fed into the multimodal large model. The model leverages its knowledge and understanding capabilities to output a category judgment. The judgment results are manually reviewed. If approved, the sample attribute labels are modified and added to the corresponding training dataset. For small sample data categories, the generative capabilities of the multimodal large model are utilized. Taking the idea of ​​generative adversarial networks (GANs) as an example, the multimodal large model acts as a generator to generate simulated image data similar to the small sample data category. The generated simulated image data is manually reviewed and added to the training dataset. ② High-confidence data processing: High-confidence data is directly used as a new training set and added to the model update dataset. When fine-tuning and optimizing the model, these newly added training data are used to fine-tune the small business model. Using the transfer learning method, some of the underlying convolutional layer parameters of the small business model are fixed, and only the parameters of the upper fully connected layer are updated. Using a gradient descent algorithm (such as stochastic gradient descent), the gradient is calculated based on the newly added training data, and the weights and biases of the fully connected layer are updated to update the small business model so that it can better adapt to new data and scenarios.

[0087] The present technical solution is further described in detail below with reference to a specific embodiment:

[0088] Assume that in a safety and security scenario in a large factory, real-time monitoring and safety management of personnel activities and equipment operations in different areas of the factory are required.

[0089] Implementation of the image acquisition and personnel detection module: Multiple safety and security scene cameras are installed in various key areas of the factory, such as production workshops, warehouses, entrances and exits. These cameras capture images at set time intervals (for example, every 5 minutes), or take early warning snapshots when abnormal situations are detected (such as people entering restricted areas) to obtain original images. Afterwards, the PaddleDetection visual detection framework is used to detect people in the captured images. In actual operation, the screening threshold for personnel detection is set to a lower level, such as 0.2 (the normal threshold is 0.5). In this way, even if the human target in the image is blurry or presents a shape similar to that of a human body, it can be detected. For example, in a dark corner of the workshop, part of the worker's body is blocked by the equipment, but because the threshold is lowered, the human body in the image is still detected, and these images will be used for sample supplementation later.

[0090] Image Screening Module Implementation: Images that have undergone person detection first enter the low-quality image screening module. An image brightness analysis algorithm is used to detect the average brightness of the image. If the brightness value is above a set overbrightness threshold (e.g., 200, ranging from 0-255) or below a set overdarkness threshold (e.g., 30), the image is identified as abnormally bright. Simultaneously, an image clarity evaluation algorithm is used to calculate the image clarity index. If the value is below a blur threshold (e.g., 0.1, ranging from 0-1), the image is identified as blurry. These images with abnormal brightness and blur are screened out and uniformly marked as unclassifiable, excluding them from subsequent model training. After the low-quality image screening is complete, human image screening is performed. A deep learning-based human classification model is used to classify images that have undergone preliminary screening as either human or non-human. If the image is identified as non-human, it is placed in the negative sample dataset. If it is identified as human, its human attribute data is passed to the next module. For example, the outline of a device in an image that was originally mistakenly detected as a person can be accurately identified as non-human in this step, avoiding interference with subsequent model training.

[0091] Implementation of the business model training and testing module: Initially, it is assumed that the factory safety and security system does not have a basic small business model. Based on the factory's actual safety and security business needs, image attribute categories are determined, such as "workers wearing work uniforms," ​​"workers not wearing work uniforms," ​​and "workers wearing only half-length work uniforms." A multimodal large model (such as one with image understanding capabilities, such as Qwen-VL) is used to classify the attributes of the selected images. The attribute classification results output by the multimodal large model are manually reviewed to determine which samples can be included in the training set. For example, images that the multimodal large model identifies as "workers wearing work uniforms" with clear images and distinct features are manually identified as high-confidence samples. These samples are organized into a small model training dataset and input into the deep learning framework to train and generate a small business model. Once the small business model is trained and operational, attribute classification tests are regularly performed on newly collected images. The test results are compared and evaluated with actual results, for example, by calculating metrics such as accuracy and recall to determine model performance.

[0092] Implementation of the business model fine-tuning and optimization module: During the operation of the small business model, data is differentiated based on the confidence of the model classification. If the model's classification confidence for a certain image is lower than the set threshold (such as 0.6), it is considered low-confidence data. In cases where the human body is partially obscured due to irregular capture angles and model recognition is difficult, such as capturing an image of a worker walking sideways in a narrow passage with only part of the body captured, the multimodal large model's understanding ability is used to analyze the capture angle problem and recommend adjusting the camera position or angle. At the same time, such images are temporarily added to the undeterminable category to avoid erroneous training. If new attributes or new examples are encountered, such as the introduction of new operating procedures in a factory, and there are warning requirements or personnel behavior recognition that have not been recognized before, the multimodal large model can analyze these images and output category judgment results. After manual review, the sample attribute labels are modified and added to the corresponding training data set. For small sample data categories, such as "workers wearing summer work clothes," which have relatively few data in the training set, resulting in low model recognition confidence, we leverage the generation capabilities of large multimodal models to generate simulated image data and add it to the training dataset. High-confidence data with confidence levels above a set threshold is directly added to the training dataset and added to the model update dataset. We regularly fine-tune and optimize the small business model to ensure it continuously adapts to changing factory safety and regulatory scenarios.

[0093] Through the above specific implementation methods in large-scale factory safety and surveillance scenarios, the system and method of the present invention can be effectively utilized to achieve efficient processing and analysis of safety and surveillance scene images, improve the factory's safety management level, and promptly discover and prevent the occurrence of safety accidents.

[0094] The safety and surveillance scene image processing and analysis device based on a multimodal large model, wherein the device comprises:

[0095] a processor configured to execute computer-executable instructions;

[0096] The memory stores one or more computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned safety and supervision scene image processing and analysis method based on the multimodal large model are implemented.

[0097] The safety and supervision scene image processing and analysis processor based on a multimodal large model, wherein the processor is configured to execute computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned safety and supervision scene image processing and analysis method based on a multimodal large model are implemented.

[0098] The computer-readable storage medium stores a computer program thereon, and the computer program can be executed by a processor to implement the steps of the above-mentioned safety and supervision scene image processing and analysis method based on a multimodal large model.

[0099] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0100] It should be understood that each part of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device.

[0101] Those skilled in the art will understand that all or part of the steps of the method for implementing the above-mentioned embodiment can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0102] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0103] Throughout this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "embodiment" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0104] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

[0105] The safety and surveillance scene image processing and analysis method, system, device, processor, and computer-readable storage medium thereof based on the multimodal large model of the present invention have the following technical advantages over the existing technology:

[0106] Improve detection accuracy: By combining a large multimodal model with a small business model, and fully leveraging the powerful understanding and generation capabilities of the large multimodal model and the professionalism of the small business model for safety and supervision scenarios, it is possible to more accurately detect targets, identify attributes, and classify safety and supervision scene images, thereby effectively improving detection accuracy and reducing misjudgments and missed judgments.

[0107] Enhanced data processing capabilities: The image screening module can effectively remove low-quality images and non-human samples, thereby improving data quality and reducing the interference of noisy data on model training. At the same time, it uses a large multimodal model to process low-confidence data and supplement new data to make model training data richer and more comprehensive, thereby enhancing the model's generalization ability.

[0108] Realize real-time and accurate early warning: This technical solution can process safety and surveillance scene images in real time, quickly identify abnormal situations and issue early warnings, helping staff to take timely measures, reduce the possibility of safety accidents, and ensure the safety of people and property in safety and surveillance scenes.

[0109] Improve model adaptability: The business model fine-tuning and optimization module can continuously update the small business model based on new data, enabling it to quickly adapt to the development and changes of safety and supervision business as well as emerging security risks and scenarios, and maintain good detection performance.

[0110] In this specification, the present invention has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and variations may be made without departing from the spirit and scope of the present invention. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive.

Claims

1. A safety and surveillance scene image processing and analysis method based on a multimodal large model, characterized in that: The method comprises the following steps: (1) Obtain images captured by the safety and security camera and identify human targets from them; (2) Based on the captured images, low-quality and non-human target images are screened out and the sample set is divided; (3) Use the multimodal large model to classify the acquired images, obtain the attribute labels of the safety and supervision scene images, and manually determine the high-confidence samples to obtain the training set, thereby generating a new business small model for image classification testing; (4) Update the current data set and perform model fine-tuning and optimization on the small business model.

2. The method for processing and analyzing safety and surveillance scene images based on a multimodal large model according to claim 1 is characterized in that: The step (1) specifically includes: (1.1) Obtain the original image captured or warned by the safety and security camera from the required analysis scene; (1.2) Use the Yolo or PaddleDetection visual detection model or framework to detect people, and set a lower screening threshold to filter out blurry, human-like images for subsequent supplementation of various types of sample data.

3. The safety and surveillance scene image processing and analysis method based on a multimodal large model according to claim 1 is characterized in that: The step (2) specifically includes: (2.1) First, low-quality images are screened to remove abnormal brightness in the dataset, including images that are too bright or too dark, as well as blurred images. These images are then processed uniformly according to task requirements and placed in the undeterminable category or temporarily excluded from the training dataset. (2.2) Perform human image screening to distinguish between human data with higher credibility and non-human data. Non-human data is added to the negative sample category, and human attribute data enters the next step of processing.

4. The method for processing and analyzing safety and surveillance scene images based on a multimodal large model according to claim 2 is characterized in that: The step (3) specifically includes: (3.1) First, determine whether a basic business model already exists. If not, proceed to step (3.2). Otherwise, proceed to step (3.4). (3.2) Using the multimodal large model to assist in generating a small business model, clearly classify attribute categories based on business needs and actual data distribution, and use the multimodal large model to perform attribute classification to obtain attribute labels for safety and security scene images; (3.3) Manually determine high-confidence sample attributes and generate a small model training data set to generate a small business model; (3.4) Attribute classification is performed on the existing business models, and corresponding result evaluation is performed. The obtained evaluation results are then processed in the next step.

5. The method for processing and analyzing safety and surveillance scene images based on a multimodal large model according to claim 4 is characterized in that: The step (4) specifically includes: (4.1) Based on the model classification confidence, the currently acquired data is divided into low confidence and high confidence parts; (4.2) For low-confidence data, use the multimodal large model to perform cause analysis and select reliable data to be added to the training set; (4.3) For high confidence data, it is added as a new training set to the small business model for data update, thereby achieving model fine-tuning and optimization.

6. The method for processing and analyzing safety and surveillance scene images based on a multimodal large model according to claim 5 is characterized in that: The step (4.2) specifically includes: (4.2.1) If there is low confidence data due to unqualified captured images, the multimodal large model can be used to adjust the camera angle and capture rules, or images containing only partial human bodies can be added to the unidentifiable category; (4.2.2) For low-confidence data caused by new attributes or new examples that have not been covered before, the multimodal large model is used to output the category judgment results, the sample attribute labels are modified accordingly, and they are added to the corresponding training dataset under manual review; (4.2.3) If there is small sample category data, use the multimodal large model to generate available sample supplementary data and add it to the training data set.

7. A safety and surveillance scene image processing and analysis system based on a multimodal large model using the method according to any one of claims 1 to 6, characterized in that: The system comprises: Image acquisition and person detection module, used to obtain the original image and identify whether the image contains human targets; An image screening module, connected to the image acquisition and person detection module, for screening out images with abnormal brightness, blurred images, and non-human images; A business model training and testing module, connected to the image screening module, is used to generate a small business model using a multimodal large model and perform classification tests based on model attributes; The business model fine-tuning and optimization module is connected to the business model training and testing module, and is used to update the training set based on the classification results and perform corresponding model fine-tuning and optimization processing.

8. A safety scene image processing and analysis device based on a multimodal large model, characterized in that: The device comprises: a processor configured to execute computer-executable instructions; A memory stores one or more computer-executable instructions, which, when executed by the processor, implement the steps of the safety and supervision scene image processing and analysis method based on a multimodal large model as described in any one of claims 1 to 6.

9. A safety and surveillance scene image processing and analysis processor based on a multimodal large model, characterized in that: The processor is configured to execute computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the safety and supervision scene image processing and analysis method based on a multimodal large model described in any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the safety and supervision scene image processing and analysis method based on a multimodal large model as described in any one of claims 1 to 6.

Images