Visual analysis method and device, electronic equipment and storage medium
Through the cooperation of fusion model and difficult sample analysis model, combined with visual big model and visual language big model, the problem of video security performance in complex scenarios is solved, the robustness and generalization ability of the model are improved, and the standardization of the model is reduced. Reliance on large-scale data.
Patent Information
- Application Number
- CN202410990251.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
AI Technical Summary
The existing video security visual analysis methods perform poorly in complex scenarios and have a high dependence on large-scale data, resulting in limited generalization capabilities of models in the absence of scenes or situations.
The fusion model is used for preliminary analysis, most security events are identified, and the difficult sample input and difficult sample analysis model are processed in depth. Combining the visual big model and the visual language big model, the analysis results are integrated to improve the robustness and generalization of the model. ability.
Through this division of labor and cooperation, misjudgment and misjudgment can be reduced, overall analysis accuracy will be improved, and dependence on large-scale data will be reduced, and the model's adaptability and generalization ability in new scenarios or new situations will be enhanced.
Smart Images

Figure CN118942004A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of image processing technology, and in particular to a visual analysis method and device, an electronic device, and a storage medium. Background Art
[0002] AI vision technology plays a core role in the field of video security. It simulates the human visual system through computers to understand and analyze video content, effectively improving the capabilities of security prevention, real-time monitoring and incident warning.
[0003] In the video security scenario, the core algorithms of AI vision mainly cover three basic categories: target classification, target detection, and image segmentation. There are two mainstream paths in the existing video security visual analysis methods: on the one hand, a single model is used to process a single task. This type of method relies on large-scale real-scene data sets for training to optimize model performance. On the other hand, a series of small models for a single task can be used to decompose complex video security problems into multiple interrelated subtasks, and these modules can be combined to collaboratively complete the overall video analysis. Although these two methods can achieve a high degree of customization and adapt to the needs of specific scenarios, they also face the need for a large amount of real-world data training, and due to the limited generalization ability of the model after task segmentation, it may perform poorly in new scenes or new situations that have not been seen. Therefore, how to reduce the dependence of video security models on large-scale data while ensuring analysis accuracy and improve the robustness and generalization ability of models in complex scenarios has become an urgent problem to be solved. Summary of the invention
[0004] The present disclosure provides a method and device for visual analysis, an electronic device and a storage medium, the main purpose of which is to reduce the reliance of video security models on large-scale data while ensuring the accuracy of analysis, and to improve the robustness and generalization ability of the model in complex scenarios.
[0005] According to a first aspect of the present disclosure, there is provided a method for visual analysis, comprising:
[0006] Preprocessing the video to be analyzed to obtain image frame data, and inputting the image frame data into the fusion model for analysis and processing to obtain a model recognition result;
[0007] Based on the model recognition result, identify the difficult-to-security samples in the image frame data, and remove the recognition results of the difficult-to-security samples from the model recognition result to obtain a first model analysis result;
[0008] Input the difficult security sample into the difficult sample analysis model for analysis and processing to obtain a second model analysis result;
[0009] The first model analysis result and the second model analysis result are integrated to obtain a visual analysis result.
[0010] In some embodiments, before inputting the image frame data into the fusion model for analysis and processing, the method further includes:
[0011] Analyze the analysis task requirements of the video to be analyzed, and determine the task categories and quantities of sub-models in the fusion model for processing the video to be analyzed, wherein the task categories include three task categories: image classification, target detection, and semantic segmentation;
[0012] The step of inputting the image frame data into a fusion model for analysis and processing to obtain a model recognition result includes:
[0013] According to the analysis task requirements, calling the sub-model of the corresponding task category to analyze and process the image frame data;
[0014] The analysis and processing results of the sub-models of different task categories in the fusion model are fused to obtain the model recognition result.
[0015] In some embodiments, the identifying the security samples in the image frame data based on the model recognition result includes:
[0016] Based on the model recognition result, extracting the maximum probability value corresponding to all classification labels in the recognition result of each of the image frame data;
[0017] The maximum probability value is used as the confidence score of the image frame data, and the image frame data with a confidence score lower than an evaluation threshold is marked as the security-difficult sample.
[0018] In some embodiments, the difficult sample analysis model includes a visual large model and a visual language large model; the inputting the difficult security sample into the difficult sample analysis model for analysis and processing to obtain a second model analysis result includes:
[0019] Identifying a model type of the fusion model, wherein the model type includes a first type and a second type;
[0020] If the model type is the first type, analyzing and processing the security difficult sample based on the visual large model to obtain a second model analysis result;
[0021] If the model type is the second type, the security difficult sample is analyzed and processed based on the visual language large model to obtain a second model analysis result.
[0022] In some embodiments, identifying the model type of the fusion model includes:
[0023] Identify the task categories and quantities of sub-models in the fusion model, where different model types are composed of sub-models of different task categories or quantities;
[0024] If the fusion model is generated by combining sub-models of a single category, or by combining sub-models of a single number and not more than two categories, then the fusion model is determined to be of the first type;
[0025] If the fusion model is generated by combining sub-models of multiple categories and multiple numbers, the fusion model is determined to be the second type.
[0026] In some embodiments, the method further comprises:
[0027] The training image frame data is input into the fusion model for training, and the training security difficult samples are input into the difficult sample analysis model for training to obtain the trained fusion model and difficult sample analysis model.
[0028] According to a second aspect of the present disclosure, there is provided a visual analysis device, comprising:
[0029] The first analysis unit is used to pre-process the video to be analyzed to obtain image frame data, and input the image frame data into the fusion model for analysis and processing to obtain a model recognition result;
[0030] an identification unit, configured to identify difficult-to-security samples in the image frame data based on the model identification result, and remove the identification result of the difficult-to-security samples from the model identification result to obtain a first model analysis result;
[0031] A second analysis unit is used to input the difficult security sample into the difficult sample analysis model for analysis and processing to obtain a second model analysis result;
[0032] An integration unit is used to integrate the first model analysis result and the second model analysis result to obtain a visual analysis result.
[0033] In some embodiments, the apparatus further comprises:
[0034] A determination unit, used for analyzing the analysis task requirements of the video to be analyzed before inputting the image frame data into the fusion model for analysis and processing, and determining the task categories and quantity of sub-models in the fusion model for processing the video to be analyzed, wherein the task categories include three task categories: image classification, target detection, and semantic segmentation;
[0035] The first analysis unit comprises:
[0036] An analysis module, used to call a sub-model of a corresponding task category to analyze and process the image frame data according to the analysis task requirements;
[0037] The fusion module is used to fuse the analysis and processing results of the sub-models of different task categories in the fusion model to obtain the model recognition result.
[0038] In some embodiments, the identification unit comprises:
[0039] An extraction module, used for extracting the maximum probability value corresponding to all classification labels in the recognition result of each of the image frame data based on the model recognition result;
[0040] A marking module is used to use the maximum probability value as the confidence score of the image frame data, and mark the image frame data with a confidence score lower than an evaluation threshold as the security-difficult sample.
[0041] In some embodiments, the difficult sample analysis model includes a visual large model and a visual language large model; the second analysis unit includes:
[0042] an identification module, used to identify a model type of the fusion model, wherein the model type includes a first type and a second type;
[0043] A first analysis module, used for analyzing and processing the security difficult sample based on the visual large model to obtain a second model analysis result when the model type is the first type;
[0044] The second analysis module is used to analyze and process the difficult security sample based on the visual language large model when the model type is the second type, so as to obtain a second model analysis result.
[0045] In some embodiments, the identification module is further configured to:
[0046] Identify the task categories and quantities of sub-models in the fusion model, where different model types are composed of sub-models of different task categories or quantities;
[0047] If the fusion model is generated by combining sub-models of a single category, or by combining sub-models of a single number and not more than two categories, then the fusion model is determined to be of the first type;
[0048] If the fusion model is generated by combining sub-models of multiple categories and multiple numbers, the fusion model is determined to be the second type.
[0049] In some embodiments, the apparatus further comprises:
[0050] The training unit is used to input the training image frame data into the fusion model for training, and input the training security difficult samples into the difficult sample analysis model for training, so as to obtain the trained fusion model and the difficult sample analysis model.
[0051] According to a third aspect of the present disclosure, there is provided an electronic device, including:
[0052] at least one processor; and
[0053] a memory communicatively connected to the at least one processor; wherein,
[0054] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.
[0055] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.
[0056] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method as described in the first aspect above.
[0057] The present disclosure provides a method and device for visual analysis, an electronic device and a storage medium, which pre-process the video to be analyzed to obtain image frame data, and input the image frame data into a fusion model for analysis and processing to obtain a model recognition result; based on the model recognition result, identify the difficult security samples in the image frame data, and remove the recognition results of the difficult security samples from the model recognition result to obtain a first model analysis result; input the difficult security samples into a difficult sample analysis model for analysis and processing to obtain a second model analysis result; integrate the first model analysis result and the second model analysis result to obtain a visual analysis result. Compared with the related art, the embodiment of the present disclosure can capture the key information in the image frame data more comprehensively by combining the fusion model and the difficult sample analysis model for analysis and processing. The fusion model performs preliminary analysis and identifies most of the security events; while the difficult sample analysis model focuses on processing difficult security samples, that is, conducting in-depth analysis of complex or ambiguous situations that are difficult for the first model to accurately identify. This division of labor and cooperation helps to reduce misjudgments and missed judgments and improve the overall analysis accuracy. By introducing a processing mechanism for difficult security samples, existing data sets can be used more efficiently. Traditional video security methods usually require a large amount of real-world data for model training, but this technical means can reduce the dependence on large-scale data sets to a certain extent. Through the preliminary screening of the fusion model and the targeted processing of the difficult sample analysis model, more accurate analysis results can be achieved with limited data resources. When processing difficult security samples, the difficult sample analysis model can focus on these complex and special situations, thereby enhancing the adaptability of the model in new scenes or new situations. This technical means helps to improve the robustness of the model in complex scenes, enabling it to better cope with various uncertainties and changes. At the same time, through the targeted processing of difficult samples, the generalization ability of the model will also be improved accordingly.
[0058] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0060] Figure 1 A flowchart of a visual analysis method provided by an embodiment of the present disclosure;
[0061] Figure 2 A flowchart of another visual analysis method provided by an embodiment of the present disclosure;
[0062] Figure 3A flowchart of another visual analysis method provided by an embodiment of the present disclosure;
[0063] Figure 4 A schematic diagram of the structure of a visual analysis device provided by an embodiment of the present disclosure;
[0064] Figure 5 A schematic diagram of the structure of another visual analysis device provided by an embodiment of the present disclosure;
[0065] Figure 6 A schematic block diagram of an exemplary electronic device provided for an embodiment of the present disclosure. DETAILED DESCRIPTION
[0066] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0067] The following describes the visual analysis method and apparatus, electronic device, and storage medium according to embodiments of the present disclosure with reference to the accompanying drawings.
[0068] Figure 1 A flowchart of a visual analysis method provided in an embodiment of the present disclosure.
[0069] like Figure 1 As shown, the method comprises the following steps:
[0070] Step 101 , pre-processing the video to be analyzed to obtain image frame data, and inputting the image frame data into a fusion model for analysis and processing to obtain a model recognition result.
[0071] In the embodiments of the present disclosure, the purpose of preprocessing is to extract key information from the video to be analyzed, eliminate noise and interference, so as to generate image data that can be processed by the fusion model, and facilitate subsequent model analysis to be more accurate and efficient. During the preprocessing process, the video is first decoded and decomposed into a series of continuous image frames. These image frames are the basic units of video analysis, and they contain all the visual information of the video content. Then, through a series of image processing techniques, such as scaling, cropping, denoising, etc., the image frames are optimized to make them more in line with the requirements of model analysis.
[0072] After preprocessing, the image frame data will be input into the fusion model for further analysis and processing. The fusion model is a complex system that integrates multiple single sub-models. It can make full use of the advantages of different models and improve the overall analysis performance. In the fusion model, the image frame data will go through a series of complex calculations and reasoning processes to obtain the model recognition results.
[0073] The recognition result of this model is an in-depth understanding and interpretation of the content of the image frame, which may include target detection, classification, tracking and other information. This information provides an important basis for subsequent security prevention and decision-making.
[0074] Step 102: Based on the model recognition result, identify the difficult-to-security samples in the image frame data, and remove the recognition results of the difficult-to-security samples from the model recognition result to obtain a first model analysis result.
[0075] In the embodiments of the present disclosure, after obtaining the model recognition results, the next key step is to identify the difficult security samples. Difficult security samples generally refer to image frame data that are difficult for the model to accurately recognize due to various factors (such as blur, occlusion, angle problems or abnormal behavior, etc.). These difficult samples may interfere with the overall analysis results and reduce the accuracy of the analysis.
[0076] In order to identify these difficult samples, a series of strategies and methods are usually adopted. First, they will be screened based on the confidence or probability score of the model recognition results, and those samples with low confidence or abnormal score distribution will be marked as potential difficult samples. Secondly, some special algorithms or technologies, such as anomaly detection algorithms and attention mechanisms in deep learning, will be combined to further analyze and determine whether these samples are truly difficult security samples.
[0077] After identifying the difficult samples for security, the corresponding recognition results are removed from the model recognition results. The purpose of this step is to eliminate the impact of difficult samples on the overall analysis results, making the remaining results more accurate and reliable. The model recognition results after removing the difficult samples are called the first model analysis results. This result mainly includes the image frame data that the model can confidently and accurately recognize, providing an important basis for subsequent security decisions.
[0078] Step 103: input the difficult security sample into the difficult sample analysis model for analysis and processing to obtain a second model analysis result.
[0079] In the embodiments of the present disclosure, when using the fusion model for visual task analysis, some samples that are difficult to be accurately identified by the existing model are often encountered, which are called "difficult security samples". These difficult samples often contain complex and changing backgrounds, blurred target features or extreme posture changes, which makes it easy for the model to misjudge or miss detection when classifying or detecting them.
[0080] The difficult sample analysis model is an advanced model that performs in-depth analysis on samples that are difficult to be accurately identified or processed by the fusion model. This may include but is not limited to large visual models, large visual language models, etc. The difficult sample analysis model can perform in-depth analysis and processing on the input security difficult samples to obtain more accurate analysis results. Specifically, by inputting security difficult samples into the model, the model will use its strong versatility and zero (few) sample capabilities to perform refined analysis and judgment on these samples. In this way, the accuracy of the model can be effectively improved, allowing it to respond more calmly when faced with complex and changing security scenarios.
[0081] In addition, this method can also effectively solve the problem of difficulty in obtaining training data for some video security scenarios. In practical applications, due to various reasons, it is often difficult to obtain sufficient and high-quality training data. By utilizing the strong versatility and zero (few) sample capabilities of large models based on massive data training, we can make up for this deficiency to a certain extent, so that the model can perform well even with little or no training data.
[0082] Step 104: Integrate the first model analysis result and the second model analysis result to obtain a visual analysis result.
[0083] In the embodiment of the present disclosure, after preliminary analysis of the image frame data, the first model analysis result is obtained. However, considering the complexity and variability of security scenes, a single model analysis result may not be able to accurately identify difficult security samples. Therefore, the difficult sample analysis model is further used to conduct in-depth analysis and processing of difficult security samples, and the second model analysis result is obtained.
[0084] In order to obtain more comprehensive and accurate visual analysis results, the analysis results of the first model are integrated with those of the second model. This integration process fully considers the characteristics and advantages of the analysis results of the two models, and finally forms a more complete and accurate visual analysis result through comparison, verification and complementarity. This integration result not only includes the basic characteristics and attributes of difficult samples, but also conducts an in-depth analysis of their specific performance in security scenarios. This can better understand the nature and characteristics of difficult samples, thereby providing more accurate and effective support for their application in the security field.
[0085] The present disclosure provides a method for visual analysis, which pre-processes a video to be analyzed to obtain image frame data, and inputs the image frame data into a fusion model for analysis and processing to obtain a model recognition result; based on the model recognition result, identifies a difficult security sample in the image frame data, and removes the recognition result of the difficult security sample from the model recognition result to obtain a first model analysis result; inputs the difficult security sample into a difficult sample analysis model for analysis and processing to obtain a second model analysis result; integrates the first model analysis result with the second model analysis result to obtain a visual analysis result. Compared with the related art, the embodiment of the present disclosure can capture the key information in the image frame data more comprehensively by combining the fusion model and the difficult sample analysis model for analysis and processing. The fusion model performs preliminary analysis and identifies most security events; while the difficult sample analysis model focuses on processing difficult security samples, that is, conducting in-depth analysis of complex or ambiguous situations that are difficult for the first model to accurately identify. This division of labor and cooperation helps to reduce misjudgments and missed judgments and improve the overall analysis accuracy. By introducing a processing mechanism for difficult security samples, existing data sets can be used more efficiently. Traditional video security methods usually require a large amount of real-world data for model training, but this technical means can reduce the dependence on large-scale data sets to a certain extent. Through the preliminary screening of the fusion model and the targeted processing of the difficult sample analysis model, more accurate analysis results can be achieved with limited data resources. When processing difficult security samples, the difficult sample analysis model can focus on these complex and special situations, thereby enhancing the adaptability of the model in new scenes or new situations. This technical means helps to improve the robustness of the model in complex scenes, enabling it to better cope with various uncertainties and changes. At the same time, through the targeted processing of difficult samples, the generalization ability of the model will also be improved accordingly.
[0086] In order to clearly illustrate the embodiments of the present disclosure, this embodiment provides a flowchart of another visual analysis method.
[0087] like Figure 2 As shown, the method comprises the following steps:
[0088] Step 201, analyze the analysis task requirements of the video to be analyzed, and determine the task categories and quantities of sub-models in the fusion model for processing the video to be analyzed, wherein the task categories include three task categories: image classification, target detection, and semantic segmentation.
[0089] Step 202: According to the analysis task requirements, call the sub-model of the corresponding task category to analyze and process the image frame data.
[0090] Step 203, fusing the analysis and processing results of the sub-models of different task categories in the fusion model to obtain the model recognition result.
[0091] Specifically, in step 201 to step 203, the fusion model combines a variety of different sub-models to cope with various analysis tasks. These sub-models mainly include three task categories: image classification, target detection, and semantic segmentation, each of which has excellent performance in its specific field. Image classification sub-models include but are not limited to deep learning models such as VGG, ResNet, InceptionNetwork, MobileNet, etc. for image classification; target detection sub-models include but are not limited to deep learning models such as Faster R-CNN, YOLO, RetinaNet, DETR, etc. for target detection; semantic segmentation sub-models include but are not limited to deep learning models such as FCN, SegNet, DeepLab, PSPNet, etc. for semantic segmentation.
[0092] Before analyzing the video to be analyzed, it is necessary to conduct an in-depth analysis of the analysis task requirements. This involves understanding the video content, clarifying the analysis objectives, and defining the tasks to be processed. Through this step, the task categories and number of sub-models that need to be included in the fusion model can be accurately determined.
[0093] Once the analysis task requirements are determined, the sub-models of the corresponding task categories can be called according to these requirements to analyze and process the image frame data. For example, if the task requirements include object detection, we need to call the sub-model that excels in object detection to identify and locate the key objects in the video. Similarly, if image classification or semantic segmentation is required, there will also be corresponding sub-models to handle these tasks.
[0094] After completing the analysis and processing of each sub-model, their results are fused. This step is crucial because it ensures that the information obtained by the sub-models of different task categories can complement and verify each other, thereby obtaining a more accurate and comprehensive model recognition result. Through fusion processing, we can maximize the advantages of each sub-model while making up for their possible limitations.
[0095] Step 204: based on the model recognition result, extract the maximum probability value corresponding to all classification labels in the recognition result of each of the image frame data.
[0096] Step 205: Use the maximum probability value as the confidence score of the image frame data, and mark the image frame data with a confidence score lower than an evaluation threshold as the difficult security sample.
[0097] Specifically, in step 204 to step 205, due to the complexity and changeability of the real scene, the prediction based on the fusion model will have a certain amount of misunderstanding. This proposal proposes the concept of "Security Hard Sample" (SHS) to classify the images according to their difficulty. In order to calculate the difficulty of the image x under the small model, as shown in formula (1), the maximum probability of all labels is calculated as the confidence score.
[0098] s(x) = max y∈Y p(y|x) (1)
[0099] Among them, p(y|x) represents the probability that x has the label y calculated by the fusion model. Samples with confidence scores below the threshold are recorded as SHS. The image corresponding to this sample will be used as the input of the hard sample analysis model.
[0100] Step 206: input the difficult security sample into the difficult sample analysis model for analysis and processing to obtain a second model analysis result.
[0101] Step 207: Integrate the first model analysis result and the second model analysis result to obtain a visual analysis result.
[0102] For the description of step 206 and step 207, please refer to the description of the above embodiment, which will not be described in detail in this embodiment.
[0103] In order to clearly illustrate the embodiments of the present disclosure, this embodiment provides a flowchart of another visual analysis method.
[0104] like Figure 3 As shown, the method comprises the following steps:
[0105] Step 301 , pre-processing the video to be analyzed to obtain image frame data, and inputting the image frame data into a fusion model for analysis and processing to obtain a model recognition result.
[0106] Step 302: Based on the model recognition result, identify the difficult-to-security samples in the image frame data, and remove the recognition results of the difficult-to-security samples from the model recognition result to obtain a first model analysis result.
[0107] For the description of step 301 to step 302, please refer to the above-mentioned embodiment of the present disclosure, which will not be described in detail in this embodiment.
[0108] Step 303: Identify the model type of the fusion model, wherein the model type includes a first type and a second type.
[0109] Specifically, in step 303, the model type of the fusion model may be identified in the following manners, but not limited to:
[0110] Identify the task category and quantity of sub-models in the fusion model. Different model types are composed of sub-models of different task categories or quantities. If the fusion model is generated by a combination of sub-models of a single category, or a combination of sub-models of a single number and not more than two categories, then the fusion model is determined to be of the first type. If the fusion model is generated by a combination of sub-models of multiple categories and multiple numbers, then the fusion model is determined to be of the second type.
[0111] Exemplarily, in the disclosed embodiment, the model types of the fusion model are divided into long-tail category (i.e., the aforementioned "first type") and perception category (i.e., the aforementioned "second type"). Long-tail category: Algorithms with low task complexity, clear task types, and relatively objective usage scenarios. For example, products such as dangerous dog identification, pedestrian detection, and road water detection can be directly decomposed into traditional classification, detection, and segmentation algorithms. This type of scenario requires high accuracy of the visual analysis algorithm, and there is often a certain deviation between the distribution of training data and real scene data, and there is a high probability of misunderstanding in actual applications.
[0112] Perception: Algorithms with high algorithm task complexity, diverse task types, and certain subjective understanding. For example, products such as road occupation, door-to-door identification, and mobile vendors require comprehensive consideration of the surrounding conditions of the target object for overall analysis and understanding. Tasks in this type of scenario are difficult and usually require a combination of multiple traditional visual analysis algorithms for prediction. It is difficult to obtain training data, and there is a risk that the lower limit of the algorithm accuracy is too low when used in real scenarios.
[0113] For example, the algorithm R of the fusion model is classified into i , Detection i and segmentation i If R can only be split into a single type of c i or i or i (such as c1c2, d1d2), then R is counted as a long-tail algorithm; if R can be split into a single number of c i or i or i If there are no more than two combinations (such as c1d1 and c1s1), R is counted as a long-tail algorithm; if R can be split into multiple numbers of multiple categories of c i or i or i If there is no combination (such as c1d1s1, c1d1d2), R is counted as a perception algorithm.
[0114] If the model type of the fusion model is the first type (long-tail class), execute step 304; if the model type of the fusion model is the second type (perception class), execute step 305.
[0115] Step 304: Analyze and process the difficult security sample based on the visual large model to obtain a second model analysis result.
[0116] Specifically, in step 304, the difficult sample analysis model includes a vision foundation model (VFM) and a vision language foundation model (VLM).
[0117] The visual large model (VFM) in the difficult sample analysis model has excellent feature extraction capabilities through pre-training with a large amount of data. This enables the model to capture key information more accurately and improve the accuracy of analysis results when processing difficult security samples. VFM is not only suitable for big data scenarios, but also performs well in small sample scenarios. This means that even when the training data is limited, the model can still maintain a high level of analysis accuracy, effectively solving the problem of difficulty in obtaining training data in the security field. By superimposing the Head of classification, detection and segmentation tasks, the difficult sample analysis model can realize multi-task parallel processing. This design not only improves the efficiency of the model, but also enables the model to cope with multiple security needs at the same time, enhancing its practicality.
[0118] Step 305: Analyze and process the difficult security sample based on the visual language large model to obtain a second model analysis result.
[0119] Specifically in step 305, the introduction of the visual language model (VLM) enables the difficult sample analysis model to have powerful visual language processing capabilities. Through the pre-training of massive image-text pairs, Vision Encoder can extract rich visual features and provide strong support for downstream task processing. In VLM, No Backbone Head is used for downstream task processing, and there is no need to train the Backbone part of the Decoder module. This design simplifies the model structure, reduces training costs, and ensures the performance of the model.
[0120] Step 306: Integrate the first model analysis result and the second model analysis result to obtain a visual analysis result.
[0121] Specifically, in step 306, after preliminary analysis of the image frame data, the first model analysis result is obtained. However, considering the complexity and variability of security scenes, a single model analysis result may not be able to accurately identify difficult security samples. Therefore, the difficult sample analysis model is further used to conduct in-depth analysis and processing of difficult security samples, and the second model analysis result is obtained.
[0122] Before using the fusion model and the difficult sample analysis model for visual task analysis, it is necessary to train the fusion model and the difficult sample analysis model so that the model can better meet the needs of the visual analysis task and improve the robustness and generalization ability of the model in complex scenarios.
[0123] As an achievable method of an embodiment of the present disclosure, model training of the fusion model and the difficult sample analysis model may be performed in the following manner but is not limited to: inputting training image frame data into the fusion model for training, and inputting training security difficult samples into the difficult sample analysis model for training, to obtain a trained fusion model and a difficult sample analysis model.
[0124] In the training phase of the model, the training image frame data is input into the fusion model so that it can be fully trained. This step is crucial to ensure the accuracy and stability of the fusion model when processing various security scenarios. Through training, the fusion model can learn and understand the key features in the image frame data, thereby improving its performance in tasks such as image classification, target detection and semantic segmentation. At the same time, the security difficult samples used for training are input into the difficult sample analysis model for training. The difficult sample analysis model has powerful feature extraction and few-sample scene analysis capabilities, and can perform refined learning for complex features and difficult-to-identify targets in security difficult samples. Through this training process, the difficult sample analysis model can more accurately identify and analyze security difficult samples, further improving its practicality in the security field. After training, the trained fusion model and difficult sample analysis model are obtained. These two models not only have powerful processing capabilities, but also can play an important role in the security field. The fusion model can comprehensively use sub-models of multiple task categories to conduct a comprehensive and accurate analysis of image frame data; while the difficult sample analysis model can perform special processing for security difficult samples to improve the accuracy and efficiency of the entire security system.
[0125] It should be noted that the embodiments of the present disclosure may include multiple steps. For the convenience of description, these steps are numbered, but these numbers do not limit the execution time slots or execution order between the steps; these steps can be implemented in any order, and the embodiments of the present disclosure do not limit this.
[0126] Corresponding to the above-mentioned visual analysis method, the present invention also provides a visual analysis device. Since the device embodiment of the present invention corresponds to the above-mentioned method embodiment, details not disclosed in the device embodiment can be referred to the above-mentioned method embodiment, and will not be repeated in the present invention.
[0127] Figure 4 A schematic diagram of a visual analysis device provided by an embodiment of the present disclosure is shown in FIG. Figure 4 As shown, including:
[0128] The first analysis unit 41 is used to pre-process the video to be analyzed to obtain image frame data, and input the image frame data into the fusion model for analysis and processing to obtain a model recognition result;
[0129] The identification unit 42 is used to identify the difficult-to-security samples in the image frame data based on the model identification result, and remove the identification result of the difficult-to-security samples from the model identification result to obtain a first model analysis result;
[0130] The second analysis unit 43 is used to input the difficult security sample into the difficult sample analysis model for analysis and processing to obtain a second model analysis result;
[0131] The integration unit 44 is used to integrate the first model analysis result and the second model analysis result to obtain a visual analysis result.
[0132] The present disclosure provides a visual analysis device, which pre-processes a video to be analyzed to obtain image frame data, and inputs the image frame data into a fusion model for analysis and processing to obtain a model recognition result; based on the model recognition result, identifies a security difficult sample in the image frame data, and removes the recognition result of the security difficult sample from the model recognition result to obtain a first model analysis result; inputs the security difficult sample into a difficult sample analysis model for analysis and processing to obtain a second model analysis result; integrates the first model analysis result and the second model analysis result to obtain a visual analysis result. Compared with the related art, the embodiment of the present disclosure can capture the key information in the image frame data more comprehensively by combining the fusion model and the difficult sample analysis model for analysis and processing. The fusion model performs preliminary analysis and identifies most of the security events; while the difficult sample analysis model focuses on processing security difficult samples, that is, conducting in-depth analysis of complex or ambiguous situations that are difficult for the first model to accurately identify. This division of labor and cooperation helps to reduce misjudgments and missed judgments and improve the overall analysis accuracy. By introducing a processing mechanism for security difficult samples, existing data sets can be used more efficiently. Traditional video security methods usually require a large amount of real-world data for model training, but this technical means can reduce the dependence on large-scale data sets to a certain extent. Through the preliminary screening of the fusion model and the targeted processing of the difficult sample analysis model, more accurate analysis results can be achieved with limited data resources. When processing difficult security samples, the difficult sample analysis model can focus on these complex and special situations, thereby enhancing the adaptability of the model in new scenes or new situations. This technical means helps to improve the robustness of the model in complex scenes, enabling it to better cope with various uncertainties and changes. At the same time, through the targeted processing of difficult samples, the generalization ability of the model will also be improved accordingly.
[0133] Furthermore, in a possible implementation of this embodiment, as Figure 5 As shown, the device also includes:
[0134] The determination unit 45 is used to analyze the analysis task requirements of the video to be analyzed before inputting the image frame data into the fusion model for analysis and processing, and determine the task categories and quantity of sub-models in the fusion model for processing the video to be analyzed, wherein the task categories include three task categories: image classification, object detection and semantic segmentation;
[0135] The first analysis unit 41 includes:
[0136] The analysis module 411 is used to call the sub-model of the corresponding task category to analyze and process the image frame data according to the analysis task requirements;
[0137] The fusion module 412 is used to fuse the analysis and processing results of the sub-models of different task categories in the fusion model to obtain the model recognition result.
[0138] Furthermore, in a possible implementation of this embodiment, as Figure 5 As shown, the identification unit 42 includes:
[0139] An extraction module 421 is used to extract the maximum probability value corresponding to all classification labels in the recognition result of each of the image frame data based on the model recognition result;
[0140] The marking module 422 is used to use the maximum probability value as the confidence score of the image frame data, and mark the image frame data with a confidence score lower than an evaluation threshold as the security-difficult sample.
[0141] Furthermore, in a possible implementation of this embodiment, as Figure 5 As shown, the difficult sample analysis model includes a visual large model and a visual language large model; the second analysis unit 43 includes:
[0142] An identification module 431, used to identify the model type of the fusion model, wherein the model type includes a first type and a second type;
[0143] A first analysis module 432 is used for analyzing and processing the security difficult sample based on the visual large model to obtain a second model analysis result when the model type is the first type;
[0144] The second analysis module 433 is used to analyze and process the security difficult sample based on the visual language large model when the model type is the second type, so as to obtain a second model analysis result.
[0145] Furthermore, in a possible implementation of this embodiment, the identification module 431 is further configured to:
[0146] Identify the task categories and quantities of sub-models in the fusion model, where different model types are composed of sub-models of different task categories or quantities;
[0147] If the fusion model is generated by combining sub-models of a single category, or by combining sub-models of a single number and not more than two categories, then the fusion model is determined to be of the first type;
[0148] If the fusion model is generated by combining sub-models of multiple categories and multiple numbers, the fusion model is determined to be the second type.
[0149] Furthermore, in a possible implementation of this embodiment, as Figure 5As shown, the device also includes:
[0150] The training unit 46 is used to input the training image frame data into the fusion model for training, and input the training security difficult samples into the difficult sample analysis model for training, so as to obtain the trained fusion model and the difficult sample analysis model.
[0151] It should be noted that the above explanation of the method embodiment is also applicable to the device of this embodiment, and the principle is the same, which is not limited in this embodiment.
[0152] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0153] Figure 6 A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0154] like Figure 6 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 502 or a computer program loaded from a storage unit 508 to a RAM (Random Access Memory) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An I / O (Input / Output) interface 505 is also connected to the bus 504.
[0155] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0156] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as a method of visual analysis. For example, in some embodiments, the method of visual analysis may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the aforementioned visual analysis method in any other appropriate manner (for example, by means of firmware).
[0157] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0158] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0159] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0160] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0161] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.
[0162] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.
[0163] It should be noted that artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.
[0164] The various numerical numbers such as first and second involved in the present disclosure are only for the convenience of description and are not used to limit the scope of the embodiments of the present disclosure, but also indicate the order of precedence.
[0165] At least one in the present disclosure may also be described as one or more, and a plurality may be two, three, four or more, which is not limited in the present disclosure. In the embodiments of the present disclosure, for a technical feature, the technical features in the technical feature are distinguished by "first", "second", "third", "A", "B", "C" and "D", etc., and there is no order of precedence or size between the technical features described by the "first", "second", "third", "A", "B", "C" and "D".
[0166] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0167] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A method of visual analysis, characterized in that: include: Preprocessing the video to be analyzed to obtain image frame data, and inputting the image frame data into the fusion model for analysis and processing to obtain a model recognition result; Based on the model recognition result, identify the difficult-to-security samples in the image frame data, and remove the recognition results of the difficult-to-security samples from the model recognition result to obtain a first model analysis result; Input the difficult security sample into the difficult sample analysis model for analysis and processing to obtain a second model analysis result; The first model analysis result and the second model analysis result are integrated to obtain a visual analysis result.
2. The method according to claim 1, characterized in that: Before inputting the image frame data into the fusion model for analysis and processing, the method further includes: Analyze the analysis task requirements of the video to be analyzed, and determine the task categories and quantities of sub-models in the fusion model for processing the video to be analyzed, wherein the task categories include three task categories: image classification, target detection, and semantic segmentation; The step of inputting the image frame data into a fusion model for analysis and processing to obtain a model recognition result includes: According to the analysis task requirements, calling the sub-model of the corresponding task category to analyze and process the image frame data; The analysis and processing results of the sub-models of different task categories in the fusion model are fused to obtain the model recognition result.
3. The method according to claim 1, characterized in that The step of identifying the difficult-to-security samples in the image frame data based on the model recognition result includes: Based on the model recognition result, extracting the maximum probability value corresponding to all classification labels in the recognition result of each of the image frame data; The maximum probability value is used as the confidence score of the image frame data, and the image frame data with a confidence score lower than an evaluation threshold is marked as the security-difficult sample.
4. The method according to claim 1, characterized in that The difficult sample analysis model includes a visual large model and a visual language large model; The step of inputting the difficult security sample into the difficult sample analysis model for analysis and processing to obtain a second model analysis result includes: Identifying a model type of the fusion model, wherein the model type includes a first type and a second type; If the model type is the first type, analyzing and processing the security difficult sample based on the visual large model to obtain a second model analysis result; If the model type is the second type, the security difficult sample is analyzed and processed based on the visual language large model to obtain a second model analysis result.
5. The method according to claim 4, characterized in that The identifying the model type of the fusion model includes: Identify the task categories and quantities of sub-models in the fusion model, where different model types are composed of sub-models of different task categories or quantities; If the fusion model is generated by combining sub-models of a single category, or by combining sub-models of a single number and not more than two categories, then the fusion model is determined to be of the first type; If the fusion model is generated by combining sub-models of multiple categories and multiple numbers, the fusion model is determined to be the second type.
6. The method according to claim 1, characterized in that The method further comprises: The training image frame data is input into the fusion model for training, and the training security difficult samples are input into the difficult sample analysis model for training to obtain the trained fusion model and difficult sample analysis model.
7. A visual analysis device, characterized in that: include: The first analysis unit is used to pre-process the video to be analyzed to obtain image frame data, and input the image frame data into the fusion model for analysis and processing to obtain a model recognition result; an identification unit, configured to identify a difficult-to-security sample in the image frame data based on the model identification result, and remove the identification result of the difficult-to-security sample from the model identification result to obtain a first model analysis result; A second analysis unit is used to input the difficult security sample into the difficult sample analysis model for analysis and processing to obtain a second model analysis result; An integration unit is used to integrate the first model analysis result and the second model analysis result to obtain a visual analysis result.
8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.
10. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 6.
