Early warning method and device based on multi-modal large model, equipment and medium

Abstract

The invention relates to an early warning method and device based on a multi-modal large model, equipment and a medium, and belongs to the technical field of deep learning. The method comprises the steps of collecting a real-time video stream; receiving the real-time video stream, and performing frame extraction operation on the real-time video stream to obtain a key frame image; calling a classification model to classify the key frame images, and screening out to-be-analyzed images needing further early warning analysis; inputting the to-be-analyzed image into a multi-modal large model, understanding the content of the to-be-analyzed image to generate image description, and generating an early warning result of the to-be-analyzed image in combination with the information in the constructed knowledge base and the image description; and outputting the obtained early warning result in real time. According to the method, the classification model is adopted to perform preliminary screening on the images, the images which do not need early warning obviously are screened out in advance, and the images which need to be further screened enter the multi-mode large model to be analyzed, so that the cost of the system can be effectively reduced.

Description

Technical Field

[0001] The present application belongs to the field of deep learning technology, and specifically relates to an early warning method, device, equipment and medium based on a multimodal large model. Background Art

[0002] Traditional early warning systems usually rely on fixed rules or algorithms based on predefined scene rules or image recognition models. When faced with complex and more semantic real-world environments, these fixed rules may not fully capture all important variables and relationships, resulting in limited analytical capabilities. Traditional early warning systems often focus on detecting significant visual features or behaviors that conform to specific patterns, but may ignore the context in which the event occurred, such as time, location, previous events, and other related information. The lack of integrated analysis of this contextual information may cause the system to be unable to correctly interpret the observed behavior, resulting in false positives or negatives. Traditional early warning systems are often designed for specific application scenarios and have poor adaptability to new situations in subsequent operations. This is because updating the system's rules or models requires time and resources, especially when these systems rely on heavily customized hardware or software. In a rapidly changing environment, this slow update speed makes it difficult to meet the needs of real-time response.

[0003] In addition, although the use of large-scale multimodal models can improve the understanding and accuracy of early warning systems, they usually require powerful computing resources to handle large amounts of data input and complex data processing tasks. This reliance on high-performance hardware significantly increases the cost of the system, making large-scale commercial deployment difficult and expensive.

[0004] Therefore, how to improve the design and functional limitations of the existing early warning system is a topic worthy of study. Summary of the invention

[0005] In view of the above analysis, the embodiments of the present invention aim to provide an early warning method, apparatus, device and medium based on a multimodal large model, aiming to overcome the shortcomings of existing early warning systems, such as limited analysis capabilities, lack of contextual understanding, difficulty in adapting to new scenarios and high hardware costs.

[0006] In one aspect of the present application, there is provided an early warning method based on a multimodal large model, comprising:

[0007] Collect real-time video streams;

[0008] Receiving the real-time video stream, performing frame extraction operation on the real-time video stream to obtain a key frame image;

[0009] Calling a classification model to classify the key frame images, and screening out images to be analyzed that require further early warning analysis;

[0010] The image to be analyzed is input into the multimodal large model, the content of the image to be analyzed is understood to generate an image description, and the information in the constructed knowledge base and the image description are combined to generate an early warning result for the image to be analyzed; wherein the knowledge base includes a scene description of the set scene, safety rules, hazard identification information, and prior knowledge of historical violation cases;

[0011] The obtained warning results are output in real time.

[0012] Optionally, calling the classification model to classify the key frame images and screening out the images to be analyzed that require further early warning analysis includes:

[0013] Pre-train the CLIP multimodal learning model, define text labels for each potential warning scenario, and use CLIP's text editor to convert the text labels into text label vectors;

[0014] Loading the CLIP multimodal learning model, converting the key frame image into an image feature vector through the CLIP image encoder;

[0015] Calculating the similarity between each image feature vector and the text label vector, and classifying the key frame image into the label category with the highest similarity based on the similarity score;

[0016] When the similarity between the key frame image and the label exceeds a preset threshold, the image is marked as an image to be analyzed that requires further early warning analysis.

[0017] Optionally, calling the classification model to classify the key frame images and screening out the images to be analyzed that require further early warning analysis includes:

[0018] Loading a pre-trained YOLO model; the YOLO model is a model that uses pre-defined object categories as warning indicators according to business requirements or security rules;

[0019] The key frame image is input into the YOLO model, the output result of the YOLO model is obtained, and the image containing the warning indicator is marked as an image to be analyzed that requires further warning analysis.

[0020] Optionally, the combining of the information in the constructed knowledge base and the image description to generate a warning result for the image to be analyzed includes:

[0021] Match the content of the image to be analyzed with the security rules in the knowledge base to determine whether any security rules have been violated. If so, determine that there is a warning situation;

[0022] Establish the corresponding relationship between hazard identification information, scenario description and risk, detect whether there are prohibited activities or items in a specific area, and if so, determine that there is a warning situation;

[0023] The content of the image to be analyzed is compared with historical violation cases to determine whether there are similar patterns. If so, it is determined that a warning situation exists.

[0024] Optionally, the image to be analyzed is input into a multimodal large model, the content of the image to be analyzed is understood to generate an image description, and the information in the constructed knowledge base and the image description are combined to generate an early warning result for the image to be analyzed, including:

[0025] Using image processing technology to compare multiple consecutive frames or multiple key frame images at predetermined time intervals;

[0026] Identify the change information between multiple key frame images in the time dimension through comparison algorithm;

[0027] A warning result for the image to be analyzed is generated based on the change information.

[0028] Optionally, the generating a warning result for the image to be analyzed based on the change information includes:

[0029] If the change information reaches the cumulative warning condition, a warning response with a higher security level than the conventional warning response is triggered.

[0030] Optionally, the real-time output of the obtained warning result includes:

[0031] The image description, warning conclusion and warning reason are output in real time as warning results.

[0032] The second aspect of the present application provides an early warning device based on a multimodal large model, comprising:

[0033] The acquisition module is used to acquire real-time video streams;

[0034] A video frame extraction module is used to receive the real-time video stream, perform frame extraction operation on the real-time video stream, and obtain a key frame image;

[0035] A classification model module is used to call the classification model to classify the key frame images and screen out the images to be analyzed that need further early warning analysis;

[0036] A multimodal large model module, used to input the image to be analyzed into the multimodal large model, understand the content of the image to be analyzed to generate an image description, and generate an early warning result for the image to be analyzed by combining the information in the constructed knowledge base and the image description; wherein the knowledge base includes a scene description of the set scene, safety rules, hazard identification information, and prior knowledge of historical violation cases;

[0037] The early warning module is used to output the early warning results in real time.

[0038] According to a third aspect of the present application, there is provided an early warning device based on a multimodal large model, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements any of the early warning methods based on the multimodal large model described above.

[0039] In a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the early warning method based on the multimodal large model according to any one of the above-mentioned methods is implemented.

[0040] The early warning method based on a multimodal large model provided in this application uses a combination of a multimodal learning model and a classification model to process and understand the association between images and related text descriptions, thereby providing a richer understanding of the scene. By using deep learning and natural language processing technology to extract and understand the semantic content in the scene, it is possible to learn and predict models of future behaviors from historical data to improve the understanding of the meaning of the scene. By updating its knowledge base and model in real time, it can quickly adapt to new threats and scenarios and process unseen data and scenarios. In addition, the present application uses a classification model to perform a preliminary screening of images, screens out images that obviously do not require early warning in advance, and then enters the multimodal large model for analysis of images that require further screening, which can effectively reduce the cost of the system and solve the problem of the system being difficult to implement. In addition, the present application also provides an early warning device, equipment and medium based on a multimodal large model with the above-mentioned technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0042] Figure 1 A flowchart of a specific implementation of the early warning method based on a multimodal large model provided in this application;

[0043] Figure 2 A flowchart of a specific implementation method for calling a classification model to classify the key frame image;

[0044] Figure 3 A flowchart of another specific implementation method for calling a classification model to classify the key frame image;

[0045] Figure 4 A structural block diagram of a specific implementation of the early warning device based on the multimodal large model provided in this application;

[0046] Figure 5 This is a structural block diagram of the early warning device based on the multimodal large model provided in this application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. It should be noted that, in the absence of conflict, the embodiments in the present disclosure and the features in the embodiments can be combined, separated, interchanged and / or rearranged with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0048] The terms used here are for the purpose of describing specific embodiments, and are not intended to be restrictive. As used here, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, it is explained that there are stated features, integral bodies, steps, operations, parts, assemblies and / or their groups, but it is not excluded that there are or add one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups. It should also be noted that, as used here, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values ​​and / or the values ​​provided that will be recognized by those of ordinary skill in the art.

[0049] The flowchart of a specific implementation method of the early warning method based on the multimodal large model provided in this application is as follows: Figure 1 As shown, the method includes:

[0050] S101: Collect real-time video stream.

[0051] Specifically, video streams can be collected in real time from surveillance cameras or other video sources as input to the early warning system. Cameras can be installed at key locations to continuously collect on-site video stream data to ensure monitoring and early warning of key areas. The camera can be a visible light camera, or of course, a camera with infrared night vision function, which does not affect the implementation of this application.

[0052] S102: receiving the real-time video stream, performing a frame extraction operation on the real-time video stream, and obtaining a key frame image.

[0053] Frame extraction for real-time video stream is the process of selecting specific still images (frames) from a continuous video stream. In digital video, there are usually multiple frames per second (such as 24, 30 or 60 frames, etc.). Frame extraction is to select some frames from these continuous frames for processing or analysis. The computational cost of processing the entire video stream is relatively high. The frame extraction operation in this step can significantly reduce the amount of data calculation and extract the key frames, providing a basis for subsequent analysis.

[0054] The specific implementation methods can be (1) timed frame extraction: by setting a fixed time interval, such as extracting a frame every minute or every 5 minutes, for routine monitoring. (2) motion-triggered frame extraction: using dynamic detection technology, extracting frames when significant changes are detected in the picture to capture possible security events. (3) multi-resolution frame extraction: extracting low-resolution frames at a low frequency under normal circumstances, and extracting high-resolution frames at a high frequency when potential risks are detected.

[0055] S103: calling a classification model to classify the key frame images, and screening out images to be analyzed that require further early warning analysis.

[0056] Use the classification model to preliminarily classify key frames and filter out images containing potential risks or events. You can use trained machine learning models, such as the CLIP multimodal learning model or the YOLO model, for preliminary screening. Using a classification model can greatly improve the ability of image classification and target detection.

[0057] The classification model is used for preliminary screening before input into the multimodal large model. The classification model can quickly identify key frames containing potentially important information. This means that only a few relevant images will be sent to the more complex and computationally expensive multimodal large model for in-depth analysis. Most images can be excluded after simple classification processing, which reduces the need for data storage and advanced processing. Through preliminary classification and screening, images that obviously do not meet the warning conditions can be excluded (such as situations that fully comply with safety regulations or completely irrelevant scenes). This not only speeds up processing, but also avoids wasting computing cycles on high-performance computing resources, which can effectively reduce hardware costs and operation and maintenance costs.

[0058] S104: Input the image to be analyzed into the multimodal large model, understand the content of the image to be analyzed to generate an image description, and combine the information in the constructed knowledge base and the image description to generate a warning result for the image to be analyzed.

[0059] The knowledge base includes scenario descriptions, safety rules, hazard identification information, and prior knowledge of historical violation cases for setting scenarios. Scenario descriptions describe various typical scenarios in detail, including environmental characteristics, common activities, and possible risk factors. These descriptions can help the model better understand the environment and context. Safety rules list the safety rules and standards related to specific scenarios, clarifying what behaviors are allowed and what behaviors are prohibited. Hazard identification information contains information that identifies potential hazards, such as flammable material signs, toxic material areas, etc., and provides detailed information about potential hazards and risk factors. Historical violation cases collect and analyze historical violation cases, including the environment, causes, consequences, and measures taken when the accident occurred, to provide case study materials.

[0060] The multimodal large model analyzes the image content and generates descriptive text to describe the activities and visible objects in the image. The multimodal large model uses the knowledge base (scene description of the set scene, safety rules, hazard identification information, historical violation cases, etc.) to evaluate whether the behavior in the image meets safety standards. Based on the image description and knowledge base information, the model determines whether there is a violation and gives specific reasons or explains the reasons for uncertainty.

[0061] This step uses a multimodal large model to conduct in-depth analysis of the screened images, generate detailed image descriptions, and combine them with information in an external knowledge base. This information is used to perform contextual analysis and comprehensive judgment to determine whether an early warning needs to be generated.

[0062] The multimodal large model receives an image as input, and the visual encoder (such as ViT) processes the input image to extract high-level visual features. The extracted visual features are aligned to the input space of the language model. This step allows the visual information to be understood and processed by the language model. Based on the aligned visual features, the large language model (such as the GPT series) generates the corresponding text description. The core of the image-to-text model is that it can "understand" the image content and convert this understanding into a human-readable text description to generate an image description.

[0063] The combining of the information in the constructed knowledge base and the image description to generate the warning result for the image to be analyzed includes:

[0064] Rule matching: Match the content of the image to be analyzed with the safety rules in the knowledge base to determine whether any safety rules have been violated. If so, it is determined that there is a warning situation. For example, if a worker without a helmet is monitored in the construction area, it is identified as a warning situation.

[0065] Risk assessment: Establish the corresponding relationship between hazard identification information, scene description and risk, detect whether there are prohibited activities or items in a specific area, and if so, determine that there is a warning situation. For example, smoking in a non-smoking area is identified as a warning situation.

[0066] Historical case comparison: Compare the content of the image to be analyzed with historical violation cases to determine whether there are similar patterns. If so, it is determined that a warning situation exists.

[0067] S105: Output the obtained warning results in real time.

[0068] Based on the analysis of the multimodal large model, the final warning results are output. The output of results is the key process of transforming complex data analysis into clear and actionable information. The output can be presented in the form of text reports, visual dashboards or real-time alerts to ensure that decision makers can quickly understand the situation and respond in a timely manner. The output content can include: real-time output of image descriptions, warning conclusions and warning reasons as warning results.

[0069] The early warning information is communicated to the monitoring personnel or system in real time so that corresponding measures can be taken. Specifically, the early warning information can be displayed or sent in real time through the user interface, automated system or mobile device to ensure the response speed and effect.

[0070] The early warning method based on a multimodal large model provided in this application uses a combination of a multimodal learning model and a classification model to process and understand the association between images and related text descriptions, thereby providing a richer understanding of the scene. By using deep learning and natural language processing technology to extract and understand the semantic content in the scene, it is possible to learn and predict models of future behaviors from historical data to improve the understanding of the meaning of the scene. By updating its knowledge base and model in real time, it can quickly adapt to new threats and scenarios and process unseen data and scenarios. In addition, this application uses a classification model to perform a preliminary screening of images, screen out images that obviously do not require early warning in advance, and then enter the multimodal large model for analysis of images that need further screening, which can effectively reduce the cost of the system and solve the problem of the difficulty of implementing the system.

[0071] The method provided in this application can be applied to multiple fields such as security monitoring, abnormal behavior detection, smart city management, etc. It can automatically identify and describe complex scenarios and provide timely warning information.

[0072] Based on the above embodiment, the classification model can specifically use the CLIP (Contrastive Language-Image Pre-training) multimodal learning model. This model combines vision and language processing capabilities. Figure 2 A flowchart of a specific implementation method of calling a classification model to classify the key frame images. The process of screening out the images to be analyzed that need further early warning analysis may specifically include:

[0073] S201: Pre-train the CLIP multimodal learning model, define a text label for each potential warning scenario, and use the CLIP text editor to convert the text label into a text label vector.

[0074] The CLIP model includes an image encoder and a text encoder. The image encoder is based on the architecture of Vision Transformer (ViT) or Convolutional Neural Network (CNN) and is used to extract image features from the input image. The text encoder is based on the Transformer model and is used to process the input text description and extract its semantic features.

[0075] S202: Load the CLIP multimodal learning model, and convert the key frame image into an image feature vector through the CLIP image encoder.

[0076] S203: Calculate the similarity between each image feature vector and the text label vector, and classify the key frame image into the label category with the highest similarity based on the similarity score.

[0077] S204: When the similarity between the key frame image and the label exceeds a preset threshold, the image is marked as an image to be analyzed that requires further warning analysis.

[0078] Using CLIP in image classification tasks allows classification without having seen a specific category, which is very useful for dealing with new or rare scenes, and solves the problem that traditional classification models require a large amount of labeled data. And guiding image classification through natural language descriptions makes classification tasks more flexible and scalable, and new classification categories can be easily added by simply providing text descriptions.

[0079] The classification model can also specifically use the YOLO model. YOLO is mainly used for target detection, but it can also be used for image classification tasks, especially in scenarios where specific objects need to be located and classified. Figure 3 Another specific implementation flow chart of calling the classification model to classify the key frame image, the process of screening out the image to be analyzed that needs further early warning analysis may specifically include:

[0080] S301: Loading a pre-trained YOLO model; the YOLO model is a model that uses predefined object categories as warning indicators based on business requirements or security rules.

[0081] S302: Input the key frame image into the YOLO model, obtain the output result of the YOLO model, and mark the image containing the warning indicator as an image to be analyzed that requires further warning analysis.

[0082] YOLO's single-stage detection architecture enables fast target detection and classification in real-time video streams, making it suitable for early warning systems that require a quick response. Completing the location and classification of targets in a single forward propagation improves processing efficiency and helps understand the spatial relationship of objects in the scene. YOLO considers the global information of the entire image, which helps understand the context of the target in the scene. In addition, it can be fine-tuned for specific scenarios (such as security monitoring) to improve the detection and classification accuracy of specific targets. When using YOLO, segmenting and screening the important parts of the relevant pictures, and then obtaining the parts of the picture that you are concerned about, can improve its adaptability and classification accuracy.

[0083] On the basis of any of the above embodiments, the early warning method provided by the present application can also detect changes in time based on a series of previous and next pictures, and realize early warning in the time dimension. This implementation method can help identify and respond to developing events or situations, such as people entering a restricted area, changes in environmental conditions, etc. The process specifically includes: inputting the image to be analyzed into a multimodal large model, understanding the content of the image to be analyzed to generate an image description, combining the information in the constructed knowledge base and the image description, and generating an early warning result for the image to be analyzed: using image processing technology to compare multiple key frame images of consecutive frames or predetermined time intervals; identifying change information between multiple key frame images in the time dimension through a comparison algorithm; and generating an early warning result for the image to be analyzed based on the change information.

[0084] After extracting multiple key frame images, the extracted key frame images are preprocessed as necessary, including resizing, normalization, and possible image enhancement, for subsequent processing. A reference image is selected or calculated as the starting point for comparison. The reference image can be the first frame at the beginning of the day, or a static scene image selected under specific conditions. The comparison algorithm is used to identify the change information between multiple key frame images in the time dimension. A threshold for change detection is set, and an early warning is triggered when the change exceeds this threshold. The setting of the threshold can depend on the sensitivity of the specific application and environmental conditions. The frequency, intensity, and pattern of changes can be analyzed in this process.

[0085] Once a significant change is detected, an alert is immediately generated. This may include sending a notification, triggering an alarm, or initiating automated response measures. If the change information reaches the cumulative alert condition, an alert response with a higher security level than the conventional alert response is triggered. For example, if several consecutive frames of images show unauthorized access to the same location, workers without helmets are found multiple times in an area, or potential hazardous material leaks are detected in a hazardous area for several consecutive days, these may be set as conditions for triggering advanced alerts. These conditions are usually based on the frequency, severity, or a combination of the two of the events. When these changes accumulate to a certain extent, it is considered that the preset cumulative alert conditions have been met. Once it is determined that the change information has reached the cumulative alert condition, an alert response with a higher security level than the conventional one will be automatically triggered. This response may include emergency notification, automatic initiation of safety protection measures, or direct intervention in manual control. This can prevent potential accidents from developing into more serious incidents and minimize risks and losses through timely advanced responses.

[0086] The following is a detailed description of the early warning method based on the multimodal large model provided by this application in conjunction with a specific example scenario. The scenario description of the example setting scenario is a construction site safety monitoring helmet scenario, and the safety rule for early warning is to determine whether the worker is wearing a helmet at the construction site. The process specifically includes:

[0087] Cameras are used to monitor the production process on the construction site and collect real-time video streams on the construction site.

[0088] For the scene of whether a helmet is worn on the construction site, a text prompt word can be set: "The worker is not wearing a helmet." This prompt word serves as the basis for model search and recognition, and describes the specific scene that needs to be detected. Use the CLIP model to convert the text prompt "The worker is not wearing a helmet" into a text vector, and also convert the extracted key frame image into an image vector. CLIP evaluates the degree of match between each image and the text prompt by comparing the similarity between the image vector and the text vector. This usually uses cosine similarity or other similarity metrics. Set a similarity threshold. When the similarity score between the image and the text prompt is greater than the preset similarity threshold, it is considered that the scene in the image meets the text description (that is, the worker in the scene may not be wearing a helmet). Traverse all key frame images and filter out images that match the description of "The worker is not wearing a helmet."

[0089] The selected images are sent to the multimodal big model of the early warning system, which can set prompt words for the multimodal big model, including: giving the work scene in the image, safety requirements, and reasons for uncertainty about whether there is a violation. The multimodal big model analyzes different construction site images based on specific safety requirements (such as the wearing of hard hats) and ultimately generates a conclusion on whether there is any violation.

[0090] For each image, based on the image description generated by the model and the safety standards in the knowledge base, a conclusion is output as to whether there is any violation. For example, for the example image, if all the workers in the image are wearing helmets, a warning message indicating no violation is output. For the example image, there are four workers, two of whom are wearing yellow helmets, one is wearing a black short-sleeved shirt with a towel around his neck, and the other is wearing a blue top and white gloves, working with his head down. Another worker stands on the left side of the image, not wearing a helmet, and seems to be handling some steel bars. The last worker only shows his arms, and it is impossible to tell whether he is wearing a helmet. Judging from the picture, only two workers are wearing helmets, which may mean that they have complied with safety regulations, while the other two may have violated the regulations, thus outputting a warning message indicating a violation.

[0091] The structural block diagram of a specific implementation of the early warning device based on the multi-modal large model provided in this application is as follows: Figure 4 As shown, the device specifically includes:

[0092] The acquisition module 100 is used to acquire real-time video streams.

[0093] The video frame extraction module 200 is used to receive the real-time video stream, perform frame extraction operation on the real-time video stream, and obtain key frame images.

[0094] The classification model module 300 is used to call the classification model to classify the key frame images and screen out the images to be analyzed that need further early warning analysis.

[0095] The multimodal large model module 400 is used to input the image to be analyzed into the multimodal large model, understand the content of the image to be analyzed to generate an image description, and combine the information in the constructed knowledge base and the image description to generate an early warning result for the image to be analyzed; wherein the knowledge base includes a scene description of the set scene, safety rules, hazard identification information, and prior knowledge of historical violation cases.

[0096] The early warning module 500 is used to output the obtained early warning results in real time.

[0097] In addition, the present application also provides an early warning device based on a multi-modal large model, such as Figure 5 As shown in the structural block diagram of the early warning device provided in the present application, it specifically includes a memory 51 and a processor 52, and the memory 51 stores a computer program. When the computer program is executed by the processor 52, any one of the above-mentioned early warning methods based on the multimodal large model is implemented.

[0098] In addition, the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements any of the above-mentioned early warning methods based on the multimodal large model.

[0099] Computer readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be configured to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0100] The professionals should also be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0101] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0102] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application and is not configured to limit the protection scope of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. An early warning method based on a multimodal large model, characterized in that: include: Collect real-time video streams; Receiving the real-time video stream, performing frame extraction operation on the real-time video stream to obtain a key frame image; Calling a classification model to classify the key frame images, and screening out images to be analyzed that require further early warning analysis; The image to be analyzed is input into the multimodal large model, the content of the image to be analyzed is understood to generate an image description, and the information in the constructed knowledge base and the image description are combined to generate an early warning result for the image to be analyzed; wherein the knowledge base includes a scene description of the set scene, safety rules, hazard identification information, and prior knowledge of historical violation cases; The obtained warning results are output in real time.

2. The early warning method based on a multimodal large model according to claim 1 is characterized in that: The calling of the classification model to classify the key frame images and screen out the images to be analyzed that need further early warning analysis includes: Pre-train the CLIP multimodal learning model, define text labels for each potential warning scenario, and use CLIP's text editor to convert the text labels into text label vectors; Loading the CLIP multimodal learning model, converting the key frame image into an image feature vector through the CLIP image encoder; Calculating the similarity between each image feature vector and the text label vector, and classifying the key frame image into the label category with the highest similarity based on the similarity score; When the similarity between the key frame image and the label exceeds a preset threshold, the image is marked as an image to be analyzed that requires further early warning analysis.

3. The early warning method based on multimodal large model according to claim 1 is characterized in that: The calling of the classification model to classify the key frame images and screen out the images to be analyzed that need further early warning analysis includes: Loading a pre-trained YOLO model; the YOLO model is a model that uses pre-defined object categories as warning indicators according to business requirements or security rules; The key frame image is input into the YOLO model, the output result of the YOLO model is obtained, and the image containing the warning indicator is marked as an image to be analyzed that requires further warning analysis.

4. The early warning method based on a multimodal large model according to any one of claims 1 to 3, characterized in that: The combining of the information in the constructed knowledge base and the image description to generate the warning result for the image to be analyzed includes: Match the content of the image to be analyzed with the security rules in the knowledge base to determine whether any security rules have been violated. If so, determine that there is a warning situation; Establish the corresponding relationship between hazard identification information, scenario description and risk, detect whether there are prohibited activities or items in a specific area, and if so, determine that there is a warning situation; The content of the image to be analyzed is compared with historical violation cases to determine whether there are similar patterns. If so, it is determined that a warning situation exists.

5. The early warning method based on a multimodal large model according to any one of claims 1 to 3, characterized in that: Inputting the image to be analyzed into the multimodal large model, understanding the content of the image to be analyzed to generate an image description, combining the information in the constructed knowledge base and the image description, generating an early warning result for the image to be analyzed, including: Using image processing technology to compare multiple consecutive frames or multiple key frame images at predetermined time intervals; Identify the change information between multiple key frame images in the time dimension through comparison algorithm; A warning result for the image to be analyzed is generated based on the change information.

6. The early warning method based on multimodal large model according to claim 5 is characterized in that: The generating of the warning result for the image to be analyzed based on the change information includes: If the change information reaches the cumulative warning condition, a warning response with a higher security level than the conventional warning response is triggered.

7. The early warning method based on a multimodal large model according to any one of claims 1 to 3, characterized in that: The real-time output of the obtained warning results includes: The image description, warning conclusion and warning reason are output in real time as warning results.

8. An early warning device based on a multimodal large model, characterized in that: include: The acquisition module is used to acquire real-time video streams; A video frame extraction module is used to receive the real-time video stream, perform frame extraction operation on the real-time video stream, and obtain a key frame image; A classification model module is used to call the classification model to classify the key frame images and screen out the images to be analyzed that need further early warning analysis; A multimodal large model module, used to input the image to be analyzed into the multimodal large model, understand the content of the image to be analyzed to generate an image description, and generate an early warning result for the image to be analyzed by combining the information in the constructed knowledge base and the image description; wherein the knowledge base includes a scene description of the set scene, safety rules, hazard identification information, and prior knowledge of historical violation cases; The early warning module is used to output the early warning results in real time.

9. An early warning device based on a multimodal large model, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the early warning method based on the multimodal large model according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the early warning method based on a multimodal large model according to any one of claims 1 to 7 is implemented.

Images