A multi-modal large model-based early warning method, device, equipment and medium

Abstract

The application relates to a pre-warning method and device based on a multi-modal large model, equipment and a medium, and belongs to the technical field of deep learning. The application collects a real-time video stream; receives the real-time video stream, performs a frame extraction operation on the real-time video stream, and obtains a key frame image; a classification model is called to classify the key frame image, and an image to be analyzed that needs to be further analyzed is screened out; the image to be analyzed is input into a multi-modal large model, the content of the image to be analyzed is understood to generate an image description, information in a constructed knowledge base and the image description are combined, a pre-warning result of the image to be analyzed is generated, and the obtained pre-warning result is output in real time. The application preliminarily screens the images by using a classification model, screens out images that obviously do not need to be pre-warned in advance, and analyzes images that need to be further screened into a multi-modal large model, so that the cost of the system can be effectively reduced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of deep learning, and specifically relates to a pre-warning method and device based on a multi-modal large model, equipment and medium. BACKGROUND

[0002] Traditional pre-warning systems usually rely on fixed rules or algorithms, which are based on pre-defined scenario rules or image recognition models. However, these fixed rules may not fully capture all important variables and relationships in complex, semantically rich real-world environments, resulting in limited analysis capabilities. Traditional pre-warning systems often focus on detecting significant visual features or behaviors that conform to specific patterns, but may overlook the context of events, such as time, location, and past events. Lack of integrated analysis of these contextual information may lead to incorrect interpretation of observed behavior, resulting in false positives or false negatives. Traditional pre-warning systems are often designed for specific application scenarios and have poor adaptability to new situations in subsequent operation. This is because updating the rules or models of the system requires time and resources, especially when the system relies on heavily customized hardware or software. In a rapidly changing environment, this slow update speed makes it difficult to meet the needs of real-time response.

[0003] In addition, although the use of graph-to-text multi-modal large models can improve the understanding ability and accuracy of the pre-warning system, it usually requires powerful computing resources to process large amounts of data input and complex data processing tasks. This dependence on high-performance hardware significantly increases the cost of the system, making large-scale commercial deployment difficult and expensive.

[0004] Therefore, how to improve the limitations of existing pre-warning systems in design and function is a topic worthy of study. SUMMARY

[0005] In view of the above analysis, the embodiments of the present application aim to provide a pre-warning method, device, equipment and medium based on a multi-modal large model, aiming to overcome the shortcomings of the existing pre-warning system, such as limited analysis capability, lack of context understanding, difficulty in adapting to new scenarios, and high hardware cost.

[0006] One aspect of the present application provides a pre-warning method based on a multi-modal large model, comprising:

[0007] Collecting a real-time video stream;

[0008] Receiving the real-time video stream, performing frame extraction operation on the real-time video stream to obtain key frame images;

[0009] Calling a classification model to classify the key frame images and screening out images to be analyzed for further pre-warning analysis;

[0010] inputting the image to be analyzed into a multi-modal large model, generating an image description of the content of the image to be analyzed, combining information in the constructed knowledge base and the image description, and generating a warning result for the image to be analyzed; wherein the knowledge base includes scene descriptions of set scenes, safety rules, dangerous identification information, and prior knowledge of historical violation cases;

[0011] outputting the obtained warning result in real time.

[0012] Optionally, the calling of the classification model to classify the key frame image and screening out the image to be analyzed for further warning analysis includes:

[0013] pre-training a CLIP multi-modal learning model to define a text label for each potential warning scene, and converting the text label into a text label vector using a text editor of the CLIP;

[0014] loading the CLIP multi-modal learning model, and converting the key frame image into an image feature vector through an image encoder of the CLIP;

[0015] calculating the similarity between each image feature vector and the text label vector, and classifying the key frame image into a label category with the highest similarity based on a similarity score;

[0016] when the similarity between the key frame image and the label exceeds a preset threshold, marking the image as an image to be analyzed for further warning analysis.

[0017] Optionally, the calling of the classification model to classify the key frame image and screening out the image to be analyzed for further warning analysis includes:

[0018] loading a pre-trained YOLO model; the YOLO model is a model that defines a pre-defined object category as a warning indicator according to business needs or safety rules;

[0019] inputting the key frame image into the YOLO model, obtaining an output result of the YOLO model, and marking an image containing the warning indicator as an image to be analyzed for further warning analysis.

[0020] Optionally, the combination of information in the constructed knowledge base and the image description to generate a warning result for the image to be analyzed includes:

[0021] matching the content of the image to be analyzed with the safety rules in the knowledge base to determine whether a safety rule is violated, and if so, determining that there is a warning situation;

[0022] Establish the corresponding relationship between the dangerous sign information, scene description and risk, detect whether there are prohibited activities or articles in a specific area, and if so, determine that there is a warning situation;

[0023] Compare the content of the image to be analyzed with historical violation cases to determine whether there is a similar pattern, and if so, determine that there is a warning situation.

[0024] Optionally, the image to be analyzed is input into a multi-modal large model to understand the content of the image to be analyzed and generate an image description. In combination with the information in the constructed knowledge base and the image description, a warning result for the image to be analyzed is generated, including:

[0025] Using image processing technology to compare multiple consecutive frames or multiple key frame images at a predetermined time interval;

[0026] By comparing the algorithm, the change information between multiple key frame images in the time dimension is identified;

[0027] Based on the change information, a warning result for the image to be analyzed is generated.

[0028] Optionally, the generation of the warning result for the image to be analyzed based on the change information includes:

[0029] If the change information meets the cumulative warning condition, a higher security level of warning response than the regular warning response is triggered.

[0030] Optionally, the real-time output of the obtained warning result includes:

[0031] The image description, warning conclusion and warning reason are output as the warning result in real time.

[0032] In a second aspect of the present application, a warning device based on a multi-modal large model is provided, including:

[0033] The acquisition module is configured to acquire a real-time video stream;

[0034] The video frame extraction module is configured to receive the real-time video stream, perform a frame extraction operation on the real-time video stream, and obtain key frame images;

[0035] The classification model module is configured to call a classification model to classify the key frame images and filter out images to be analyzed for further warning analysis;

[0036] The multi-modal large model module is configured to input the image to be analyzed into a multi-modal large model, generate an image description by understanding the content of the image to be analyzed, and generate an early warning result of the image to be analyzed by combining information in the constructed knowledge base and the image description. The knowledge base includes scene descriptions of set scenes, safety rules, dangerous identification information, and prior knowledge of historical violation cases.

[0037] The early warning module is configured to output the obtained early warning result in real time.

[0038] In a third aspect, the present application provides a multi-modal large model-based early warning device, including a memory and a processor. The memory stores a computer program, and the computer program is executed by the processor to implement any of the multi-modal large model-based early warning methods described above.

[0039] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. The computer program is executed by a processor to implement any of the multi-modal large model-based early warning methods described above.

[0040] The multi-modal large model-based early warning method provided by the present application uses a combination of multi-modal learning models and classification models to process and understand the association between images and related text descriptions, thereby providing richer scene understanding. By using deep learning and natural language processing techniques to extract and understand semantic content in the scene, the model can learn from historical data and predict future behavior to improve understanding of the meaning of the scene. By updating its knowledge base and model in real time, it can quickly adapt to new threats and scenarios, and handle unseen data and scenarios. In addition, the classification model is used to preliminarily screen the images, and the images that do not need to be warned are screened out in advance. The images that need to be further screened are analyzed in the multi-modal large model, which can effectively reduce the cost of the system and solve the problem of difficult landing of the system. In addition, the present application also provides a multi-modal large model-based early warning device, equipment and medium with the above technical effects. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments or the prior art, the following will briefly introduce the drawings needed in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0042] Figure 1 A flowchart of a specific embodiment of the multi-modal large model-based early warning method provided by the present application;

[0043] Figure 2 A specific implementation flowchart for calling a classification model to classify the key frame image;

[0044] Figure 3 Another specific implementation flowchart for calling a classification model to classify the key frame image;

[0045] Figure 4 A structural block diagram of one specific implementation of the early warning device based on the multi-modal large model provided in the present application;

[0046] Figure 5 A structural block diagram of the early warning device based on the multi-modal large model provided in the present application. DETAILED DESCRIPTION

[0047] To make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments. It should be noted that, in the case of no conflict, the embodiments and the features in the embodiments in the present disclosure can be combined, separated, interchanged and / or rearranged. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present application.

[0048] The terms used herein are for the purpose of describing specific embodiments and are not intended to be limiting. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. Further, when the terms “comprises,” “comprising,” “includes,” “including,” and the like are used in the following description and claims, such terms are intended to be inclusive in a manner that the recited features, integers, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should also be noted that, as used herein, the terms “substantially,” “approximately,” and other similar terms are used as terms of approximation and not as terms of degree, such that they are used to account for inherent deviations in measurements, calculations, and / or provided values that would be recognized by those of ordinary skill in the art.

[0049] A flowchart of one specific implementation of the early warning method based on the multi-modal large model provided in the present application is shown in Figure 1 The method comprises:

[0050] S101: Collect a real-time video stream.

[0051] Specifically, the video stream can be collected in real time from a surveillance camera or other video source as input for the early warning system. The camera can be installed at key locations to continuously collect video stream data on site, ensuring monitoring and early warning of key areas. The camera can use a visible light camera, or of course a camera with infrared night vision function, which does not affect the implementation of the present application.

[0052] S102: Receive the real-time video stream, and perform frame extraction on the real-time video stream to obtain key frame images.

[0053] Frame extraction on the real-time video stream is a process of selecting specific still pictures (frames) from continuous video streams. In digital video, there are usually multiple frames per second (such as 24 frames, 30 frames, or 60 frames, etc.), and frame extraction is to select some frames for processing or analysis from these continuous frames. The computational cost of processing the entire video stream is relatively high, and the use of frame extraction in this step can significantly reduce the data computation, extracting key frames to provide a basis for subsequent analysis.

[0054] The specific implementation can be (1) timed frame extraction: by setting a fixed time interval, such as extracting a frame every minute or every 5 minutes, for regular monitoring. (2) motion-triggered frame extraction: using dynamic detection technology, when a significant change in the picture is detected, frame extraction is performed, for capturing possible security events. (3) multi-resolution frame extraction: under normal circumstances, low-resolution frames are extracted at a low frequency, and when potential risks are detected, high-resolution frames are extracted at a high frequency.

[0055] S103: Call the classification model to classify the key frame images, and filter out the images to be analyzed for further early warning analysis.

[0056] The classification model is used to preliminarily classify the key frames and filter out images containing potential risks or events. A trained machine learning model, such as the CLIP multi-modal learning model or the YOLO model, can be used for preliminary screening. The use of a classification model can greatly improve the ability of image classification and target detection.

[0057] The classification model is used for preliminary screening before inputting into the multi-modal large model, which can quickly identify key frames containing potentially important information. This means that only a small number of relevant images will be sent to the more complex and computationally expensive multi-modal large model for in-depth analysis. Most images can be excluded after simple classification, which reduces the need for data storage and advanced processing. Through preliminary classification and screening, images that do not meet the early warning conditions can be excluded (such as completely safe conditions or completely irrelevant scenes). This not only speeds up the processing, but also avoids wasting computing cycles on high-performance computing resources, effectively reducing hardware costs and operational costs.

[0058] S104: input the image to be analyzed into a multi-modal large model, understand the content of the image to be analyzed to generate an image description, combine the information in the constructed knowledge base and the image description to generate a warning result for the image to be analyzed.

[0059] The knowledge base includes scene descriptions of set scenes, safety rules, dangerous identification information, and prior knowledge of historical violation cases. Scene descriptions describe various typical scenes in detail, including environmental characteristics, common activities, and possible risk factors. These descriptions can help the model better understand the environment and context. Safety rules list safety rules and standards related to specific scenes, clearly stating what behaviors are allowed and what behaviors are prohibited. Dangerous identification information contains information identifying potential dangers, such as flammable material signs, toxic substance areas, etc., providing detailed information about potential dangers and risk factors. Historical violation cases collect and analyze historical violation cases, including the environment, causes, consequences, and measures taken in accidents, to provide case learning materials.

[0060] The multi-modal large model analyzes the image content and generates descriptive text describing the activities and visible objects in the image. The multi-modal large model uses the knowledge base (scene descriptions of set scenes, safety rules, dangerous identification information, historical violation cases, etc.) to evaluate whether the behavior in the image conforms to safety standards. Based on the image description and knowledge base information, the model determines whether there is a violation of operation and gives specific reasons or explanations of uncertainty.

[0061] This step uses a multi-modal large model to analyze the selected images in depth, generating detailed image descriptions, and combining them with information from an external knowledge base to perform context analysis and comprehensive judgment to determine whether a warning needs to be generated.

[0062] The multi-modal large model receives images as input, and a visual encoder (such as ViT) processes the input images to extract high-level visual features. Align the extracted visual features to the input space of the language model, which allows visual information to be understood and processed by the language model. Based on the aligned visual features, a large language model (such as the GPT series) generates corresponding text descriptions. The core of the image-to-text model is its ability to "understand" image content and convert this understanding into human-readable text descriptions, generating image descriptions.

[0063] The combination of information in the constructed knowledge base and the image description to generate a warning result for the image to be analyzed includes:

[0064] Rule matching: match the content of the image to be analyzed with safety rules in the knowledge base to determine if there are safety rules that have been violated, if so, it is determined that there is a warning situation. For example, if it is monitored that there are workers without safety helmets in the construction area, it is identified that there is a warning situation.

[0065] Risk assessment: establish the correspondence between hazard identification information, scene description and risk, detect whether there are prohibited activities or articles in a specific area, if so, it is determined that there is a warning situation. For example, smoking in a non-smoking area is identified as a warning situation.

[0066] Historical case comparison: compare the content of the image to be analyzed with historical violation cases to determine if there is a similar pattern, if so, it is determined that there is a warning situation.

[0067] S105: output the obtained warning result in real time.

[0068] Based on the analysis of the multi-modal large model, the final warning result is output. The result output is the key process of converting complex data analysis into clear and operable information. The output form can be in the form of a text report, a visual dashboard or a real-time alarm, ensuring that decision makers can quickly understand the situation and respond in a timely manner. The output content can include: image description, warning conclusion and warning reason as the real-time output of the warning result.

[0069] Real-time transmission of warning information to monitoring personnel or systems for appropriate measures. Specifically, the warning information can be displayed or sent in real time through user interfaces, automated systems or mobile devices, etc. to ensure response speed and effectiveness.

[0070] The multi-modal large model-based warning method provided in the present application uses a combination of multi-modal learning models and classification models to process and understand the association between images and related text descriptions, providing richer scene understanding. Using deep learning and natural language processing techniques to extract and understand semantic content in the scene, the model can learn from historical data and predict future behavior to improve understanding of scene meaning. By updating its knowledge base and model in real time, it can quickly adapt to new threats and scenarios, handle unseen data and scenarios. In addition, the classification model is used to preliminarily screen images, and images that do not need to be warned are screened out in advance, and images that need to be further screened are analyzed in the multi-modal large model, which can effectively reduce the cost of the system and solve the problem of system difficulty in landing.

[0071] The method provided in the present application can be applied to security monitoring, abnormal behavior detection, intelligent city management and other fields, and can automatically identify and describe complex scenes to provide timely warning information.

[0072] On the basis of the above embodiments, the classification model can specifically select a CLIP (Contrastive Language-Image Pre-training) multi-modal learning model. This model combines visual and language processing capabilities. Referring to Figure 2 A specific implementation flowchart for calling the classification model to classify the key frame images, the process of screening out the images to be analyzed that need further early warning analysis can specifically include:

[0073] S201: Pre-training the CLIP multi-modal learning model, defining a text label for each potential early warning scene, and converting the text label into a text label vector using the text editor of CLIP.

[0074] The CLIP model includes an image encoder and a text encoder. The image encoder is based on the VisionTransformer (ViT) or Convolutional Neural Network (CNN) architecture, which is used to extract image features from the input image. The text encoder is based on the Transformer model, which is used to process the input text description and extract its semantic features.

[0075] S202: Load the CLIP multi-modal learning model, and convert the key frame images into image feature vectors through the image encoder of CLIP.

[0076] S203: Calculate the similarity between each image feature vector and the text label vector, and classify the key frame images into the label category with the highest similarity based on the similarity score.

[0077] S204: When the similarity between the key frame image and the label exceeds a preset threshold, mark the image as an image to be analyzed that needs further early warning analysis.

[0078] Using CLIP in image classification tasks can classify without seeing specific categories, which is very useful for handling emerging or rare scenes, solving the problem of requiring a large amount of labeled data for traditional classification models. And through natural language description to guide image classification, makes the classification task more flexible and scalable, can easily add new classification categories, only need to provide text description.

[0079] The classification model can also specifically select a YOLO model. YOLO is mainly used for object detection, but it can also be used for image classification tasks, especially in scenarios that require locating and classifying specific objects. Referring to Figure 3 Another specific implementation flowchart for calling the classification model to classify the key frame images, the process of screening out the images to be analyzed that need further early warning analysis can specifically include:

[0080] S301: load a pre-trained YOLO model; the YOLO model is a model that takes predefined object categories as early warning indicators according to business needs or safety rules.

[0081] S302: input the key frame image into the YOLO model, obtain the output result of the YOLO model, and mark the image containing the early warning indicator as an image to be analyzed for further early warning analysis.

[0082] The single-stage detection architecture of YOLO enables fast target detection and classification in real-time video streams, making it suitable for early warning systems that require fast response. The positioning and classification of targets are completed in one forward propagation, improving processing efficiency and helping to understand the spatial relationship of objects in the scene. YOLO considers the global information of the entire image, which helps to understand the context of the target in the scene. In addition, it can be fine-tuned for specific scenarios such as security monitoring to improve the detection and classification accuracy of specific targets. When using YOLO, the important parts of the relevant pictures are segmented and screened, and then the parts of interest in the pictures are obtained, which can improve the adaptability and classification accuracy.

[0083] On the basis of any of the above embodiments, the early warning method provided by the present application can also detect changes in time based on a series of before and after pictures, and realize early warning in the time dimension. This implementation can help identify and respond to developing events or conditions, such as personnel entering restricted areas, changes in environmental conditions, etc. The process specifically includes: inputting the image to be analyzed into a multi-modal large model, understanding the content of the image to be analyzed to generate an image description, combining information in the constructed knowledge base and the image description, and generating an early warning result for the image to be analyzed: comparing multiple key frame images in a continuous frame or a predetermined time interval using image processing techniques; identifying change information between multiple key frame images in the time dimension through a comparison algorithm; and generating an early warning result for the image to be analyzed based on the change information.

[0084] After extracting multiple key frame images, necessary preprocessing is performed on the extracted key frame images, including size adjustment, normalization, and possible image enhancement, for subsequent processing. A reference image is selected or calculated as the starting point for comparison. The reference image can be the first frame at the beginning of the day, or a static scene image selected under certain conditions. Change information between multiple key frame images in the time dimension is identified through a comparison algorithm. A threshold for change detection is set, and an early warning is triggered when the change exceeds this threshold. The threshold can be set depending on the sensitivity of the specific application and environmental conditions. The frequency, intensity, and pattern of changes can be analyzed in this process.

[0085] Once a significant change is detected, an immediate alert is generated. This can include sending a notification, triggering an alarm, or initiating an automated response measure. If the change information reaches a cumulative alert condition, a higher security level alert response is triggered than the regular alert response. For example, if consecutive frames of images show unauthorized access at the same location, multiple instances of workers not wearing safety helmets in a region, or potential hazardous substance leaks detected in a certain dangerous area for consecutive days, these can be set as conditions to trigger a high-level alert. These conditions are usually based on the frequency, severity, or a combination of both of the events. When these changes accumulate to a certain extent, it is considered that the preset cumulative alert condition is reached. Once it is determined that the change information reaches the cumulative alert condition, a higher security level alert response than the regular alert response is automatically triggered. This response can include emergency notifications, automatic activation of safety protection measures, or direct intervention in manual control. This can prevent potential accidents from developing into more serious incidents, minimizing risks and losses through timely high-level responses.

[0086] The multi-modal large model-based early warning method provided in the present application will be described in detail below in combination with a specific example scene. The scene description of the example setting scene is a construction site safety monitoring safety helmet scene, and the safety rule of the early warning is to judge whether the workers wear safety helmets on the construction site. The process specifically includes:

[0087] A camera is used to monitor the production process on the construction site to collect real-time video streams of the construction site.

[0088] For the scene of whether to wear safety helmets on the construction site, a text prompt word "worker not wearing safety helmet" can be set. This prompt word serves as the basis for model search and identification, describing the specific scene to be detected. The CLIP model is used to convert the text prompt "worker not wearing safety helmet" into a text vector, and the extracted key frame images are also converted into image vectors. CLIP evaluates the matching degree of each image with the text prompt by comparing the similarity of the image vector and the text vector. This usually uses cosine similarity or other similarity measurement methods. A similarity threshold is set, and when the similarity score of the image and the text prompt is greater than the similarity preset threshold, it is considered that the scene in the image matches the text description (i.e., the worker in the scene may not wear a safety helmet). All key frame images are traversed to screen out images that match the description of "worker not wearing safety helmet".

[0089] The screened images are sent to the multi-modal large model of the early warning system, and a multi-modal large model prompt word can be set, specifically including: giving the work scene in the image, the safety requirement, and the reason for uncertainty whether there is a violation. The multi-modal large model analyzes different construction site images according to specific safety requirements (such as the wearing of safety helmets), and finally generates a conclusion about whether there is a violation of operation.

[0090] For each image, based on the image description generated by the model and the safety standards in the knowledge base, a conclusion is output whether there is a violation. For example, for the case of the example picture: the workers in the picture all wear safety helmets, the output is no warning information of violation. For the case of the example picture: there are four workers, two of whom wear yellow safety helmets, one wears a black short-sleeved shirt with a towel around his neck, and the other wears a blue shirt and white gloves, and is working with his head down. Another worker stands on the left side of the picture without a safety helmet, and seems to be handling some steel bars. The last worker only shows his arm and it is impossible to judge whether he wears a safety helmet. From the picture, only two workers wear safety helmets, which may mean that they comply with safety regulations, while the other two may violate the regulations, thereby outputting warning information with violation.

[0091] The structural diagram of one specific embodiment of the early warning device based on the multi-modal large model provided in the present application is shown in Figure 4 The device specifically comprises:

[0092] The acquisition module 100 is configured to acquire a real-time video stream.

[0093] The video frame extraction module 200 is configured to receive the real-time video stream, perform a frame extraction operation on the real-time video stream, and obtain a key frame image.

[0094] The classification model module 300 is configured to call a classification model to classify the key frame image and filter out a to-be-analyzed image that needs further early warning analysis.

[0095] The multi-modal large model module 400 is configured to input the to-be-analyzed image into a multi-modal large model, understand the content of the to-be-analyzed image to generate an image description, combine information in a constructed knowledge base and the image description, and generate an early warning result for the to-be-analyzed image. The knowledge base includes scene description, safety rules, dangerous identification information, and prior knowledge of historical violation cases of the set scene.

[0096] The early warning module 500 is configured to output the obtained early warning result in real time.

[0097] In addition, the present application also provides an early warning device based on a multi-modal large model, as shown in Figure 5 The structural diagram of the early warning device provided in the present application is shown, which specifically comprises a memory 51 and a processor 52, the memory 51 stores a computer program, and the computer program is executed by the processor 52 to realize the early warning method based on the multi-modal large model described above.

[0098] In addition, the application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement any of the early warning methods based on the multi-modal large model.

[0099] Computer readable storage media includes permanent and non-permanent, removable and non-removable media, which can be realized by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be configured to store information that can be accessed by a computing device. According to the definition herein, computer readable media does not include transitory computer readable media, such as modulated data signals and carriers.

[0100] The skilled person should also further realize that the units and algorithm steps of the examples described in connection with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in the above description in general terms. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0101] The steps of the methods or algorithms described in connection with the embodiments disclosed herein can be implemented in hardware, software modules executed by a processor, or a combination of both. The software modules can be placed in random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.

[0102] The above detailed description of the specific implementation is further detailed for the purpose of the application, technical solutions and beneficial effects, and it should be understood that the above detailed description is only the specific implementation of the application, and is not configured to limit the protection scope of the application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application should be included in the protection scope of the application.

Claims

1. A pre-warning method based on a multi-modal large model, characterized in that, The method comprises: collecting a real-time video stream; receiving the real-time video stream, performing frame extraction on the real-time video stream to obtain a key frame image; calling a classification model to classify the key frame image, and screening out a to-be-analyzed image that needs further early warning analysis; inputting the to-be-analyzed image into a multi-modal large model, understanding the content of the to-be-analyzed image to generate an image description, combining information in a constructed knowledge base and the image description to generate an early warning result for the to-be-analyzed image; wherein the knowledge base comprises scene description, safety rules, dangerous identification information, and prior knowledge of historical violation cases of a set scene; the combination of information in the constructed knowledge base and the image description to generate the early warning result for the to-be-analyzed image comprises: matching the content of the to-be-analyzed image with the safety rules in the knowledge base to determine whether a safety rule is violated, and if so, determining that there is an early warning situation; establishing a corresponding relationship between dangerous identification information, scene description and risk, detecting whether there is a prohibited activity or object in a specific area, and if so, determining that there is an early warning situation; comparing the content of the to-be-analyzed image with historical violation cases to determine whether there is a similar pattern, and if so, determining that there is an early warning situation; outputting the obtained early warning result in real time. 2.The early warning method based on a multi-modal large model according to claim 1, wherein, The calling of the classification model to classify the key frame image and screen out the to-be-analyzed image that needs further early warning analysis comprises: pre-training a CLIP multi-modal learning model, defining a text label for each potential early warning scene, and converting the text label into a text label vector using a CLIP text editor; loading the CLIP multi-modal learning model, converting the key frame image into an image feature vector through a CLIP image encoder; calculating the similarity between each image feature vector and the text label vector, and classifying the key frame image into the label category with the highest similarity based on the similarity score; when the similarity between the key frame image and the label exceeds a preset threshold, marking the image as a to-be-analyzed image that needs further early warning analysis. 3.The early warning method based on a multi-modal large model according to claim 1, wherein, The calling of the classification model to classify the key frame image and screen out the to-be-analyzed image that needs further early warning analysis comprises: loading a pre-trained YOLO model; the YOLO model is a model that defines a pre-defined object category as an early warning indicator according to business needs or safety rules; inputting the key frame image into the YOLO model to obtain an output result of the YOLO model, and marking an image containing the early warning indicator as a to-be-analyzed image that needs further early warning analysis.

4. The early warning method based on a multi-modal large model according to any one of claims 1 to 3, characterized in that, The inputting of the to-be-analyzed image into the multi-modal large model, the understanding of the content of the to-be-analyzed image to generate an image description, and the combination of information in the constructed knowledge base and the image description to generate an early warning result for the to-be-analyzed image comprise: comparing a plurality of key frame images in a continuous frame or a plurality of key frame images in a predetermined time interval by using image processing technology; identifying change information between the plurality of key frame images in the time dimension by using a comparison algorithm; generating an early warning result for the to-be-analyzed image based on the change information. 5.The early warning method based on a multi-modal large model according to claim 4, characterized in that, The generating a pre-warning result for the image to be analyzed based on the change information comprises: If the change information reaches a cumulative pre-warning condition, triggering a pre-warning response with a higher security level than a regular pre-warning response.

6. The early warning method based on a multi-modal large model according to any one of claims 1 to 3, characterized in that, The real-time output of the obtained pre-warning result comprises: Real-time output of the image description, pre-warning conclusion and pre-warning reason as the pre-warning result.

7. A multi-modal large model-based early warning device, characterized in that, Comprise: The acquisition module is used for acquiring a real-time video stream; The video frame extraction module is used for receiving the real-time video stream, performing a frame extraction operation on the real-time video stream, and obtaining a key frame image; The classification model module is used for calling a classification model to classify the key frame image and filter out an image to be analyzed that needs further pre-warning analysis; The multi-modal large model module is used for inputting the image to be analyzed into a multi-modal large model, understanding the content of the image to be analyzed to generate an image description, combining information in a constructed knowledge base and the image description, and generating a pre-warning result for the image to be analyzed; wherein the knowledge base comprises scene description, safety rules, dangerous identification information and prior knowledge of historical violation cases of a set scene; the combining information in the constructed knowledge base and the image description to generate the pre-warning result for the image to be analyzed comprises: matching the content of the image to be analyzed with the safety rules in the knowledge base, judging whether a safety rule is violated, and if so, determining that there is a pre-warning situation; establishing a corresponding relationship between dangerous identification information, scene description and risk, detecting whether there is a prohibited activity or object in a specific area, and if so, determining that there is a pre-warning situation; comparing the content of the image to be analyzed with historical violation cases, judging whether there is a similar mode, and if so, determining that there is a pre-warning situation; The pre-warning module is used for real-time output of the obtained pre-warning result.

8. A multi-modal large model-based early warning device, characterized in that, Comprise a memory and a processor, the memory stores a computer program, the computer program is executed by the processor to realize the pre-warning method based on the multi-modal large model according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and the computer program is executed by a processor to realize the pre-warning method based on the multi-modal large model according to any one of claims 1-6.

Images