Scene monitoring method and device based on multi-modal large model, equipment and medium

Abstract

The invention discloses a scene monitoring method and device based on a multi-modal large model, equipment and a medium, and the method comprises the steps: obtaining a first video stream of a target scene, and determining a corresponding video frame image set according to the first video stream; inputting the video frame image set into a multi-mode large model deployed at an end side to obtain corresponding video scene description information, and uploading the first video stream and the video scene description information to a cloud platform; matching a corresponding video scene recognition model on a cloud platform according to the video scene description information, and performing scene recognition on the first video stream according to the video scene recognition model to obtain a scene recognition result; and adjusting the video scene description information according to the scene recognition result, and judging whether to give an alarm or not according to the adjusted video scene description information. The method improves the efficiency and accuracy of scene monitoring, and can be widely applied to the technical field of video monitoring.

Description

Technical Field

[0001] The present invention relates to the field of video surveillance technology, and in particular to a scene monitoring method, device, equipment and medium based on a multimodal large model. Background Art

[0002] As the construction of smart cities is further advanced, the number of IoT devices such as cameras is growing exponentially. How to efficiently and accurately identify the information collected by cameras is crucial to urban governance. At present, traditional visual monitoring solutions based on cameras have been widely used in various fields such as production and life. For example, license plate recognition cameras installed at highway checkpoints can accurately identify the license plates of passing vehicles; high-speed cameras installed in mountainous areas can capture and monitor mountain fires; garbage monitoring cameras installed in streets and alleys. This type of camera installed with traditional AI algorithms can achieve monitoring and classification in traditional single scenes, but it is still essentially unable to understand the video. For example, garbage detection cameras installed in streets and alleys have no way to understand whether vicious fights have occurred on the streets; license plate recognition cameras installed at traffic checkpoints cannot identify whether traffic accidents have occurred. With the development of multimodal large model technology, it is possible to integrate multiple information such as surveillance video and audio for places such as communities, parks, and factories to achieve more accurate intelligent monitoring and early warning.

[0003] The scene monitoring solutions in the prior art mainly include the following:

[0004] 1) AI camera monitoring. Mainly through the installation of traditional CV models on camera equipment, target objects are identified in the fields of security, fire protection, etc. In actual scene monitoring, AI cameras are used to monitor violations and abnormal phenomena; when an abnormality occurs, the event is verified and confirmed by retrieving relevant camera videos to verify and confirm the reported event; when a violation or event occurs, email, voice call, SMS, corporate WeChat, LED screen and smart speaker notifications and broadcasts can be linked; alarm events are graded and warned, and the responsible person is pushed for the first time. If it is not handled within the deadline, it will be pushed to the relevant leaders, and production safety reports can be made every week and every month; after the event is pushed, the photos and videos are manually checked and confirmed.

[0005] 2) AI box associated camera monitoring. AI box is mainly used to AI traditional cameras, so that they have the ability to monitor video images. It is mostly used to transform traditional cameras. The handling processes such as event monitoring, verification and confirmation, linkage alarm, alarm push, and completion and verification are consistent with the above. Taking the security scene as an example, traditional security solves the problem of "seeing" and "seeing clearly", while intelligent security solves the problem of "understanding". In the past, people relied on manual methods to view videos, but now intelligent security will record useful information such as "traffic hustle and bustle" and filter out useless information such as "slight disturbance".

[0006] 3) Cloud multimodal model monitoring. By extracting key frames from the video and transferring them to a large multimodal model on the cloud for recognition, this method can understand image information to a certain extent and recognize some events. However, since it is recognized through key frames, the information utilization rate is very low. Secondly, the model scale is large, generally above 30b, and the hardware consumption is large, so it can only be deployed on the cloud. The real-time performance of image recognition is low, and alarms cannot be generated in time.

[0007] In summary, the existing scene monitoring solutions have the following disadvantages:

[0008] 1) There are many false alarms and the efficiency of manual verification is low. The alarms generated by monitoring need to be manually pulled to check the video stream, and cannot be processed automatically and efficiently. Therefore, in actual applications, due to the lack of sufficient manpower for verification, a large amount of video alarm data is not fully utilized.

[0009] 2) Traditional AI algorithm cameras installed with CV models can only achieve "seeing" and "seeing clearly" but cannot solve the problem of "understanding". It is difficult for a single CV model to meet the requirements for complex multi-scene judgment.

[0010] 3) The large-scale multimodal models commonly used in the industry have high resource consumption and can only be deployed in the cloud. At the same time, in order to support a large number of monitoring devices, it brings huge challenges to the real-time local alarms.

[0011] Terminology explanation:

[0012] MultiModal Large Language Models (MM-LLMs): Multimodal large models.

[0013] Computer Vision(CV): Computer vision.

[0014] Prompt Engineering(PE): Prompt Engineering. Summary of the invention

[0015] The purpose of the present invention is to solve one of the technical problems existing in the prior art to at least a certain extent.

[0016] To this end, an object of an embodiment of the present invention is to provide a scene monitoring method based on a multimodal large model, which improves the efficiency and accuracy of scene monitoring.

[0017] Another object of an embodiment of the present invention is to provide a scene monitoring device based on a multimodal large model.

[0018] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present invention include:

[0019] On the one hand, an embodiment of the present invention provides a scene monitoring method based on a multimodal large model, comprising the following steps:

[0020] Acquire a first video stream of a target scene, and determine a corresponding video frame image set according to the first video stream;

[0021] Inputting the video frame image set into a multimodal large model deployed on the terminal side to obtain corresponding video scene description information, and then uploading the first video stream and the video scene description information to a cloud platform;

[0022] Matching a corresponding video scene recognition model on the cloud platform according to the video scene description information, and performing scene recognition on the first video stream according to the video scene recognition model to obtain a scene recognition result;

[0023] The video scene description information is adjusted according to the scene recognition result, and it is determined whether to issue an alarm according to the adjusted video scene description information.

[0024] Further, in one embodiment of the present invention, determining a corresponding video frame image set according to the first video stream specifically includes:

[0025] Segmenting the first video stream according to a preset time length to obtain a plurality of video segments;

[0026] Extracting video frames of each of the video clips according to a preset sampling frequency to obtain the video frame image set corresponding to each of the video clips;

[0027] The image set number of the video frame image set is determined according to the start time, the end time and the corresponding camera code of the video segment.

[0028] Further, in one embodiment of the present invention, the step of inputting the video frame image set into a multimodal large model deployed on the terminal side to obtain corresponding video scene description information specifically includes:

[0029] Generate a first prompt sentence according to a preset first prompt word template;

[0030] Inputting the video frame image set and the first prompt sentence into the multimodal large model to obtain the video scene description information;

[0031] The video scene description information includes video content description, event type description and event nature description.

[0032] Further, in one embodiment of the present invention, matching the corresponding video scene recognition model on the cloud platform according to the video scene description information specifically includes:

[0033] Obtaining a scene recognition model library stored in the cloud platform;

[0034] The target model type is determined according to the event type description, and matching is performed in the scene recognition model library according to the target model type to obtain the corresponding video scene recognition model.

[0035] Furthermore, in one embodiment of the present invention, the scene recognition model library is constructed by the following steps:

[0036] Acquire the video scene recognition models of multiple subdivision fields that have been pre-trained, and determine the model type labels of the corresponding video scene recognition models according to the subdivision fields;

[0037] Using the model type label as the key value and the corresponding video scene recognition model as the value value to generate a model key-value pair;

[0038] The scene recognition model library is constructed according to the model key-value pairs.

[0039] Further, in an embodiment of the present invention, the adjusting the video scene description information according to the scene recognition result specifically includes:

[0040] Determining whether the scene recognition result is consistent with the event type description;

[0041] When the scene recognition result is consistent with the event type description, the video content description is supplemented according to the scene recognition result to obtain adjusted video scene description information;

[0042] When the scene recognition result is inconsistent with the event type description, scene recognition is performed on the first video stream according to the scene recognition large model deployed by the cloud platform to obtain adjusted video scene description information.

[0043] Further, in some optional embodiments, judging whether to issue an alarm according to the adjusted video scene description information specifically includes:

[0044] Generate a second prompt sentence according to a preset second prompt word template;

[0045] Inputting the adjusted video scene description information and the second prompt sentence into the alarm decision model deployed on the cloud platform to obtain an alarm decision description;

[0046] Determine whether to issue an alarm based on the alarm decision description. If an alarm is issued, determine the alarm cause description based on the alarm decision description.

[0047] On the other hand, an embodiment of the present invention provides a scene monitoring device based on a multimodal large model, comprising:

[0048] A video frame extraction module, used to obtain a first video stream of a target scene, and determine a corresponding video frame image set according to the first video stream;

[0049] A device-side recognition module, configured to input the video frame image set into a multimodal large model deployed on the device side, obtain corresponding video scene description information, and then upload the first video stream and the video scene description information to a cloud platform;

[0050] A cloud platform recognition module, used to match a corresponding video scene recognition model on the cloud platform according to the video scene description information, and perform scene recognition on the first video stream according to the video scene recognition model to obtain a scene recognition result;

[0051] The alarm decision module is used to adjust the video scene description information according to the scene recognition result, and determine whether to issue an alarm according to the adjusted video scene description information.

[0052] On the other hand, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein the program, when executed by the processor, realizes the scene monitoring method based on the multimodal large model as described above.

[0053] On the other hand, an embodiment of the present invention further provides a storage medium, which is a computer-readable storage medium for computer-readable storage, and stores one or more programs, and the one or more programs can be executed by one or more processors to implement the scene monitoring method based on the multimodal large model as described above.

[0054] The advantages and beneficial effects of the present invention will be partly given in the following description, partly become apparent from the following description, or be understood through the practice of the present invention:

[0055] The embodiment of the present invention obtains the first video stream of the target scene, determines the corresponding video frame image set according to the first video stream, inputs the video frame image set into the multimodal large model deployed on the end side, obtains the corresponding video scene description information, and then uploads the first video stream and the video scene description information to the cloud platform, matches the corresponding video scene recognition model on the cloud platform according to the video scene description information, and performs scene recognition on the first video stream according to the video scene recognition model to obtain the scene recognition result, adjusts the video scene description information according to the scene recognition result, and determines whether to issue an alarm according to the adjusted video scene description information. The embodiment of the present invention first uses the lightweight multimodal large model deployed on the end side to perform preliminary recognition on the collected video image to obtain the video scene description information, and then matches the video scene recognition model of the corresponding subdivided field on the cloud platform according to the video scene description information, and uses the high-precision video scene recognition model deployed on the cloud platform to further perform scene recognition on the video image, so that the video scene description information can be adjusted according to the scene recognition result to obtain more accurate and detailed video scene description information, and then an alarm decision can be made according to the adjusted video scene description information, thereby improving the efficiency and accuracy of scene monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solution in the embodiments of the present invention, the following introduction is made to the drawings required for use in the embodiments of the present invention. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solution of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0057] Figure 1 A flowchart of a scene monitoring method based on a multimodal large model provided by an embodiment of the present invention;

[0058] Figure 2 A step flow chart of step S101 provided in an embodiment of the present invention;

[0059] Figure 3 A step flow chart of step S102 provided in an embodiment of the present invention;

[0060] Figure 4 A step flow chart of step S103 provided in an embodiment of the present invention;

[0061] Figure 5 A flowchart of the steps for constructing a scene recognition model library provided by an embodiment of the present invention;

[0062] Figure 6 A step flow chart of step S104 provided in an embodiment of the present invention;

[0063] Figure 7 Another step flow chart of step S104 provided in an embodiment of the present invention;

[0064] Figure 8 A schematic diagram of the structure of a scene monitoring device based on a multimodal large model provided in an embodiment of the present invention;

[0065] Fig. 9 A schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention;

[0066] Fig.10 A schematic diagram of the structure of a storage medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0067] Embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limitations on the present application. It should be noted that, although the functional module division is performed in the system schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order from the module division in the system schematic diagram or the flow chart. For the step numbers in the following embodiments, they are only set for the convenience of explanation, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0068] In the description of the present invention, the meaning of "a plurality" is two or more. If there is a description of the first or the second, it is only for the purpose of distinguishing the technical features, and it cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features. In addition, unless otherwise defined, all technical and scientific terms used in this document have the same meaning as those commonly understood by technicians in the technical field of this application. The terms used in this document are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0069] The scene monitoring method based on the multimodal large model provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a set-top box, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the scene monitoring method based on the multimodal large model, etc., but is not limited to the above forms.

[0070] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0071] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0072] like Figure 1 FIG. 1 is a flowchart of a scene monitoring method based on a multimodal large model provided by an embodiment of the present invention, referring to FIG. Figure 1The embodiment of the present invention provides a scene monitoring method based on a multimodal large model, which specifically includes the following steps:

[0073] S101, obtaining a first video stream of a target scene, and determining a corresponding video frame image set according to the first video stream;

[0074] S102: Input the video frame image set into the multimodal large model deployed on the terminal side to obtain corresponding video scene description information, and then upload the first video stream and the video scene description information to the cloud platform;

[0075] S103, matching a corresponding video scene recognition model on the cloud platform according to the video scene description information, and performing scene recognition on the first video stream according to the video scene recognition model to obtain a scene recognition result;

[0076] S104: Adjust the video scene description information according to the scene recognition result, and determine whether to issue an alarm according to the adjusted video scene description information.

[0077] The embodiment of the present invention improves the detection and recognition capabilities of AI camera monitoring scenes by coupling a small-scale lightweight multimodal model deployed on the end side with a high-precision video scene recognition model deployed in different segments on the cloud platform, so that the video stream data can be fully utilized on the end-side device to achieve cognitive understanding of complex scenes. The prompt project is used to achieve adaptive algorithm matching for key scene and general scene monitoring, so as to achieve the purpose of further identifying the high-precision video scene recognition model of the cloud platform for key single scenes, thereby realizing the verification of the end-side monitoring results and being able to cope with the identification and detection of undefined sudden disaster events in complex and sudden scenes. In addition, the embodiment of the present invention uses the recognition results of the end side and the cloud platform for comprehensive analysis and judgment and provides an alarm strategy, which can greatly increase the processing efficiency compared to manual judgment, and provide support for the judgment analysis of multi-service monitoring and early warning of massive monitoring equipment.

[0078] The embodiment of the present invention first uses the lightweight multimodal large model deployed on the terminal side to perform preliminary recognition on the collected video images to obtain video scene description information, and then matches the video scene recognition model of the corresponding segmented field on the cloud platform according to the video scene description information, and uses the high-precision video scene recognition model deployed on the cloud platform to perform further scene recognition on the video images, so that the video scene description information can be adjusted according to the scene recognition result to obtain more accurate and detailed video scene description information, and then an alarm decision can be made according to the adjusted video scene description information, thereby improving the efficiency and accuracy of scene monitoring.

[0079] It should be noted that the focus of the present invention is on the model deployment of the cloud platform and the terminal side and the process of mutual collaboration to achieve scene monitoring. The specific model training process is not the focus of the embodiments of the present invention. The training of relevant models can adopt the training methods of corresponding models in the prior art, or directly use the relevant models already available in the prior art, which will not be elaborated in the embodiments of the present invention.

[0080] like Figure 2 FIG. 1 is a flowchart of step S101 provided in an embodiment of the present invention, referring to FIG. Figure 2 As an optional implementation, determining a corresponding video frame image set according to the first video stream specifically includes:

[0081] S1011, dividing the first video stream into multiple video segments according to a preset time length;

[0082] S1012, extracting video frames of each video segment according to a preset sampling frequency to obtain a video frame image set corresponding to each video segment;

[0083] S1013: Determine an image set number of a video frame image set according to the start time, the end time and the corresponding camera code of the video segment.

[0084] Specifically, the video stream is divided into multiple video segments according to the time length T, where T is less than or equal to 24 hours; the video frames of the video segments are extracted at 15 frames / s and packaged into a group of image sets images; the image sets are numbered according to the rule of "start time + end time + camera code + unique value + number", for example, images-202409111420-111520-vabc-123.

[0085] like Figure 3 FIG. 1 is a flow chart of step S102 provided in an embodiment of the present invention, referring to FIG. Figure 3 As an optional implementation, the video frame image set is input into the multimodal large model deployed on the client side to obtain corresponding video scene description information, which specifically includes:

[0086] S1021, generating a first prompt sentence according to a preset first prompt word template;

[0087] S1022, inputting the video frame image set and the first prompt sentence into the multimodal large model to obtain video scene description information;

[0088] The video scene description information includes video content description, event type description and event nature description.

[0089] Specifically, the images set is fed into the multimodal large model, the preset prompt words are edited, the model is guided to output its understanding of the video, a detailed description of the video is generated, and then the first video stream and video scene description information are uploaded to the cloud platform.

[0090] For example, input the images image set, and input the prompt word: ""msgs":[{"role":"user","content":"Classify the events in the video as neutral, negative or positive\nAnswer strictly in the following format. For example: [Video content]: A fight occurred; [Event type]: Fire; [Event nature]: Negative. \n[Video content]: XXXXXXXX (detailed description of the video content); [Label]: XXXX (only one can be selected: a fire, b traffic congestion, c flooding, d injury, e garbage accumulation, f cannot be defined); [Event classification]: XX (can only answer positive, negative or neutral). "." The model returns information: ""result":["[Video content]: A fire occurred, grass and trees beside the road burned, and no casualties occurred; [Event type]: Fire; [Event nature]: Negative. "."

[0091] In the above example, the information returned by the model includes video content description, event type description, and event nature description. Then, using "images-202409111420-111520-vabc-123" as the text number and "result" as the text information, the video scene description information of the corresponding video clip can be obtained.

[0092] like Figure 4 FIG. 1 is a flow chart of step S103 provided in an embodiment of the present invention, referring to FIG. Figure 4 As an optional implementation, matching a corresponding video scene recognition model on the cloud platform according to the video scene description information specifically includes:

[0093] S1031. Obtain a scene recognition model library stored in the cloud platform;

[0094] S1032: Determine the target model type according to the event type description, and match the target model type in the scene recognition model library to obtain a corresponding video scene recognition model.

[0095] Specifically, the cloud platform is deployed with a scene recognition model library, which includes high-precision video scene recognition models of different model types, such as fire recognition models, traffic congestion recognition models, flood recognition models, personal injury recognition models, and waste recognition models. These high-precision video scene recognition models can not only accurately identify the corresponding video scenes, but also quantify the degree, for example, the severity of the fire, the degree of traffic congestion, etc.

[0096] The corresponding high-precision video scene recognition model is matched according to the video scene description information uploaded by the terminal side. For example, if the event type is described as "fire", the target model type is determined to be "fire recognition", and then the high-precision video scene recognition model of the corresponding model type is searched in the scene recognition model library for further scene recognition, so as to determine whether a fire has occurred and the scope, source, severity, and whether there are casualties, etc. of the fire.

[0097] like Figure 5 FIG. 1 is a flowchart of a step of constructing a scene recognition model library provided by an embodiment of the present invention, referring to FIG. Figure 5 As an optional implementation, the scene recognition model library is constructed by the following steps:

[0098] S201, obtaining pre-trained video scene recognition models for multiple subdivision fields, and determining model type labels of corresponding video scene recognition models according to the subdivision fields;

[0099] S202, using the model type label as the key value and the corresponding video scene recognition model as the value value to generate a model key-value pair;

[0100] S203: Build a scene recognition model library based on the model key-value pairs.

[0101] like Figure 6 FIG. 1 is a flow chart of step S104 provided in an embodiment of the present invention, referring to FIG. Figure 6 As an optional implementation, the video scene description information is adjusted according to the scene recognition result, which specifically includes:

[0102] S1041, determining whether the scene recognition result is consistent with the event type description;

[0103] S1042: When the scene recognition result is consistent with the event type description, the video content description is completed according to the scene recognition result to obtain adjusted video scene description information;

[0104] S1043. When the scene recognition result is inconsistent with the event type description, scene recognition is performed on the first video stream according to the scene recognition large model deployed on the cloud platform to obtain adjusted video scene description information.

[0105] Specifically, it is determined whether the scene recognition result output by the high-precision video scene recognition model is consistent with the event type description uploaded on the end side, so as to adjust the video scene description information. For example, the video scene description information uploaded on the end side is "[Video content]: A fire occurred, grass and trees beside the road burned, and no casualties occurred; [Event type]: Fire; [Nature of event]: Negative", and the scene recognition result output by the high-precision video scene recognition model is a multi-label result "A fire occurred, the fire was large, it was a natural fire, a second-level fire, and there were no casualties", which means that the scene recognition result is consistent with the event type description, and the scene recognition result can be used to adjust the scene recognition result. The other tags of the video content description are used to complete the video content description to obtain the adjusted video scene description information; for another example, the video scene description information uploaded by the terminal side is "[Video content]: A fire occurred, grass and trees beside the road were burned, and no casualties occurred; [Event type]: Fire; [Nature of event]: Negative", and the scene recognition result output by the high-precision video scene recognition model is a single-label result "No fire occurred", which means that the scene recognition result is inconsistent with the event type description. At this time, it is necessary to re-perform scene recognition on the first video stream according to the high-precision general scene recognition large model deployed on the cloud platform to obtain the adjusted video scene description information.

[0106] like Figure 7 FIG. 1 is another step flow chart of step S104 provided in an embodiment of the present invention, referring to FIG. Figure 7 As an optional implementation, determining whether to issue an alarm according to the adjusted video scene description information specifically includes:

[0107] S1044, generating a second prompt sentence according to a preset second prompt word template;

[0108] S1045, inputting the adjusted video scene description information and the second prompt sentence into the alarm decision model deployed on the cloud platform to obtain an alarm decision description;

[0109] S1046. Determine whether to issue an alarm based on the alarm decision description. If an alarm is issued, determine the alarm cause description based on the alarm decision description.

[0110] Specifically, the adjusted video scene description information is input into the comprehensive decision-making model for analysis. For example, construct prompt: "[Video content]: A fire broke out, grass and trees beside the road burned, the fire was large, natural fire, secondary fire, no casualties occurred; [Event type]: Fire; [Event nature]: Negative. ". Determine whether to issue an alarm based on the above content and answer in the following format: Alarm: XXX (only answer "yes" or "no"), Alarm reason: XXXXXX (description of alarm reason). For example: Alarm: Yes, Alarm reason: A fight has caused casualties. ", the decision-making model returns the result: "Alarm: Yes, Alarm reason: The spread of fire will cause loss of life and property, and the relevant departments need to be notified in time for processing. ", then generate a work order dispatch, the relevant department personnel receive the order and send people to handle it, and check again through the camera whether the disposal is completed.

[0111] The above is an explanation of the method steps of the embodiment of the present invention. It can be recognized that the embodiment of the present invention first uses the lightweight multimodal large model deployed on the terminal side to perform preliminary recognition on the collected video images to obtain video scene description information, and then matches the video scene recognition model of the corresponding subdivision field on the cloud platform according to the video scene description information, and uses the high-precision video scene recognition model deployed on the cloud platform to further recognize the scene of the video image, so that the video scene description information can be adjusted according to the scene recognition result to obtain more accurate and detailed video scene description information, and then the alarm decision can be made according to the adjusted video scene description information, thereby improving the efficiency and accuracy of scene monitoring.

[0112] like Figure 8 FIG. 1 is a schematic diagram of the structure of a scene monitoring device based on a multimodal large model provided by an embodiment of the present invention, referring to FIG. Figure 8 , an embodiment of the present invention provides a scene monitoring device based on a multimodal large model, comprising:

[0113] A video frame extraction module, used to obtain a first video stream of a target scene, and determine a corresponding video frame image set according to the first video stream;

[0114] The end-side recognition module is used to input the video frame image set into the multimodal large model deployed on the end-side to obtain the corresponding video scene description information, and then upload the first video stream and the video scene description information to the cloud platform;

[0115] A cloud platform recognition module, used to match a corresponding video scene recognition model on the cloud platform according to the video scene description information, and perform scene recognition on the first video stream according to the video scene recognition model to obtain a scene recognition result;

[0116] The alarm decision module is used to adjust the video scene description information according to the scene recognition result, and determine whether to issue an alarm according to the adjusted video scene description information.

[0117] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0118] The embodiment of the present invention also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned scene monitoring method based on the multimodal large model. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0119] like Fig. 9 FIG. 1 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Fig. 9 , an embodiment of the present invention provides an electronic device, including:

[0120] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention;

[0121] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the scene monitoring method based on the multimodal large model of the embodiment of the present invention;

[0122] Input / output interface 903, used to implement information input and output;

[0123] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0124] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0125] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0126] like Fig.10 FIG. 1 is a schematic diagram of a storage medium according to an embodiment of the present invention. Fig.10 The embodiment of the present invention further provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs 1001, and the one or more programs 1001 can be executed by one or more processors to implement the above-mentioned scene monitoring method based on the multimodal large model.

[0127] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0128] The embodiment of the present invention also discloses a computer program product or a computer program, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes Figure 1 The method shown.

[0129] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the above-mentioned boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. The optional embodiment is expected, wherein the order of various operations is changed and the sub-operation described as a part of a larger operation is performed independently.

[0130] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified to the contrary, one or more of the above-mentioned functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0131] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the above methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0132] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0133] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the above-mentioned program is printed, since the above-mentioned program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or processing in other suitable ways as necessary, and then stored in a computer memory.

[0134] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0135] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0136] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0137] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A scene monitoring method based on a multimodal large model, characterized in that: The following steps are involved: Acquire a first video stream of a target scene, and determine a corresponding video frame image set according to the first video stream; Inputting the video frame image set into a multimodal large model deployed on the terminal side to obtain corresponding video scene description information, and then uploading the first video stream and the video scene description information to a cloud platform; Matching a corresponding video scene recognition model on the cloud platform according to the video scene description information, and performing scene recognition on the first video stream according to the video scene recognition model to obtain a scene recognition result; The video scene description information is adjusted according to the scene recognition result, and whether to issue an alarm is determined according to the adjusted video scene description information.

2. The scene monitoring method based on a multimodal large model according to claim 1, characterized in that: The determining a corresponding video frame image set according to the first video stream specifically includes: Segmenting the first video stream according to a preset time length to obtain a plurality of video segments; Extracting video frames of each of the video clips according to a preset sampling frequency to obtain the video frame image set corresponding to each of the video clips; The image set number of the video frame image set is determined according to the start time, the end time and the corresponding camera code of the video segment.

3. The scene monitoring method based on a multimodal large model according to claim 1, characterized in that: The step of inputting the video frame image set into the multimodal large model deployed on the terminal side to obtain corresponding video scene description information specifically includes: Generate a first prompt sentence according to a preset first prompt word template; Inputting the video frame image set and the first prompt sentence into the multimodal large model to obtain the video scene description information; The video scene description information includes video content description, event type description and event nature description.

4. The scene monitoring method based on a multimodal large model according to claim 3 is characterized in that: The matching of the corresponding video scene recognition model on the cloud platform according to the video scene description information specifically includes: Obtaining a scene recognition model library stored in the cloud platform; The target model type is determined according to the event type description, and matching is performed in the scene recognition model library according to the target model type to obtain the corresponding video scene recognition model.

5. The scene monitoring method based on a multimodal large model according to claim 4 is characterized in that: The scene recognition model library is constructed by the following steps: Acquire the video scene recognition models of multiple subdivision fields that have been pre-trained, and determine the model type labels of the corresponding video scene recognition models according to the subdivision fields; Using the model type label as the key value and the corresponding video scene recognition model as the value value to generate a model key-value pair; The scene recognition model library is constructed according to the model key-value pairs.

6. The scene monitoring method based on a multimodal large model according to claim 3 is characterized in that: The step of adjusting the video scene description information according to the scene recognition result specifically includes: Determining whether the scene recognition result is consistent with the event type description; When the scene recognition result is consistent with the event type description, the video content description is supplemented according to the scene recognition result to obtain adjusted video scene description information; When the scene recognition result is inconsistent with the event type description, scene recognition is performed on the first video stream according to the scene recognition large model deployed by the cloud platform to obtain adjusted video scene description information.

7. A scene monitoring method based on a multimodal large model according to any one of claims 1 to 6, characterized in that: The step of determining whether to issue an alarm according to the adjusted video scene description information specifically includes: Generate a second prompt sentence according to a preset second prompt word template; Inputting the adjusted video scene description information and the second prompt sentence into the alarm decision model deployed on the cloud platform to obtain an alarm decision description; Determine whether to issue an alarm based on the alarm decision description. If an alarm is issued, determine the alarm cause description based on the alarm decision description.

8. A scene monitoring device based on a multimodal large model, characterized in that: include: A video frame extraction module, used to obtain a first video stream of a target scene, and determine a corresponding video frame image set according to the first video stream; A device-side recognition module, configured to input the video frame image set into a multimodal large model deployed on the device side, obtain corresponding video scene description information, and then upload the first video stream and the video scene description information to a cloud platform; A cloud platform recognition module, used to match a corresponding video scene recognition model on the cloud platform according to the video scene description information, and perform scene recognition on the first video stream according to the video scene recognition model to obtain a scene recognition result; The alarm decision module is used to adjust the video scene description information according to the scene recognition result, and determine whether to issue an alarm according to the adjusted video scene description information.

9. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the scene monitoring method based on a multimodal large model as described in any one of claims 1 to 7 are realized.

10. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the scene monitoring method based on a multimodal large model as described in any one of claims 1 to 7.

Images