Abnormal event early warning method, device and equipment based on video data

Abstract

The invention provides an abnormal event early warning method, device and equipment based on video data, which can be applied to the field of artificial intelligence. The method comprises the following steps: acquiring to-be-processed video data, wherein the to-be-processed video data comprises a to-be-processed video, and time data and place data related to the to-be-processed video; processing a to-be-processed video by using the trained event recognition model to obtain an initial event type of the to-be-processed video; under the condition that the initial event type belongs to the abnormal event type, determining an early warning prompt text according to the initial event type, the time data and the place data; transmitting the early warning prompt text and visual features to a cloud, wherein the visual features are obtained by performing feature extraction on the to-be-processed video by the monitoring terminal; processing the early warning prompt text and the visual features by using a big language model of the cloud to obtain an abnormal event analysis result, the abnormal event analysis result comprising an abnormal event type; and matching the abnormal event type with a preset emergency plan to determine an emergency strategy.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to an abnormal event early warning method, device and equipment based on video data. Background Art

[0002] With the widespread application of video surveillance systems in the fields of public security, traffic safety, etc., video abnormal event detection and early warning have become an important part of smart city construction. However, massive surveillance video data poses a huge challenge to real-time analysis and rapid response. In addition, video data involves citizens' privacy. While rapid response is required in abnormal early warning, citizens' privacy in the video screen must also be protected. Summary of the invention

[0003] In view of the above problems, the present invention provides a method, device and equipment for early warning of abnormal events based on video data.

[0004] According to a first aspect of the present invention, there is provided a method for warning abnormal events based on video data, comprising: obtaining video data to be processed, the video data to be processed comprising the video to be processed, time data and location data related to the video to be processed; processing the video to be processed using a trained event recognition model to obtain an initial event type of the video to be processed; when the initial event type belongs to an abnormal event type, determining a warning prompt text according to the initial event type, the time data and the location data; transmitting the warning prompt text and visual features to a cloud, the visual features being obtained by feature extraction of the video to be processed by a monitoring terminal; processing the warning prompt text and visual features using a large language model on the cloud to obtain an abnormal event analysis result, the abnormal event analysis result comprising the abnormal event type, the visual features being obtained by feature extraction of the video to be processed; matching the abnormal event type with a preset emergency plan to determine an emergency strategy.

[0005] According to an embodiment of the present invention, a large language model on the cloud is used to process warning prompt text and visual features to obtain abnormal event analysis results, including: fusing the warning prompt text and visual features to obtain multimodal fusion features; determining N text word features and M visual word features according to the multimodal fusion features, where M is an integer greater than 0 and N is an integer greater than 0; for each visual word feature, processing the visual word feature and the N text word features based on an attention mechanism to obtain a target visual word feature; for each text word feature, processing the text word feature and the M visual word features based on an attention mechanism to obtain a target text word feature; and using a multi-head prediction function to process the N target visual word features and the M target text word features to obtain abnormal event analysis results.

[0006] According to an embodiment of the present invention, visual word-meta features and N text word-meta features are processed based on an attention mechanism to obtain target visual word-meta features, including: processing visual word-meta features and N text word-meta features based on an attention mechanism to obtain visual attention weights; performing weighted summation of the N text word-meta features according to the visual attention weights to obtain text representation features; and using a feedforward network to process text representation features and visual word-meta features to obtain target visual word-meta features.

[0007] According to an embodiment of the present invention, text word features and M visual word features are processed based on an attention mechanism to obtain target text word features, including: processing text word features and M visual word features based on an attention mechanism to obtain text attention weights; performing weighted summation of the M visual word features according to the text attention weights to obtain visual representation features; and using a feedforward network to process the visual representation features and text word features to obtain target text word features.

[0008] According to an embodiment of the present invention, before matching the abnormal event type with the preset emergency plan to determine the emergency strategy, it also includes: generating structured warning information based on the abnormal event analysis results, the structured warning information including the abnormal event type, the abnormal occurrence time period and the abnormal location; and sending the structured warning information to the target personnel.

[0009] According to an embodiment of the present invention, the trained event recognition model is trained based on the following steps: obtaining sample video data and video label data, the video label data including abnormal event type labels, time labels and location labels; iteratively performing the following operations until the loss function value converges: inputting the sample video data into the initial event recognition model, outputting the training event type, training time information and training location information; using the cross entropy loss function to process the training event type, training time information and training location information with the video label data to obtain the loss function value.

[0010] The second aspect of the present invention provides an abnormal event warning device based on video data, including: an acquisition module, used to acquire video data to be processed, the video data to be processed includes the video to be processed, time data and location data related to the video to be processed; an initial event type acquisition module, used to use the trained event recognition model to process the video to be processed to obtain the initial event type of the video to be processed; a warning prompt text determination module, used to determine the warning prompt text according to the initial event type, time data and location data when the initial event type belongs to the abnormal event type; a transmission module, used to transmit the warning prompt text and visual features to the cloud, the visual features are obtained by the monitoring terminal performing feature extraction on the video to be processed; an abnormal event analysis result acquisition module, used to use the large language model on the cloud to process the warning prompt text and visual features to obtain the abnormal event analysis result, the abnormal event analysis result includes the abnormal event type; a matching module, used to match the abnormal event type with a preset emergency plan to determine the emergency strategy.

[0011] A third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0012] The fourth aspect of the present invention further provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the above method when executed by a processor.

[0013] The fifth aspect of the present invention also provides a computer program product, including a computer program or instruction, which implements the steps of the above method when executed by a processor. According to an embodiment of the present invention, the video to be processed is analyzed in real time by using a trained event recognition model to obtain an initial event type. When the initial event type belongs to an abnormal event type, the warning prompt text can be determined based on the initial event type, time data and location data to ensure the response speed, and the warning prompt text and visual features are transmitted to the cloud. The visual features are obtained by feature extraction of the video to be processed by the monitoring terminal. The warning prompt text and visual features are processed by a large language model on the cloud to obtain abnormal event analysis results, and the abnormal event type is matched with a preset emergency plan to determine emergency measures. This method does not transmit the original video screen back to the cloud, but communicates through visual features with highly compressed parameters, while improving the analysis capability and protecting the privacy of personnel to the maximum extent. The lightweight event recognition model is deployed on the monitoring terminal to conduct preliminary screening of abnormal behaviors, overcoming the shortcomings of large language models deployed on edge devices, such as high resource overhead and high inference latency. The large language model on the cloud is responsible for in-depth analysis of suspicious events, leveraging its powerful semantic understanding and data association capabilities to improve the accuracy and comprehensiveness of abnormal event identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0015] Figure 1 An application scenario diagram showing a method and device for early warning of abnormal events based on video data according to an embodiment of the present invention;

[0016] Figure 2 A flowchart of an abnormal event early warning method based on video data according to an embodiment of the present invention is shown;

[0017] Figure 3 A schematic diagram showing an abnormal event early warning method based on video data according to another embodiment of the present invention is shown;

[0018] Figure 4 It shows a structural block diagram of an abnormal event early warning device based on video data according to an embodiment of the present invention;

[0019] Figure 5 A block diagram of an electronic device suitable for implementing an abnormal event early warning method based on video data according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0020] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0021] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0022] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0023] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0024] With the widespread application of video surveillance systems in the fields of public security, traffic safety, etc., video abnormal event detection and early warning have become an important part of smart city construction. However, massive surveillance video data poses a huge challenge to real-time analysis and rapid response. In addition, video data involves citizens' privacy, and how to balance efficiency and privacy protection in abnormal early warning also needs to be solved urgently.

[0025] In view of this, the present invention provides an abnormal event warning method based on video data, an abnormal event warning device and equipment based on video data. The method comprises: obtaining video data to be processed, the video data to be processed includes the video to be processed, time data and location data related to the video to be processed; using the trained event recognition model to process the video to be processed to obtain the initial event type of the video to be processed; when the initial event type belongs to the abnormal event type, determining the warning prompt text according to the initial event type, time data and location data; using the cloud-based large language model to process the warning prompt text and visual features to obtain the abnormal event analysis result, the abnormal event analysis result includes the abnormal event type, and the visual features are obtained by feature extraction of the video to be processed; matching the abnormal event type with the preset emergency plan to determine the emergency strategy.

[0026] It should be noted that the abnormal event warning method based on video data and the abnormal event warning device based on video data provided by the present invention can be used in the field of artificial intelligence, and can also be used in any field other than the field of artificial intelligence, such as the field of urban traffic control. Therefore, the application field of the abnormal event warning method based on video data and the abnormal event warning device based on video data provided by the present invention is not limited.

[0027] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0028] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present invention provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0029] Figure 1 A diagram showing an application scenario of the abnormal event early warning method based on video data according to an embodiment of the present invention.

[0030] like Figure 1 As shown, the application scenario according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0031] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0032] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0033] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0034] It should be noted that the abnormal event warning method based on video data provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the abnormal event warning device based on video data provided in the embodiment of the present invention can generally be set in the server 105. The abnormal event warning method based on video data provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Correspondingly, the abnormal event warning device based on video data provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0035] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0036] Figure 2 A flow chart of an abnormal event early warning method based on video data according to an embodiment of the present invention is shown.

[0037] like Figure 2 As shown, the abnormal event early warning method based on video data of this embodiment includes operations S210 to S260, and the abnormal event early warning method based on video data can be executed by an electronic device.

[0038] In operation S210, to-be-processed video data is acquired, where the to-be-processed video data includes the to-be-processed video, and time data and location data related to the to-be-processed video.

[0039] In operation S220, the video to be processed is processed using the trained event recognition model to obtain an initial event type of the video to be processed.

[0040] In operation S230, when the initial event type belongs to an abnormal event type, a warning prompt text is determined according to the initial event type, time data, and location data.

[0041] In operation S240, the warning prompt text and visual features are transmitted to the cloud, and the visual features are obtained by extracting features from the video to be processed by the monitoring terminal. In operation S250, the warning prompt text and visual features are processed using a large language model on the cloud to obtain an abnormal event analysis result, which includes an abnormal event type, and the visual features are obtained by extracting features from the video to be processed.

[0042] In operation S260, the abnormal event type is matched with a preset emergency plan to determine an emergency strategy.

[0043] According to an embodiment of the present invention, video data to be processed may be obtained, and the video data to be processed may include the video to be processed, time data and location data related to the video to be processed. The time data may include the start time and end time of the video. The location data may include the location of the monitoring scene, such as a street, a square, a station, a campus, a supermarket, etc., but is not limited thereto, and the embodiments of the present invention do not limit this.

[0044] According to the embodiments of the present invention, the trained event recognition model is used to process the video to be processed, and the initial event type of the video to be processed can be obtained. The initial event type may include daily activities, work-related behaviors, traffic behaviors, crowds, falls, climbs, etc., but is not limited to these. The embodiments of the present invention do not limit the specific categories of the initial event type. , using the event recognition model Perform frame-by-frame detection. Output initial event type Abnormal event type When Suspicious abnormal behaviors are detected at all times, and the next step of complex event analysis is initiated, which is to use the large language model in the cloud for further processing.

[0045] According to the embodiment of the present invention, the abnormal event types may include gathering of people, fighting, intrusion, fire, etc., but are not limited thereto. The embodiment of the present invention does not limit the specific categories of abnormal event types. , the system extracts 5 frames of image input every 1 second Perform anomaly detection. If the same abnormal event type is detected within three consecutive frames, it is considered that there is suspicious abnormal behavior and a real-time warning can be issued to improve the response speed. In the case where the initial event type belongs to the abnormal event type, the warning prompt text is determined based on the initial event type, time data and location data. After a suspicious abnormal event is detected, the event recognition model will and preliminary analysis information The data are packaged into warning prompt text, as shown in formula (1).

[0046] (1);

[0047] in, Indicates warning prompt text, abnormal fragment , including the duration before and after video content. Indicates the use of natural language to describe the initial detected event information, which can include the initial event type, time data and location data, such as: "A suspected fighting incident occurred in the South Gate Square of the campus, lasting from about 20xx-05-01 14:32:00 to 20xx-05-01 14:33:30" includes the duration before and after = 5 seconds of video content.

[0048] According to an embodiment of the present invention, the method can use an event recognition model to analyze surveillance video data in real time, preliminarily identify abnormal behaviors such as gatherings of people and fighting, and generate early warning prompts containing key information of the event type, ensuring that the system can respond quickly to potential risks.

[0049] According to an embodiment of the present invention, by using a large language model in the cloud to process the warning prompt text and visual features, an abnormal event analysis result can be obtained, and the abnormal event analysis result can include the abnormal event type. The visual feature is obtained by extracting features from the video to be processed, and the video to be processed can be processed by a visual feature extractor to obtain the visual feature, as shown in formula (2).

[0050] (2);

[0051] in, Represents visual features, Represents a pre-trained visual feature extractor, such as ResNet101, etc. are the parameters of the visual feature extractor.

[0052] According to the embodiments of the present invention, the cloud-based large language model can be used to perform more fine-grained event analysis on suspicious clips (videos whose initial event type belongs to the abnormal event type). The identified abnormal clips can be integrated into a "task template" and input into the large language model for visual semantic understanding. The large language model models the spatiotemporal dependency of video clips through the self-attention mechanism, mines the event evolution context, and combines with the external knowledge base to infer and judge the nature of the event, and finally outputs a structured description covering the cause, process, and personnel information of the event.

[0053] According to an embodiment of the present invention, the abnormal event type can be matched with the preset emergency plan to determine the emergency strategy. Emergency plans for multiple common abnormal event types can be predefined and stored in the policy library of the system. For example, when the abnormal event type When a fire occurs, the emergency measures recommended by the system may include: 1) immediately dispatching firefighters to the scene; 2) notifying on-site personnel to evacuate in an orderly manner; 3) cutting off the power supply to the fire area. Through the predefined emergency response strategy, the loss of personnel can be minimized and the negative impact of abnormal events can be minimized.

[0054] In addition, this method is applicable to but not limited to various video surveillance scenarios such as smart communities and smart parks. The abnormal warning method of this method can provide solid technical support for tasks such as community security management and internal security of enterprises, and help build a safer and smarter working and living environment.

[0055] According to an embodiment of the present invention, by using the trained event recognition model to analyze the video to be processed in real time, the initial event type is obtained. When the initial event type belongs to the abnormal event type, the warning prompt text can be determined according to the initial event type, time data and location data to ensure the response speed. The warning prompt text and visual features are transmitted to the cloud by the large language model on the cloud. The visual features are obtained by feature extraction of the video to be processed by the monitoring terminal. The large language model on the cloud is used to process the warning prompt text and visual features to obtain the abnormal event analysis results, and the abnormal event type is matched with the preset emergency plan to determine the emergency measures. This method does not transmit the original video screen back to the cloud, but communicates through the visual features with highly compressed parameters, which maximizes the protection of personnel privacy while improving the analysis ability. The lightweight event recognition model is deployed on the monitoring terminal for preliminary abnormal behavior screening, which overcomes the shortcomings of large resource overhead and high inference delay of the large language model deployed on the edge device; and the large language model on the cloud is responsible for in-depth analysis of suspicious events, giving full play to its powerful semantic understanding and data association capabilities, and improving the accuracy and comprehensiveness of abnormal event recognition.

[0056] According to an embodiment of the present invention, a large language model on the cloud is used to process warning prompt text and visual features to obtain abnormal event analysis results, including: fusing the warning prompt text and visual features to obtain multimodal fusion features; determining N text word features and M visual word features according to the multimodal fusion features, where M is an integer greater than 0 and N is an integer greater than 0; for each visual word feature, processing the visual word feature and the N text word features based on an attention mechanism to obtain a target visual word feature; for each text word feature, processing the text word feature and the M visual word features based on an attention mechanism to obtain a target text word feature; and using a multi-head prediction function to process the N target visual word features and the M target text word features to obtain abnormal event analysis results.

[0057] According to an embodiment of the present invention, the warning prompt text can be and visual features Fusion to obtain multimodal fusion features , as shown in formula (3).

[0058] (3);

[0059] According to an embodiment of the present invention, in order to better integrate visual information and text information, the present invention adopts a cross-modal attention mechanism to update feature representation. Based on the multimodal fusion features, N text word features can be determined. and M visual word features , M is an integer greater than 0, and N is an integer greater than 0. For each visual word-meta feature, the visual word-meta feature and the N text word-meta features are processed based on the attention mechanism to obtain the target visual word-meta feature. For each text word-meta feature, the text word-meta feature and the M visual word-meta features are processed based on the attention mechanism to obtain the target text word-meta feature.

[0060] According to an embodiment of the present invention, a multi-head prediction function is used to process N target visual word features and M target text word features to obtain an abnormal event analysis result. The large language model performs multi-task prediction at the output layer to obtain an abnormal event analysis result. The abnormal event analysis result may include event type , Occurrence time ,Place Etc. The scene in The final output at the moment is shown in formula (4):

[0061] (4);

[0062] in, and They represent the target visual word features and the target text word features respectively. is the multi-head prediction function of the output layer.

[0063] According to an embodiment of the present invention, each frame of image can be scaled to a resolution of 224×224, input to ResNet101, and the output of the penultimate fully connected layer is extracted as a 2048-dimensional feature vector of the frame. It is represented as a F×2048 feature matrix, where F represents the number of images.

[0064] According to an embodiment of the present invention, a higher semantic fusion and event understanding capability is achieved by bidirectionally updating and multi-round interaction of visual word-meta features and text word-meta features. Bidirectional update, for example, for each visual word-meta feature, the visual word-meta feature and all text word-meta features are processed based on the attention mechanism to obtain the target visual word-meta feature; similarly, the text word-meta feature is updated to achieve bidirectional update. The updated target visual word-meta features and target text word-meta features can continue to be bidirectionally updated to achieve multi-round interaction, thereby achieving a higher semantic fusion and event understanding capability, and can ensure the accuracy of large language model analysis while using only a small amount of text information (warning prompt text), achieving a good balance between privacy protection and analysis effect.

[0065] According to an embodiment of the present invention, visual word-meta features and N text word-meta features are processed based on an attention mechanism to obtain target visual word-meta features, including: processing visual word-meta features and N text word-meta features based on an attention mechanism to obtain visual attention weights; performing weighted summation of the N text word-meta features according to the visual attention weights to obtain text representation features; and using a feedforward network to process text representation features and visual word-meta features to obtain target visual word-meta features.

[0066] According to an embodiment of the present invention, in In the attention layer, visual word features can be calculated Attention weights for all text word features. and N text word features After processing, the visual attention weight is obtained, as shown in formula (5).

[0067] (5);

[0068] in, represents the visual attention weight, is the visual query matrix, A key matrix for the text.

[0069] According to an embodiment of the present invention, attention weights can be used The weighted sum of the text word features is obtained relative to the visual word The text representation of is the text representation feature. By weighted summing the N text word features according to the visual attention weight, the text representation feature can be obtained, as shown in formula (6).

[0070] (6);

[0071] in, represents the text representation feature, A matrix of values ​​representing the text.

[0072] According to an embodiment of the present invention, a feed-forward network FFN is used to process text representation features. and visual word features , the target visual word features can be obtained, as shown in formula (7).

[0073] (7);

[0074] in, Represents the target visual word features.

[0075] According to an embodiment of the present invention, a cross-modal attention mechanism can be used to update text word-meta features and visual word-meta features. For each update, the visual word-meta features can be used to update the text word-meta features, and the text word-meta features can be used to update the visual word-meta features. During the update process, the visual word-meta features and the text word-meta features can be updated multiple times, and the embodiment of the present invention does not limit the number of updates.

[0076] According to an embodiment of the present invention, the visual word-meta features can be updated with the text word-meta features for fine-grained alignment, overcoming the limitations of single-modal representation and enhancing the depth and breadth of the large language model's understanding of events. The visual word-meta features and the text word-meta features are updated alternately in multiple cross-modal attention layers, so that the semantic representations of the two are fully aligned and integrated.

[0077] According to an embodiment of the present invention, text word features and M visual word features are processed based on an attention mechanism to obtain target text word features, including: processing text word features and M visual word features based on an attention mechanism to obtain text attention weights; performing weighted summation of the M visual word features according to the text attention weights to obtain visual representation features; and using a feedforward network to process the visual representation features and text word features to obtain target text word features.

[0078] According to an embodiment of the present invention, similarly, the text word-meta features can also be updated by visual word-meta features. The text word-meta features and M visual word-meta features are processed based on the attention mechanism to obtain the text attention weight; the M visual word-meta features are weighted and summed according to the text attention weight to obtain the visual representation feature; the visual representation feature and the text word-meta features are processed using a feedforward network to obtain the target text word-meta features.

[0079] According to an embodiment of the present invention, visual word-meta features can be used to update text word-meta features, fully tapping into the complementary information of vision and text, overcoming the limitations of single-modality representation, and enhancing the depth and breadth of event understanding of a large language model.

[0080] According to an embodiment of the present invention, before matching the abnormal event type with the preset emergency plan to determine the emergency strategy, it also includes: generating structured warning information based on the abnormal event analysis results, the structured warning information including the abnormal event type, the abnormal occurrence time period and the abnormal location; and sending the structured warning information to the target personnel.

[0081] According to an embodiment of the present invention, structured warning information can be generated based on the abnormal event analysis results. ,The structured warning information can include the type of abnormal event, the time period of abnormal occurrence, and the abnormal location, and the formal expression is shown in formula (8).

[0082] (8);

[0083] in, Indicates the type of abnormal event. Indicates the time of occurrence, Indicates the location where the event occurred.

[0084] According to an embodiment of the present invention, this method only uploads the warning prompt text and visual features generated by the event recognition model to the cloud-based large language model, avoiding privacy leakage caused by uploading the original video. The structured warning information can be sent to target personnel, including but not limited to relevant security personnel or competent departments, so that timely response measures can be taken. It does not contain any image or video data, which effectively protects the privacy of people in the monitoring scene.

[0085] According to an embodiment of the present invention, the trained event recognition model is trained based on the following steps: obtaining sample video data and video label data, the video label data including abnormal event type labels, time labels and location labels; iteratively performing the following operations until the loss function value converges: inputting the sample video data into the initial event recognition model, outputting the training event type, training time information and training location information; using the cross entropy loss function to process the training event type, training time information and training location information with the video label data to obtain the loss function value.

[0086] According to an embodiment of the present invention, sample video data and video label data may be obtained, and the video label data may include abnormal event type labels, time labels, and location labels. A data set may be defined ,in Indicates A surveillance video clip. Indicates the corresponding event label, including the event type ,time and location ,Right now , N represents the number of surveillance video clips. Sample video data can be collected from surveillance cameras in P different scenes. ,in Indicates To improve data diversity, monitoring scenes include but are not limited to streets, squares, stations, campuses, supermarkets, etc. Professional labelers review each video , identify possible abnormal event fragments For each segment , the annotator gives the corresponding event label , forming labeled data pairs . Abnormal event type Can be divided into the following kind: ,in Respectively represent the predefined Types of abnormal events, such as gatherings of people, fights, intrusions, etc. Location tags Indicates the location of the event in text form, such as "Nanmen Square", "First Canteen Entrance", etc. Time Label The start and end time of the abnormal event:

[0087] According to an embodiment of the present invention, the following operations can be iteratively performed until the loss function value converges: the sample video data is input into the initial event recognition model, and the training event type, training time information and training location information can be output; the training event type, training time information and training location information and the video label data are processed using the cross entropy loss function to obtain the loss function value. All video clips in the event recognition model Conduct training. The optimization objective of training is as shown in formula (9):

[0088] (9);

[0089] in, For Model In the input snippet The type of abnormal event. Represents the cross entropy loss function, which is used to measure the difference between the predicted abnormal event type and the true label.

[0090] According to an embodiment of the present invention, the event recognition model can be trained using the target detection algorithm lightweight YOLO, which is characterized by small computational complexity and few parameters, and is suitable for deployment in edge computing devices such as surveillance cameras. The event recognition model input is a 224×224 red, green, and blue (RGB) image. The backbone network uses MobileNetV2, and the output layer divides the image into 7×7 units, each unit predicts 2 bounding boxes. The pre-trained public object dataset is used to initialize the model parameters, and then fine-tuned on the dataset D for 50 iterations. The batch size can be set to 32, the learning rate can be set to 0.01, and the optimizer can use Adaptive Moment Estimation (Adam).

[0091] Figure 3 A schematic diagram of an abnormal event early warning method based on video data according to another embodiment of the present invention is shown.

[0092] like Figure 3 As shown, firstly, the video data to be processed 301 is obtained, and the video data to be processed is detected in real time using the event recognition model, and the initial event type 303 can be obtained. When the initial event type belongs to the abnormal event type, the warning prompt text 304 is determined according to the initial event type, time data and location data. The video data to be processed 301 can be processed using an image encoder to obtain visual features 302. The visual features 302 and the warning prompt text 304 are processed using a large language model on the cloud to obtain abnormal event analysis results. Specifically, the visual features 302 and the warning prompt text 304 are processed based on the cross-modal attention mechanism 305 to obtain target visual word-unit features and target text word-unit features. The target visual word-unit features and target text word-unit features are processed using a multi-head prediction function 306 to obtain abnormal event analysis results, which include risk levels 308 and risk warning descriptions 307. The preset emergency plan may include multiple emergency response strategies 309, each of which has a corresponding confidence range, and can be matched from the preset emergency plan based on the risk level 308 and the risk warning description 307 to determine the final emergency response strategy 310. Finally, the emergency response strategy 310 is used to optimize the description of the warning prompt text 304, thereby improving the accuracy of abnormal behavior recognition.

[0093] Figure 4 A structural block diagram of an abnormal event warning device based on video data according to an embodiment of the present invention is shown.

[0094] like Figure 4 As shown, the abnormal event warning device based on video data of this embodiment includes an acquisition module 410, an initial event type acquisition module 420, a warning prompt text determination module 430, a transmission module 440, an abnormal event analysis result acquisition module 450 and a matching module 460.

[0095] The acquisition module 410 is used to acquire the video data to be processed, where the video data to be processed includes the video to be processed, and time data and location data related to the video to be processed.

[0096] The initial event type obtaining module 420 is used to process the video to be processed using the trained event recognition model to obtain the initial event type of the video to be processed.

[0097] The warning prompt text determination module 430 is used to determine the warning prompt text according to the initial event type, time data and location data when the initial event type belongs to the abnormal event type.

[0098] The transmission module 440 is used to transmit the warning prompt text and visual features to the cloud, where the visual features are obtained by the monitoring terminal performing feature extraction on the video to be processed.

[0099] The abnormal event analysis result obtaining module 450 is used to use the large language model in the cloud to process the warning prompt text and visual features to obtain the abnormal event analysis results. The abnormal event analysis results include the abnormal event type. The visual features are obtained by feature extraction of the video to be processed.

[0100] The matching module 460 is used to match the abnormal event type with the preset emergency plan to determine the emergency strategy.

[0101] According to an embodiment of the present invention, by using the trained event recognition model to analyze the video to be processed in real time, the initial event type is obtained. When the initial event type belongs to the abnormal event type, the warning prompt text can be determined according to the initial event type, time data and location data to ensure the response speed, and then the warning prompt text and visual features are processed by the large language model on the cloud to obtain the abnormal event analysis results, and the abnormal event type is matched with the preset emergency plan to determine the emergency measures. The present invention does not transmit the original video screen back to the cloud, but communicates through the visual features with highly compressed parameters, which maximizes the protection of personnel privacy while improving the analysis ability. The lightweight event recognition model is deployed on the monitoring terminal for preliminary abnormal behavior screening, which overcomes the shortcomings of large language model deployment on edge devices, such as large resource overhead and high inference delay; while the large language model on the cloud is responsible for in-depth analysis of suspicious events, exerting its powerful semantic understanding and data association capabilities, and improving the accuracy and comprehensiveness of abnormal event recognition.

[0102] According to an embodiment of the present invention, the abnormal event analysis result obtaining module 440 includes: a fusion unit, a determination unit, an update unit, a target text word-unit feature obtaining unit and an abnormal event analysis result obtaining unit.

[0103] The fusion unit is used to fuse the warning prompt text and the visual features to obtain multimodal fusion features.

[0104] The determination unit is used to determine N text word features and M visual word features according to the multimodal fusion features, where M is an integer greater than 0 and N is an integer greater than 0.

[0105] The updating unit is used to process the visual word-meta feature and N text word-meta features based on the attention mechanism for each visual word-meta feature to obtain a target visual word-meta feature.

[0106] The target text word-meta feature obtaining unit is used to process the text word-meta feature and M visual word-meta features based on the attention mechanism for each text word-meta feature to obtain the target text word-meta feature.

[0107] The abnormal event analysis result obtaining unit is used to process N target visual word-unit features and M target text word-unit features using a multi-head prediction function to obtain the abnormal event analysis result.

[0108] According to an embodiment of the present invention, the updating unit includes: a visual attention weight obtaining subunit, a text representation feature obtaining subunit and a target visual word-meta feature obtaining subunit.

[0109] The visual attention weight obtaining subunit is used to process the visual word-unit features and N text word-unit features based on the attention mechanism to obtain the visual attention weight.

[0110] The text representation feature obtaining subunit is used to perform weighted summation on N text word features according to the visual attention weight to obtain the text representation feature.

[0111] The target visual word-meta feature obtaining subunit is used to process the text representation features and the visual word-meta features using a feed-forward network to obtain the target visual word-meta feature.

[0112] According to an embodiment of the present invention, the updating unit includes: a text attention weight obtaining subunit, a visual representation feature obtaining subunit and a target text word unit feature obtaining subunit.

[0113] The text attention weight obtaining subunit is used to process the text word meta-features and M visual word meta-features based on the attention mechanism to obtain the text attention weights.

[0114] The visual representation feature obtaining subunit is used to perform weighted summation of M visual word-unit features according to the text attention weight to obtain the visual representation feature.

[0115] The target text word-unit feature obtaining subunit is used to process the visual representation features and the text word-unit features by using a feed-forward network to obtain the target text word-unit features.

[0116] According to an embodiment of the present invention, the abnormal event warning device based on video data further includes: a generating module and a sending module.

[0117] The generation module is used to generate structured warning information based on the analysis results of abnormal events. The structured warning information includes the type of abnormal event, the time period when the abnormal event occurred, and the abnormal location.

[0118] The sending module is used to send structured warning information to target personnel.

[0119] According to an embodiment of the present invention, the initial event type obtaining module includes: an obtaining unit and an iterating unit.

[0120] The acquisition unit is used to acquire sample video data and video label data, wherein the video label data includes an abnormal event type label, a time label and a location label.

[0121] The iteration unit is used to iteratively perform the following operations until the loss function value converges: input the sample video data into the initial event recognition model, output the training event type, training time information and training location information; use the cross entropy loss function to process the training event type, training time information and training location information and the video label data to obtain the loss function value.

[0122] According to an embodiment of the present invention, any multiple modules among the acquisition module 410, the initial event type acquisition module 420, the warning prompt text determination module 430, the transmission module 440, the abnormal event analysis result acquisition module 450 and the matching module 460 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the acquisition module 410, the initial event type acquisition module 420, the warning prompt text determination module 430, the transmission module 440, the abnormal event analysis result acquisition module 450 and the matching module 460 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware and firmware or by a suitable combination of any of them. Alternatively, at least one of the acquisition module 410, the initial event type acquisition module 420, the warning prompt text determination module 430, the transmission module 440, the abnormal event analysis result acquisition module 450 and the matching module 460 can be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function can be executed.

[0123] Figure 5 A block diagram of an electronic device suitable for implementing an abnormal event early warning method based on video data according to an embodiment of the present invention is shown.

[0124] like Figure 5As shown, the electronic device according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 to a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0125] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0126] According to an embodiment of the present invention, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the input / output (I / O) interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage portion 508 as needed.

[0127] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0128] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0129] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program includes a program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the abnormal event early warning method based on video data provided by the embodiment of the present invention.

[0130] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 501. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0131] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0132] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0133] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0134] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0135] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0136] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A method for early warning of abnormal events based on video data, characterized in that: include: Acquire video data to be processed, wherein the video data to be processed includes the video to be processed, time data and location data related to the video to be processed; Processing the video to be processed using the trained event recognition model to obtain an initial event type of the video to be processed; In the case where the initial event type belongs to an abnormal event type, determining a warning prompt text according to the initial event type, the time data and the location data; Transmitting the warning prompt text and visual features to the cloud, wherein the visual features are obtained by extracting features from the video to be processed by the monitoring terminal; Processing the warning prompt text and the visual features using the cloud-based large language model to obtain an abnormal event analysis result, wherein the abnormal event analysis result includes an abnormal event type; The abnormal event type is matched with the preset emergency plan to determine the emergency strategy.

2. The method according to claim 1, characterized in that The using of the cloud-based large language model to process the warning prompt text and the visual features to obtain abnormal event analysis results includes: Fusing the warning prompt text with the visual feature to obtain a multimodal fusion feature; Determine N text word features and M visual word features according to the multimodal fusion features, where M is an integer greater than 0 and N is an integer greater than 0; For each of the visual word-meta features, the visual word-meta features and the M text word-meta features are processed based on an attention mechanism to obtain a target visual word-meta feature; For each of the text word-unit features, the text word-unit features and the N visual word-unit features are processed based on an attention mechanism to obtain a target text word-unit feature; The N target visual word-unit features and the M target text word-unit features are processed using a multi-head prediction function to obtain the abnormal event analysis result.

3. The method according to claim 2, characterized in that The step of processing the visual word-meta feature and the M text word-meta features based on the attention mechanism to obtain a target visual word-meta feature includes: Processing the visual word-unit features and the M text word-unit features based on an attention mechanism to obtain a visual attention weight; Performing weighted summation on the M text word features according to the visual attention weights to obtain a text representation feature; The text representation features and the visual word-meta features are processed using a feed-forward network to obtain target visual word-meta features.

4. The method according to claim 2 or 3, characterized in that: The processing of the text word-meta features and the N visual word-meta features based on the attention mechanism to obtain the target text word-meta features includes: Processing the text word-unit features and the N visual word-unit features based on the attention mechanism to obtain text attention weights; Performing weighted summation on the N visual word-element features according to the text attention weights to obtain a visual representation feature; The visual representation features and the text word features are processed using a feedforward network to obtain target text word features.

5. The method according to claim 1, characterized in that Before matching the abnormal event type with the preset emergency plan to determine the emergency strategy, the method further includes: Generate structured warning information according to the abnormal event analysis result, wherein the structured warning information includes the abnormal event type, abnormal occurrence time period and abnormal location; and The structured warning information is sent to the target person.

6. The method according to claim 1, characterized in that The trained event recognition model is trained based on the following steps: Acquire sample video data and video label data, wherein the video label data includes an abnormal event type label, a time label, and a location label; Iteratively perform the following operations until the loss function value converges: Inputting the sample video data into an initial event recognition model, and outputting training event type, training time information, and training location information; The training event type, training time information, training location information and the video label data are processed using a cross entropy loss function to obtain a loss function value.

7. An abnormal event warning device based on video data, characterized in that: include: An acquisition module, used for acquiring video data to be processed, wherein the video data to be processed includes the video to be processed, time data and location data related to the video to be processed; An initial event type obtaining module is used to process the video to be processed using the trained event recognition model to obtain the initial event type of the video to be processed; A warning prompt text determination module, used for determining a warning prompt text according to the initial event type, the time data and the location data when the initial event type belongs to an abnormal event type; A transmission module, used for transmitting the warning prompt text and visual features to the cloud, wherein the visual features are obtained by extracting features from the video to be processed by the monitoring terminal; An abnormal event analysis result obtaining module is used to use the cloud-based large language model to process the warning prompt text and the visual features to obtain an abnormal event analysis result, wherein the abnormal event analysis result includes an abnormal event type; The matching module is used to match the abnormal event type with a preset emergency plan to determine an emergency strategy.

8. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images