Multi-modal intelligent analysis method and system based on law enforcement supervision

Abstract

The invention relates to the technical field of law enforcement intelligent alarm, in particular to a multi-mode intelligent analysis method and system based on law enforcement supervision. The method comprises the following steps: acquiring a law enforcement scene video file from a law enforcement supervision recorder platform, and performing audio and video segmentation and law enforcement behavior information identification analysis to obtain law enforcement scene audio behavior information data and law enforcement scene image behavior information data; performing timestamp and spatial position extraction processing on the law enforcement scene video file, performing multi-modal time-space synchronization association analysis, and performing law enforcement behavior sequence connection processing and event behavior intention recognition analysis at the same time to obtain law enforcement scene event behavior intention information data; and obtaining a preset law enforcement event behavior knowledge graph, and carrying out behavior normative score calculation and nonstandard behavior intelligent alarm optimization analysis to generate a law enforcement scene event nonstandard behavior optimization suggestion report. According to the invention, the accuracy of law enforcement nonstandard behavior supervision alarm can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent warning for law enforcement, and in particular to a multimodal intelligent analysis method and system for law enforcement supervision. Background Art

[0002] In the field of modern law enforcement, the application of multimodal intelligent analysis technology is becoming increasingly important. By integrating multiple information sources such as video, audio and data logs, and using advanced machine learning and deep learning algorithms, the collected and integrated data sets are processed and fused in real time to extract useful information and patterns. At the same time, intelligent analysis models are used to identify abnormal behaviors, illegal operations or other situations that require attention, and generate detailed analysis reports. Based on the analysis results, real-time feedback and improvement suggestions are provided to help law enforcement personnel improve law enforcement efficiency and standardization. However, traditional law enforcement supervision data analysis methods are insufficient in coverage and timeliness, and often fail to respond in a timely manner and fully cover all key monitoring points of law enforcement on-site behaviors. Summary of the invention

[0003] Based on this, it is necessary for the present invention to provide a multimodal intelligent analysis method and system for law enforcement supervision to solve at least one of the above-mentioned technical problems.

[0004] To achieve the above purpose, a multimodal intelligent analysis method based on law enforcement supervision includes the following steps:

[0005] Step S1: Obtain the law enforcement scene video file from the law enforcement supervisory recorder platform, and perform audio and video segmentation processing on the law enforcement scene video file to obtain the law enforcement scene audio segment and the law enforcement scene image video frame segment; perform law enforcement behavior information recognition and analysis on the law enforcement scene audio segment and the law enforcement scene image video frame segment respectively to obtain the law enforcement scene audio behavior information data and the law enforcement scene image behavior information data;

[0006] Step S2: extracting the timestamp and spatial position of the law enforcement scene video file to obtain the timestamp and spatial position information of the law enforcement scene process; performing multimodal spatiotemporal synchronous correlation analysis on the law enforcement scene audio behavior information data and the law enforcement scene image behavior information data based on the timestamp and spatial position information of the law enforcement scene process to obtain multimodal spatiotemporal synchronous law enforcement behavior correlation event data;

[0007] Step S3: Perform law enforcement behavior sequence connection processing on the multimodal spatiotemporal synchronous law enforcement behavior associated event data to generate a law enforcement scene event behavior sequence connection diagram; perform event behavior intention recognition analysis on the law enforcement scene event behavior sequence connection diagram to obtain law enforcement scene event behavior intention information data;

[0008] Step S4: Obtain a preset law enforcement event behavior knowledge graph, and perform a behavioral normativeness score calculation on the law enforcement scene event behavior intention information data based on the law enforcement event behavior knowledge graph to obtain a law enforcement scene behavior intention normativeness score value; perform non-normative behavior intelligent alarm optimization analysis on the corresponding law enforcement scene events in the law enforcement scene event behavior intention information data based on the law enforcement scene behavior intention normativeness score value to generate a non-normative behavior optimization suggestion report for law enforcement scene events.

[0009] Further, step S1 includes the following steps:

[0010] Step S11: Obtaining the law enforcement scene video file from the law enforcement supervisory recorder platform;

[0011] Step S12: Performing audio and video synchronization processing on the law enforcement scene video file to obtain an audio and video synchronized video of the law enforcement scene;

[0012] Step S13: extracting video content from the law enforcement scene audio and video synchronization video to obtain video content information data of the law enforcement scene; based on the video content information data of the law enforcement scene, identifying and analyzing the start and end time of different law enforcement events in the law enforcement scene audio and video synchronization video to obtain the start time and end time of different law enforcement events in the law enforcement scene video;

[0013] Step S14: performing audio and video segmentation processing on the law enforcement scene audio and video synchronization video according to the start time and end time of different law enforcement events in the law enforcement scene video to obtain the law enforcement scene audio segment and the law enforcement scene image video frame segment;

[0014] Step S15: Perform law enforcement behavior information recognition and analysis on the law enforcement scene audio clips and the law enforcement scene image video frame clips respectively to obtain law enforcement scene audio behavior information data and law enforcement scene image behavior information data.

[0015] Further, step S15 includes the following steps:

[0016] Step S151: using Speechbrain technology to perform law enforcement audio voiceprint feature analysis on the law enforcement scene audio clip to obtain law enforcement scene audio voiceprint feature data;

[0017] Step S152: performing audio speech signal channel deconstruction processing on different speakers in the audio clip of the law enforcement scene based on the audio voiceprint feature data of the law enforcement scene, and obtaining audio speech signal sub-channels of different speakers in the audio clip of the law enforcement scene;

[0018] Step S153: performing law enforcement behavior event triggering recognition analysis on the audio speech signal sub-channels of different speakers in the law enforcement scene audio to obtain the law enforcement behavior event triggering points of the law enforcement scene audio;

[0019] Step S154: performing law enforcement audio behavior information recognition and analysis on the law enforcement scene audio clip based on the law enforcement scene audio law enforcement behavior event trigger point to obtain law enforcement scene audio behavior information data;

[0020] Step S155: Use the object detection algorithm Yolov8 and the tracking technology Bytetrack to perform law enforcement image behavior information recognition and analysis on the law enforcement scene image video frame segments to obtain law enforcement scene image behavior information data.

[0021] Further, step S155 includes the following steps:

[0022] The object detection algorithm Yolov8 is used to perform object target recognition analysis on the law enforcement scene image video frame fragments to obtain the object detection target of the law enforcement scene image frame;

[0023] Performing object boundary region marking processing on the object detection target of the law enforcement scene image frame to obtain the law enforcement scene object boundary region marking target;

[0024] The tracking technology Bytetrack is used to track and identify the motion trajectory of the marked target in the boundary area of ​​the law enforcement scene within the video frame of the law enforcement scene image to generate the motion trajectory of the marked target of the law enforcement scene object;

[0025] The target behavior pattern of the object is analyzed on the motion trajectory of the object mark at the law enforcement scene to obtain the law enforcement behavior patterns of the object marks at different law enforcement scenes; based on the law enforcement behavior patterns of the object marks at different law enforcement scenes, the corresponding target boundary area mark of the law enforcement scene is classified and processed to obtain the law enforcement behavior actions of the object marks at different law enforcement scenes;

[0026] Based on the law enforcement behavior patterns and law enforcement behavior actions of different law enforcement scene object targets, law enforcement image behavior information recognition and analysis are performed on the law enforcement scene image video frame fragments to obtain law enforcement scene image behavior information data.

[0027] Further, step S2 includes the following steps:

[0028] Step S21: extracting the timestamp of the law enforcement scene video file to obtain the law enforcement scene process timestamp;

[0029] Step S22: performing a three-dimensional space coordinate system conversion on each frame of the video image in the law enforcement scene video file to obtain a three-dimensional space coordinate system for each frame of the video image in the law enforcement scene video;

[0030] Step S23: performing law enforcement scene spatial coordinate extraction processing on the three-dimensional spatial coordinate system of each frame image in the law enforcement scene video to obtain the law enforcement scene spatial coordinates of each frame image in the law enforcement scene video;

[0031] Step S24: performing spatial position marking processing on the law enforcement scene spatial coordinates of each frame of the law enforcement scene video to obtain the law enforcement scene process spatial position information;

[0032] Step S25: Based on the law enforcement scene process timestamp and the law enforcement scene process spatial position information, a multimodal spatiotemporal synchronous correlation analysis is performed on the law enforcement scene audio behavior information data and the law enforcement scene image behavior information data to obtain multimodal spatiotemporal synchronous law enforcement behavior correlation event data.

[0033] Further, step S25 includes the following steps:

[0034] Step S251: constructing a spatiotemporal synchronization benchmark for the law enforcement scene process timestamp and the law enforcement scene process spatial location information to obtain a spatiotemporal data synchronization benchmark framework for the law enforcement scene process;

[0035] Step S252: Based on the law enforcement scene process spatiotemporal data synchronization benchmark framework, audio event timing synchronization arrangement processing is performed on the law enforcement scene audio behavior information data to obtain the law enforcement scene audio event timing synchronization sequence data;

[0036] Step S253: performing image behavior space mapping processing on the image behavior information data of the law enforcement scene based on the law enforcement scene process spatiotemporal data synchronization benchmark framework to obtain the law enforcement scene image behavior space position mapping data;

[0037] Step S254: performing multimodal event-behavior interaction correlation analysis on the audio event time synchronization sequence data of the law enforcement scene and the image behavior spatial position mapping data of the law enforcement scene to obtain the multimodal event-behavior interaction correlation relationship between the audio event and the image behavior of the law enforcement scene;

[0038] Step S255: Based on the multimodal event-behavior interactive correlation relationship between the audio events and image behaviors at the law enforcement scene, the law enforcement scene audio behavior information data and the law enforcement scene image behavior information data are subjected to law enforcement behavior related event extraction processing to obtain multimodal spatiotemporal synchronized law enforcement behavior related event data.

[0039] Further, step S3 includes the following steps:

[0040] Step S31: extracting the behavior sequences of law enforcement personnel and parties involved in the multimodal spatiotemporal synchronous law enforcement behavior related event data to obtain the behavior sequences of law enforcement personnel in the law enforcement behavior events and the behavior sequences of parties involved in the law enforcement behavior events;

[0041] Step S32: performing behavior description matching analysis between the behavior description nodes in the behavior sequence of the law enforcement personnel in the law enforcement behavior event and the behavior description nodes in the behavior sequence of the parties involved in the law enforcement behavior event, so as to obtain the law enforcement event matching relationship between the law enforcement personnel behavior description nodes and the party behavior description nodes;

[0042] Step S33: Based on the law enforcement event matching relationship between the law enforcement officer behavior description node and the party behavior description node, the law enforcement behavior sequence connection processing is performed between the behavior description nodes in the law enforcement officer behavior sequence of the law enforcement behavior event and the behavior description nodes in the party behavior sequence of the law enforcement behavior event, so as to generate a law enforcement scene event behavior sequence connection diagram;

[0043] Step S34: Perform event behavior intention recognition analysis on the law enforcement scene event behavior sequence connection diagram to obtain law enforcement scene event behavior intention information data.

[0044] Further, step S4 includes the following steps:

[0045] Step S41: Obtaining a preset law enforcement event behavior knowledge graph;

[0046] Step S42: performing a behavior normative rule retrieval and extraction process on the law enforcement event behavior knowledge graph to obtain a behavior normative rule standard for law enforcement events;

[0047] Step S43: performing behavior intention semantic feature analysis on the behavior intention information data of the law enforcement scene event to obtain the behavior intention semantic feature data of the law enforcement scene event;

[0048] Step S44: Based on the law enforcement event behavior normative rule standard, the law enforcement scene event behavior intention semantic feature data is adapted to the normative rule and the behavior normative score is calculated to obtain the law enforcement scene behavior intention normative score value;

[0049] Step S45: Based on the law enforcement scene behavior intention standardization score value, the non-standard behavior intelligent alarm optimization analysis is performed on the corresponding law enforcement scene events in the law enforcement scene event behavior intention information data to generate a law enforcement scene event non-standard behavior optimization suggestion report.

[0050] Further, step S45 includes the following steps:

[0051] Step S451: Compare and judge the normative score value of the law enforcement scene behavior intention according to the preset normative score threshold. When the normative score value of the law enforcement scene behavior intention is greater than or equal to the preset normative score threshold, the corresponding law enforcement scene event in the law enforcement scene event behavior intention information data is determined as a normative behavior event; when the normative score value of the law enforcement scene behavior intention is less than the preset normative score threshold, the corresponding law enforcement scene event in the law enforcement scene event behavior intention information data is determined as an irregular behavior event;

[0052] Step S452: Performing irregular behavior intelligent alarm processing on the irregular behavior event determined as irregular behavior according to the preset irregular law enforcement behavior alarm rule to generate the alarm information of the irregular behavior event of this supervision;

[0053] Step S453: performing law enforcement behavior problem identification and analysis on the alarm information of the current supervision irregular behavior event to obtain a report on the current supervision irregular law enforcement behavior problem;

[0054] Step S454: Process the report on the problem of irregular law enforcement behavior in this supervision with suggestions for optimizing law enforcement behavior to generate a report on optimization suggestions for irregular behavior in law enforcement on-site events.

[0055] Furthermore, the present invention also provides a multimodal intelligent analysis system for law enforcement supervision, which is used to execute the multimodal intelligent analysis method for law enforcement supervision as described above. The multimodal intelligent analysis system for law enforcement supervision includes:

[0056] The law enforcement audio and video behavior recognition and analysis module is used to obtain the law enforcement scene video file from the law enforcement supervisory recorder platform, and perform audio and video segmentation processing on the law enforcement scene video file to obtain the law enforcement scene audio segment and the law enforcement scene image video frame segment; perform law enforcement behavior information recognition and analysis on the law enforcement scene audio segment and the law enforcement scene image video frame segment respectively, so as to obtain the law enforcement scene audio behavior information data and the law enforcement scene image behavior information data;

[0057] The law enforcement behavior multimodal spatiotemporal correlation analysis module is used to extract the timestamp and spatial position of the law enforcement scene video file to obtain the law enforcement scene process timestamp and law enforcement scene process spatial position information; based on the law enforcement scene process timestamp and law enforcement scene process spatial position information, the law enforcement scene audio behavior information data and the law enforcement scene image behavior information data are subjected to multimodal spatiotemporal synchronous correlation analysis to obtain multimodal spatiotemporal synchronous law enforcement behavior correlation event data;

[0058] The law enforcement event behavior intention recognition and analysis module is used to perform law enforcement behavior sequence connection processing on the multi-modal spatiotemporal synchronous law enforcement behavior related event data to generate a law enforcement scene event behavior sequence connection diagram; perform event behavior intention recognition and analysis on the law enforcement scene event behavior sequence connection diagram to obtain law enforcement scene event behavior intention information data;

[0059] The module for intelligent warning of irregular behaviors in law enforcement events is used to obtain a preset knowledge graph of law enforcement event behaviors, and to calculate the behavioral normativeness score of the behavior intention information data of law enforcement scene events based on the knowledge graph of law enforcement event behaviors, so as to obtain the law enforcement scene behavior intention normativeness score value; based on the law enforcement scene behavior intention normativeness score value, the module performs an optimization analysis of irregular behaviors of intelligent warnings on corresponding law enforcement scene events in the behavior intention information data of law enforcement scene events, so as to generate an optimization suggestion report on irregular behaviors of law enforcement scene events.

[0060] Beneficial effects of the present invention:

[0061] 1. The multimodal intelligent analysis method for law enforcement supervision proposed by the present invention has the beneficial effect of obtaining video files of the law enforcement scene from the law enforcement supervision recorder platform, which can ensure comprehensive records and reviews of law enforcement activities. These video files provide real and comprehensive visual evidence for the law enforcement process. The acquisition of these video files enables every detail of the law enforcement process to be recorded, whether it is the behavior of law enforcement personnel or the reaction of the law enforcement object, which can be fully captured. This detailed record not only helps the subsequent review and investigation, but also enhances the credibility and legitimacy of law enforcement activities, thereby helping to identify potential problems in the law enforcement process and ensure that law enforcement operations meet norms and standards. By performing audio and video segmentation processing on the video files of the law enforcement scene, the video data can be decomposed into smaller audio clips and image video frame clips. This segmentation processing can divide the entire video content into parts with independent meanings, each of which corresponds to a specific law enforcement event, so that targeted analysis and review can be carried out more conveniently. By processing audio and image data separately, we can get a clearer picture of the specific situation in each event. For example, audio clips can be used to analyze the content of conversations and voice emotions, while video frame clips can be used to observe visual details and behavioral performance. This method improves the accuracy and efficiency of data analysis, allowing each law enforcement event to be independently reviewed and evaluated, thereby gaining a more comprehensive understanding of the law enforcement process. At the same time, by performing law enforcement behavior information recognition and analysis on the audio clips and image video frame clips of the law enforcement scene, specific law enforcement behavior information data can be extracted. The analysis of audio clips can identify the language, sound features and conversation content, helping to understand the communication and behavioral motivations in the law enforcement process, while the analysis of image video frame clips can identify visual behavioral features, such as law enforcement actions, on-site environment and personnel location. This behavior information recognition can provide detailed empirical data for law enforcement activities and help support a comprehensive evaluation of the law enforcement process. By combining the analysis of audio and image data, we can more accurately understand the behavior and law enforcement effects of law enforcement personnel and ensure the transparency and compliance of the law enforcement process. This multi-angle behavior information recognition not only improves the review capability of the law enforcement process, but also strengthens the control and optimization of law enforcement quality. Secondly, by extracting timestamps from on-site law enforcement video files, the time nodes of law enforcement events in the video can be accurately recorded and tracked. The timestamp provides an accurate time record for events in the law enforcement process, so that subsequent event analysis can be performed according to the time series and the order of events can be restored. It can also help synchronize law enforcement behaviors in the video with other data sources (such as audio and sensor data), compare and analyze the time consistency of data in different modalities, and ensure precise alignment between multimodal data, thereby providing basic data protection for the transparency and reliability of subsequent processing.By marking the spatial coordinate position of each frame of video image in the law enforcement scene video file, detailed marks and annotations can be added to the spatial coordinates in each frame of the image, providing intuitive information about the law enforcement scene. Through spatial position marking, law enforcement personnel and analysts can clearly understand the location of each object and event in the image and its specific significance at the scene. This processing can help identify important locations, such as crime scenes, evidence locations, or locations of relevant personnel, making event analysis and investigation more systematic and efficient. Spatial position marking also supports the generation of visual spatial layout diagrams, which is convenient for intuitively displaying the spatial structure and object distribution of the scene. In addition, based on the law enforcement scene process timestamp and law enforcement scene process spatial position information, multimodal spatiotemporal synchronous correlation analysis is performed on the law enforcement scene audio behavior information data and the law enforcement scene image behavior information data. It can combine timestamps and spatial position information, synchronize multiple data sources such as audio and images in time and space, and provide a comprehensive view of law enforcement events, so that the visual information in the video image can be effectively aligned with the sound information in the audio, so that analysts can comprehensively consider different data sources in time and space, so as to more accurately understand the occurrence and development of events. Through spatiotemporal synchronization, the association between different modal data can be achieved, the key behaviors and interactions in the event can be identified, and the in-depth analysis and understanding of complex law enforcement events can be supported. The comprehensive use of multimodal data can improve the comprehensiveness and accuracy of law enforcement behavior information, help analyze law enforcement behavior events from different angles, and provide a richer basis for the subsequent processing process. It can improve the coverage and timeliness of law enforcement behavior event analysis, so that it can respond in time and fully cover all law enforcement scene behavior conditions. Then, by connecting the law enforcement behavior sequence connection processing of the multimodal spatiotemporal synchronization law enforcement behavior association event data, the event behavior sequence connection diagram is connected to generate. The main advantage of this process is that the matching relationship in the behavior sequence is presented in a visual way, which helps to better understand the dynamic process of the event. The behavior sequence connection diagram can intuitively show the time sequence, interaction relationship and key nodes of the behaviors of all parties in the event, making the entire process of the event clearer. Through the connection diagram, the correlation and causal relationship of the behavior can be identified, and the key event points and important behavior chains in the event can be found. This graphical display not only helps analysts quickly grasp the overall picture of the event, but also provides intuitive data support for the subsequent processing process. We also conduct event behavior intention recognition analysis on the law enforcement scene event behavior sequence connection diagram, aiming to interpret the potential intention and purpose of each behavior in the law enforcement scene event behavior sequence connection diagram. The main advantage of this analysis is that it can deeply explore and understand the motivation and intention behind the behavior, and provide deep insights into the cause and background of the incident. Through the intention recognition of the behavior sequence, the real purpose of law enforcement officers and parties in the incident can be revealed, and whether the behavior conforms to the established norms and legal requirements can be judged.For example, it can identify whether the intention of law enforcement officers is to perform their duties legally or whether there is excessive behavior; at the same time, it can also analyze whether the parties have defensive, evasive or other intentions. This analysis not only improves the depth of understanding of the incident, but also helps to evaluate the compliance and legality of the law enforcement process, and provides data support for improving law enforcement behavior strategies and training. Finally, by obtaining the preset law enforcement event behavior knowledge graph, this process involves integrating and establishing a systematic knowledge base containing various law enforcement events and behaviors, including historical law enforcement behavior case data, behavioral norms, law enforcement procedures and other information. By constructing such a knowledge graph, it is possible to achieve in-depth analysis of the corresponding law enforcement behavior events and provide law enforcement personnel with an intuitive reference model, which not only helps to identify and understand the norms and trends of various law enforcement behaviors, but also can provide a basis for the subsequent formulation of more scientific law enforcement standards and optimization of law enforcement strategies. By calculating the behavior normativeness score of the behavior intention information data of the law enforcement scene event based on the law enforcement event behavior knowledge graph, the standardized evaluation of law enforcement behavior can be achieved. By matching the semantic features of the behavior intention with the normative rule standards in the knowledge graph, the standardization degree of the behavior can be objectively evaluated and the corresponding normative score value can be calculated. This score can help quickly identify behaviors that do not meet the norms and provide law enforcement personnel with clear improvement directions. In addition, by performing non-standard behavior intelligent alarm optimization analysis on the corresponding law enforcement scene events in the behavior intention information data of the law enforcement scene event based on the normative score value of the law enforcement scene behavior intention, it can automatically detect and identify non-standard behaviors and generate targeted optimization suggestion reports. This process can not only remind law enforcement personnel of existing problems in real time, but also provide specific measures and suggestions for improvement. The optimization analysis report will help law enforcement personnel adjust and improve law enforcement behavior to ensure that it meets the established normative standards, which will significantly improve the standardization and efficiency of the law enforcement process and reduce the occurrence of errors and non-standard law enforcement behaviors.

[0062] 2. The multimodal intelligent analysis system for law enforcement supervision proposed in the present invention is generally composed of a law enforcement audio and video behavior recognition and analysis module, a law enforcement behavior multimodal spatiotemporal correlation analysis module, a law enforcement event behavior intention recognition and analysis module, and a law enforcement event irregular behavior intelligent alarm module. It can implement any multimodal intelligent analysis method for law enforcement supervision described in the present invention, and is used to combine the operations between computer programs running on each module to implement the multimodal intelligent analysis method for law enforcement supervision. The internal structures of the system cooperate with each other, which can greatly reduce duplication of work and manpower investment, and can quickly and effectively provide a more accurate and efficient multimodal intelligent analysis process for law enforcement supervision, thereby simplifying the operation flow of the multimodal intelligent analysis system for law enforcement supervision. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments thereof made with reference to the following drawings:

[0064] Figure 1 It is a schematic diagram of the steps of the multimodal intelligent analysis method for law enforcement supervision according to the present invention;

[0065] Figure 2 for Figure 1 Detailed step flow diagram of step S1;

[0066] Figure 3 for Figure 2 Detailed step flow chart of step S15 in FIG. DETAILED DESCRIPTION

[0067] The technical method of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by technicians in this field without creative work are within the scope of protection of the present invention.

[0068] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. The functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor methods and / or microcontroller methods.

[0069] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.

[0070] To achieve this, please refer to Figures 1 to 3 The present invention provides a multimodal intelligent analysis method for law enforcement supervision, the method comprising the following steps:

[0071] Step S1: Obtain the law enforcement scene video file from the law enforcement supervisory recorder platform, and perform audio and video segmentation processing on the law enforcement scene video file to obtain the law enforcement scene audio segment and the law enforcement scene image video frame segment; perform law enforcement behavior information recognition and analysis on the law enforcement scene audio segment and the law enforcement scene image video frame segment respectively to obtain the law enforcement scene audio behavior information data and the law enforcement scene image behavior information data;

[0072] Step S2: extracting the timestamp and spatial position of the law enforcement scene video file to obtain the timestamp and spatial position information of the law enforcement scene process; performing multimodal spatiotemporal synchronous correlation analysis on the law enforcement scene audio behavior information data and the law enforcement scene image behavior information data based on the timestamp and spatial position information of the law enforcement scene process to obtain multimodal spatiotemporal synchronous law enforcement behavior correlation event data;

[0073] Step S3: Perform law enforcement behavior sequence connection processing on the multimodal spatiotemporal synchronous law enforcement behavior associated event data to generate a law enforcement scene event behavior sequence connection diagram; perform event behavior intention recognition analysis on the law enforcement scene event behavior sequence connection diagram to obtain law enforcement scene event behavior intention information data;

[0074] Step S4: Obtain a preset law enforcement event behavior knowledge graph, and perform a behavioral normativeness score calculation on the law enforcement scene event behavior intention information data based on the law enforcement event behavior knowledge graph to obtain a law enforcement scene behavior intention normativeness score value; perform non-normative behavior intelligent alarm optimization analysis on the corresponding law enforcement scene events in the law enforcement scene event behavior intention information data based on the law enforcement scene behavior intention normativeness score value to generate a non-normative behavior optimization suggestion report for law enforcement scene events.

[0075] In the embodiment of the present invention, please refer to Figure 1 As shown, it is a schematic diagram of the steps of the multimodal intelligent analysis method for law enforcement supervision of the present invention. In this example, the multimodal intelligent analysis method for law enforcement supervision includes the following steps:

[0076] Step S1: Obtain the law enforcement scene video file from the law enforcement supervisory recorder platform, and perform audio and video segmentation processing on the law enforcement scene video file to obtain the law enforcement scene audio segment and the law enforcement scene image video frame segment; perform law enforcement behavior information recognition and analysis on the law enforcement scene audio segment and the law enforcement scene image video frame segment respectively to obtain the law enforcement scene audio behavior information data and the law enforcement scene image behavior information data;

[0077] In an embodiment of the present invention, by obtaining the corresponding law enforcement video original from the law enforcement supervisory recorder platform, the platform usually has the function of storing and managing law enforcement records, after user authentication and authority verification, log in to the file management system of the platform, select the "video record" column on the system interface, browse and filter the required law enforcement video original, and use the search function provided by the platform to find a specific video record according to the date, time, case number or other keywords, so as to obtain the law enforcement scene video file. In addition, by determining the start time and end time of each law enforcement event audio and video in the previously obtained law enforcement scene video file, the corresponding law enforcement event in the law enforcement scene video file is segmented according to the start time and end time previously identified, so as to use a video editing tool (such as FFmpeg or Adobe Premiere Pro) to import the video file and set the cutting parameters according to the start time and end time to segment the image video track, thereby generating multiple image video clips, each clip corresponding to a law enforcement event, and at the same time, by cutting the audio track in the same time period, it is ensured that each video clip has a corresponding audio clip, thereby obtaining the law enforcement scene audio clip and the law enforcement scene image video frame clip. Then, by using speech recognition technology (such as Speechbrain technology), the previously segmented audio clips of the law enforcement scene are analyzed for behavioral information recognition, so as to convert the audio content into text, analyze the keywords and context in the language, and extract the corresponding law enforcement behavior information from it, so as to obtain the law enforcement scene audio behavior information data. At the same time, by using image recognition algorithms (such as object detection algorithm Yolov8 and tracking technology Bytetrack) to analyze each frame of the law enforcement scene image video frame fragment, the actions, objects and scene changes of law enforcement personnel and parties are identified, and the analysis results are organized into structured data, including the behavior description of each event, the participants, key actions and other information, which provide detailed law enforcement behavior records, and finally obtain the law enforcement scene image behavior information data.

[0078] Step S2: extracting the timestamp and spatial position of the law enforcement scene video file to obtain the timestamp and spatial position information of the law enforcement scene process; performing multimodal spatiotemporal synchronous correlation analysis on the law enforcement scene audio behavior information data and the law enforcement scene image behavior information data based on the timestamp and spatial position information of the law enforcement scene process to obtain multimodal spatiotemporal synchronous law enforcement behavior correlation event data;

[0079] In an embodiment of the present invention, a video parsing tool such as OpenCV is used to read the law enforcement scene video file frame by frame, and the time information of each frame is obtained by calling the get method of the video stream to accurately record the timestamp of each frame, thereby obtaining the law enforcement scene process timestamp. At the same time, by using the camera calibration technology, each frame of the video image in the previously extracted law enforcement scene video file is converted into a three-dimensional space coordinate system, and the camera's internal and external parameters are calibrated to ensure that the relationship between the image coordinates and the actual world coordinates can be accurately mapped, and a corresponding three-dimensional coordinate system is generated for each frame of the image, and the spatial position corresponding to the identified object in the video is extracted from the previously converted three-dimensional space coordinate system, thereby obtaining the law enforcement scene process spatial position information. Then, by using the data fusion tool, the previously extracted law enforcement scene process timestamps and law enforcement scene process spatial position information are processed for spatiotemporal synchronization, so as to align the timestamps and spatial position information, so that the time and spatial position of each law enforcement event can correspond synchronously, thereby integrating and constructing a spatiotemporal data synchronization benchmark framework, and by using the previously constructed spatiotemporal data synchronization benchmark framework, the law enforcement scene audio behavior information data is processed for temporal synchronization, so as to match the timestamps of the audio events with the timestamps in the synchronization framework, so as to ensure that the temporal sequence of the audio events is consistent with the actual situation at the scene, and by combining the previously constructed spatiotemporal data synchronization benchmark framework, the image behavior features in the law enforcement scene image behavior information data are matched with the spatial position information, so as to ensure that the spatial information in the image data is consistent with the position in the actual scene, and by comprehensively using data analysis and machine learning tools, the previously determined audio event time series synchronization data and image Interactive association analysis is performed on the behavior spatial position matching data to fuse the audio event with the image behavior data, thereby identifying and extracting the interactive association relationship between the audio event and the image behavior. Subsequently, the corresponding law enforcement scene audio behavior information data and the law enforcement scene image behavior information data are processed by combining the interactive association relationship between the audio event and the image behavior obtained by the previous analysis. The law enforcement behavior related events are extracted and processed to establish a behavior association model, combine the interactive information in the audio event and image behavior data, and process them using a machine learning algorithm (such as a classifier or a clustering algorithm) to identify the associated events between the audio event and the image behavior, such as the law enforcement related events between a specific audio event (such as an alarm sound) and the corresponding image behavior (such as personnel movement). Finally, multimodal spatiotemporal synchronous law enforcement behavior related event data is obtained, which includes the law enforcement behavior of law enforcement personnel and the parties involved in the current law enforcement event.

[0080] Step S3: Perform law enforcement behavior sequence connection processing on the multimodal spatiotemporal synchronous law enforcement behavior associated event data to generate a law enforcement scene event behavior sequence connection diagram; perform event behavior intention recognition analysis on the law enforcement scene event behavior sequence connection diagram to obtain law enforcement scene event behavior intention information data;

[0081] In an embodiment of the present invention, all the behavior records of law enforcement officers and parties involved are extracted from the multimodal spatiotemporal synchronous law enforcement behavior associated event data obtained by previous analysis. These records come from surveillance cameras, sensors, voice records and other data acquisition devices, including the approach, questioning, inspection, punishment and other behaviors of law enforcement officers and the parking, explanation, cooperation, resistance and other behaviors of the parties. The behavior records are divided into law enforcement officer behavior sequences and party behavior sequences by using data tags or event identifiers, and the behavior description nodes in the law enforcement officer behavior sequence obtained by previous analysis and the behavior description nodes in the party behavior sequence are compared. The similarity score of each pair of behavior description nodes is calculated by a matching algorithm to determine their matching relationship. For example, by setting a Threshold, when the similarity score exceeds the threshold, the two nodes are considered to be matched successfully, and the results of the matching analysis are recorded in a matching relationship table to determine the matching relationship between each pair of matched law enforcement officer and party behavior description nodes. At the same time, by combining the matching relationship between the law enforcement officer and party behavior description nodes obtained by the previous matching analysis, the corresponding behavior description nodes in the law enforcement officer behavior sequence and the corresponding behavior description nodes in the party behavior sequence are matched and connected, so as to connect the matched behavior nodes in chronological order by using graph construction technology in graph theory (such as adjacency matrix or adjacency list), each behavior node is used as a node in the graph, and the connected behavior descriptions are used as edges, so as to construct a connection graph of the law enforcement scene event behavior sequence. Then, by constructing a behavior intention model, the model is based on known behavior patterns and intention templates and can be trained using machine learning techniques (such as classification algorithms or sequence labeling models) to identify different behavior intentions. For example, by training a support vector machine (SVM) classifier to predict the intention type of each behavior sequence, and according to the previously constructed behavior intention model, each behavior node and behavior sequence in the law enforcement scene event behavior sequence connection graph is subjected to intention recognition, so as to apply the behavior intention model to the nodes in the graph, and determine the corresponding behavior intention by calculating the intention probability of each behavior sequence. For example, the behavior sequence is analyzed to see whether it meets specific law enforcement intentions (such as warning, dissuasion, punishment, arrest, escape, etc.), and finally the law enforcement scene event behavior intention information data is obtained.

[0082] Step S4: Obtain a preset law enforcement event behavior knowledge graph, and perform a behavioral normativeness score calculation on the law enforcement scene event behavior intention information data based on the law enforcement event behavior knowledge graph to obtain a law enforcement scene behavior intention normativeness score value; perform non-normative behavior intelligent alarm optimization analysis on the corresponding law enforcement scene events in the law enforcement scene event behavior intention information data based on the law enforcement scene behavior intention normativeness score value to generate a non-normative behavior optimization suggestion report for law enforcement scene events.

[0083] In an embodiment of the present invention, a knowledge graph of corresponding law enforcement event behaviors is extracted from a preset knowledge base, and the graph contains behavioral patterns of various law enforcement events and their related normative provisions, so as to convert law enforcement behavior documents, normative clauses and case data from different sources into structured knowledge using natural language processing technology. The data source can be a legal database, law enforcement records and expert domain knowledge. The behavior knowledge graph is integrated and constructed through the visual interface of the graph construction platform, so as to obtain the law enforcement event behavior knowledge graph. Secondly, the previously obtained law enforcement event behavior knowledge graph is processed by applying a rule extraction method. First, the standard for retrieving behavioral normative rules is defined, and the search is performed using a rule engine. The rule engine extracts rules related to behavioral norms by parsing the attributes of nodes and edges in the graph. As a result, a rule list containing detailed behavioral normative standards is generated. The list may include the code of conduct, processing procedures and corresponding normative basis of law enforcement personnel, and the previously analyzed law enforcement scene event behavior intention information data is statistically analyzed by using natural language processing and machine learning models to analyze the semantic features of behavioral intentions, so as to achieve word segmentation of the description text of the scene event. , part-of-speech tagging and syntactic analysis, and extract the characteristic data of behavioral intentions through semantic analysis tools, such as the semantic features of the purpose, motivation and context of law enforcement behavioral intentions, and adapt the corresponding semantic characteristic data of law enforcement on-site event behavioral intentions to normative rules by combining the behavioral normative rule standards extracted by previous retrieval, so as to use matching algorithms such as cosine similarity or Euclidean distance to calculate the similarity between the behavioral intention feature vector and the normative rules, and generate a normative score based on the calculated similarity score by using a normative rule adapter. This score reflects the degree of compliance between the on-site behavioral intentions and the normative rules, thereby obtaining the law enforcement on-site behavioral intention normative score value. Then, by using the previously quantified and calculated normative score of the law enforcement scene behavior intention, the corresponding law enforcement scene events in the law enforcement scene event behavior intention information data are analyzed, so as to set a threshold and mark the behaviors with scores below the threshold as non-normative behaviors, and apply intelligent alarm methods to further analyze these non-normative behaviors, use anomaly detection algorithms to identify potential problem areas, and optimize the analysis, including generating a detailed optimization suggestion report. The report content includes the identified non-normative behavior types, existing normative risks, and improvement suggestions. The report uses data visualization tools to display the analysis results and suggestions to help law enforcement personnel adjust their law enforcement behaviors and ensure the standardization of the law enforcement process, and finally generate an optimization suggestion report on non-normative behaviors in law enforcement scene events.

[0084] Further, step S1 includes the following steps:

[0085] Step S11: Obtaining the law enforcement scene video file from the law enforcement supervisory recorder platform;

[0086] Step S12: Performing audio and video synchronization processing on the law enforcement scene video file to obtain an audio and video synchronized video of the law enforcement scene;

[0087] Step S13: extracting video content from the law enforcement scene audio and video synchronization video to obtain video content information data of the law enforcement scene; based on the video content information data of the law enforcement scene, identifying and analyzing the start and end time of different law enforcement events in the law enforcement scene audio and video synchronization video to obtain the start time and end time of different law enforcement events in the law enforcement scene video;

[0088] Step S14: performing audio and video segmentation processing on the law enforcement scene audio and video synchronization video according to the start time and end time of different law enforcement events in the law enforcement scene video to obtain the law enforcement scene audio segment and the law enforcement scene image video frame segment;

[0089] Step S15: Perform law enforcement behavior information recognition and analysis on the law enforcement scene audio clips and the law enforcement scene image video frame clips respectively to obtain law enforcement scene audio behavior information data and law enforcement scene image behavior information data.

[0090] As an embodiment of the present invention, refer to Figure 2 As shown, Figure 1 Detailed step flow diagram of step S1 in FIG. 1 , in this embodiment, step S1 includes the following steps:

[0091] Step S11: Obtaining the law enforcement scene video file from the law enforcement supervisory recorder platform;

[0092] In an embodiment of the present invention, the corresponding law enforcement video original is obtained from the law enforcement inspector recorder platform, which usually has the function of storing and managing law enforcement records. After user authentication and authority verification, log in to the file management system of the platform, and select the "Video Record" column on the system interface to browse and filter the required law enforcement video originals. Using the retrieval function provided by the platform, you can find specific video records based on date, time, case number or other keywords. After selecting the target video, click the "Download" button to save the video file to the local storage device. The video file usually uses a standard video format such as MP4 or AVI, and contains complete recording data of the law enforcement scene. It is necessary to ensure that the downloaded file is complete and not damaged, and finally obtain the law enforcement scene video file.

[0093] Step S12: Performing audio and video synchronization processing on the law enforcement scene video file to obtain an audio and video synchronized video of the law enforcement scene;

[0094] In an embodiment of the present invention, the audio and video in the previously acquired law enforcement scene video file are synchronized to import the video file downloaded from the law enforcement inspection recorder platform into the video processing software, and a tool that supports audio and video synchronization is selected, such as Adobe Premiere Pro or FFmpeg. In the software, the video and audio tracks are separated to check whether there is a time deviation. If there is a deviation, synchronization is achieved by adjusting the start time of the audio track. The specific operation includes using the software's timeline tool to align the audio track with the video track to ensure that the sound and picture are consistent during playback. After the synchronization is completed, the adjusted audio and video are merged into one file and saved in a new video file format to finally obtain a law enforcement scene audio and video synchronized video.

[0095] Step S13: extracting video content from the law enforcement scene audio and video synchronization video to obtain video content information data of the law enforcement scene; based on the video content information data of the law enforcement scene, identifying and analyzing the start and end time of different law enforcement events in the law enforcement scene audio and video synchronization video to obtain the start time and end time of different law enforcement events in the law enforcement scene video;

[0096] In an embodiment of the present invention, the law enforcement scene audio and video synchronized video obtained after audio and video synchronization is previously subjected to content extraction processing, so as to parse the video frame by frame by using video analysis software, such as OpenCV or a custom video analysis tool, to extract specific content information in the law enforcement video, thereby obtaining law enforcement scene video content information data. At the same time, by combining the law enforcement scene video content information data obtained by the previous analysis, the law enforcement scene audio and video synchronized video is subjected to identification and classification processing of different law enforcement events, so as to obtain the law enforcement events contained in the law enforcement scene video, and according to the visual features of different law enforcement events and the voice content in the audio, the timestamp of each law enforcement event is marked and recorded, from which the start time and end time of each law enforcement event are determined, and finally the start time and end time of different law enforcement events in the law enforcement scene video are obtained.

[0097] Step S14: performing audio and video segmentation processing on the law enforcement scene audio and video synchronization video according to the start time and end time of different law enforcement events in the law enforcement scene video to obtain the law enforcement scene audio segment and the law enforcement scene image video frame segment;

[0098] In an embodiment of the present invention, the audio and video segmentation processing is performed on the corresponding law enforcement events in the law enforcement scene audio and video synchronized video according to the start time and end time of different law enforcement events in the law enforcement scene video previously identified, so as to use a video editing tool (such as FFmpeg or Adobe Premiere Pro) to import the video file and set the cutting parameters according to the start time and the end time to split the video track, thereby generating multiple video clips, each clip corresponding to a law enforcement event, and at the same time, by cutting the audio track in the same time period, it is ensured that each video clip has a corresponding audio clip, thereby generating audio clips and video frame clips, each clip is independently saved as a different file, and finally the law enforcement scene audio clips and the law enforcement scene image video frame clips are obtained.

[0099] Step S15: Perform law enforcement behavior information recognition and analysis on the law enforcement scene audio clips and the law enforcement scene image video frame clips respectively to obtain law enforcement scene audio behavior information data and law enforcement scene image behavior information data.

[0100] In an embodiment of the present invention, speech recognition technology (such as Speechbrain technology) is used to identify and analyze the behavior information of the law enforcement scene audio clips obtained after previous segmentation, so as to convert the audio content into text, analyze the keywords and context in the language, and extract the corresponding law enforcement behavior information from it, so as to obtain the law enforcement scene audio behavior information data. At the same time, by using image recognition algorithms (such as object detection algorithm Yolov8 and tracking technology Bytetrack) to analyze each frame of the law enforcement scene image video frame fragment, the actions, objects and scene changes of law enforcement personnel and parties are identified, and the analysis results are organized into structured data, including the behavior description of each event, the participants, key actions and other information, which provide detailed law enforcement behavior records, and finally obtain the law enforcement scene image behavior information data.

[0101] Further, step S15 includes the following steps:

[0102] Step S151: using Speechbrain technology to perform law enforcement audio voiceprint feature analysis on the law enforcement scene audio clip to obtain law enforcement scene audio voiceprint feature data;

[0103] Step S152: performing audio speech signal channel deconstruction processing on different speakers in the audio clip of the law enforcement scene based on the audio voiceprint feature data of the law enforcement scene, and obtaining audio speech signal sub-channels of different speakers in the audio clip of the law enforcement scene;

[0104] Step S153: performing law enforcement behavior event triggering recognition analysis on the audio speech signal sub-channels of different speakers in the law enforcement scene audio to obtain the law enforcement behavior event triggering points of the law enforcement scene audio;

[0105] Step S154: performing law enforcement audio behavior information recognition and analysis on the law enforcement scene audio clip based on the law enforcement scene audio law enforcement behavior event trigger point to obtain law enforcement scene audio behavior information data;

[0106] Step S155: Use the object detection algorithm Yolov8 and the tracking technology Bytetrack to perform law enforcement image behavior information recognition and analysis on the law enforcement scene image video frame segments to obtain law enforcement scene image behavior information data.

[0107] As an embodiment of the present invention, refer to Figure 3 As shown, Figure 2 Detailed step flow diagram of step S15 in FIG. 1 , in this embodiment, step S15 includes the following steps:

[0108] Step S151: using Speechbrain technology to perform law enforcement audio voiceprint feature analysis on the law enforcement scene audio clip to obtain law enforcement scene audio voiceprint feature data;

[0109] In an embodiment of the present invention, the Speechbrain technology is used to perform voiceprint feature analysis on the law enforcement scene audio clips previously obtained after slicing, so that the law enforcement scene audio clips are input into the Speechbrain framework. The framework provides rich audio processing functions, and by using its pre-trained voiceprint recognition model, the voiceprint features in the audio clips can be extracted. These voiceprint features are based on the spectral features in the audio signal and the specific patterns of the speech samples, and are obtained by feature vectorization processing. In a specific implementation, the audio signal is first preprocessed, including denoising, normalization and other steps to ensure the quality of the input data, and the voiceprint recognition model is used to extract the feature vectors of each speaker. These vectors represent the voiceprint features of the speaker, and finally the law enforcement scene audio voiceprint feature data is obtained.

[0110] Step S152: performing audio speech signal channel deconstruction processing on different speakers in the audio clip of the law enforcement scene based on the audio voiceprint feature data of the law enforcement scene, and obtaining audio speech signal sub-channels of different speakers in the audio clip of the law enforcement scene;

[0111] In an embodiment of the present invention, the signal channels of different speakers in the audio clips of the law enforcement scene are deconstructed by combining the voiceprint feature data of the law enforcement scene audio obtained by previous analysis, so that the voiceprint feature data can be applied to the separation algorithm of the audio signal, such as blind source separation (BSS) technology based on voiceprint features, and by performing blind source separation on the audio signal, the mixed audio signal can be deconstructed into multiple sub-channels, each sub-channel corresponding to the audio signal of an independent speaker. Specifically, the algorithm will extract the speech components of each speaker in the audio signal according to the voiceprint features, and assign these components to different audio speech signal sub-channels, and finally obtain the audio speech signal sub-channels of different speakers in the law enforcement scene audio.

[0112] Step S153: performing law enforcement behavior event triggering recognition analysis on the audio speech signal sub-channels of different speakers in the law enforcement scene audio to obtain the law enforcement behavior event triggering points of the law enforcement scene audio;

[0113] In an embodiment of the present invention, the law enforcement behavior event triggering situation is identified and analyzed by the audio speech signal sub-channels of different speakers in the law enforcement scene audio obtained after previous deconstruction, so as to process the audio signal in each sub-channel using an audio event detection algorithm. These algorithms can identify key events in the audio signal based on a specific acoustic model, such as commands, calls, conflict sounds, etc. In specific implementation, an event recognition model is first trained to enable it to recognize specific behavior event sound features, and each audio speech signal sub-channel is input into the trained model. The model will output the trigger time point of each event and its related information. These trigger points are recorded and used to analyze specific behavior events at the law enforcement scene, and finally the audio law enforcement behavior event trigger points at the law enforcement scene are obtained.

[0114] Step S154: performing law enforcement audio behavior information recognition and analysis on the law enforcement scene audio clip based on the law enforcement scene audio law enforcement behavior event trigger point to obtain law enforcement scene audio behavior information data;

[0115] In an embodiment of the present invention, the behavior information of the corresponding audio clips at the law enforcement scene is identified and analyzed by combining the previously identified law enforcement behavior event trigger points of the audio law enforcement behavior at the law enforcement scene, so that the audio clips are segmented according to the trigger points, and each audio segment is analyzed around an event, and each audio segment is processed by using a behavior information recognition model. The model can recognize and classify audio features related to specific law enforcement behaviors, such as sound intensity, pitch changes, etc. In a specific implementation, features are first extracted for each event clip, including frequency features, duration features, etc. of the sound, and these features are input into a classifier. The classifier matches these features with predefined law enforcement behavior categories, outputs specific behavior information data for each event, and finally obtains the law enforcement scene audio behavior information data.

[0116] Step S155: Use the object detection algorithm Yolov8 and the tracking technology Bytetrack to perform law enforcement image behavior information recognition and analysis on the law enforcement scene image video frame segments to obtain law enforcement scene image behavior information data.

[0117] In an embodiment of the present invention, the object detection algorithm Yolov8 and the tracking technology Bytetrack are used to perform recognition and analysis of law enforcement behaviors in the law enforcement scene image video frame segments obtained after previous slicing, so as to detect objects in the image video frames by using the Yolov8 algorithm. Yolov8, as an advanced target detection algorithm, can efficiently identify various objects in the image and generate corresponding bounding boxes. In a specific implementation, each frame of the image is input into the Yolov8 model, and the model will output the detected objects and their position information, and track these objects by applying the Bytetrack tracking technology to ensure continuous tracking of the movement of each target in multiple frames. Bytetrack combines target detection and tracking technology to generate motion behavior information of each object target by associating the position and motion trajectory of the object in consecutive frames, including the object's behavior category, timestamp, position and other related contextual information, and finally obtains the law enforcement scene image behavior information data.

[0118] Further, step S155 includes the following steps:

[0119] The object detection algorithm Yolov8 is used to perform object target recognition analysis on the law enforcement scene image video frame fragments to obtain the object detection target of the law enforcement scene image frame;

[0120] In an embodiment of the present invention, the object detection algorithm YOLOv8 (You Only Look Once version 8) is used to identify and analyze the object target of each frame image in the law enforcement scene image video frame segment, wherein YOLOv8 is a deep learning model that can recognize and classify various objects in the image after training. In this process, the algorithm will use a convolutional neural network to extract features from the image and generate a bounding box to mark each detected object in the image. Each bounding box will contain the location coordinates of the object and the category label of the object, and finally the object detection target of the law enforcement scene image frame is obtained.

[0121] Preferably, the object detection target of the law enforcement scene image frame is subjected to object boundary region marking processing to obtain the law enforcement scene object boundary region marking target;

[0122] In an embodiment of the present invention, the object detection target of the law enforcement scene image frame previously detected is further processed, mainly to mark the boundary area of ​​the object. This process involves converting the bounding box coordinates output by YOLOv8 into the area mark in the actual image. In the specific implementation process, the bounding box coordinates are first mapped to the pixel coordinate system of the image, and then the boundary area of ​​each object is clearly marked in the image by drawing the bounding box and adding labels. The result of the boundary area marking will be presented in the form of image overlay, so that each object has a clear visual boundary in the image, and finally the law enforcement scene object boundary area marking target is obtained.

[0123] Preferably, the tracking technology Bytetrack is used to track, identify and analyze the motion trajectory of the marked target in the boundary area of ​​the law enforcement scene object in the law enforcement scene image video frame segment to generate the motion trajectory of the marked target of the law enforcement scene object;

[0124] In an embodiment of the present invention, the tracking technology Bytetrack is used to track and identify the motion trajectory of the previously determined law enforcement scene object boundary area marking target within the law enforcement scene image video frame segment, wherein Bytetrack is an efficient multi-target tracking algorithm that can handle target tracking problems in complex scenes, and uses the initial bounding box of each object as the tracking target. By continuously analyzing the video frames, the Bytetrack algorithm calculates the motion trajectory of the object between each frame, and updates the position of each target by combining the appearance features and motion patterns of the object, ensuring accurate tracking of the object's moving path, and recording it in the form of continuous coordinate points and timestamps, and finally generating the motion trajectory of the law enforcement scene object marking target.

[0125] Preferably, the motion trajectory of the object marked target at the law enforcement scene is analyzed for the object target behavior pattern to obtain the law enforcement behavior patterns of the object targets at different law enforcement scenes; based on the law enforcement behavior patterns of the object targets at different law enforcement scenes, the corresponding law enforcement scene object boundary area marked targets are classified for law enforcement behavior actions to obtain the law enforcement behavior actions of the object targets at different law enforcement scenes;

[0126] In an embodiment of the present invention, the object behavior pattern is analyzed by using a machine learning model or a statistical method to analyze the motion trajectory of the previously tracked object mark target at the law enforcement scene, so as to extract the motion pattern of the object at the law enforcement scene by statistically analyzing the motion trajectory data of the object, which includes analyzing the moving speed, direction change and continuity of the trajectory of the object, such as the law of the moving path, the dwell time and its interaction pattern with other objects, so as to obtain the law enforcement behavior pattern of the object target at different law enforcement scenes. At the same time, the law enforcement action behavior of the corresponding law enforcement scene object boundary area mark target is classified by combining the law enforcement behavior pattern of the object target at the law enforcement scene obtained by the previous analysis, so as to establish a classification model based on the behavior pattern analysis results to classify the behavior action of the object in the video frame, and the classification process involves comparing the motion trajectory and behavior pattern of each target, and assigning it to a predefined behavior category, such as "patrol", "stationary" or "contact", etc. In specific implementation, a feature extraction method is used to extract key behavior features from the trajectory data, and a classification algorithm (such as a support vector machine or a neural network) is used for processing, so as to output the behavior action category of each object target, and finally obtain the law enforcement behavior action of the object target at different law enforcement scenes.

[0127] Preferably, based on the law enforcement behavior patterns and law enforcement behavior actions of different law enforcement scene object targets, law enforcement scene image behavior information recognition and analysis is performed on the law enforcement scene image video frame segments to obtain law enforcement scene image behavior information data.

[0128] In an embodiment of the present invention, a comprehensive law enforcement image behavior information recognition and analysis is performed on the image video frame segments of the entire law enforcement scene by combining the law enforcement behavior patterns and law enforcement behavior actions of different law enforcement scene object targets obtained through previous analysis, so as to integrate the object's behavior classification information into the video frame data and use information fusion technology to combine the object's action information with the image data to generate detailed behavior information data. In a specific implementation, the behavior classification result of each object is aligned with the video frame, and the behavior patterns of objects in the entire scene and their interactions are analyzed. The generated behavior information data will include the object's behavior category, timestamp, location and other relevant contextual information, and ultimately the law enforcement scene image behavior information data is obtained.

[0129] Further, step S2 includes the following steps:

[0130] Step S21: extracting the timestamp of the law enforcement scene video file to obtain the law enforcement scene process timestamp;

[0131] In an embodiment of the present invention, by using a video parsing tool, such as OpenCV, the law enforcement scene video file is read frame by frame, and by calling the get method of the video stream, the time information of each frame is obtained to accurately record the timestamp of each frame. In the actual implementation process, the law enforcement scene video file needs to be converted into a frame sequence, and the system clock is used to record the playback time of each frame, and finally the law enforcement scene process timestamp is obtained.

[0132] Step S22: performing a three-dimensional space coordinate system conversion on each frame of the video image in the law enforcement scene video file to obtain a three-dimensional space coordinate system for each frame of the video image in the law enforcement scene video;

[0133] In an embodiment of the present invention, a camera calibration technology is used to convert each frame of video image in the previously extracted law enforcement scene video file into a three-dimensional space coordinate system. The camera's internal and external parameters need to be calibrated to ensure that the relationship between the image coordinates and the actual world coordinates can be accurately mapped. Through the known camera parameters, a perspective transformation is applied to convert the pixel coordinates of the image into coordinates in a three-dimensional space coordinate system. This can be accomplished by using functions in a computer vision library, such as cv2.projectPoints, and generating a corresponding three-dimensional coordinate system for each frame of the image, ultimately obtaining a three-dimensional space coordinate system for each frame of the law enforcement scene video.

[0134] Step S23: performing law enforcement scene spatial coordinate extraction processing on the three-dimensional spatial coordinate system of each frame image in the law enforcement scene video to obtain the law enforcement scene spatial coordinates of each frame image in the law enforcement scene video;

[0135] In an embodiment of the present invention, the spatial position corresponding to the identified object in the video is extracted from the three-dimensional spatial coordinate system of each frame image in the previously converted law enforcement scene video, thereby extracting the coordinates of key points related to the law enforcement behavior, ensuring that the extracted coordinates accurately reflect the actual situation at the law enforcement scene, and finally obtaining the law enforcement scene spatial coordinates of each frame image in the law enforcement scene video.

[0136] Step S24: performing spatial position marking processing on the law enforcement scene spatial coordinates of each frame of the law enforcement scene video to obtain the law enforcement scene process spatial position information;

[0137] In an embodiment of the present invention, the spatial coordinates of the law enforcement scene in each frame of the law enforcement scene video previously extracted are marked with their spatial positions, and different marking rules are defined to mark the spatial coordinates in each frame. For example, a graphic processing tool is used to convert the spatial coordinates into a visual point cloud model, and different law enforcement behaviors are color-coded or shape-marked, and finally the spatial position information of the law enforcement scene process is obtained by marking.

[0138] Step S25: Based on the law enforcement scene process timestamp and the law enforcement scene process spatial position information, a multimodal spatiotemporal synchronous correlation analysis is performed on the law enforcement scene audio behavior information data and the law enforcement scene image behavior information data to obtain multimodal spatiotemporal synchronous law enforcement behavior correlation event data.

[0139] In an embodiment of the present invention, a data fusion tool is used to perform spatiotemporal synchronization processing on the previously extracted timestamps of the law enforcement scene process and the spatial position information of the law enforcement scene process, so as to align the timestamps and the spatial position information, so that the time and spatial position of each law enforcement event can be synchronously corresponding, and the timestamps and the spatial position information are integrated to construct a spatiotemporal data synchronization reference framework. At the same time, the audio behavior information data of the law enforcement scene is synchronously arranged in time sequence by using the previously constructed spatiotemporal data synchronization reference framework, so as to match the timestamps of the audio events with the timestamps in the synchronization framework, so as to ensure that the temporal sequence of the audio events is consistent with the actual situation at the scene, and the image behavior features in the image behavior information data of the law enforcement scene are matched with the spatial position information by combining the previously constructed spatiotemporal data synchronization reference framework, so as to ensure that the spatial information in the image data is consistent with the position in the actual scene, and the audio event time sequence synchronization data and the image behavior spatial position matching data determined previously are interactively analyzed by using data analysis and machine learning tools in combination, so as to fuse the audio events with the image behavior data, establish a multimodal data set containing time and space information, and apply a multimodal fusion algorithm, such as a fusion neural network. Networks), analyze the relationship between audio events and image behaviors, identify and extract the interactive correlation relationship between audio events and image behaviors, and then, by combining the interactive correlation relationship between audio events and image behaviors obtained by previous analysis, perform correlation connection extraction processing on the corresponding law enforcement scene audio behavior information data and law enforcement scene image behavior information data of law enforcement behavior related events, so as to establish a behavior association model, combine the interactive information in the audio event and image behavior data, and process them by using machine learning algorithms (such as classifiers or clustering algorithms) to identify the associated events between audio events and image behaviors, such as the law enforcement associated events between specific audio events (such as alarm sounds) and corresponding image behaviors (such as personnel movement), and finally obtain multimodal spatiotemporal synchronous law enforcement behavior associated event data, which includes the law enforcement behavior of law enforcement personnel and parties under the current law enforcement event.

[0140] Further, step S25 includes the following steps:

[0141] Step S251: constructing a spatiotemporal synchronization benchmark for the law enforcement scene process timestamp and the law enforcement scene process spatial location information to obtain a spatiotemporal data synchronization benchmark framework for the law enforcement scene process;

[0142] In an embodiment of the present invention, a data fusion tool is used to perform spatiotemporal synchronization processing on the previously extracted law enforcement scene process timestamp and the law enforcement scene process spatial position information to align the timestamp and spatial position information so that the time and spatial position of each law enforcement event can correspond synchronously. The specific operation includes using a data alignment algorithm, such as dynamic time warping (DTW) or an interpolation algorithm, to match the timestamp data with the spatial position data, and to construct a spatiotemporal data synchronization reference framework, integrating the timestamp and spatial position information into a spatiotemporal data synchronization model to ensure data consistency in subsequent analysis, and finally obtaining a spatiotemporal data synchronization reference framework for the law enforcement scene process.

[0143] Step S252: Based on the law enforcement scene process spatiotemporal data synchronization benchmark framework, audio event timing synchronization arrangement processing is performed on the law enforcement scene audio behavior information data to obtain the law enforcement scene audio event timing synchronization sequence data;

[0144] In an embodiment of the present invention, the audio behavior information data at the law enforcement scene is synchronously arranged in time sequence by utilizing the previously constructed spatiotemporal data synchronization reference framework of the law enforcement scene process, so as to extract the audio event behavior data from the recording device, and convert it into time series data through audio processing software (such as Audacity), and based on the spatiotemporal data synchronization reference framework of the law enforcement scene process, the timestamps of the audio events are matched with the timestamps in the synchronization framework to ensure that the temporal order of the audio events is consistent with the actual situation at the scene, and the audio event detection algorithm is applied to identify and mark key events in the audio, such as shouting, warning sounds or intervention sounds, and at the same time arrange these audio events in chronological order, so as to finally obtain the time series synchronization sequence data of the audio events at the law enforcement scene.

[0145] Step S253: performing image behavior space mapping processing on the image behavior information data of the law enforcement scene based on the law enforcement scene process spatiotemporal data synchronization benchmark framework to obtain the law enforcement scene image behavior space position mapping data;

[0146] In an embodiment of the present invention, by combining with the previously constructed law enforcement scene process spatiotemporal data synchronization reference framework, the image behavior features in the law enforcement scene image behavior information data are matched with the spatial position information to ensure that the spatial information in the image data is consistent with the position in the actual scene. The specific operations include the conversion of the image coordinate system and the spatial position coordinate system, and the spatial mapping of the image features. The spatial mapping algorithm (such as the feature point matching algorithm) is used to align the behavior position in the image with the spatial position data in the law enforcement scene process spatiotemporal data synchronization reference framework, and finally the law enforcement scene image behavior spatial position mapping data is obtained.

[0147] Step S254: performing multimodal event-behavior interaction correlation analysis on the audio event time synchronization sequence data of the law enforcement scene and the image behavior spatial position mapping data of the law enforcement scene to obtain the multimodal event-behavior interaction correlation relationship between the audio event and the image behavior of the law enforcement scene;

[0148] In an embodiment of the present invention, data analysis and machine learning tools are used in combination to perform interactive correlation analysis on previously determined law enforcement scene audio event time synchronization sequence data and law enforcement scene image behavior spatial position mapping data, so as to fuse the audio event and image behavior data, establish a multimodal data set containing time and space information, and apply a multimodal fusion algorithm, such as a fusion neural network (Multi-modal Fusion Neural Networks), to analyze the relationship between audio events and image behaviors, identify and extract the correlation between audio events and image behaviors, such as whether the sound and image behaviors occur in the same time period, generate a data report of the interactive correlation relationship of multimodal event behaviors, to display the interactive correlation relationship of audio events and image behavior data, and finally obtain the multimodal event behavior interactive correlation relationship between audio events and image behaviors at the law enforcement scene.

[0149] Step S255: Based on the multimodal event-behavior interactive correlation relationship between the audio events and image behaviors at the law enforcement scene, the law enforcement scene audio behavior information data and the law enforcement scene image behavior information data are subjected to law enforcement behavior related event extraction processing to obtain multimodal spatiotemporal synchronized law enforcement behavior related event data.

[0150] In an embodiment of the present invention, the corresponding law enforcement scene audio behavior information data and the law enforcement scene image behavior information data are processed by extracting the associated connection of law enforcement behavior associated events by combining the multimodal event behavior interactive association relationship between the law enforcement scene audio events and the image behavior obtained by previous analysis, so as to establish a behavior association model, combine the interactive information in the audio event and image behavior data, and process them by using a machine learning algorithm (such as a classifier or a clustering algorithm) to identify the associated event status between the audio event and the image behavior, such as the law enforcement associated event between a specific audio event (such as an alarm sound) and the corresponding image behavior (such as personnel movement), and finally obtain multimodal spatiotemporal synchronous law enforcement behavior associated event data, which includes the law enforcement behavior of law enforcement personnel and the parties involved in the current law enforcement event.

[0151] Further, step S3 includes the following steps:

[0152] Step S31: extracting the behavior sequences of law enforcement personnel and parties involved in the multimodal spatiotemporal synchronous law enforcement behavior related event data to obtain the behavior sequences of law enforcement personnel in the law enforcement behavior events and the behavior sequences of parties involved in the law enforcement behavior events;

[0153] In an embodiment of the present invention, all the behavior records of law enforcement officers and parties involved are extracted from the multimodal spatiotemporal synchronous law enforcement behavior associated event data obtained by previous analysis. These records come from surveillance cameras, sensors, voice records and other data acquisition devices, including the approach, questioning, inspection, punishment and other behaviors of law enforcement officers and the parking, explanation, cooperation, resistance and other behaviors of the parties. The behavior records are divided into law enforcement officer behaviors and party behaviors by using data tags or event identifiers. This can be achieved through regular expression matching or rule engines, and records with specific labels are classified into corresponding sequences. For example, law enforcement officer behaviors can be marked as "OfficerAction" and party behaviors can be marked as "SuspectAction". Each behavior record is decomposed into specific behavior description nodes, each node should contain the timestamp of the behavior, the action type and related context information. At the same time, the behavior description nodes are sorted according to the timestamp to generate an ordered behavior sequence, ensuring that each behavior sequence accurately reflects the behavior dynamics of law enforcement officers and parties, and finally the law enforcement officer behavior sequence of law enforcement behavior events and the party behavior sequence of law enforcement behavior events are obtained.

[0154] Step S32: performing behavior description matching analysis between the behavior description nodes in the behavior sequence of the law enforcement personnel in the law enforcement behavior event and the behavior description nodes in the behavior sequence of the parties involved in the law enforcement behavior event, so as to obtain the law enforcement event matching relationship between the law enforcement personnel behavior description nodes and the party behavior description nodes;

[0155] In an embodiment of the present invention, by standardizing the behavior description nodes in the behavior sequence of law enforcement personnel in the law enforcement behavior event obtained by previous analysis and the behavior description nodes in the behavior sequence of the parties involved in the law enforcement behavior event, natural language processing technology (such as word embedding or text similarity calculation) can be used to convert different behavior descriptions into a unified semantic representation, and by constructing a vector representation of the behavior description, at the same time, by using a matching algorithm (such as cosine similarity calculation, dynamic time warping algorithm or template-based matching method), the law enforcement personnel behavior description nodes and the party behavior description nodes are compared, and each pair of behavior description nodes is calculated through the algorithm. The similarity score is determined to determine the matching relationship. For example, by setting a threshold, when the similarity score exceeds the threshold, the two nodes are considered to be matched successfully, and the results of the matching analysis are recorded in a matching relationship table, thereby determining the matching relationship between each pair of matching law enforcement personnel and party behavior description nodes, and finally obtaining the law enforcement event matching relationship between the law enforcement personnel behavior description node and the party behavior description node.

[0156] Step S33: Based on the law enforcement event matching relationship between the law enforcement officer behavior description node and the party behavior description node, the law enforcement behavior sequence connection processing is performed between the behavior description nodes in the law enforcement officer behavior sequence of the law enforcement behavior event and the behavior description nodes in the party behavior sequence of the law enforcement behavior event, so as to generate a law enforcement scene event behavior sequence connection diagram;

[0157] In an embodiment of the present invention, by combining the law enforcement event matching relationship between the law enforcement personnel behavior description node and the party behavior description node obtained by previous matching analysis, matching and connection processing is performed between the corresponding behavior description nodes in the law enforcement personnel behavior sequence of the law enforcement behavior event and the corresponding behavior description nodes in the party behavior sequence of the law enforcement behavior event, so as to connect the matched behavior nodes in chronological order by using graph construction technology in graph theory (such as adjacency matrix or adjacency list), each behavior node is used as a node in the graph, and the connected behavior descriptions are used as edges, so as to construct a behavior sequence connection graph, display the event development process at the law enforcement scene, and ensure that each pair of matching nodes can be correctly connected together to form a complete behavior sequence, and finally connect to generate a law enforcement scene event behavior sequence connection graph.

[0158] Step S34: Perform event behavior intention recognition analysis on the law enforcement scene event behavior sequence connection diagram to obtain law enforcement scene event behavior intention information data.

[0159] In an embodiment of the present invention, a behavior intention model is constructed. The model is based on known behavior patterns and intention templates and can be trained using machine learning techniques (such as classification algorithms or sequence labeling models) to identify different behavior intentions. For example, a support vector machine (SVM) classifier is trained to predict the intention type of each behavior sequence, and the intention of each behavior node and behavior sequence in the law enforcement scene event behavior sequence connection graph is identified according to the previously constructed behavior intention model, so that the behavior intention model is applied to the nodes in the graph, and the corresponding behavior intention is determined by calculating the intention probability of each behavior sequence. For example, the behavior sequence is analyzed to see whether it meets specific law enforcement intentions (such as warning, dissuasion, punishment, arrest, escape, etc.), and finally the law enforcement scene event behavior intention information data is obtained.

[0160] Further, step S4 includes the following steps:

[0161] Step S41: Obtaining a preset law enforcement event behavior knowledge graph;

[0162] In an embodiment of the present invention, a knowledge graph of corresponding law enforcement event behaviors is extracted from a preset knowledge base, and the graph contains behavioral patterns of various law enforcement events and their related normative provisions, so as to utilize natural language processing technology to convert law enforcement behavior documents, normative clauses and case data from different sources into structured knowledge. The data source may be a legal database, law enforcement records and expert domain knowledge. Through the visual interface of the graph construction platform, the behavioral knowledge graph is integrated and constructed to form a knowledge graph containing law enforcement behavior nodes and their relationships, and finally the law enforcement event behavior knowledge graph is obtained.

[0163] Step S42: performing a behavior normative rule retrieval and extraction process on the law enforcement event behavior knowledge graph to obtain a behavior normative rule standard for law enforcement events;

[0164] In an embodiment of the present invention, a previously acquired law enforcement incident behavior knowledge graph is processed by applying a rule extraction method. First, a standard for retrieving behavioral normative rules is defined, and a rule engine is used for retrieval. The rule engine extracts rules related to behavioral norms by parsing the attributes of nodes and edges in the graph. During the retrieval process, the behavioral patterns in the graph are compared with the normative clauses, and behavioral rules that meet the norms are extracted. As a result, a rule list containing detailed behavioral norms standards is generated. The list may include the code of conduct, processing procedures and corresponding normative basis for law enforcement personnel, and ultimately the behavioral normative rule standards for law enforcement incidents are obtained.

[0165] Step S43: performing behavior intention semantic feature analysis on the behavior intention information data of the law enforcement scene event to obtain the behavior intention semantic feature data of the law enforcement scene event;

[0166] In an embodiment of the present invention, natural language processing and machine learning models are used to perform statistical analysis on the semantic features of behavioral intentions of the previously analyzed law enforcement scene event behavioral intention information data, so as to achieve word segmentation, part-of-speech tagging and syntactic analysis of the description text of the scene event, and extract feature data of behavioral intentions, such as the purpose, motivation and context of law enforcement behavior intentions, through semantic analysis tools. The machine learning model can use a pre-trained language model (such as BERT or GPT) for feature extraction, and screen out the most representative semantic features through a feature selection algorithm, and finally obtain the semantic feature data of law enforcement scene event behavioral intentions.

[0167] Step S44: Based on the law enforcement event behavior normative rule standard, the law enforcement scene event behavior intention semantic feature data is adapted to the normative rule and the behavior normative score is calculated to obtain the law enforcement scene behavior intention normative score value;

[0168] In an embodiment of the present invention, the normative rules are adapted to the corresponding semantic feature data of the behavior intention of the law enforcement scene event by combining the normative rule standards of the law enforcement event behavior extracted previously, so as to calculate the similarity between the behavior intention feature vector and the normative rule by using a matching algorithm such as cosine similarity or Euclidean distance, and generate a normative score based on the calculated similarity score by using a normative rule adapter. This score reflects the degree of compliance between the on-site behavior intention and the normative rules. The calculation process involves weighting the weight and matching degree of each rule, so as to quantify the calculated behavior normative score value to evaluate the standardization degree of the on-site behavior intention, and finally obtain the law enforcement scene behavior intention normative score value.

[0169] Step S45: Based on the law enforcement scene behavior intention standardization score value, the non-standard behavior intelligent alarm optimization analysis is performed on the corresponding law enforcement scene events in the law enforcement scene event behavior intention information data to generate a law enforcement scene event non-standard behavior optimization suggestion report.

[0170] In an embodiment of the present invention, the law enforcement scene events corresponding to the law enforcement scene event behavior intention information data are analyzed by using the law enforcement scene behavior intention normative score values ​​obtained by previous quantitative calculation, so as to mark the behaviors with score values ​​lower than the threshold as non-normative behaviors by setting a threshold, and apply the intelligent alarm method to further analyze these non-normative behaviors, use the anomaly detection algorithm to identify potential problem areas, and optimize the analysis thereof, including generating a detailed optimization suggestion report, the report content including the identified non-normative behavior types, existing normative risks and improvement suggestions, the report using data visualization tools to display the analysis results and suggestions, so as to help law enforcement personnel adjust their law enforcement behaviors, ensure the standardization of the law enforcement process, and finally generate an optimization suggestion report on non-normative behaviors of law enforcement scene events.

[0171] Further, step S45 includes the following steps:

[0172] Step S451: Compare and judge the normative score value of the law enforcement scene behavior intention according to the preset normative score threshold. When the normative score value of the law enforcement scene behavior intention is greater than or equal to the preset normative score threshold, the corresponding law enforcement scene event in the law enforcement scene event behavior intention information data is determined as a normative behavior event; when the normative score value of the law enforcement scene behavior intention is less than the preset normative score threshold, the corresponding law enforcement scene event in the law enforcement scene event behavior intention information data is determined as an irregular behavior event;

[0173] In an embodiment of the present invention, a comparison is made between the normative score values ​​of the law enforcement scene behavior intention obtained by the previous quantitative calculation according to a preset normative score threshold (which is set according to the minimum requirements for normative behavior, for example, set to 80 points). If the normative score value of the law enforcement scene behavior intention is greater than or equal to the normative score threshold of 80 points, it means that the corresponding law enforcement scene behavior event is relatively standardized, and the corresponding law enforcement scene event in the law enforcement scene event behavior intention information data is determined to be a normative behavior event; and if the normative score value of the law enforcement scene behavior intention is less than the normative score threshold of 80 points, it means that the corresponding law enforcement scene behavior event is not normative, and the corresponding law enforcement scene event in the law enforcement scene event behavior intention information data is determined to be a non-normative behavior event.

[0174] Step S452: Performing irregular behavior intelligent alarm processing on the irregular behavior event determined as irregular behavior according to the preset irregular law enforcement behavior alarm rule to generate the alarm information of the irregular behavior event of this supervision;

[0175] In an embodiment of the present invention, relevant records of all events determined to be irregular behavior events are extracted from previous judgment results, and these records include detailed information of law enforcement behavior events, such as the time, location, and standardization score of the law enforcement behavior events, and the extracted irregular behavior events are processed according to preset irregular law enforcement behavior alarm rules. The irregular law enforcement behavior alarm rules include indicators such as the severity of the behavior and the number of repeated violations. These data will be analyzed, and corresponding alarm signals will be generated in response. At the same time, the alarm signals will be converted into standardized alarm information, including the details of the law enforcement behavior events, alarm levels, recommended handling measures, etc. For example, if the standardization score of an event is 75 points and involves repeated violations, a high-level alarm information will be generated, and finally the alarm information of the irregular behavior event of this supervision will be generated. This information includes a detailed description and alarm level of the irregular behavior event generated through supervision.

[0176] Step S453: performing law enforcement behavior problem identification and analysis on the alarm information of the current supervision irregular behavior event to obtain a report on the current supervision irregular law enforcement behavior problem;

[0177] In an embodiment of the present invention, by conducting a detailed analysis of the law enforcement behavior problems of the current supervision irregular behavior event alarm information generated by the previous alarm, the frequency, type, law enforcement personnel involved and other factors of the alarm event will be statistically analyzed to identify potential problems, thereby identifying frequently occurring problems or systematic violation problems, and organizing the identified problems into corresponding problem reports, which include the type of irregular behavior problems, the law enforcement events involved, the frequency and their potential causes, and finally obtaining a report on the current supervision irregular law enforcement behavior problems.

[0178] Step S454: Process the report on the problem of irregular law enforcement behavior in this supervision with suggestions for optimizing law enforcement behavior to generate a report on optimization suggestions for irregular behavior in law enforcement on-site events.

[0179] In an embodiment of the present invention, specific irregular law enforcement behavior problems are extracted from the report on irregular law enforcement behavior problems in this supervision obtained by previous analysis, such as improper handling procedures for violations by a certain type of party or insufficient training of law enforcement personnel. The existing law enforcement process is modeled by using a process optimization tool (such as business process management software or a process modeling tool), and improvement suggestions are made based on the identified problems, including redesigning process steps, adding training links, or adjusting law enforcement resource allocation, so as to prepare a detailed optimization suggestion report, which includes a problem description, optimization suggestions, expected results, and implementation steps. The report is submitted to management as a reference for improving law enforcement behavior, and ultimately an optimization suggestion report on irregular behavior in law enforcement on-site events is generated.

[0180] Furthermore, the present invention also provides a multimodal intelligent analysis system for law enforcement supervision, which is used to execute the multimodal intelligent analysis method for law enforcement supervision as described above. The multimodal intelligent analysis system for law enforcement supervision includes:

[0181] The law enforcement audio and video behavior recognition and analysis module is used to obtain the law enforcement scene video file from the law enforcement supervisory recorder platform, and perform audio and video segmentation processing on the law enforcement scene video file to obtain the law enforcement scene audio segment and the law enforcement scene image video frame segment; perform law enforcement behavior information recognition and analysis on the law enforcement scene audio segment and the law enforcement scene image video frame segment respectively, so as to obtain the law enforcement scene audio behavior information data and the law enforcement scene image behavior information data;

[0182] The law enforcement behavior multimodal spatiotemporal correlation analysis module is used to extract the timestamp and spatial position of the law enforcement scene video file to obtain the law enforcement scene process timestamp and law enforcement scene process spatial position information; based on the law enforcement scene process timestamp and law enforcement scene process spatial position information, the law enforcement scene audio behavior information data and the law enforcement scene image behavior information data are subjected to multimodal spatiotemporal synchronous correlation analysis to obtain multimodal spatiotemporal synchronous law enforcement behavior correlation event data;

[0183] The law enforcement event behavior intention recognition and analysis module is used to perform law enforcement behavior sequence connection processing on the multi-modal spatiotemporal synchronous law enforcement behavior related event data to generate a law enforcement scene event behavior sequence connection diagram; perform event behavior intention recognition and analysis on the law enforcement scene event behavior sequence connection diagram to obtain law enforcement scene event behavior intention information data;

[0184] The module for intelligent warning of irregular behaviors in law enforcement events is used to obtain a preset knowledge graph of law enforcement event behaviors, and to calculate the behavioral normativeness score of the behavior intention information data of law enforcement scene events based on the knowledge graph of law enforcement event behaviors, so as to obtain the law enforcement scene behavior intention normativeness score value; based on the law enforcement scene behavior intention normativeness score value, the module performs an optimization analysis of irregular behaviors of intelligent warnings on corresponding law enforcement scene events in the behavior intention information data of law enforcement scene events, so as to generate an optimization suggestion report on irregular behaviors of law enforcement scene events.

[0185] Therefore, the embodiments should be regarded as illustrative and non-restrictive from all points, and the scope of the present invention is limited by the appended claims rather than the above description, and it is therefore intended that all changes falling within the meaning and range of equivalent elements of the application documents are included in the present invention.

[0186] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.

Claims

1. A multimodal intelligent analysis method for law enforcement supervision, characterized in that: The following steps are involved: Step S1: Obtain the law enforcement scene video file from the law enforcement supervisory recorder platform, and perform audio and video segmentation processing on the law enforcement scene video file to obtain the law enforcement scene audio segment and the law enforcement scene image video frame segment; perform law enforcement behavior information recognition and analysis on the law enforcement scene audio segment and the law enforcement scene image video frame segment respectively to obtain the law enforcement scene audio behavior information data and the law enforcement scene image behavior information data; Step S2: extracting the timestamp and spatial position of the law enforcement scene video file to obtain the timestamp and spatial position information of the law enforcement scene process; Based on the timestamp of the law enforcement scene process and the spatial position information of the law enforcement scene process, a multimodal spatiotemporal synchronous correlation analysis is performed on the audio behavior information data and the image behavior information data of the law enforcement scene to obtain multimodal spatiotemporal synchronous law enforcement behavior correlation event data; Step S3: performing law enforcement behavior sequence connection processing on the multi-modal spatiotemporal synchronous law enforcement behavior associated event data to generate a law enforcement scene event behavior sequence connection diagram; Conduct event behavior intention recognition analysis on the event behavior sequence connection diagram at the law enforcement scene to obtain the event behavior intention information data at the law enforcement scene; Step S4: obtaining a preset law enforcement event behavior knowledge graph, and calculating the behavior normativeness score of the law enforcement scene event behavior intention information data based on the law enforcement event behavior knowledge graph to obtain the law enforcement scene behavior intention normativeness score value; Based on the normative score value of the behavior intention at the law enforcement scene, an intelligent alarm optimization analysis of irregular behavior is performed on the corresponding law enforcement scene events in the behavior intention information data of the law enforcement scene events to generate an optimization suggestion report on irregular behavior of law enforcement scene events.

2. The multimodal intelligent analysis method for law enforcement supervision according to claim 1 is characterized in that: Step S1 includes the following steps: Step S11: Obtaining the law enforcement scene video file from the law enforcement supervisory recorder platform; Step S12: Performing audio and video synchronization processing on the law enforcement scene video file to obtain an audio and video synchronized video of the law enforcement scene; Step S13: extracting video content from the law enforcement scene audio and video synchronization video to obtain video content information data of the law enforcement scene; based on the video content information data of the law enforcement scene, identifying and analyzing the start and end time of different law enforcement events in the law enforcement scene audio and video synchronization video to obtain the start time and end time of different law enforcement events in the law enforcement scene video; Step S14: performing audio and video segmentation processing on the law enforcement scene audio and video synchronization video according to the start time and end time of different law enforcement events in the law enforcement scene video to obtain the law enforcement scene audio segment and the law enforcement scene image video frame segment; Step S15: Perform law enforcement behavior information recognition and analysis on the law enforcement scene audio clips and the law enforcement scene image video frame clips respectively to obtain law enforcement scene audio behavior information data and law enforcement scene image behavior information data.

3. The multimodal intelligent analysis method for law enforcement supervision according to claim 2 is characterized in that: Step S15 includes the following steps: Step S151: using Speechbrain technology to perform law enforcement audio voiceprint feature analysis on the law enforcement scene audio clip to obtain law enforcement scene audio voiceprint feature data; Step S152: performing audio speech signal channel deconstruction processing on different speakers in the audio clip of the law enforcement scene based on the audio voiceprint feature data of the law enforcement scene, and obtaining audio speech signal sub-channels of different speakers in the audio clip of the law enforcement scene; Step S153: performing law enforcement behavior event triggering recognition analysis on the audio speech signal sub-channels of different speakers in the law enforcement scene audio to obtain the law enforcement behavior event triggering points of the law enforcement scene audio; Step S154: performing law enforcement audio behavior information recognition and analysis on the law enforcement scene audio clip based on the law enforcement scene audio law enforcement behavior event trigger point to obtain law enforcement scene audio behavior information data; Step S155: Use the object detection algorithm Yolov8 and the tracking technology Bytetrack to perform law enforcement image behavior information recognition and analysis on the law enforcement scene image video frame segments to obtain law enforcement scene image behavior information data.

4. The multimodal intelligent analysis method for law enforcement supervision according to claim 3 is characterized in that: Step S155 includes the following steps: The object detection algorithm Yolov8 is used to perform object target recognition analysis on the law enforcement scene image video frame fragments to obtain the object detection target of the law enforcement scene image frame; Performing object boundary region marking processing on the object detection target of the law enforcement scene image frame to obtain the law enforcement scene object boundary region marking target; The tracking technology Bytetrack is used to track and identify the motion trajectory of the marked target in the boundary area of ​​the law enforcement scene within the video frame of the law enforcement scene image to generate the motion trajectory of the marked target of the law enforcement scene object; The target behavior pattern of the object is analyzed on the motion trajectory of the object mark at the law enforcement scene to obtain the law enforcement behavior patterns of the object marks at different law enforcement scenes; based on the law enforcement behavior patterns of the object marks at different law enforcement scenes, the corresponding target boundary area mark of the law enforcement scene is classified and processed to obtain the law enforcement behavior actions of the object marks at different law enforcement scenes; Based on the law enforcement behavior patterns and law enforcement behavior actions of different law enforcement scene object targets, law enforcement image behavior information recognition and analysis are performed on the law enforcement scene image video frame fragments to obtain law enforcement scene image behavior information data.

5. The multimodal intelligent analysis method for law enforcement supervision according to claim 1 is characterized in that: Step S2 includes the following steps: Step S21: extracting the timestamp of the law enforcement scene video file to obtain the law enforcement scene process timestamp; Step S22: performing a three-dimensional space coordinate system conversion on each frame of the video image in the law enforcement scene video file to obtain a three-dimensional space coordinate system for each frame of the video image in the law enforcement scene video; Step S23: performing law enforcement scene spatial coordinate extraction processing on the three-dimensional spatial coordinate system of each frame image in the law enforcement scene video to obtain the law enforcement scene spatial coordinates of each frame image in the law enforcement scene video; Step S24: performing spatial position marking processing on the law enforcement scene spatial coordinates of each frame of the law enforcement scene video to obtain the law enforcement scene process spatial position information; Step S25: Based on the law enforcement scene process timestamp and the law enforcement scene process spatial position information, a multimodal spatiotemporal synchronous correlation analysis is performed on the law enforcement scene audio behavior information data and the law enforcement scene image behavior information data to obtain multimodal spatiotemporal synchronous law enforcement behavior correlation event data.

6. The multimodal intelligent analysis method for law enforcement supervision according to claim 5 is characterized in that: Step S25 includes the following steps: Step S251: constructing a spatiotemporal synchronization benchmark for the law enforcement scene process timestamp and the law enforcement scene process spatial location information to obtain a spatiotemporal data synchronization benchmark framework for the law enforcement scene process; Step S252: Based on the law enforcement scene process spatiotemporal data synchronization benchmark framework, audio event timing synchronization arrangement processing is performed on the law enforcement scene audio behavior information data to obtain the law enforcement scene audio event timing synchronization sequence data; Step S253: performing image behavior space mapping processing on the image behavior information data of the law enforcement scene based on the law enforcement scene process spatiotemporal data synchronization benchmark framework to obtain the law enforcement scene image behavior space position mapping data; Step S254: performing multimodal event-behavior interaction correlation analysis on the audio event time synchronization sequence data of the law enforcement scene and the image behavior spatial position mapping data of the law enforcement scene to obtain the multimodal event-behavior interaction correlation relationship between the audio event and the image behavior of the law enforcement scene; Step S255: Based on the multimodal event-behavior interactive correlation relationship between the audio events and image behaviors at the law enforcement scene, the law enforcement scene audio behavior information data and the law enforcement scene image behavior information data are subjected to law enforcement behavior related event extraction processing to obtain multimodal spatiotemporal synchronized law enforcement behavior related event data.

7. The multimodal intelligent analysis method for law enforcement supervision according to claim 1 is characterized in that: Step S3 includes the following steps: Step S31: extracting the behavior sequences of law enforcement personnel and parties involved in the multimodal spatiotemporal synchronous law enforcement behavior related event data to obtain the behavior sequences of law enforcement personnel in the law enforcement behavior events and the behavior sequences of parties involved in the law enforcement behavior events; Step S32: performing behavior description matching analysis between the behavior description nodes in the behavior sequence of the law enforcement personnel in the law enforcement behavior event and the behavior description nodes in the behavior sequence of the parties involved in the law enforcement behavior event, so as to obtain the law enforcement event matching relationship between the law enforcement personnel behavior description nodes and the party behavior description nodes; Step S33: Based on the law enforcement event matching relationship between the law enforcement officer behavior description node and the party behavior description node, the law enforcement behavior sequence connection processing is performed between the behavior description nodes in the law enforcement officer behavior sequence of the law enforcement behavior event and the behavior description nodes in the party behavior sequence of the law enforcement behavior event, so as to generate a law enforcement scene event behavior sequence connection diagram; Step S34: Perform event behavior intention recognition analysis on the law enforcement scene event behavior sequence connection diagram to obtain law enforcement scene event behavior intention information data.

8. The multimodal intelligent analysis method for law enforcement supervision according to claim 1 is characterized in that: Step S4 includes the following steps: Step S41: Obtaining a preset law enforcement event behavior knowledge graph; Step S42: performing a behavior normative rule retrieval and extraction process on the law enforcement event behavior knowledge graph to obtain a behavior normative rule standard for law enforcement events; Step S43: performing behavior intention semantic feature analysis on the behavior intention information data of the law enforcement scene event to obtain the behavior intention semantic feature data of the law enforcement scene event; Step S44: Based on the law enforcement event behavior normative rule standard, the law enforcement scene event behavior intention semantic feature data is adapted to the normative rule and the behavior normative score is calculated to obtain the law enforcement scene behavior intention normative score value; Step S45: Based on the law enforcement scene behavior intention standardization score value, the non-standard behavior intelligent alarm optimization analysis is performed on the corresponding law enforcement scene events in the law enforcement scene event behavior intention information data to generate a law enforcement scene event non-standard behavior optimization suggestion report.

9. The multimodal intelligent analysis method for law enforcement supervision according to claim 8 is characterized in that: Step S45 includes the following steps: Step S451: Compare and judge the normative score value of the law enforcement scene behavior intention according to the preset normative score threshold. When the normative score value of the law enforcement scene behavior intention is greater than or equal to the preset normative score threshold, the corresponding law enforcement scene event in the law enforcement scene event behavior intention information data is determined as a normative behavior event; when the normative score value of the law enforcement scene behavior intention is less than the preset normative score threshold, the corresponding law enforcement scene event in the law enforcement scene event behavior intention information data is determined as an irregular behavior event; Step S452: Performing irregular behavior intelligent alarm processing on the irregular behavior event determined as irregular behavior according to the preset irregular law enforcement behavior alarm rule to generate the alarm information of the irregular behavior event of this supervision; Step S453: performing law enforcement behavior problem identification and analysis on the alarm information of the current supervision irregular behavior event to obtain a report on the current supervision irregular law enforcement behavior problem; Step S454: Process the report on the problem of irregular law enforcement behavior in this supervision with suggestions for optimizing law enforcement behavior to generate a report on optimization suggestions for irregular behavior in law enforcement on-site events.

10. A multimodal intelligent analysis system for law enforcement supervision, characterized in that: Used to execute the multimodal intelligent analysis method for law enforcement supervision as claimed in claim 1, the multimodal intelligent analysis system for law enforcement supervision comprises: The law enforcement audio and video behavior recognition and analysis module is used to obtain the law enforcement scene video file from the law enforcement supervisory recorder platform, and perform audio and video segmentation processing on the law enforcement scene video file to obtain the law enforcement scene audio segment and the law enforcement scene image video frame segment; perform law enforcement behavior information recognition and analysis on the law enforcement scene audio segment and the law enforcement scene image video frame segment respectively, so as to obtain the law enforcement scene audio behavior information data and the law enforcement scene image behavior information data; The law enforcement behavior multimodal spatiotemporal correlation analysis module is used to extract the timestamp and spatial position of the law enforcement scene video file to obtain the law enforcement scene process timestamp and law enforcement scene process spatial position information; based on the law enforcement scene process timestamp and law enforcement scene process spatial position information, the law enforcement scene audio behavior information data and the law enforcement scene image behavior information data are subjected to multimodal spatiotemporal synchronous correlation analysis to obtain multimodal spatiotemporal synchronous law enforcement behavior correlation event data; The law enforcement event behavior intention recognition and analysis module is used to perform law enforcement behavior sequence connection processing on the multi-modal spatiotemporal synchronous law enforcement behavior related event data to generate a law enforcement scene event behavior sequence connection diagram; perform event behavior intention recognition and analysis on the law enforcement scene event behavior sequence connection diagram to obtain law enforcement scene event behavior intention information data; The module for intelligent warning of irregular behaviors in law enforcement events is used to obtain a preset knowledge graph of law enforcement event behaviors, and to calculate the behavioral normativeness score of the behavior intention information data of law enforcement scene events based on the knowledge graph of law enforcement event behaviors, so as to obtain the law enforcement scene behavior intention normativeness score value; based on the law enforcement scene behavior intention normativeness score value, the module performs an optimization analysis of irregular behaviors of intelligent warnings on the corresponding law enforcement scene events in the behavior intention information data of law enforcement scene events, so as to generate an optimization suggestion report on irregular behaviors of law enforcement scene events.

Images