A system and method for fast checking of video violations of a specific place

Abstract

The application discloses a kind of specific place monitoring video rule-breaking behavior quick check system and method, comprising: front-end dispatching control module, for receiving and executing the command input by user, corresponding module is called to carry out task processing;Place attention parameter configuration module, for configuring place attention parameter of place entrance access control data acquisition unit and video monitoring data acquisition unit;Video acquisition module, for obtaining the monitoring video used for checking by the way of import or direct video stream;Video analysis module, for carrying out multi-path parallel analysis to monitoring video, obtain video structured data as video analysis result information;Data storage module, and associated analysis module, for the associated analysis of video structured data and rule-breaking behavior.The system of the application can carry out associated analysis to specific place monitoring video, and the automatic detection and directional search of rule-breaking behavior, and this method reduces the time consumption and application complexity of secret-related place monitoring video inspection.

Description

Technical Field

[0001] This invention belongs to the field of video analysis technology, specifically relating to a rapid detection system and method for violations in surveillance video of specific locations. Background Technology

[0002] Classified locations and other specific areas require strict management. Mismanagement failures and violations often lead to serious consequences. Therefore, rigorous monitoring is necessary to prevent violations and facilitate rapid accountability should they occur. While many companies have installed surveillance cameras in classified areas, analyzing the vast amounts of information recorded by these cameras presents a significant challenge due to various objective limitations.

[0003] Traditional video analysis methods rely on manual analysis, which has drawbacks such as high workload, slow analysis speed, and poor analysis results. Summary of the Invention

[0004] In view of the above-mentioned problems in the existing technology, and considering the accuracy and timeliness required for the analysis of surveillance video in specific locations, this invention proposes a rapid inspection system and method for violations in surveillance video in specific locations. It can perform accurate and rapid analysis of surveillance video, improve the efficiency and accuracy of violation retrieval in confidential locations. The system and method of this invention are reasonably designed, overcome the shortcomings of the existing technology, and have good effects.

[0005] The technical solution of this invention is: a rapid detection system for violations in surveillance video of specific locations, comprising:

[0006] The front-end scheduling and control module is used to receive and execute user-input commands and call the corresponding modules for task processing.

[0007] The venue attention parameter configuration module is used to configure the venue attention parameters of the venue entrance access control data acquisition unit and the venue attention parameters of the video surveillance data acquisition unit;

[0008] The video acquisition module is used to acquire the surveillance video to be inspected by importing or directly streaming video streams.

[0009] The video analysis module is used to perform multi-channel parallel analysis on surveillance videos to obtain structured video data as video analysis results.

[0010] The video retrieval module is used to retrieve surveillance videos of a specified type based on information sent from the front end.

[0011] The data storage module is used to store surveillance video information and video analysis results.

[0012] The correlation analysis module is used for correlation analysis between structured video data and violations.

[0013] Furthermore, the aforementioned location attention parameter configuration module is used to receive the input of confidential personnel and event parameters provided by the front-end scheduling and control module, and to manage personnel, surveillance cameras, and confidential events; import the face list and roster of confidential personnel and store them in the data storage module; set the types of violations; and configure the location attention parameters related to video analysis.

[0014] Furthermore, the video acquisition module is used to acquire the surveillance video to be inspected by importing or directly streaming video streams; specifically, it includes:

[0015] It supports users importing videos or directly connecting to the video surveillance data acquisition unit to download surveillance videos in real time from the network recorder; the video acquisition module obtains the duration, frame rate, resolution, and video encoding format information of the imported video and sends the information to the front-end scheduling and control module; it also includes:

[0016] The video transcoding module is used to acquire basic information about the surveillance video and analyze the video encoding format. If the video encoding format is not supported for playback, it will be transcoded. Then, the imported videos are automatically categorized according to the cameras and stored in the data storage module.

[0017] Furthermore, the video analysis module is used to read videos from the video acquisition module and perform multi-channel video analysis sequentially. During the video analysis process, the system analyzes the video frame by frame according to the configuration information related to the location attention parameters, and writes the structured data obtained after analysis into the video analysis report. During the analysis process, the video analysis progress is fed back in stages for the progress bar display of the front-end scheduling and control module. After the video analysis is completed, the video with structured analysis completed is stored through the data storage module, and the video analysis report is exported and stored in the data storage module.

[0018] Furthermore, it also includes:

[0019] The video retrieval module retrieves video summary content based on information provided by the user from the front-end scheduling and control module, according to specified requirements, from the surveillance videos that have undergone structured analysis.

[0020] Furthermore, the correlation analysis module, used for correlation analysis between video structured data and violations, performs correlation analysis after discretizing the features extracted by deep learning, specifically including:

[0021] Report Association Analysis Module: Used to import various reports such as video analysis reports and access control reports. The video analysis reports are reports exported after video analysis of surveillance videos. If the imported report does not exist in the database, it will be stored through the data storage module.

[0022] Violation Correlation Analysis Module: After manual confirmation, unidentified violations are marked. The violation correlation analysis module extracts and discretizes their features, and then performs correlation analysis between the violation and the discretized features.

[0023] Report merging module: Used to export and display compliance reports, and simultaneously store them in the database.

[0024] According to another aspect of the present invention, a method for rapid inspection of violations in surveillance video of a specific location is also proposed, comprising the following steps:

[0025] Step S1, the location attention parameter configuration step, configures the location attention parameters of the location entrance access control data acquisition unit and the location attention parameters of the video surveillance data acquisition unit;

[0026] Step S2, video acquisition step, acquire the surveillance video to be inspected by importing or directly streaming the video;

[0027] Step S3, video analysis step, involves performing multi-channel parallel analysis on the surveillance video to obtain structured video data as the video analysis result information;

[0028] Step S4, data storage step, storing surveillance video information and video analysis results information.

[0029] Step S5, the correlation analysis step, involves performing correlation analysis between the structured video data and the violations.

[0030] Furthermore, step S1 includes:

[0031] It receives input parameters of classified personnel and events from the front-end scheduling and control module, and performs personnel management, surveillance camera management, and classified event management; it imports the face list and roster of classified personnel and stores them in the data storage module; it sets the types of violations; and it configures the location attention parameters related to video analysis.

[0032] Furthermore, step S2 includes:

[0033] Users can import videos or directly interface with the video surveillance data acquisition unit to download surveillance videos in real time from the network recorder; the video acquisition module obtains the duration, frame rate, resolution, and video encoding format information of the imported video and sends the information to the front-end scheduling and control module; it also includes:

[0034] This is used to acquire basic information about surveillance videos and analyze the video encoding format. If the video encoding format is not supported for playback, it is transcoded. Then, the imported videos are automatically categorized according to the cameras and stored in the data storage module.

[0035] Furthermore, step S3 includes:

[0036] The system reads video from the video acquisition module and performs multi-channel video analysis sequentially. During video analysis, the system analyzes the video frame by frame based on the configuration information related to the location's focus parameters. The structured data obtained after analysis is written into the video analysis report. During the analysis process, the system provides periodic feedback on the video analysis progress, which is displayed on the progress bar of the front-end scheduling and control module. After the video analysis is completed, the structured video is stored through the data storage module, and the video analysis report is exported and stored in the data storage module.

[0037] Furthermore, the aforementioned violations refer to one or more of the following: unauthorized personnel or non-classified personnel entering the premises; exceeding the number of personnel allowed to enter or the number of personnel not matching the registered number; illegally carrying information carriers; and abnormal monitoring information.

[0038] Abnormal monitoring information can be caused by one or more of the following: video signal interruption, video signal obstruction, or abnormal lighting.

[0039] Furthermore, step S3 includes:

[0040] Step S31: Use the video acquisition module to open offline video or read video content from the live stream of the surveillance camera;

[0041] Step S32: Decode the read video stream to obtain the image of each frame of the video.

[0042] Step S33: For a decoded frame of image, use an object detection algorithm to analyze it, detect faces, people, illegal behaviors, and information carrier targets in the image, and record the coordinates of the targets in the image;

[0043] Step S34: Check the configuration file of the location attention parameters. When the configuration file requires real-time face comparison, whenever a target is detected, the feature extraction algorithm is used to process it and compare it with the black and white face list in the data storage module. The similarity of the feature comparison is evaluated in the form of a score. The feature extraction result of a target is a multi-dimensional floating-point array.

[0044] Step S35: For the target detected in a frame of image, the target tracking algorithm is called for analysis. Whenever a target is detected, the algorithm associates the target across multiple frames based on the target detection results in its adjacent frames.

[0045] Step S36: For each frame of image obtained by decoding, repeat steps S03-S05 until the video analysis process ends.

[0046] Step S37: Specifically, during the analysis of video frames, if the algorithm determines that the monitoring information is abnormal, the abnormal information is written into the report file, and an abnormal information warning is issued at the same time.

[0047] Furthermore, step S35 includes:

[0048] S351: If a correlation is successfully established between adjacent frames for a target, it is determined that the target is still in the video frame, and the next frame is analyzed sequentially along the timeline; or if a correlation cannot be established between adjacent frames for a target, it is determined that the target has left the image, and the following steps are performed:

[0049] S352: Record the moment the target first appears in the video and the moment it leaves the video frame;

[0050] S353: Compare all frames that the target appears in the video and select the best frame. The best frame is defined as the frame that occupies the largest area when the target appears in the video.

[0051] S354: Extract features from the target.

[0052] S355: Perform structured processing on the video based on the time the target appears and the time it leaves the frame.

[0053] S3516: All the information mentioned above shall be stored in the database according to the type of target being detected by the data storage module.

[0054] Furthermore, step S5 includes:

[0055] Step S51, Report Association Analysis: Import various reports such as video analysis reports and access control reports. The video analysis reports are reports exported after video analysis of the surveillance video. If the imported reports do not exist in the database, they are stored through the data storage module.

[0056] Step S52, Violation Correlation Analysis: After manual confirmation, unidentified violations are marked. The violation correlation analysis module extracts and discretizes their features, and performs correlation analysis between the violations and the discretized features.

[0057] Step S53, Report Merging: Export and display the compliance report, and simultaneously store it in the database.

[0058] Furthermore, following step S3, the following steps are also included:

[0059] The video retrieval process involves searching the analyzed surveillance videos in the data storage module based on the set search target, including the following search methods:

[0060] Search by name, face, or body condition;

[0061] Search by time period, camera name, and registered item;

[0062] Search by type of violation;

[0063] Search based on the type of information carrier;

[0064] Search based on images provided by the user;

[0065] Export personnel trajectory retrieval and visualization information.

[0066] Furthermore, in the video retrieval step, the retrieval results are categorized according to retrieval requirements, sorted by timeline, and contain video summary segments of the target to be detected.

[0067] Furthermore, the information carrier includes one or more of the following: a bag, a computer, a mobile phone, and a file carrier.

[0068] Furthermore, the video retrieval steps also include the retrieval and visualization of personnel entry and exit trajectories, including the following information: entry and exit time, entry and exit frequency, and personnel snapshots.

[0069] Furthermore, the site-specific parameter configuration steps also include:

[0070] Configure classified personnel management information, including a list of classified personnel, photos of classified personnel, and registration and communication information for classified personnel;

[0071] Configure monitoring information, including information on the interface with the monitoring platform and camera settings. Attached Figure Description

[0072] Figure 1 : A system block diagram of a method for rapid inspection of violations in surveillance video of a specific location according to the present invention;

[0073] Figure 2 : Flowchart of a method for rapid inspection of violations in surveillance video of a specific location according to the present invention. Detailed Implementation

[0074] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0075] According to embodiments of the present invention, a method and system for rapid inspection of violations in surveillance video of specific locations are proposed.

[0076] The specific locations mentioned above, in this embodiment, refer to places with high confidentiality requirements, especially classified locations, which are different from ordinary monitoring locations. Classified locations have special regulations regarding the environment and personnel, so general monitoring methods cannot be applied to classified locations.

[0077] The system of the present invention can adopt a front-end client and back-end server architecture. A communication connection is established between the front-end client and the back-end server architecture. The front-end client is also connected to surveillance cameras and other devices, as well as access control devices of the venue, through a monitoring platform. The monitoring platform can be set as an independent module or component, or it can be integrated into the front-end client and connected to the cameras of various monitoring venues wirelessly or wiredly.

[0078] The front-end client is used to control and schedule other modules of the system according to preset configuration and user operation, and to display the service scheduling status and analysis results of each module.

[0079] Furthermore, in one embodiment of the present invention, a rapid detection system for violations in surveillance video of specific locations is proposed, such as... Figure 1 As shown, it includes:

[0080] The front-end scheduling and control module is used to receive and execute user-input commands and call the corresponding modules for task processing.

[0081] The venue attention parameter configuration module is used to configure the venue attention parameters of the venue entrance access control data acquisition unit and the venue attention parameters of the video surveillance data acquisition unit;

[0082] The video acquisition module is used to acquire the surveillance video to be inspected by importing or directly streaming video streams.

[0083] The video analysis module is used to perform multi-channel parallel analysis on surveillance videos to obtain structured video data as video analysis results.

[0084] The video retrieval module is used to retrieve surveillance videos of a specified type based on information sent from the front end.

[0085] The data storage module is used to store surveillance video information and video analysis results.

[0086] The aforementioned front-end scheduling and control module, location attention parameter configuration module, video acquisition module, video analysis module, video retrieval module, and data storage module can all be located on the front-end client; according to other optional embodiments of the present invention, the data storage module, video retrieval module, etc., can also be located on the server side;

[0087] The server may also be equipped with a feature training and learning module for training the recognition of classified image features;

[0088] Specifically, the location attention parameter configuration module is used to receive the input of confidential personnel and event parameters from the front-end scheduling and control module, and to manage personnel, surveillance cameras, and confidential events; import the face list and roster of confidential personnel and store them in the data storage module; set the types of violations; and configure the location attention parameters related to video analysis.

[0089] According to one embodiment of the present invention, in the classified location, the flow of people in the surveillance video is less than in the surveillance video of a general public place. Considering the characteristics of classified locations, this embodiment of the invention requires analysis and labeling of every person appearing in the location, including personnel information, number of people, number of entries and exits, whether they are carrying unusual items, and whether they exhibit unusual behavior. Therefore, this invention requires personnel management, surveillance camera management, and classified event management for classified personnel; and imports a face list and a list of names of classified personnel, storing them in a data storage module for convenient subsequent video correlation analysis.

[0090] In one embodiment of the present invention, the entrance to a confidential location contains a storage area for mobile phones and other electronic communication devices. The present invention uses video recognition to identify individuals who have not performed electronic device storage operations, and then focuses on and identifies these individuals. The video analysis module can analyze the trajectory of individuals after entering the location, whether they pass through the electronic communication device storage area, and whether they linger there for a period of time. If not, these individuals are given special attention and identified. Therefore, the location attention parameter configuration module can configure detection parameters for such behavior, such as the location of the communication device storage area, trajectory parameters, and dwell time parameters, where the dwell time is, for example, 10 seconds.

[0091] This invention targets classified locations where both the interior and exterior are equipped with surveillance cameras. Given the small internal space of these locations, the invention uses video surveillance to identify whether individuals entering the classified area are violating regulations, including carrying prohibited items and other violations. Therefore, the location monitoring parameter configuration module can configure parameters for prohibited items, such as whether they include computers, mobile phones, bags, etc.; as well as limits on the number of people allowed and the walking speed range for individuals entering the location.

[0092] Furthermore, the video acquisition module is used to acquire the surveillance video to be inspected by importing or directly streaming video streams; specifically, it includes:

[0093] It supports users to import videos or directly connect to the video surveillance data acquisition unit to download surveillance videos in real time from the network recorder; the video surveillance data acquisition unit is set in the front-end client and is used to communicate with the monitoring platform to obtain video data from the surveillance camera.

[0094] The video acquisition module obtains the duration, frame rate, resolution, and video encoding format information of the imported video and sends the information to the front-end scheduling and control module; optionally, in some cases, the system may also include:

[0095] The video transcoding module is used to acquire basic information of the surveillance video and analyze the video encoding format. If the video encoding format does not support playback, it is transcoded. Then, the imported video is automatically categorized according to the camera and stored in the data storage module. The automatic categorization according to the camera refers to classifying and storing video data captured by different cameras.

[0096] Furthermore, the video analysis module is used to read videos from the video acquisition module and perform multi-channel video analysis sequentially. During the video analysis process, the system performs frame-by-frame structured analysis of the video based on the configuration information related to the location's attention parameters, and writes the structured data obtained after the structured analysis into the video analysis report. During the analysis process, the video analysis progress is fed back in stages for the progress bar display of the front-end scheduling and control module. After the video analysis is completed, the video with the structured analysis completed is stored through the data storage module, and the video analysis report is exported and stored in the data storage module.

[0097] Furthermore, it also includes:

[0098] The video retrieval module retrieves video summary content based on information provided by the user from the front-end scheduling and control module, according to specified requirements, from the surveillance videos that have undergone structured analysis.

[0099] Furthermore, the correlation analysis module, used for correlation analysis between video structured data and violations, performs correlation analysis after discretizing the features extracted by deep learning, specifically including:

[0100] Report Association Analysis Module: Used to import various reports such as video analysis reports and access control reports. The video analysis reports are reports exported after video structured analysis of surveillance videos. If the imported report does not exist in the database, it will be stored through the data storage module.

[0101] The violation association analysis module is used to label unidentified violations after manual confirmation, extract and discretize the features of such unidentified violations, and perform association analysis between the violations and the discretized features.

[0102] Report merging module: Used to export and display compliance reports, and simultaneously store them in the database.

[0103] According to another aspect of the invention, such as Figure 2 As shown, a method for rapid inspection of violations in surveillance videos of specific locations is also proposed, including the following steps:

[0104] Step S1, the location attention parameter configuration step, configures the location attention parameters of the location entrance access control data acquisition unit and the location attention parameters of the video surveillance data acquisition unit;

[0105] In this step, system settings can be configured, especially the settings of location-related parameters, as well as personnel management, camera management, and classified event management from the front-end client through the parameter configuration module; import the face list and roster of personnel with classified information and store them in the data storage module; connect the system to the monitoring platform and initialize the cameras; initialize the types of violations; and initialize the location-related parameter configuration information for frequency analysis.

[0106] In the aforementioned classified locations, the flow of people in the surveillance videos is less than in typical public places. Given the characteristics of classified locations, this embodiment of the invention requires the analysis and labeling of every person appearing in the location, including personnel information, number of people, number of entries and exits, whether they are carrying any unusual or prohibited items, and whether they exhibit any abnormal behavior. Therefore, this invention requires personnel management, surveillance camera management, and classified event management for classified personnel; and it imports a list of faces and names of classified personnel, storing them in the data storage module for subsequent video correlation analysis.

[0107] The violations referred to in this invention include one or more of the following: unauthorized personnel or non-classified personnel entering the premises, exceeding the number of personnel entering the premises or not matching the number of registered personnel, failing to store electronic devices in the electronic device storage location, illegally carrying information carriers, and abnormal monitoring information.

[0108] The prohibited items include: mobile phones, cameras, computers, and other video recording equipment;

[0109] This invention targets classified locations where both the interior and exterior are equipped with surveillance cameras. Given the small internal space of these locations, the invention uses video surveillance to identify whether individuals entering the classified area are violating regulations, including carrying prohibited items and other violations. Therefore, the location monitoring parameter configuration module can configure parameters for prohibited items, such as whether they include computers, mobile phones, bags, etc.; as well as limits on the number of people allowed and the walking speed range for individuals entering the location.

[0110] In the classified location described in this invention, the entrance to the classified location has a storage area for mobile phones and other electronic communication devices for storing items. When personnel bring items in, they need to store them. To store the devices, an access control card needs to be swiped to open the storage box, and the box door is randomly opened to store the electronic devices. Retrieving the items requires swiping an access control card. This access control information is stored and generated, including access control card information and storage time.

[0111] This invention uses video recognition to identify individuals who have not performed electronic device storage operations. These individuals are then given special attention and identified. The video analysis module analyzes the individual's trajectory after entering the location, determining whether they passed the electronic communication device storage area and remained there for a period of time. If not, these individuals are given special attention and identified. Therefore, in this step, the location attention parameter configuration module can configure detection parameters for this behavior, such as the location of the communication device storage area, trajectory parameters, and dwell time parameters, for example, a dwell time of 10 seconds.

[0112] Anomalies in monitoring information include one or more of the following: video signal interruption, video signal obstruction, and abnormal lighting. The location attention parameter configuration module can be used to configure the types of anomalies mentioned above for focused attention.

[0113] Step S2, video acquisition step, acquire the surveillance video to be inspected by importing or directly streaming the video;

[0114] In this embodiment, video import can be performed by batch importing monitoring videos from the front-end client or the connected monitoring platform through the video acquisition module. The video acquisition module obtains information such as the duration, frame rate, resolution, and video encoding format of the imported videos and sends this information to the front-end client. If the video encoding format is not supported for playback, the system automatically transcodes the video using the video transcoding module. After the above processing is completed, the imported videos are automatically categorized according to the cameras and stored in the data storage module.

[0115] Step S3, the video analysis step, is used to perform multi-channel parallel analysis on the surveillance video to obtain structured video data as video analysis result information;

[0116] The video analysis module reads videos from the video acquisition module and performs multi-channel video analysis sequentially. During video analysis, the system analyzes the video frame by frame according to configuration information and writes the analysis information into a video analysis report. Simultaneously, the video analysis module communicates with the backend service to provide periodic feedback on the video analysis progress, which is displayed on the front-end progress bar. After the video analysis is completed, the system stores the structured video through the data storage module and also exports the video analysis report and stores it in the data storage module.

[0117] Optionally, after step S3, a video retrieval step may also be included, in which, based on the information provided by the user from the front end, the video retrieval module performs a video summary retrieval of the analyzed surveillance video according to specified requirements.

[0118] Step S4, data storage step, is used to store surveillance video information and video analysis results.

[0119] Step S5, the correlation analysis step, is used for correlation analysis between video structured data and violations.

[0120] The aforementioned correlation analysis, based on user video structured data and manual violation judgments, generates corresponding correlation analysis results, including:

[0121] Report association analysis steps: This step is used to import various reports such as video analysis reports and access control reports. The video analysis reports are reports exported after video structured analysis of surveillance videos. If the imported reports do not exist in the database, they are stored through the data storage module.

[0122] The steps of the violation association analysis are as follows: After manual confirmation, unidentified violations are labeled, their features are extracted and discretized, and the violation is associated with the discretized features.

[0123] Report merging steps: Used to export and display compliance reports, and simultaneously store them in the database.

[0124] Optionally, it also includes system management steps, which use parameter configuration and data storage modules to update system settings and back up and manage data from the front end.

[0125] According to one embodiment of the present invention, steps S1-S5 can be executed to check the surveillance video of the confidential location based on the information input by the user from the front end. In particular, the user can also retrieve the content by inputting it. Therefore, the video retrieval operation can be performed after the video data has been structured and analyzed.

[0126] Step S6: The monitoring and inspection operation of classified locations is completed.

[0127] According to an embodiment of the present invention, step S3, the video analysis step, is used to perform multi-channel parallel analysis on the surveillance video to obtain structured video data as video analysis result information, as follows:

[0128] Step S31: Use the video acquisition module to open offline video or read video content from the live stream of the surveillance camera.

[0129] Step S32: Decode the read video stream to obtain the image of each frame of the video.

[0130] Step S33: Use the frame difference method to determine whether the target appears in the frame.

[0131] If a target is detected, proceed with steps S04-S06 for that frame.

[0132] If no target is found, skip the frame and return to step S02.

[0133] Target detection is performed on each frame of the image, and personnel entering classified areas are identified using blacklists and whitelists.

[0134] Step S33: Check the configuration file. When the configuration file requires real-time face comparison, whenever a target is detected, a feature extraction algorithm is used to process it, and its features are compared with the black-and-white face list located in the data storage module. The similarity of the features is evaluated in the form of a score. The configuration file is generated through the location attention parameter configuration module.

[0135] If the similarity score is greater than a certain requirement, the feature matching algorithm determines that the feature matching similarity is greater than a certain threshold, and determines that the two feature arrays describe the same target.

[0136] If the similarity score is less than or equal to a certain requirement, and the feature comparison similarity is less than or equal to a threshold, then it is determined that the two feature arrays do not describe the same target.

[0137] Based on the results of the feature comparison algorithm, individuals on the blacklist and whitelist are identified.

[0138] Step S34: For the detection of illegal items and pedestrian targets, a predetermined algorithm is used for feature extraction, such as the YOLO5s feature extraction algorithm.

[0139] In one embodiment, publicly available and photographed images of illegal items are collected via the internet to construct an illegal items dataset.

[0140] In addition, it collects publicly available pedestrian databases online to construct pedestrian datasets.

[0141] According to an embodiment of the present invention, the YOLO5s model is used to train the model on datasets of illegal items (mobile phones, cameras, computers and other camera devices) and pedestrian datasets, and the trained YOLO5s model is used to extract features from illegal items.

[0142] Step S35: For a target detected in a frame of image, the target tracking algorithm is invoked for analysis. Whenever a target is detected, the algorithm correlates the target across multiple frames based on the target detection results in its neighboring frames. This further includes:

[0143] Step S351: If a connection for a target is successfully established between adjacent frames, it is determined that the target is still in the video frame, and the next frame is analyzed in chronological order.

[0144] If it is impossible to establish a correlation between a target and adjacent frames, and the target is determined to have left the image, the following steps are performed:

[0145] Step S352: Record the moment the target first appears in the video and the moment it leaves the video frame.

[0146] Step S353: Compare all frames appearing in the video with the target and select the best frame. Specifically, the best frame is defined as the frame that occupies the largest area when the target appears in the video.

[0147] Step S354: Extract features from the target.

[0148] Step S355: Based on the time the target appears and the time it leaves the frame, the video is structured, all frames of the target's entry and exit process are analyzed, the trajectory of the target person is generated, and the person is considered to have stored the item if the dwell time at the storage location is greater than a predetermined time.

[0149] Video structuring refers to a deep learning-based extraction technique for video content. The machine extracts information from each frame of the video and stores it in a corresponding structured database. In this invention, video structuring extracts information such as people, number of people, feature analysis, and spatial coordinates from the video frame by frame.

[0150] Step S36: Record pedestrian features under the corresponding face information using the YOLO5s model, that is, collect pedestrian features while recognizing the face image.

[0151] Step S37: Merge the identified target entry and exit feature information into the target's directory, and save the image to the storage device, indexed as the personnel entering and leaving.

[0152] This embodiment uses deep learning to structure the video data and stores the generated fields in a task analysis report. As an example, the structured data might be:

[0153] The data includes: target person name, person screenshot, whether it matches the access card, entry time, exit time, whether there are any anomalies in the video, number of people appearing, type of prohibited items, correlation analysis to determine if there is a violation, and violation judgment result. Among these, by acquiring access control data, it is possible to determine the relevant violation situation during video analysis by combining the access control data with the data.

[0154] Alternatively, in other embodiments, if the access card data has not yet been obtained, then:

[0155] Target Person Name - Person Screenshot - Entry Time - Exit Time - Are there any anomalies in the video? - Number of people appearing - Type of prohibited items - Correlation analysis to determine if it is a violation - Violation determination result;

[0156] Alternatively, in other embodiments, if the focus is on information about prohibited items being carried, then it could be:

[0157] Target Person Name - Person Screenshot - Entry Time - Exit Time - Whether Carrying Illegal Items - Type of Illegal Items - Whether Stored - Is the Track Abnormal? - Is the Stay Time Abnormal? - Correlation Analysis: Is it a Violation? - Violation Judgment Result;

[0158] All the information mentioned above is stored in the database according to the type of target being detected, using the data storage module.

[0159] Step S38: For each decoded image frame, repeat steps S34-S37 until the video analysis process ends. Specifically, during the analysis of video frames, if the algorithm determines that the monitoring information is abnormal, the abnormal information is written to the report file, and an abnormal information warning is issued simultaneously.

[0160] According to an embodiment of the present invention, the video retrieval step involves using a video retrieval module to search for analyzed surveillance videos in a data storage module based on a set retrieval target, including the following retrieval methods:

[0161] Search by name, face, or body condition;

[0162] Search by time period, camera name, and registered item;

[0163] Search by type of violation;

[0164] Search based on the type of information carrier;

[0165] Search based on images provided by the user;

[0166] Export personnel trajectory retrieval and visualization information;

[0167] Preferably, in the video retrieval step, the retrieval results are categorized according to retrieval requirements, sorted by timeline, and include video summary segments of the target to be detected.

[0168] Preferably, the information carrier in this invention includes one or more of the following: a bag, a computer, a mobile phone, and a file carrier.

[0169] Preferably, the retrieval and visualization of personnel entry and exit trajectories includes the following information: entry and exit time, entry and exit frequency, and personnel snapshots.

[0170] In particular, in one implementation, image retrieval can also be performed based on user-provided images, with the following specific steps:

[0171] Step S031: The user imports the images to be detected from the front end and determines the video to be retrieved.

[0172] Step S032: Perform target detection and feature extraction on the image using a feature extraction algorithm.

[0173] Step S033: Connect to the database through the data storage module. For each target detected in the video analysis step and stored in the database, compare its features with the target image. The similarity is evaluated in the form of a score.

[0174] If the similarity score is greater than a certain requirement, the feature comparison algorithm determines that the feature comparison similarity is greater than a certain threshold, and therefore determines that the two feature arrays describe the same target.

[0175] If the similarity score is less than or equal to the requirement, and the feature comparison similarity is less than or equal to the threshold, then it is determined that the two feature arrays do not describe the same target.

[0176] Step S034: Perform the above steps sequentially until the features of the target corresponding to the video to be queried and the image to be queried are completely compared in the database.

[0177] Step S035: Based on the determination result in step S033, for all targets that are determined to be the same target, output their corresponding video summary segments as query results.

[0178] According to one embodiment of the present invention, in step S5, the correlation analysis step, correlation analysis is performed between the video structured data and the violation behavior. Preferably, the correlation analysis process includes the following steps:

[0179] Step S51: Import various reports such as video analysis reports and access control reports. Among them, the video analysis report is the report exported after the required video is analyzed by the structured analysis step (i.e., it contains structured data). In addition, if the imported report does not exist in the database, it can be stored through the data storage module.

[0180] In one embodiment, the video analysis report data includes: person image, name, entry time, exit time, number of people entering, list of personnel, violations, and person in charge.

[0181] The access control report data includes: captured image, cardholder number, cardholder name, cardholder department, card type, door number, cardholder's electronic device card swipe record, storage time, card swipe time, door opening time, operation time, operator number, and operator name.

[0182] After importing the access control data report, the data is stored in the data storage module using the table header as an index.

[0183] In one embodiment, surveillance video needs to be uploaded first for video-based person analysis. Specifically, this includes: first, the video acquisition module and video transcoding module import the video into the data storage module. After importation, the video analysis module performs video structuring and stores the structured information in a person analysis report.

[0184] Step S52, correlation analysis of violations to establish correlations, followed by steps S521-S524 in sequence:

[0185] Step S521: The YOLO5 model is used to train the model on datasets of prohibited items (mobile phones, cameras, computers, and other camera devices) and pedestrians. The training data consists of publicly available and photographed images of prohibited items collected from the internet to construct the prohibited item dataset; a publicly available pedestrian database is collected from the internet to construct the pedestrian dataset.

[0186] Step S522: Using the features extracted by the trained YOLO5s model, illegal items appearing in the surveillance video are detected;

[0187] Step S523: For violations where no illegal items were captured, the YOLO5s model outputs corresponding features but fails to identify them as violations. Therefore, some violations are manually labeled, and the features are discretized using chi-square binning. For example, after manual confirmation, the unidentified violations are labeled.

[0188] Step S524 involves finding relevant features using a predetermined algorithm to compensate for the lack of detection of violations, thereby enabling the identification of some violations that were not detected in previous videos. This includes extracting and discretizing these features, and then performing correlation analysis between the violations and the discretized features.

[0189] Furthermore, during the correlation analysis process, according to an embodiment of the present invention, the method for determining violations is as follows:

[0190] 1) The cardholder's information does not match the access card data, or the cardholder is a blacklisted individual;

[0191] 2) The number of people entering a classified area exceeds a certain limit;

[0192] 3) Internal surveillance cameras in classified areas detected prohibited items, such as cameras, mobile phones, and backpacks;

[0193] 4) Personnel failed to close the door of the classified area when performing classified operations (the time obtained through video analysis will be compared with the time recorded by the access control system; if the door was not closed, the access control system will not record the closing time).

[0194] The preferred behaviors that require special attention are as follows:

[0195] 1) Unidentified personnel entering classified areas;

[0196] 2) Personnel entering classified areas failed to store electronic devices;

[0197] First, the extracted feature values ​​are discretized. The chi-square binning method is used to discretize the features extracted by the YOLO5s pre-trained model. The training data includes violations without detected illegal items, normal behavior, and violations with detected illegal items. The feature values ​​are discretized, and the threshold is set as β, the minimum number of bins Z, and the maximum number of bins M.

[0198] Chi-square binning is a bottom-up merging method. First, each distinct value of the numerical feature is treated as an interval. The chi-square statistic is calculated for each pair of adjacent intervals. A threshold β is set. If the value is less than the threshold, the intervals are merged. This process is repeated recursively until no chi-square statistic is found that is less than the threshold.

[0199] Optionally, the predetermined association analysis method adopts the Apriori algorithm;

[0200] This invention mines features related to violations through association analysis. The YOLO5s model, after training, extracts features that detect illegal items appearing in the frame. However, for violations where no illegal items are captured, the YOLO5s model outputs corresponding features but fails to identify them as violations. Therefore, some violations are manually labeled, and features are discretized using chi-square binning. The Apriori algorithm is then used to find correlated features to compensate for the inaccurate detection of violations. The specific algorithm description is as follows:

[0201] Step S5241: Manually label images of violations for which illegal items could not be identified; output structured features through deep learning (the structured features refer to storing the feature values ​​output by the model as structured data), and use the discretized feature values ​​as the associated field X (representing a class of events). The associated field X includes feature 1, feature 2, ...;

[0202] The relationship between field X and violation Y:

[0203] Support: Support(X,Y) = P(X,Y) = num(xy) / num(allsamples)

[0204] Minimum support: Minimum support is an artificially defined threshold that represents the lowest statistical importance of an itemset.

[0205] Where num(xy) represents the number of times event x and event y occur simultaneously, X represents a class of events, x represents a specific event, and num(allsamples) represents all occurring events. The confidence score is calculated as follows:

[0206] Confidence level: Cinfidence(XY) = P(x|Y) = P(xy) / P(y);

[0207] Minimum confidence level: The minimum confidence level is also a manually defined threshold, representing the lowest reliability of an association rule. Only when both support and confidence reach the minimum support and minimum confidence level can an association rule be considered a strong rule.

[0208] Input: The input is a set of scanned data D, which is a set of behavioral features of people extracted from a large number of collected videos, including violations and non-violations;

[0209] The scanned data set D includes discretized feature values, which are the set of a fixed number of feature values ​​output by the model after feature extraction from the surveillance video and the discretized feature values. The support threshold is α.

[0210] Output: The largest frequent k-itemset;

[0211] Frequent itemsets are those that satisfy the minimum support requirement.

[0212] Step S5242: Scan the entire feature dataset D to obtain all the data that have appeared, and use them as candidate frequent 1-itemsets.

[0213] Step S52243: Mining frequent k-itemsets;

[0214] Step S5244 further scans the data to calculate the support of candidate frequent k-itemsets;

[0215] Step S5245 removes data sets whose support is below a threshold from the candidate frequent k-itemsets, obtaining frequent k-itemsets. If the obtained frequent k-itemsets are empty, the set of frequent k-1 itemsets is directly returned as the algorithm result, and the algorithm ends. If the obtained frequent k-itemsets contain only one item, the set of frequent k-itemsets is directly returned as the algorithm result, and the algorithm ends. Otherwise, proceed to the next step.

[0216] Step S5246 generates candidate frequent k+1 itemsets by connecting frequent k-itemsets.

[0217] Step S5247: Let k = k + 1, then proceed to step S5243.

[0218] Step S53, Report Merging: Export and display the compliance report, and simultaneously store it in the database.

[0219] The generated personnel analysis report can be merged with the access control data report to generate a compliance report. The compliance report combines the access control and personnel analysis results for each target, which will identify illegal items and illegal personnel, the number of illegal acts, and whether they need to be given special attention, generating corresponding structured data.

[0220] Through the above embodiments, this invention can analyze videos captured by cameras in classified locations, achieving structured monitoring video. By training a YOLO5 model for prohibited items, it enables the identification and classification of such items. Identifying prohibited items and individuals indicates the occurrence of violations. Unidentified personnel and those not stored on electronic devices are given special attention. After video feature extraction, the chi-square binning method is used for feature discretization. The discretized features X are then correlated with manually labeled violations using Apriori analysis to identify other features associated with the violations, increasing the violation identification rate and thus achieving efficient management of classified locations. Simultaneously, the developed video inspection system reduces the time consumption and application complexity of video inspection in classified locations, improving the accuracy and efficiency of classified location management.

[0221] Preferably, in one embodiment, the system management step includes the following sub-steps: updating system settings information and managing and backing up data.

[0222] Preferably, the system management steps include the following system settings information:

[0223] Information on personnel handling classified information, including a list of personnel handling classified information, photos of personnel handling classified information, and information on personnel handling classified information and their communication.

[0224] Monitoring information, including information on the connection with the monitoring platform and camera settings;

[0225] Preferably, in the system management steps, the data managed in data management and backup includes: raw monitoring data, structured video data, violation event data, system setting data, and data report data.

[0226] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes will be obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A rapid detection system for violations in surveillance video of specific locations, characterized in that, include: The front-end scheduling and control module is used to receive and execute user-input commands and call the corresponding modules for task processing. The venue attention parameter configuration module is used to configure the venue attention parameters of the venue entrance access control data acquisition unit and the venue attention parameters of the video surveillance data acquisition unit; The video acquisition module is used to acquire the surveillance video to be inspected by importing or directly streaming video streams. The video analysis module is used to perform multi-channel parallel analysis on surveillance videos to obtain structured video data as video analysis results. The data storage module is used to store surveillance video information and video analysis results. The correlation analysis module is used for correlation analysis between structured video data and violations. This module performs correlation analysis after discretizing the features extracted through deep learning, specifically including: Report Association Analysis Module: Used to import various reports such as video analysis reports and access control reports. The video analysis reports are reports exported after video analysis of surveillance videos. If the imported reports do not exist in the database, they are stored through the data storage module. Violation Correlation Analysis Module: After manual verification, unidentified violations are marked. This module extracts and discretizes their features, then performs correlation analysis between the violations and the discretized features. Specifically, it includes: Association analysis is used to mine features related to violations. Based on the features extracted by the trained YOLO5s model, illegal items appearing in the camera are detected. For violation images where no illegal items are captured, some violation images are manually labeled. For the labeled images, deep learning is used to output structured features, and the structured features are discretized using chi-square binning. The discretized features are then correlated with the manually labeled violations using the Apriori algorithm to compensate for the lack of violation detection. Report merging module: Used to export and display compliance reports, and simultaneously store them in the database; The generated personnel analysis report and access control data report are merged to generate a compliance report. The compliance report combines the access control and personnel analysis results for each target, which will identify illegal items and illegal personnel, the number of illegal persons, and whether they need to be given special attention, generating corresponding structured data.

2. The rapid detection system for violations in surveillance video of a specific location according to claim 1, characterized in that: The aforementioned location attention parameter configuration module is used to receive input parameters of confidential personnel and events provided by the front-end scheduling and control module, and to manage personnel, surveillance cameras, and confidential events; import the face list and roster of confidential personnel and store them in the data storage module; set the types of violations; and configure location attention parameters related to video analysis.

3. The rapid detection system for violations in surveillance video of a specific location according to claim 1, characterized in that: The video acquisition module is used to acquire the surveillance video to be inspected by importing or directly streaming video; specifically, it includes: It supports users importing videos or directly connecting to the video surveillance data acquisition unit to download surveillance videos in real time from the network recorder; the video acquisition module obtains the duration, frame rate, resolution, and video encoding format information of the imported video and sends the information to the front-end scheduling and control module; it also includes: The video transcoding module is used to acquire basic information about the surveillance video and analyze the video encoding format. If the video encoding format is not supported for playback, it will be transcoded. Then, the imported videos are automatically categorized according to the cameras and stored in the data storage module.

4. The rapid detection system for violations in surveillance video of a specific location according to claim 1, characterized in that, The video analysis module is used to read videos from the video acquisition module and perform multi-channel video analysis sequentially. During the video analysis, the system analyzes the video frame by frame according to the configuration information related to the location's focus parameters, and writes the structured data obtained after analysis into the video analysis report. During the analysis process, the system provides phased feedback on the video analysis progress, which is displayed on the progress bar of the front-end scheduling and control module. After the video analysis is completed, the structured video is stored through the data storage module, and the video analysis report is exported and stored in the data storage module.

5. The rapid detection system for violations in surveillance video of a specific location according to claim 1, characterized in that, Also includes: The video retrieval module retrieves video summary content based on information provided by the user from the front-end scheduling and control module, according to specified requirements, from the surveillance videos that have undergone structured analysis.

6. A method for rapid inspection of violations in surveillance video of a specific location, characterized in that, Includes the following steps: Step S1, the location attention parameter configuration step, configures the location attention parameters of the location entrance access control data acquisition unit and the location attention parameters of the video surveillance data acquisition unit; Step S2, video acquisition step, acquire the surveillance video to be inspected by importing or directly streaming the video; Step S3, video analysis step, involves performing multi-channel parallel analysis on the surveillance video to obtain structured video data as the video analysis result information; Step S4, data storage step, storing surveillance video information and video analysis results; Step S5, the correlation analysis step, involves performing correlation analysis between the structured video data and the violation; step S5 includes: Step S51, Report Association Analysis: Import various reports such as video analysis report and access control report. The video analysis report is the report exported after video analysis of the surveillance video. If the imported report does not exist in the database, it will be stored through the data storage module. Step S52, Violation Correlation Analysis: After manual confirmation, unidentified violations are marked. The violation correlation analysis module extracts and discretizes their features, and then performs correlation analysis between the violations and the discretized features; specifically including: Association analysis is used to mine features related to violations. Based on the features extracted by the trained YOLO5s model, illegal items appearing in the camera are detected. For violation images where no illegal items are captured, some violation images are manually labeled. For the labeled images, deep learning is used to output structured features, and the structured features are discretized using chi-square binning. The discretized features are then correlated with the manually labeled violations using the Apriori algorithm to compensate for the lack of violation detection. The generated personnel analysis report and access control data report are merged to generate a compliance report. The compliance report combines the access control and personnel analysis results for each target, which will identify illegal items and illegal personnel, the number of illegal persons, and whether they need to be given special attention, generating corresponding structured data.

7. The method for rapid inspection of violations in surveillance video of a specific location according to claim 6, characterized in that, Step S1 includes: It receives input parameters of classified personnel and events from the front-end scheduling and control module, and performs personnel management, surveillance camera management, and classified event management; it imports the face list and roster of classified personnel and stores them in the data storage module; it sets the types of violations; and it configures the location attention parameters related to video analysis.

8. The method for rapid inspection of violations in surveillance video of a specific location according to claim 6, characterized in that, Step S2 includes: Users can import videos or directly interface with the video surveillance data acquisition unit to download surveillance videos in real time from the network recorder; the video acquisition module obtains the duration, frame rate, resolution, and video encoding format information of the imported video and sends the information to the front-end scheduling and control module; it also includes: This is used to acquire basic information about surveillance videos and analyze the video encoding format. If the video encoding format is not supported for playback, it is transcoded. Then, the imported videos are automatically categorized according to the cameras and stored in the data storage module.

9. The method for rapid inspection of violations in surveillance video of a specific location according to claim 6, characterized in that, Step S3 includes: The system reads video from the video acquisition module and performs multi-channel video analysis sequentially. During video analysis, the system analyzes the video frame by frame based on the configuration information related to the location's focus parameters. The structured data obtained after analysis is written into the video analysis report. During the analysis process, the system provides periodic feedback on the video analysis progress, which is displayed on the progress bar of the front-end scheduling and control module. After the video analysis is completed, the structured video is stored through the data storage module, and the video analysis report is exported and stored in the data storage module.

10. A method for rapid inspection of violations in surveillance video of a specific location according to claim 6, characterized in that: The aforementioned violations refer to one or more of the following: unauthorized personnel or non-classified personnel entering the premises; exceeding the number of personnel entering the premises or not matching the number of registered personnel; illegally carrying information carriers; and abnormal monitoring information. Abnormal monitoring information can be caused by one or more of the following: video signal interruption, video signal obstruction, or abnormal lighting.

11. The method for rapid inspection of violations in surveillance video of a specific location according to claim 6, characterized in that, Step S3 includes: Step S31: Open the offline video or read the video content from the real-time streaming video of the surveillance camera through the video acquisition module; Step S32: Decode the read video stream to obtain the image of each frame of the video; Step S33: For a decoded frame of image, use an object detection algorithm to analyze it, detect faces, people, illegal behaviors, and information carrier targets in the image, and record the coordinates of the targets in the image; Step S34: Check the configuration file of the location attention parameters. When the configuration file requires real-time face comparison, whenever a target is detected, the feature extraction algorithm is used to process it and compare it with the black and white face list located in the data storage module. Based on the feature comparison algorithm judgment result, the black and white list persons are judged. The similarity of feature comparison is evaluated in the form of a score. The feature extraction result of a target is a multi-dimensional floating-point array. Step S35: For the target detected in a frame of image, the target tracking algorithm is called for analysis. Whenever a target is detected, the algorithm associates the target across multiple frames based on the target detection results in its adjacent frames. Step S36: For each frame of image obtained by decoding, repeat steps S33-S35 until the video analysis process ends; Step S37: During the analysis of video frames, if the algorithm determines that the monitoring information is abnormal, the abnormal information is written into the report file, and an abnormal information warning is issued at the same time.

12. The method for rapid inspection of violations in surveillance video of a specific location according to claim 11, characterized in that, Step S35 includes: S351, If ​​a correlation is successfully established between adjacent frames for a target, it is determined that the target is still in the video frame, and the next frame is analyzed sequentially along the timeline; or if a correlation cannot be established between adjacent frames for a target, it is determined that the target has left the image, and the following steps are performed: S352, records the moment the target first appears in the video and the moment it leaves the video frame; S353, compare all frames that the target appears in the video and select the best frame. The best frame is defined as the frame that occupies the largest area when the target appears in the video. S354, extract features from the target; S355 performs structured processing on the video based on the time the target appears and the time it leaves the frame; S356. All the above information is stored in the database according to the type of target being detected through the data storage module.

13. The method for rapid inspection of violations in surveillance video of a specific location according to claim 6, characterized in that, Following step S3, the method further includes: The video retrieval process involves searching the analyzed surveillance videos in the data storage module based on the set search target, including the following search methods: Search by name, face, or body condition; Search by time period, camera name, and registered item; Search by type of violation; Search based on the type of information carrier; Search based on images provided by the user; Export personnel trajectory retrieval and visualization information.

14. The method for rapid inspection of violations in surveillance video of a specific location according to claim 13, characterized in that, The video retrieval process involves classifying the results according to the retrieval requirements, sorting them by timeline, and including video summary segments of the target to be detected.

15. The method for rapid inspection of violations in surveillance video of a specific location according to claim 10, characterized in that, The information carrier includes one or more of the following: a bag, a computer, a mobile phone, and a file carrier.

16. The method for rapid inspection of violations in surveillance video of a specific location according to claim 13, characterized in that, The video retrieval process, including the retrieval and visualization of personnel entry and exit trajectories, includes the following information: entry and exit time, entry and exit frequency, and personnel snapshots.

17. The method for rapid inspection of violations in surveillance video of a specific location according to claim 7, characterized in that, The site-specific parameter configuration steps also include: Configure classified personnel management information, including a list of classified personnel, photos of classified personnel, and registration and communication information for classified personnel; Configure monitoring information, including information on the interface with the monitoring platform and camera settings.

Images