System for identifying illegal car door pulling behavior based on large model multi-mode

Abstract

The invention relates to the technical field of safety monitoring, in particular to a system for identifying illegal car door pulling behaviors based on large model multi-mode. Comprising a monitoring picture data acquisition module, an audio data acquisition module, a data preprocessing module, a data analysis module, a judgment module, a video research and judgment module and a processing and pushing module, and the data preprocessing module is used for preprocessing picture data and audio data and fusing the picture data and the audio data; the data analysis module is used for analyzing illegal car door pulling behaviors in the monitoring area at the time according to the fused data; the video research and judgment module is used for sending out a manual research and judgment request and obtaining a research and judgment result; and the disposal pushing module is used for pushing the research and judgment result to disposal personnel to carry out a disposal process. Through the mode, illegal car door pulling behaviors can be identified and recorded in a night environment, illegal behaviors can be effectively prevented and attacked, and property safety of the public is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of safety monitoring, and in particular to a system for identifying illegal door pulling behaviors based on large model multimodality. Background Art

[0002] With the continuous increase in the number of cars, vehicle safety issues have become increasingly prominent. Among them, there are illegal people who illegally pull car doors at night, causing property losses to car owners and affecting social order.

[0003] With the development of society, urban construction is accelerating, the demand for public safety is increasing, and the coverage area of ​​various surveillance cameras has been significantly expanded. Although surveillance equipment is everywhere in the streets and alleys, most surveillance systems currently rely mainly on post-event video playback to obtain evidence when investigating illegal personnel pulling car doors. This passive surveillance mode has significant shortcomings:

[0004] 1. Poor real-time performance: Most existing systems cannot identify and alarm in real time. Reviewing the video afterwards not only takes a lot of time, but it may also be difficult to detect illegal activities due to blind spots in monitoring or unclear video.

[0005] 2. Passive evidence collection: Looking for video records to obtain evidence after the fact, which often causes property damage and cannot prevent the crime in time.

[0006] 3. Insufficient reliability of evidence: When playing back the video, it is difficult to extract highly traceable information from saboteurs with a strong anti-reconnaissance awareness, making it difficult to hold the saboteurs accountable.

[0007] Therefore, it is necessary to propose a system that can identify and record illegal door-pulling behaviors in nighttime environments. Summary of the invention

[0008] The purpose of the present invention is to provide a system for identifying illegal door-pulling behaviors based on large-model multimodality, which can identify and record illegal door-pulling behaviors in a night environment, effectively prevent and combat illegal behaviors, and protect the property safety of the public.

[0009] To achieve the above-mentioned purpose, the present invention adopts a system for identifying illegal door-pulling behaviors based on large-model multimodality, including a monitoring screen data acquisition module, an audio data acquisition module, a data preprocessing module, a data analysis module, a judgment module, a video analysis module and a disposal push module, wherein the data preprocessing module is respectively connected to the monitoring screen data acquisition module, the audio data acquisition module and the data analysis module, the judgment module is connected to the data analysis module, and the video analysis module is respectively connected to the judgment module and the disposal push module;

[0010] The monitoring screen data acquisition module is used to acquire video data of the monitoring area and extract static picture data from the video data;

[0011] The audio data acquisition module is used to acquire audio data of the monitoring area;

[0012] The data preprocessing module is used to perform preprocessing operations on the image data and the audio data respectively, and to fuse the image data and the audio data according to time;

[0013] The data analysis module is used to analyze the illegal door pulling behaviors in the monitoring area at that time based on the fused data, and output the analysis results;

[0014] The judgment module is used to set the logic value and match the logic value for the analysis result;

[0015] The video analysis module is used to issue a manual analysis request and obtain the analysis result;

[0016] The handling push module is used to push the analysis and judgment results to the handling personnel to carry out the handling process.

[0017] Among them, the data preprocessing module includes an image processing unit, an audio processing unit, a real-time time division unit and a fusion unit. The image processing unit is connected to the monitoring screen data acquisition module, the audio processing unit is connected to the audio data acquisition module, and the fusion unit is respectively connected to the image processing unit, the audio processing unit, the real-time time division unit and the data analysis module.

[0018] The image processing unit is used to perform preprocessing operations on the image data; the preprocessing process includes grayscale, noise removal, contrast enhancement, and normalization;

[0019] The audio processing unit is used to perform preprocessing operations on the audio data; wherein the preprocessing process includes digitization, noise suppression, feature extraction, and standardization;

[0020] The real-time time division unit is used to provide real-time time data;

[0021] The fusion unit is used to fuse the image data and the audio data according to the real-time time data.

[0022] Among them, the data analysis module includes a data set acquisition unit, a question and answer data set unit, a model selection unit, a model fine-tuning unit, a model verification unit and an application analysis unit. The question and answer data set unit is respectively connected to the data set acquisition unit and the model selection unit, the model fine-tuning unit is connected to the model selection unit, the model verification unit is respectively connected to the model fine-tuning unit and the application analysis unit, and the application analysis unit is connected to the data preprocessing module.

[0023] The data set acquisition unit is used to acquire historical image data and audio data and obtain the data set by statistics;

[0024] The question and answer dataset unit is used to make the dataset into a question and answer dataset;

[0025] The model selection unit is used to select a recognition model;

[0026] The model fine-tuning unit is used to fine-tune the recognition model;

[0027] The model verification unit is used to train and verify the recognition model;

[0028] The application analysis unit is used to analyze the illegal door pulling behavior in the monitoring area at that time based on the fused data, and output the analysis result data.

[0029] Among them, the judgment module includes an analysis result data acquisition unit, a logic value matching unit, a logic value conversion unit and an analysis and judgment operation control unit. The analysis result data acquisition unit is connected to the data analysis module, the logic value conversion unit is respectively connected to the analysis result data acquisition unit, the logic value matching unit and the analysis and judgment operation control unit, and the analysis and judgment operation control unit is connected to the video analysis module.

[0030] Wherein, the logic value matching unit is used to set the logic value and assign the corresponding preset analysis result to the logic value;

[0031] The analysis result data acquisition unit is used to acquire analysis result data;

[0032] The logic value conversion unit is used to match the logic value for the analysis result data;

[0033] The analysis and operation control unit is used to output an operation request according to the matching logic value.

[0034] Wherein, the system for identifying illegal door-pulling behaviors based on large model multimodality further comprises a data retention module, and the data retention module is connected to the data preprocessing module;

[0035] The data retention module is used to store data whose identification result is illegal door pulling behavior.

[0036] The data retention module includes a data transmission acquisition unit, a division unit and a storage unit. The data transmission acquisition unit is connected to the data preprocessing module, and the division unit is connected to the data transmission acquisition unit and the storage unit respectively.

[0037] Wherein, the data transmission and acquisition unit is used to acquire data whose identification result is illegal door pulling behavior;

[0038] The division unit is used to establish a main file and divide the picture data and the audio data into the main file according to time;

[0039] The storage unit is used to store the main file.

[0040] The present invention discloses a system for identifying illegal door-pulling behaviors based on large-model multimodality, which adopts the monitoring screen data acquisition module, the audio data acquisition module, the data preprocessing module, the data analysis module, the judgment module, the video analysis module and the disposal push module; wherein the monitoring screen data acquisition module is used to acquire the video data of the monitoring area and extract the static picture data in the video data; the audio data acquisition module is used to acquire the audio data of the monitoring area; the data preprocessing module is used to preprocess the picture data and the audio data respectively, and fuse the picture data and the audio data according to time; the data analysis module is used to analyze the illegal door-pulling behaviors in the monitoring area at the time according to the fused data, and output the analysis results; the judgment module is used to set the logical value and match the logical value for the analysis result; the video analysis module is used to issue a manual analysis request and obtain the analysis result; the disposal push module is used to push the analysis result to the disposal personnel to carry out the disposal process; it can identify and record illegal door-pulling behaviors in a night environment, effectively prevent and crack down on illegal behaviors, and protect the property safety of the public. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 This is a structural principle diagram of the system for identifying illegal door pulling behavior based on large model multimodality of the present invention.

[0043] Figure 2It is a structural principle diagram of the data preprocessing module of the present invention.

[0044] Figure 3 It is a structural principle diagram of the data retention module of the present invention.

[0045] 101-monitoring screen data acquisition module, 102-audio data acquisition module, 200-data preprocessing module, 201-image processing unit, 202-audio processing unit, 203-real-time time division unit, 204-fusion unit, 300-data analysis module, 301-data set acquisition unit, 302-question and answer data set unit, 303-model selection unit, 304-model fine-tuning unit, 305-model verification unit, 306-application analysis unit, 400-judgment module, 401-analysis result data acquisition unit, 402-logical value matching unit, 403-logical value conversion unit, 404-analysis and judgment operation control unit, 500-video analysis and judgment module, 600-disposal push module, 700-data retention module, 701-data transmission and acquisition unit, 702-division unit, 703-storage unit. DETAILED DESCRIPTION

[0046] See also Figure 1 to Figure 3 ,in Figure 1 This is the structural principle diagram of the system for identifying illegal door-pulling behaviors based on large-model multimodality. Figure 2 This is the structural principle diagram of the data preprocessing module. Figure 3 It is the structural principle diagram of the data retention module.

[0047] The present invention provides a system for identifying illegal door-pulling behaviors based on a large model multimodality, comprising a monitoring screen data acquisition module 101, an audio data acquisition module 102, a data preprocessing module 200, a data analysis module 300, a judgment module 400, a video analysis module 500 and a disposal push module 600, wherein the data preprocessing module 200 is respectively connected to the monitoring screen data acquisition module 101, the audio data acquisition module 102 and the data analysis module 300, the judgment module 400 is connected to the data analysis module 300, and the video analysis module 500 is respectively connected to the judgment module 400 and the disposal push module 600;

[0048] The monitoring screen data acquisition module 101 is used to acquire video data of the monitoring area and extract static picture data from the video data;

[0049] The audio data acquisition module 102 is used to acquire audio data of the monitoring area;

[0050] The data preprocessing module 200 is used to perform preprocessing operations on the image data and the audio data respectively, and to fuse the image data and the audio data according to time;

[0051] The data analysis module 300 is used to analyze the illegal door pulling behaviors in the monitoring area at that time based on the fused data, and output the analysis results;

[0052] The judgment module 400 is used to set the logic value and match the logic value for the analysis result;

[0053] The video analysis module 500 is used to issue a manual analysis request and obtain an analysis result;

[0054] The handling push module 600 is used to push the analysis and judgment results to the handling personnel to carry out the handling process.

[0055] In this embodiment, the monitoring screen data acquisition module 101, the audio data acquisition module 102, the data preprocessing module 200, the data analysis module 300, the judgment module 400, the video analysis module 500 and the disposal push module 600 are used; wherein the monitoring screen data acquisition module 101 acquires the video data of the monitoring area and extracts the static image data in the video data; the audio data acquisition module 102 acquires the audio data of the monitoring area; the data preprocessing module 200 preprocesses the image data and the audio data respectively, and fuses the image data and the audio data according to time; the data analysis module 300 analyzes the illegal door pulling behavior in the monitoring area at that time according to the fused data, and outputs the analysis result; the judgment module 400 sets the logical value and matches the logical value for the analysis result; the video analysis module 500 issues a manual analysis request and obtains the analysis result; the disposal push module 600 pushes the analysis result to the disposal personnel to carry out the disposal process; it can identify and record illegal door pulling behaviors in a night environment, effectively prevent and crack down on illegal behaviors, and protect the property safety of the public.

[0056] Furthermore, the data preprocessing module 200 includes an image processing unit 201, an audio processing unit 202, a real-time time division unit 203 and a fusion unit 204, the image processing unit 201 is connected to the monitoring screen data acquisition module 101, the audio processing unit 202 is connected to the audio data acquisition module 102, and the fusion unit 204 is respectively connected to the image processing unit 201, the audio processing unit 202, the real-time time division unit 203 and the data analysis module 300.

[0057] Furthermore, the image processing unit 201 is used to perform preprocessing operations on the image data; wherein the preprocessing process includes grayscale conversion, noise removal, contrast enhancement, and normalization;

[0058] The audio processing unit 202 is used to perform preprocessing operations on the audio data; wherein the preprocessing process includes digitization, noise suppression, feature extraction, and standardization;

[0059] The real-time time division unit 203 is used to provide real-time time data;

[0060] The fusion unit 204 is used to fuse the image data and the audio data according to the real-time time data.

[0061] In this embodiment, the image processing unit 201 performs preprocessing operations on the image data; the preprocessing process includes graying, noise removal, contrast enhancement, and normalization; the audio processing unit 202 performs preprocessing operations on the audio data; the preprocessing process includes digitization, noise suppression, feature extraction, and standardization; the real-time time division unit 203 provides real-time time data; the fusion unit 204 fuses the image data and the audio data according to the real-time time data.

[0062] Furthermore, the data analysis module 300 includes a data set acquisition unit 301, a question and answer data set unit 302, a model selection unit 303, a model fine-tuning unit 304, a model verification unit 305 and an application analysis unit 306, the question and answer data set unit 302 is respectively connected to the data set acquisition unit 301 and the model selection unit 303, the model fine-tuning unit 304 is connected to the model selection unit 303, the model verification unit 305 is respectively connected to the model fine-tuning unit 304 and the application analysis unit 306, and the application analysis unit 306 is connected to the data preprocessing module 200.

[0063] Furthermore, the data set acquisition unit 301 is used to acquire historical image data and audio data and obtain a data set by statistics;

[0064] The question and answer data set unit 302 is used to make the data set into a question and answer data set;

[0065] The model selection unit 303 is used to select a recognition model;

[0066] The model fine-tuning unit 304 is used to fine-tune the recognition model;

[0067] The model verification unit 305 is used to train and verify the recognition model;

[0068] The application analysis unit 306 is used to analyze the illegal door pulling behavior in the monitored area at this time for the fused data, and output the analysis result data.

[0069] In this embodiment, the dataset acquisition unit 301 acquires historical picture data and audio data, and statistically obtains a dataset to improve the recognition model's understanding ability of complex night environment scenes.

[0070] The Q&A dataset unit 302 makes the dataset into a Q&A dataset in the following Json Q&A format:

[0072] {

[0073] "id":"identity_0",

[0074] "conversations":

[0075] {

[0076] "from":"user",

[0077] "value":"Picture1: xxx.jpeg\nIs there a phenomenon of people pulling the car door in the picture? If so, please mark the position where people pull the car door and describe the information of the car, and return it to me in the following Json format.

[0078] {

[0079] "alarm_boxes":[{

[0080] "lfx":10,

[0081] "lfy":20,

[0082] "rbx":30,

[0083] "rby":40,

[0084] "car_plate":"Zhejiang A98765",

[0085] "car_type":"SUV",

[0086] "car_color":"black"}]

[0087] },

[0088] {

[0089] "from":"assistant",

[0090] ​"value":

[0091] {"alarm_boxes":

[0092] {

[0093] "lfx": 11,

[0094] "lfy": 22,

[0095] "rbx": 33,

[0096] "rby": 44,

[0097] "car_plate": "Zhe A 98XXX",

[0098] "car_type": "SUV",

[0099] "car_color": "green"}]

[0100] }}

[0101] }

[0103] The model selection unit 303 selects an identification model and fine-tunes a base model with different numbers of parameters based on the complexity of the vertical domain application scenario. In this embodiment, the Max version with a larger number of parameters is used for fine-tuning. This model has a score of 63.7 in text-oriented visual question answering and has strong graphic and text understanding capabilities.

[0104] The model fine-tuning unit 304 performs fine-tuning processing on the identification model. Based on DeepSpeed and FSD technologies, a multi-server multi-GPUs training script is written to perform full-scale fine-tuning training while consuming as few computer resources as possible.

[0105] The model verification unit 305 trains and verifies the identification model, verifies the model effect on a new data set, selects relatively excellent weights for saving, and applies them to the inference of actual projects.

[0106] The application analysis unit 306 analyzes the illegal door-pulling behavior in the monitored area at this time for the fused data and outputs the analysis result data.

[0107] ​Furthermore, the judgment module 400 includes an analysis result data acquisition unit 401, a logic value matching unit 402, a logic value conversion unit 403 and an analysis and judgment operation control unit 404, the analysis result data acquisition unit 401 is connected to the data analysis module 300, the logic value conversion unit 403 is respectively connected to the analysis result data acquisition unit 401, the logic value matching unit 402 and the analysis and judgment operation control unit 404, and the analysis and judgment operation control unit 404 is connected to the video analysis module 500.

[0108] Furthermore, the logic value matching unit 402 is used to set the logic value and assign the corresponding preset analysis result to the logic value;

[0109] The analysis result data acquisition unit 401 is used to acquire analysis result data;

[0110] The logic value conversion unit 403 is used to match the logic value for the analysis result data;

[0111] The analysis and operation control unit 404 is used to output an operation request according to the matching logic value.

[0112] In this embodiment, the logic value matching unit 402 sets a logic value and assigns a corresponding preset analysis result to the logic value; the analysis result data acquisition unit 401 acquires the analysis result data; the logic value conversion unit 403 matches the logic value to the analysis result data; and the analysis and judgment operation control unit 404 outputs an operation request based on the matching logic value.

[0113] Furthermore, the system for identifying illegal door-pulling behaviors based on large-model multimodality further includes a data retention module 700, and the data retention module 700 is connected to the data preprocessing module 200;

[0114] The data retention module 700 is used to store data whose identification result is illegal door pulling behavior.

[0115] Furthermore, the data retention module 700 includes a data transmission acquisition unit 701, a division unit 702 and a storage unit 703, the data transmission acquisition unit 701 is connected to the data preprocessing module 200, and the division unit 702 is connected to the data transmission acquisition unit 701 and the storage unit 703 respectively.

[0116] Furthermore, the data transmission acquisition unit 701 is used to acquire data whose identification result is illegal door pulling behavior;

[0117] The division unit 702 is used to establish a main file and divide the image data and the audio data into the main file according to time;

[0118] The storage unit 703 is used to store the main file.

[0119] In this embodiment, the data transmission acquisition unit 701 acquires data whose identification result is illegal door pulling behavior; the division unit 702 establishes a master file, and divides the image data and audio data into the master file according to time; the storage unit 703 stores the master file, which can subsequently serve as evidence of the illegal door pulling by the offender, effectively preventing and combating illegal acts, and protecting the property safety of the public.

[0120] What is disclosed above is only one or more preferred embodiments of the present invention, which certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of implementing the above embodiments and making equivalent changes according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A system for identifying illegal door-pulling behaviors based on a large model and multi-modality, characterized in that: It includes a monitoring screen data acquisition module, an audio data acquisition module, a data preprocessing module, a data analysis module, a judgment module, a video analysis module and a disposal push module, wherein the data preprocessing module is respectively connected to the monitoring screen data acquisition module, the audio data acquisition module and the data analysis module, the judgment module is connected to the data analysis module, and the video analysis module is respectively connected to the judgment module and the disposal push module; The monitoring screen data acquisition module is used to acquire video data of the monitoring area and extract static picture data from the video data; The audio data acquisition module is used to acquire audio data of the monitoring area; The data preprocessing module is used to perform preprocessing operations on the image data and the audio data respectively, and to fuse the image data and the audio data according to time; The data analysis module is used to analyze the illegal door pulling behaviors in the monitoring area at that time based on the fused data, and output the analysis results; The judgment module is used to set the logic value and match the logic value for the analysis result; The video analysis module is used to issue a manual analysis request and obtain the analysis result; The handling push module is used to push the analysis and judgment results to the handling personnel to carry out the handling process.

2. The system for identifying illegal door-pulling behaviors based on large-model multimodality as claimed in claim 1 is characterized in that: The data preprocessing module includes an image processing unit, an audio processing unit, a real-time time division unit and a fusion unit. The image processing unit is connected to the monitoring screen data acquisition module, the audio processing unit is connected to the audio data acquisition module, and the fusion unit is respectively connected to the image processing unit, the audio processing unit, the real-time time division unit and the data analysis module.

3. The system for identifying illegal door-pulling behaviors based on large-model multimodality as claimed in claim 2 is characterized in that: The image processing unit is used to perform preprocessing operations on the image data; The preprocessing process includes grayscale, noise removal, contrast enhancement, and normalization; The audio processing unit is used to perform pre-processing operations on the audio data; The preprocessing process includes digitization, noise suppression, feature extraction, and standardization; The real-time time division unit is used to provide real-time time data; The fusion unit is used to fuse the image data and the audio data according to the real-time time data.

4. The system for identifying illegal door-pulling behaviors based on large-model multimodality as claimed in claim 1 is characterized in that: The data analysis module includes a data set acquisition unit, a question and answer data set unit, a model selection unit, a model fine-tuning unit, a model verification unit and an application analysis unit. The question and answer data set unit is respectively connected to the data set acquisition unit and the model selection unit, the model fine-tuning unit is connected to the model selection unit, the model verification unit is respectively connected to the model fine-tuning unit and the application analysis unit, and the application analysis unit is connected to the data preprocessing module.

5. The system for identifying illegal door-pulling behaviors based on large-model multimodality as claimed in claim 4 is characterized in that: The data set acquisition unit is used to acquire historical image data and audio data and obtain a data set by statistics; The question and answer dataset unit is used to make the dataset into a question and answer dataset; The model selection unit is used to select a recognition model; The model fine-tuning unit is used to fine-tune the recognition model; The model verification unit is used to train and verify the recognition model; The application analysis unit is used to analyze the illegal door pulling behavior in the monitoring area at that time based on the fused data, and output the analysis result data.

6. The system for identifying illegal door-pulling behaviors based on large-model multimodality as claimed in claim 1 is characterized in that: The judgment module includes an analysis result data acquisition unit, a logic value matching unit, a logic value conversion unit and an analysis and judgment operation control unit. The analysis result data acquisition unit is connected to the data analysis module, the logic value conversion unit is respectively connected to the analysis result data acquisition unit, the logic value matching unit and the analysis and judgment operation control unit, and the analysis and judgment operation control unit is connected to the video analysis module.

7. The system for identifying illegal door-pulling behaviors based on large-model multimodality as claimed in claim 6 is characterized in that: The logic value matching unit is used to set the logic value and assign the corresponding preset analysis result to the logic value; The analysis result data acquisition unit is used to acquire analysis result data; The logic value conversion unit is used to match the logic value for the analysis result data; The analysis and operation control unit is used to output an operation request according to the matching logic value.

8. The system for identifying illegal door-pulling behaviors based on large-model multimodality as claimed in claim 1 is characterized in that: The system for identifying illegal door-pulling behaviors based on a large model and multiple modes further comprises a data retention module, and the data retention module is connected to the data preprocessing module; The data retention module is used to store data whose identification result is illegal door pulling behavior.

9. The system for identifying illegal door-pulling behaviors based on large-model multimodality as claimed in claim 8, characterized in that: The data retention module includes a data transmission acquisition unit, a division unit and a storage unit. The data transmission acquisition unit is connected to the data preprocessing module, and the division unit is connected to the data transmission acquisition unit and the storage unit respectively.

10. The system for identifying illegal door-pulling behaviors based on large-model multimodality as claimed in claim 9, characterized in that: The data transmission acquisition unit is used to acquire data whose identification result is illegal door pulling behavior; The division unit is used to establish a main file and divide the picture data and the audio data into the main file according to time; The storage unit is used to store the main file.

Images