Large-model multi-mode private wire charging behavior identification system

Abstract

The invention relates to the technical field of safety monitoring, in particular to a behavior recognition system for private wire charging based on large model multi-mode. Comprising a picture data acquisition module, a picture area identification module, a data transmission module, a data initial analysis module, a research and judgment request module, a research and judgment module, a disposal push module and a data storage module. The picture area identification module is used for carrying out area division on a monitoring area according to picture data and identifying a picture area to be monitored; the data initial analysis module analyzes a private wire charging behavior of the to-be-monitored picture area; the research and judgment request module sends out a manual research and judgment request; the research and judgment module carries out manual research and judgment and obtains a research and judgment result; the data storage module stores data that the research and judgment result is a private wire charging behavior; through the above mode, behavior identification of private wire charging is realized, and the monitoring efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of security monitoring technology, and in particular to a system for identifying the behavior of privately pulling electric wires for charging based on a large model and multiple modes. Background Art

[0002] With the development of society, urban construction is accelerating, and the demand for public safety is increasing. Among them, there are illegal acts of privately pulling wires to charge electric vehicles on the roadside and in the alleys. This behavior has the danger of causing fire. At the same time, if someone touches the charging point, it is easy to get an electric shock, which poses a safety hazard.

[0003] The existing monitoring of the behavior of charging by pulling private wires relies on monitoring equipment to transmit images and personnel patrols, which has problems such as difficulty in detection, late detection, and untimely processing; therefore, it is very necessary to propose a system that is conducive to identifying the behavior of charging by pulling private wires and improving monitoring efficiency. Summary of the invention

[0004] The purpose of the present invention is to provide a system for identifying the behavior of privately pulling electric wires for charging based on large models and multi-modality, which solves the problems of relying on monitoring equipment to transmit images and personnel patrolling when monitoring the behavior of privately pulling electric wires for charging, resulting in difficulty in detection, late detection and untimely processing.

[0005] To achieve the above-mentioned purpose, the present invention adopts a system for identifying the behavior of privately pulling wires for charging based on a large model multimodality, including a screen data acquisition module, a screen area recognition module, a data transmission module, a data initial analysis module, a judgment request module, a judgment module, a disposal push module and a data storage module, wherein the screen area recognition module is respectively connected to the screen data acquisition module and the data transmission module, the data initial analysis module is respectively connected to the data transmission module and the judgment request module, the judgment module is connected to the judgment request module, and the disposal push module is respectively connected to the judgment module and the data storage module;

[0006] The picture data acquisition module is used to acquire the video data of the monitoring area and extract the static picture data and audio data in the video data;

[0007] The picture area identification module is used to divide the monitoring area into regions according to the picture data and identify the picture area to be monitored;

[0008] The data transmission module is used to transmit the identified image area to be monitored;

[0009] The data initial analysis module is used to analyze the charging behavior of privately pulled wires in the area of ​​the monitored screen and output the analysis results;

[0010] The analysis and judgment request module is used to issue a manual analysis and judgment request;

[0011] The analysis module is used to perform manual analysis and obtain the analysis results;

[0012] The handling push module is used to push the analysis and judgment results to the handling personnel to carry out the handling process;

[0013] The data storage module is used to store data whose results are determined to be the behavior of charging by pulling electric wires privately.

[0014] Among them, the picture area identification module includes a video frame extraction unit, a region division unit and a region selection unit, the video frame extraction unit is connected to the picture data acquisition module and the region division unit respectively, and the region selection unit is connected to the region division unit and the data transmission module.

[0015] Wherein, the video frame extraction unit is used to extract static picture data from the video data;

[0016] The region division unit is used to divide the image data into multiple sub-regions;

[0017] The area selection unit is used to select a sub-area in which private wire charging can be carried out from among the multiple sub-areas.

[0018] Among them, the initial data analysis module includes a data set acquisition unit, a question and answer data set unit, a model selection unit, a model fine-tuning unit, a model verification unit and an application analysis unit. The question and answer data set unit is connected to the data set acquisition unit, the model selection unit is connected to the question and answer data set unit, the model fine-tuning unit is respectively connected to the model selection unit and the model verification unit, and the application analysis unit is respectively connected to the model verification unit, the data transmission module and the analysis and judgment request module.

[0019] The data set acquisition unit is used to acquire video data of the monitoring area and obtain the data set by statistics;

[0020] The question and answer dataset unit is used to make the dataset into a question and answer dataset;

[0021] The model selection unit is used to select a recognition model;

[0022] The model fine-tuning unit is used to fine-tune the recognition model;

[0023] The model verification unit is used to train and verify the recognition model;

[0024] The application analysis unit is used to analyze the charging behavior of privately pulled wires in the area of ​​the screen to be monitored and output the analysis result data.

[0025] Among them, the analysis and judgment request module includes an analysis data acquisition unit, a logic value judgment unit, a logic value output unit and a request trigger unit. The analysis data acquisition unit is respectively connected to the data initial analysis module and the logic value judgment unit, the logic value output unit is connected to the logic value judgment unit, and the request trigger unit is respectively connected to the logic value judgment unit and the analysis and judgment module.

[0026] Wherein, the analysis data acquisition unit is used to acquire analysis result data;

[0027] The logic value judgment unit is used to assign a logic value to the analysis result data;

[0028] The logic value output unit is used to output a logic value;

[0029] The request triggering unit is used to trigger a manual analysis request according to a logical value.

[0030] The data storage module includes a time extraction unit, a data division unit and a storage unit. The time extraction unit is connected to the disposal push module and the screen data acquisition module respectively, and the data division unit is connected to the time extraction unit and the storage unit respectively.

[0031] Wherein, the time extraction unit is used to extract the time data of the identification result of the behavior of charging by pulling the electric wire privately;

[0032] The data division unit is used to establish a main file and divide the video data into the main file according to the time data;

[0033] The storage unit is used to store the main file.

[0034] The present invention discloses a system for identifying the behavior of privately pulled electric wires for charging based on large-model multimodality, wherein the picture data acquisition module is used to acquire video data of a monitoring area and extract static picture data and audio data from the video data; the picture area identification module is used to divide the monitoring area into regions according to the picture data and identify the picture area to be monitored; the data transmission module is used to transmit the identified picture area to be monitored; the data initial analysis module is used to analyze the behavior of privately pulled electric wires for charging in the picture area to be monitored and output the analysis result; the analysis request module is used to issue a manual analysis request; the analysis module is used to perform manual analysis and obtain the analysis result; the disposal push module is used to push the analysis result to the disposal personnel to carry out the disposal process; the data storage module is used to store data whose analysis result is the behavior of privately pulled electric wires for charging. Through the above-mentioned method, the behavior identification conducive to privately pulled electric wires for charging is realized and the monitoring efficiency is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 It is a schematic diagram of the principle of the system for identifying the behavior of privately pulling wires for charging based on large-model multimodality of the present invention.

[0037] Figure 2 It is a schematic diagram of the principle of the picture area recognition module of the present invention.

[0038] Figure 3 It is a schematic diagram of the principle of the data initial analysis module of the present invention.

[0039] Figure 4 It is a schematic diagram of the principle of the analysis and judgment request module of the present invention.

[0040] Figure 5 It is a principle schematic diagram of the data storage module of the present invention.

[0041] 100-screen data acquisition module, 200-screen area identification module, 201-video frame extraction unit, 202-area division unit, 203-area selection unit, 300-data transmission module, 400-initial data analysis module, 401-data set acquisition unit, 402-question and answer data set unit, 403-model selection unit, 404-model fine-tuning unit, 405-model verification unit, 406-application analysis unit, 500-analysis and judgment request module, 501-analysis data acquisition unit, 502-logical value judgment unit, 503-logical value output unit, 504-request trigger unit, 600-analysis and judgment module, 700-disposal push module, 800-data storage module, 801-time extraction unit, 802-data division unit, 803-storage unit. DETAILED DESCRIPTION

[0042] See also Figure 1 to Figure 5 ,in Figure 1 This is a schematic diagram of the principle of a system for identifying the behavior of privately pulling wires for charging based on a large model and multi-modality. Figure 2 This is a schematic diagram of the principle of the picture area recognition module. Figure 3 This is a schematic diagram of the principle of the data initial analysis module. Figure 4 This is a schematic diagram of the principle of the research and judgment request module. Figure 5 This is a schematic diagram of the data storage module.

[0043] The present invention provides a system for identifying the behavior of privately pulling wires for charging based on a large model multimodality, comprising a screen data acquisition module 100, a screen area recognition module 200, a data transmission module 300, a data initial analysis module 400, a judgment request module 500, a judgment module 600, a disposal push module 700 and a data storage module 800, wherein the screen area recognition module 200 is respectively connected to the screen data acquisition module 100 and the data transmission module 300, the data initial analysis module 400 is respectively connected to the data transmission module 300 and the judgment request module 500, the judgment module 600 is connected to the judgment request module 500, and the disposal push module 700 is respectively connected to the judgment module 600 and the data storage module 800;

[0044] The picture data acquisition module 100 is used to acquire the video data of the monitoring area and extract the static picture data and audio data in the video data;

[0045] The picture area identification module 200 is used to divide the monitoring area into regions according to the picture data and identify the picture area to be monitored;

[0046] The data transmission module 300 is used to transmit the identified image area to be monitored;

[0047] The data initial analysis module 400 is used to analyze the charging behavior of privately pulled wires in the monitored image area and output the analysis results;

[0048] The analysis request module 500 is used to issue a manual analysis request;

[0049] The analysis module 600 is used to perform manual analysis and obtain the analysis result;

[0050] The handling push module 700 is used to push the analysis and judgment results to the handling personnel to carry out the handling process;

[0051] The data storage module 800 is used to store data indicating that the result of the analysis is the behavior of charging by pulling electric wires privately.

[0052] In this embodiment, the screen data acquisition module 100 acquires the video data of the monitoring area and extracts static image data and audio data from the video data; the screen area identification module 200 divides the monitoring area into regions based on the image data and identifies the screen area to be monitored; the data transmission module 300 transmits the identified screen area to be monitored; the data initial analysis module 400 analyzes the privately pulled wire charging behavior in the screen area to be monitored and outputs the analysis result; the analysis request module 500 issues a manual analysis request; the analysis module 600 performs manual analysis and obtains the analysis result; the disposal push module 700 pushes the analysis result to the disposal personnel to carry out the disposal process; the data storage module 800 stores the data whose analysis result is the privately pulled wire charging behavior. Through the above method, the behavior identification that is conducive to the privately pulled wire charging is realized and the monitoring efficiency is improved.

[0053] Furthermore, the picture area identification module 200 includes a video frame extraction unit 201, a region division unit 202 and a region selection unit 203, the video frame extraction unit 201 is connected to the picture data acquisition module 100 and the region division unit 202 respectively, and the region selection unit 203 is connected to the region division unit 202 and the data transmission module 300.

[0054] Further, the video frame extraction unit 201 is used to extract static picture data from the video data;

[0055] The region division unit 202 is used to divide the image data into multiple sub-regions;

[0056] The area selection unit 203 is used to select a sub-area from among the multiple sub-areas where private wire charging can be performed.

[0057] In this embodiment, the video frame extraction unit 201 extracts static image data from the video data; the area division unit 202 divides the image data to obtain multiple sub-areas; the area selection unit 203 selects a sub-area in which private charging by pulling electric wires can be carried out from the multiple sub-areas, and the selection rule may be: selecting the roadside and the side of the corridor, and excluding the center of the road.

[0058] Furthermore, the data initial analysis module 400 includes a data set acquisition unit 401, a question and answer data set unit 402, a model selection unit 403, a model fine-tuning unit 404, a model verification unit 405 and an application analysis unit 406. The question and answer data set unit 402 is connected to the data set acquisition unit 401, the model selection unit 403 is connected to the question and answer data set unit 402, the model fine-tuning unit 404 is respectively connected to the model selection unit 403 and the model verification unit 405, and the application analysis unit 406 is respectively connected to the model verification unit 405, the data transmission module 300 and the analysis and judgment request module 500.

[0059] Furthermore, the data set acquisition unit 401 is used to acquire video data of the monitoring area and obtain the data set by statistics;

[0060] The question and answer data set unit 402 is used to make the data set into a question and answer data set;

[0061] The model selection unit 403 is used to select a recognition model;

[0062] The model fine-tuning unit 404 is used to fine-tune the recognition model;

[0063] The model verification unit 405 is used to train and verify the recognition model;

[0064] The application analysis unit 406 is used to analyze the charging behavior of privately pulled wires in the monitored image area and output analysis result data.

[0065] In this embodiment, the data set acquisition unit 401 acquires video data of the monitoring area and obtains the data set through statistics.

[0066] The question and answer data set unit 402 creates the data set into a question and answer data set, and creates a Json question and answer in the following format: [

[0068] {

[0069] "id":"identity_0",

[0070] "conversations":[

[0071] {

[0072] "from":"user",

[0073] "value":"Picture 1: xxx.jpeg\nIs there any exposed wires and charging based on the electric frequency vehicle in the picture? If so, please mark the location of the violation and return it to me in the following Json format.

[0074] {

[0075] "alarm_boxes":[{

[0076] "lfx":10,

[0077] "lfy":20,

[0078] "rbx":30,

[0079] "rby":40}]

[0080] },

[0081] {

[0082] "from":"assistant",

[0083] "value":

[0084] {"alarm_boxes":[

[0085] {

[0086] "lfx":11,

[0087] "lfy":22,

[0088] "rbx":33,

[0089] "rby":44}]

[0090] }}

[0091] } ]

[0093] The model selection unit 403 selects a recognition model; among them, a base model with different parameter amounts is selected for fine-tuning based on the complexity of the vertical field application scenario. In this embodiment, a Max version with a larger parameter amount is used for fine-tuning. The model has a score of 63.7 in text-oriented visual question answering and has a strong ability to understand pictures and texts.

[0094] The model fine-tuning unit 404 fine-tunes the recognition model, writes a multi-server multi-GPUs training script based on DeepSpeed ​​and FSD technologies, and performs full fine-tuning training while consuming as few computer resources as possible.

[0095] The model verification unit 405 trains and verifies the recognition model; verifies the model effect under the new data set, selects relatively excellent weights to save, and applies them to the reasoning of the actual project.

[0096] The application analysis unit 406 analyzes the charging behavior of privately pulled wires in the monitored image area and outputs analysis result data.

[0097] Furthermore, the analysis and judgment request module 500 includes an analysis data acquisition unit 501, a logic value judgment unit 502, a logic value output unit 503 and a request trigger unit 504, the analysis data acquisition unit is respectively connected to the data initial analysis module 400 and the logic value judgment unit 502, the logic value output unit 503 is connected to the logic value judgment unit 502, and the request trigger unit 504 is respectively connected to the logic value judgment unit 502 and the analysis and judgment module 600.

[0098] Furthermore, the analysis data acquisition unit 501 is used to acquire analysis result data;

[0099] The logic value judgment unit 502 is used to assign a logic value to the analysis result data;

[0100] The logic value output unit 503 is used to output a logic value;

[0101] The request triggering unit 504 is used to trigger a manual analysis request according to a logic value.

[0102] In this embodiment, the logic value judgment unit 502 assigns a logic value to the analysis result data. For example, if the analysis result shows that there is private charging behavior by pulling wires, the logic value is set to 1; if the analysis result shows that there is no private charging behavior by pulling wires, the logic value is set to 0; the logic value output unit 503 outputs a logic value; the request trigger unit 504 triggers a manual judgment request according to the logic value. For example, a logic value of 1 triggers a manual judgment request, and a logic value of 0 continues monitoring.

[0103] Furthermore, the data storage module 800 includes a time extraction unit 801, a data division unit 802 and a storage unit 803, the time extraction unit 801 is respectively connected to the disposal push module 700 and the screen data acquisition module 100, and the data division unit 802 is respectively connected to the time extraction unit 801 and the storage unit 803.

[0104] Furthermore, the time extraction unit 801 is used to extract the time data of the identification result of the behavior of charging by pulling the electric wire privately;

[0105] The data division unit 802 is used to establish a main file and divide the video data into the main file according to the time data;

[0106] The storage unit 803 is used to store the main file.

[0107] In this embodiment, the time extraction unit 801 is used to extract the time data of the identification result of the behavior of privately pulling wires for charging; the data division unit 802 is used to establish a master file and divide the video data into the master file according to the time data; the storage unit 803 is used to store the master file; the master file can subsequently become evidence of the illegal person's privately pulling wires for charging, effectively preventing and combating illegal acts, and protecting the property and life safety of the public.

[0108] What is disclosed above is only one or more preferred embodiments of the present invention, which certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of implementing the above embodiments and making equivalent changes according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A system for identifying the behavior of charging by pulling private wires based on large models and multi-modality, characterized in that: It includes a picture data acquisition module, a picture area recognition module, a data transmission module, a data initial analysis module, a judgment request module, a judgment module, a disposal push module and a data storage module, wherein the picture area recognition module is respectively connected to the picture data acquisition module and the data transmission module, the data initial analysis module is respectively connected to the data transmission module and the judgment request module, the judgment module is connected to the judgment request module, and the disposal push module is respectively connected to the judgment module and the data storage module; The picture data acquisition module is used to acquire the video data of the monitoring area and extract the static picture data and audio data in the video data; The picture area identification module is used to divide the monitoring area into regions according to the picture data and identify the picture area to be monitored; The data transmission module is used to transmit the identified image area to be monitored; The data initial analysis module is used to analyze the charging behavior of privately pulled wires in the area of ​​the monitored screen and output the analysis results; The analysis and judgment request module is used to issue a manual analysis and judgment request; The analysis module is used to perform manual analysis and obtain the analysis results; The handling push module is used to push the analysis and judgment results to the handling personnel to carry out the handling process; The data storage module is used to store data whose results are determined to be the behavior of charging by pulling electric wires privately.

2. The system for identifying the behavior of charging by pulling private wires based on large model multimodality as claimed in claim 1, characterized in that: The picture area recognition module includes a video frame extraction unit, a region division unit and a region selection unit. The video frame extraction unit is connected to the picture data acquisition module and the region division unit respectively, and the region selection unit is connected to the region division unit and the data transmission module.

3. The system for identifying the behavior of charging by privately pulling electric wires based on large model multimodality as claimed in claim 2, characterized in that: The video frame extraction unit is used to extract static picture data from the video data; The region division unit is used to divide the image data into multiple sub-regions; The area selection unit is used to select a sub-area from among the multiple sub-areas where private wire charging can be performed.

4. The system for identifying the behavior of charging by privately pulling electric wires based on large model multimodality as claimed in claim 1, characterized in that: The initial data analysis module includes a data set acquisition unit, a question and answer data set unit, a model selection unit, a model fine-tuning unit, a model verification unit and an application analysis unit. The question and answer data set unit is connected to the data set acquisition unit, the model selection unit is connected to the question and answer data set unit, the model fine-tuning unit is respectively connected to the model selection unit and the model verification unit, and the application analysis unit is respectively connected to the model verification unit, the data transmission module and the analysis and judgment request module.

5. The system for identifying the behavior of charging by pulling private wires based on large model multimodality as claimed in claim 4, characterized in that: The data set acquisition unit is used to acquire video data of the monitoring area and obtain the data set by statistics; The question and answer dataset unit is used to make the dataset into a question and answer dataset; The model selection unit is used to select a recognition model; The model fine-tuning unit is used to fine-tune the recognition model; The model verification unit is used to train and verify the recognition model; The application analysis unit is used to analyze the charging behavior of privately pulled wires in the monitored image area and output analysis result data.

6. The system for identifying the behavior of charging by pulling private wires based on large model multimodality as claimed in claim 1, characterized in that: The analysis and judgment request module includes an analysis data acquisition unit, a logic value judgment unit, a logic value output unit and a request trigger unit. The analysis data acquisition unit is respectively connected to the data initial analysis module and the logic value judgment unit, the logic value output unit is connected to the logic value judgment unit, and the request trigger unit is respectively connected to the logic value judgment unit and the analysis and judgment module.

7. The system for identifying the behavior of charging by pulling private wires based on large model multimodality as claimed in claim 6, characterized in that: The analysis data acquisition unit is used to acquire analysis result data; The logic value judgment unit is used to assign a logic value to the analysis result data; The logic value output unit is used to output a logic value; The request triggering unit is used to trigger a manual analysis request according to a logical value.

8. The system for identifying the behavior of charging by pulling private wires based on large model multimodality as claimed in claim 1, characterized in that: The data storage module includes a time extraction unit, a data division unit and a storage unit. The time extraction unit is connected to the disposal push module and the screen data acquisition module respectively, and the data division unit is connected to the time extraction unit and the storage unit respectively.

9. The system for identifying the behavior of charging by pulling private wires based on large-model multimodality as claimed in claim 8, characterized in that: The time extraction unit is used to extract the time data of the identification result of the behavior of charging by pulling the electric wire privately; The data division unit is used to establish a main file and divide the video data into the main file according to the time data; The storage unit is used to store the main file.

Images