A system and method for mutual supplementation and verification of audio and video target stream identification

Abstract

The present invention relates to the field of traffic analysis target detection, and specifically to a system and method for mutually complementary verification of audio and video target stream identification, including a network port acquisition module, a full traffic analysis module, etc., which detects, infers and evaluates audio and video frame by frame in a streaming manner. When the threshold set with the target is reached, the result is dynamically generated to achieve the effect of mutual complementation of target detection. The dual-model reasoning of the voiceprint recognition module and the image target detection module is integrated to increase the accuracy of the target, improve the target detection accuracy, and reduce the false alarm rate through deep learning. In the scenario of traffic analysis, custom-trained audio and video models are introduced to accurately identify the targets of interest. Compared with the basic target recognition of a single model and the post-result set processing recognition, the targets of interest can be proofread more accurately and in real time. Multi-model reasoning improves the efficiency of recognition, increases the adaptability of actual scenarios, and reduces labor costs.

Description

Technical Field

[0001] The present invention relates to the field of traffic analysis target detection, and in particular to a system and method for mutual supplementation and verification of audio and video target stream identification. Background Art

[0002] There are two existing media target recognition solutions:

[0003] (1) Single-model target recognition refers to providing a set of video or audio targets. For example, cameras generate image target recognition results in real time and then output them to the interface. The characteristic of this method is that it allows the system to have single-model indicator recognition capabilities in traffic analysis scenarios, and data segmentation and post-processing can filter out some indicators of interest. However, if the image is blurred or there is severe noise, the score will be very low, and the target may be lost, requiring subsequent manual processing.

[0004] (2) Post-processing and identification of result sets refers to re-analyzing the result sets generated by the corresponding system, such as audio and video result files, to identify keywords, feature values, etc. The characteristics of this method are to use tools or manual methods to re-screen and analyze the result sets. It can improve the accuracy of the results to a certain extent, but the efficiency is relatively low, the labor cost is relatively high, and the result sets need to be stored.

[0005] A Chinese patent with publication number CN114565636A discloses a target association method based on multiple cameras. The accuracy of target recognition is improved by associating the streaming data of each camera with the target and integrating them according to corresponding threshold conditions. However, the model is single and in an emergency, it is impossible to determine the state of the target. Once the person is out of the camera, the target will be lost.

[0006] At the same time, a Chinese patent with publication number CN118485628A discloses a YOLOv7 method, system and device for detecting defects in colored cloth. It prepares data sets and images in advance, and then performs target recognition based on the YOLO model. However, it analyzes the data result set and needs to store data, which results in problems such as exchanging time for space. Summary of the invention

[0007] In view of the above problems, the present invention provides a system and method for mutual supplementation verification of audio and video target stream identification.

[0008] The adopted technical solution is an audio and video target flow recognition and mutual complementation verification system, including a network port acquisition module, a full flow analysis module, a voiceprint recognition module, an image target detection module and a calculation module;

[0009] The network port acquisition module is used to collect network port traffic in real time and forward it to the data bus for use by subsequent modules;

[0010] The full traffic analysis module is used to analyze the traffic collected by the network port collection module, generate communication logs and restore audio and video files;

[0011] The voiceprint recognition module is used to identify the speaker in the audio file generated by the full flow analysis module, and evaluate the speaker recognition weight to obtain the score weight of the audio;

[0012] The image target detection module is used to extract input image features from the video file generated by the full flow analysis module to obtain a score weight of the video;

[0013] The calculation module generates an overall score based on the audio score weight and the video score weight obtained by the voiceprint recognition module and the image target detection module, obtains a final score through a fitting function, and generates a result set based on the final score.

[0014] Optionally, the full traffic analysis module distinguishes the VOIP protocol family protocols through protocol identification, including the SIP signaling stream and the RTP media stream that actually transmits the audio and video stream.

[0015] Optionally, the full-flow analysis module decodes the corresponding matching media stream through the encoding format carried in the SIP, and the encoding format includes AAC, AMR, EVS, G711, G7221, G722, G723, G726, G728, G729, ILBC, OPUS, SILK, SPEEX, VP8, H261, H263, H264, and H265.

[0016] Optionally, the voiceprint recognition module is based on the EcapaTdnn model, and the audio file obtains a real-time inference audio_score score through the EcapaTdnn model.

[0017] Optionally, the image target detection module is based on the YOLOv7 target detection model, and can detect and classify multiple targets in the image through one forward propagation. The video file obtains the implementation reasoning video_score score through the YOLOv7 target detection model.

[0018] Optionally, the calculation module performs weighted processing on the audio_score and the video_score to obtain the Media_score, and the Media_score formula is as follows:

[0019] media_score =(a*audio_score+v*video_score) / 2;

[0020] Where a is the weighting coefficient of audio_score, v is the weighting coefficient of video_score, and the final score Final_score is calculated by Media_score according to the non-threaded fitting formula, and the Final_score formula is as follows:

[0021] f(media_score) = 1 / (1+e -5*(media_score*2-1) ).

[0022] Optionally, when Final_score exceeds a threshold, a result set is generated.

[0023] The present application also provides a method for mutual complementation verification of audio and video target stream identification, which is implemented based on a system for mutual complementation verification of audio and video target stream identification, and includes the following steps:

[0024] S1. The network port collection module collects network port traffic and transmits it to the full traffic analysis module;

[0025] S2. The full traffic analysis module analyzes the traffic collected by the network port collection module, generates communication logs and restores audio and video files;

[0026] S3. Training and prediction of the EcapaTdnn model to generate audio_score scores for audio files;

[0027] S4. Training and prediction of the YOLOv7 model, generating video_score scores from video files;

[0028] S5. The calculation module performs weighted processing on the audio_score and the video_score to obtain the Media_score, and calculates the final score Final_score according to the non-threaded fitting formula;

[0029] S6. Generate target recognition warning processing based on the final score Final_score.

[0030] The benefits of the present invention include:

[0031] 1. Through streaming, the audio (voiceprint) and video (image) are detected, inferred and evaluated frame by frame. When the threshold set with the target is reached, the results are dynamically generated to achieve the effect of complementary target detection. YOLOv7 and EcapaTdnn dual-model reasoning are used to integrate to increase the accuracy of the target, improve the accuracy of target detection, and reduce the false alarm rate through deep learning. In simple terms, it is to introduce custom-trained audio and video models in the scenario of traffic analysis to accurately identify the targets of interest. Compared with the basic target recognition and post-result set processing recognition of a single model, the targets of interest can be proofread more accurately and in real time. Multi-model reasoning improves the efficiency of recognition, increases the adaptability of actual scenarios, and reduces labor costs.

[0032] 2. It can solve the problem of post-restoration of the result set under current traffic analysis and the problem of manual efficiency. By combining audio and video, it improves the accuracy of target alarms and reduces misjudgments. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of a module of a system for mutual complementation and verification of audio and video target stream recognition;

[0034] Figure 2 A flow chart of a method for mutual complementary verification of audio and video target stream identification;

[0035] Figure 3 It is the fitting diagram. DETAILED DESCRIPTION

[0036] The following describes the implementation of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific implementations, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0037] It should be noted that the illustrations provided in the following embodiments are only used to schematically illustrate the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0038] like Figure 1 As shown, an audio and video target flow recognition and mutual complementation verification system includes a network port acquisition module, a full flow analysis module, a voiceprint recognition module, an image target detection module and a calculation module;

[0039] The network port acquisition module is used to collect network port traffic in real time and forward it to the data bus for use by subsequent modules;

[0040] The full traffic analysis module is used to analyze the traffic collected by the network port collection module, generate communication logs and restore audio and video files;

[0041] The voiceprint recognition module is used to identify the speaker in the audio file generated by the full flow analysis module, and evaluate the speaker recognition weight to obtain the score weight of the audio;

[0042] The image target detection module is used to extract input image features from the video file generated by the full flow analysis module to obtain a score weight of the video;

[0043] The calculation module generates an overall score based on the audio score weight and the video score weight obtained by the voiceprint recognition module and the image target detection module, obtains a final score through a fitting function, and generates a result set based on the final score.

[0044] The purpose of this design is to detect, infer and evaluate audio (voiceprint) and video (image) frame by frame in a streaming manner. When the threshold set with the target is reached, the result is generated dynamically to achieve the effect of complementary target detection.

[0045] The voiceprint recognition module and the image target detection module are used for dual model reasoning, which are integrated to increase the accuracy of the target and improve the accuracy of target detection. In addition, the false alarm rate can be reduced through deep learning. In simple terms, it is to introduce custom trained audio and video models in the scenario of traffic analysis to accurately identify the targets of interest. Compared with the basic target recognition and post-result set processing recognition of a single model, the targets of interest can be proofread more accurately and in real time. Multi-model reasoning improves the efficiency of recognition, increases the adaptability of actual scenarios, and reduces labor costs. It can solve the problems of post-result set restoration and labor efficiency in the current traffic analysis. Combined with audio and video, it improves the accuracy of target alarms and reduces misjudgments.

[0046] At the same time, in this embodiment, the full flow analysis module distinguishes the VOIP protocol family protocols through protocol identification, including the SIP signaling stream and the RTP media stream that actually transmits the audio and video stream.

[0047] Furthermore, in this embodiment, the full traffic analysis module decodes the corresponding matching media stream through the encoding format carried in the SIP, and the encoding format includes AAC, AMR, EVS, G711, G7221, G722, G723, G726, G728, G729, ILBC, OPUS, SILK, SPEEX, VP8, H261, H263, H264, and H265.

[0048] At the same time, the voiceprint recognition module is based on the EcapaTdnn model, and the audio file obtains the real-time inference audio_score score through the EcapaTdnn model.

[0049] Furthermore, the image target detection module is based on the YOLOv7 target detection model. It can detect and classify multiple targets in the image through one forward propagation. The video file obtains the inference video_score score through the YOLOv7 target detection model.

[0050] like Figure 3 As shown, the calculation module performs weighted processing on the audio_score and the video_score to obtain the Media_score, and the Media_score formula is as follows:

[0051] media_score =(a*audio_score+v*video_score) / 2;

[0052] Where a is the weighting coefficient of audio_score, v is the weighting coefficient of video_score, and the final score Final_score is calculated by Media_score according to the non-threaded fitting formula, and the Final_score formula is as follows:

[0053] f(media_score) = 1 / (1+e -5*(media_score*2-1) )

[0054] like Figure 2 As shown, this embodiment provides a method for mutual complementation verification of audio and video target stream identification, which is implemented based on a system for mutual complementation verification of audio and video target stream identification, and includes the following steps:

[0055] S1. The network port collection module collects network port traffic and transmits it to the full traffic analysis module;

[0056] S2. The full traffic analysis module analyzes the traffic collected by the network port collection module, generates communication logs and restores audio and video files;

[0057] S3. Training and prediction of the EcapaTdnn model to generate audio_score scores for audio files;

[0058] S4. Training and prediction of the YOLOv7 model, generating video_score scores from video files;

[0059] S5. The calculation module performs weighted processing on the audio_score and the video_score to obtain the Media_score, and calculates the final score Final_score according to the non-threaded fitting formula;

[0060] S6. Generate target recognition warning processing based on the final score Final_score.

[0061] It should be pointed out that in the S3 step, you can first download the Chinese dataset, such as downloading it from https: / / github.com / fighting41love / zhvoice. This dataset contains voice data of 3242 people and more than 1,130,000 voice data in mp3 format. In order to speed up reading, it is converted into wav format for easy reading.

[0062] Then, data annotation is performed to create a data list in the format of <voice file path\tvoice classification label>

[0063] For example:

[0064] dataset / zhvoice / zhmagicdata / 5_895 / 5_895_20170614203758.wav1;

[0065] dataset / zhvoice / zhmagicdata / 5_895 / 5_895_20170614214007.wav2;

[0066] dataset / zhvoice / zhmagicdata / 5_941 / 5_941_20170613151344.wav3;

[0067] dataset / zhvoice / zhmagicdata / 5_941 / 5_941_20170614221329.wav4;

[0068] dataset / zhvoice / zhmagicdata / 5_941 / 5_941_20170616153308.wav5;

[0069] dataset / zhvoice / zhmagicdata / 5_968 / 5_968_20170614162657.wav6;

[0070] dataset / zhvoice / zhmagicdata / 5_968 / 5_968_20170622194003.wav7;

[0071] dataset / zhvoice / zhmagicdata / 5_968 / 5_968_20170707200554.wav8;

[0072] dataset / zhvoice / zhmagicdata / 5_970 / 5_970_20170616000122.wav9;

[0073] Next, we train the model using train.py. This project supports multiple audio preprocessing methods, which can be specified through the parameter feature_method. melspectrogram is the Mel spectrum, and spectrogram is the spectrogram.

[0074] For example: python train.py --gpus=0,1;

[0075] Finally, voiceprint comparison is performed using python infer_recognition.py.

[0076] It should also be pointed out that in the dataset prepared for training the EcapaTdnn model, additional audio data of the target of interest, such as A's voice, can be added, so that the trained model can recognize target A.

[0077] In step S4, first select the data set and then label the data, using labelimg to label the data, as shown below:

[0078] <object-class>Classification index 0,1,2...;

[0079] <x_center> The coordinates of the upper left point x account for the proportion of the overall X;

[0080] <y_center> The coordinate of the upper left y point accounts for the proportion of the overall Y;

[0081] <width>The ratio of the target width to the overall width W;

[0082] <height>The ratio of the target height to the overall height H.

[0083] Then use the model training, such as:

[0084] Use python train.py;

[0085] --workers 8;

[0086] --device 0 ;

[0087] --batch-size 32 ;

[0088] --data data / custom.yaml ;

[0089] --img 640 640 ;

[0090] --cfg cfg / training / yolov7-custom.yaml;

[0091] --weights 'yolov7_training.pt';

[0092] --name yolov7-custom ;

[0093] --hyp data / hyp.scratch.custom.yaml;

[0094] Then perform inference on the model:

[0095] python detect.py;

[0096] --weights yolov7.pt;

[0097] --conf 0.25;

[0098] --img-size 640;

[0099] --source inference / images / test.jpg.

[0100] Similarly, in the dataset prepared for training the YOLOv7 target detection model, additional image data of the target of interest, such as the image of B, can be added, so that the trained model can recognize target B.

[0101] The purpose of this design is that the design idea of ​​this method is to integrate audio and video target recognition for media restoration streams under traffic analysis. You only need to train the corresponding model in advance to obtain stream data under traffic analysis, and then the two models jointly explore the target and identify the target in multiple dimensions to improve the accuracy of recognition and reduce manual verification. In addition, this idea can be deployed on any audio and video streaming restoration device to make inferences, without taking up a lot of memory, storing a lot of window data, and affecting the performance of the device.

[0102] Finally, it should be noted that the above are only preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.< / height> < / width>

Claims

1. A system for mutual supplementation and verification of audio and video target stream identification, characterized in that: It includes network port acquisition module, full traffic analysis module, voiceprint recognition module, image target detection module and calculation module; The network port acquisition module is used to collect network port traffic in real time and forward it to the data bus for use by subsequent modules; The full traffic analysis module is used to analyze the traffic collected by the network port collection module, generate communication logs and restore audio and video files; The voiceprint recognition module is used to identify the speaker in the audio file generated by the full flow analysis module, and evaluate the speaker recognition weight to obtain the score weight of the audio; The image target detection module is used to extract input image features from the video file generated by the full flow analysis module to obtain a score weight of the video; The calculation module generates an overall score based on the audio score weight and the video score weight obtained by the voiceprint recognition module and the image target detection module, obtains a final score through a fitting function, and generates a result set based on the final score; The full flow analysis module distinguishes the VOIP protocol family protocols through protocol identification, including the SIP signaling stream and the RTP media stream that actually transmits the audio and video stream; The voiceprint recognition module is based on the EcapaTdnn model, and the audio file obtains the real-time inference audio_score score through the EcapaTdnn model; The image target detection module is based on the YOLOv7 target detection model, and can detect and classify multiple targets in an image through one forward propagation. The video file obtains the implementation reasoning video_score score through the YOLOv7 target detection model.

2. According to claim 1, a system for mutually complementary verification of audio and video target stream identification is characterized in that: The full traffic analysis module decodes the corresponding matching media stream through the encoding format carried in SIP, and the encoding format includes AAC, AMR, EVS, G711, G7221, G722, G723, G726, G728, G729, ILBC, OPUS, SILK, SPEEX, VP8, H261, H263, H264, and H265.

3. The audio and video target stream identification and mutual supplement verification system according to claim 1 is characterized in that: The calculation module performs weighted processing on the audio_score and the video_score to obtain the Media_score, and the Media_score formula is as follows: media_score =(a*audio_score+v*video_score) / 2; Where a is the weighting coefficient of audio_score, v is the weighting coefficient of video_score, and the final score Final_score is calculated by Media_score according to the non-threaded fitting formula, and the Final_score formula is as follows: f(media_score) = 1 / (1+e -5*(media_score*2-1) )。 4. The audio and video target stream identification and mutual supplement verification system according to claim 3 is characterized in that: When Final_score exceeds the threshold, a result set is generated.

5. A method for mutual complementation verification of audio and video target stream identification, based on the audio and video target stream identification mutual complementation verification system in claim 3 or 4, characterized in that: The following steps are involved: S1. The network port collection module collects network port traffic and transmits it to the full traffic analysis module; S2. The full traffic analysis module analyzes the traffic collected by the network port collection module, generates communication logs and restores audio and video files; S3. Training and prediction of the EcapaTdnn model to generate audio_score scores for audio files; S4. Training and prediction of the YOLOv7 model, generating video_score scores from video files; S5. The calculation module performs weighted processing on the audio_score and the video_score to obtain the Media_score, and calculates the final score Final_score according to the non-threaded fitting formula; S6. Generate target recognition warning processing based on the final score Final_score.

Images