Video stream pushing method and system based on target detection visual annotation

By using the YOLOv5 model for object detection and labeling in the video surveillance system, the problems of inefficiency and error-prone traditional annotation methods are solved, efficient and accurate video object detection and labeling are achieved, and video playback performance and quality are improved.

CN119071522APending Publication Date: 2024-12-03GUIZHOU FANLIAN INFORMATION TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411103559.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The traditional visual annotation method is used to label a single image frame by frame. After the label is marked, the stream is encoded and pushed through frame by frame, resulting in poor performance. The video frame rate after the player plays the visual annotation is low, and manual workload is increased, making it easy to cause labeling errors. Impacts video quality.

Method used

Use a video surveillance camera to collect urban environment videos, preprocess and train the YOLOv5 model for object detection, generate visual target annotations, encode and synchronously push to the client through the RTP protocol, the client decodes and marks them in real time in video frames, and stores them in in the database.

Benefits of technology

It improves the accuracy and efficiency of video object detection, improves the frame rate performance of video playback after labeling, reduces labeling errors, and improves the quality of the overall visual annotation video.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071522A_ABST
    Figure CN119071522A_ABST
Patent Text Reader

Abstract

The invention discloses a video stream pushing method and system based on target detection visual annotation, and relates to the technical field of YY. The method comprises the steps that a video monitoring camera is used for collecting and preprocessing urban basic environment videos; and acquiring a city specific environment video to train a YOLOv5 model, carrying out city basic environment video target detection based on the YOLOv5 model, generating a visual target label, and carrying out optimization. The method comprises the following steps: collecting a city basic environment video and a city specific environment video, carrying out YOLOv5 model training by using the city specific environment video, carrying out target detection on the city basic environment video based on the trained YOLOv5 model to generate target labeling information, embedding the target labeling information into an RTP packet header file, and carrying out synchronous coding transmission with the video. And after decoding, the target labeling information is labeled in the video, so that the accuracy and efficiency of video target detection are improved, and the playing frame rate performance of the labeled video is greatly improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of video annotation and streaming, and in particular to a video streaming method and system based on target detection and visual annotation. Background Art

[0002] In recent years, with the rapid development of computer vision and deep learning technologies, video surveillance and target detection technologies have been widely used in smart city construction. Video surveillance systems collect a large amount of video data through cameras, providing important information support for urban management. However, traditional visual annotation can only annotate a single image frame by frame, and then encode and stream it frame by frame after annotation. This method has poor performance. The fps of the video played by the player after visual annotation may not even reach 15 frames. In addition, annotating a single image frame by frame and encoding and streaming it frame by frame after annotation greatly increases the workload, which is prone to annotation errors, thus affecting the quality of the overall visual annotation video. Summary of the invention

[0003] In view of the problems existing in the above-mentioned existing video streaming methods and systems based on target detection and visual annotation, the present invention is proposed.

[0004] Therefore, the problem to be solved by the present invention is that traditional visual annotation can only annotate a single image frame by frame, and then encode and stream it frame by frame. The performance of this method is very poor. The fps of the video played by the player after visual annotation may not even reach 15 frames. In addition, annotating a single image frame by frame greatly increases the manual workload and is prone to annotation errors, thereby affecting the quality of the overall visual annotation video.

[0005] To solve the above technical problems, the present invention provides the following technical solutions: a video streaming method based on target detection and visual annotation, which includes: using a video surveillance camera to collect and preprocess a city's basic environment video; obtaining a city's specific environment video to train a YOLOv5 model, performing city's basic environment video target detection based on the YOLOv5 model, generating and optimizing a visual target annotation; encoding the target annotation information into an RTP packet header, compressing and encoding the city's basic environment video, and synchronously streaming the RTP packet header and the encoded video to a client through the RTP protocol; the client decodes the encoded video and the target annotation information, and annotates the target annotation information in the video frame in real time; the client stores the decoded video and the target annotation information.

[0006] As a preferred solution of the video streaming method based on target detection and visual annotation described in the present invention, the use of video surveillance cameras to collect urban basic environment videos and perform preprocessing refers to randomly selecting urban basic environments and using high-definition surveillance cameras to shoot videos, adjusting the frame rate of the collected urban basic environment videos to 30fps, and using a Gaussian filter for denoising, converting the denoised video frames into RGB format, and adjusting the size to 640*640 pixels.

[0007] As a preferred solution of the video streaming method based on target detection and visual annotation described in the present invention, wherein: the acquisition of urban specific environment video training YOLOv5 model, the urban basic environment video target detection based on the YOLOv5 model refers to selecting a video shot in an urban specific environment, preprocessing the urban specific environment video, and annotating the target objects in the specific environment to generate training data, enhancing the training data by rotation, scaling and flipping, inputting the training data into the YOLOv5 model, defining a loss function and an Adam optimizer to iteratively train the model parameters until the loss is minimized and then stopping the iteration to obtain a trained YOLOv5 model, the YOLOv5 model divides the input video image into S*S grid units, and each grid unit predicts B target bounding box parameters, target category probability and target bounding box confidence P, and the YOLOv5 model includes three parts: Backbone, Neck and Head:

[0008] Backbone uses the CSPDarknet53 structure, which includes multiple convolutional layers, a normalization layer, and an activation function layer;

[0009] Neck uses the PANet structure, which includes upsampling layers, convolutional layers, and feature fusion layers;

[0010] Head uses YOLO Layer, which contains multiple convolutional layers;

[0011] The preprocessed urban basic environment video is input into the Backbone part to extract video image features, the extracted video image features are input into the Neck part for feature fusion, and the fused features are input into the Head part to output B target bounding box parameters, target category probability and target bounding box confidence P, and the target label b is determined according to the target category probability.

[0012] As a preferred solution of the video streaming method based on target detection visual annotation of the present invention, wherein: the generating and optimizing of visual target annotation refers to extracting the target bounding box coordinates (x, y, w, h) and the target label b from the target bounding box parameters and target category output by the YOLOv5 model, wherein x and y are the coordinates of the center point of the target bounding box, w and h are the width and height of the target bounding box, generating the target annotation according to the target label b and the target bounding box, and attaching timestamp information to each target annotation;

[0013] Calculate the coordinates of the upper left corner and lower right corner of the target bounding box according to the coordinates of the center point and the width and height of the target bounding box, and calculate the intersection and union ratio of adjacent target bounding boxes:

[0014]

[0015] Where IoU is the intersection over union ratio, Y and Z are adjacent target bounding boxes, is the intersection area of ​​Y and Z, is the area of ​​the union of Y and Z;

[0016] The intersection-over-union ratio threshold is set. When the intersection-over-union ratio of adjacent target bounding boxes is greater than the intersection-over-union ratio threshold, the confidence P of the adjacent target bounding boxes is compared, the target bounding boxes with low confidence and the corresponding target annotations are removed, and all target bounding boxes are traversed to complete the target annotation optimization.

[0017] As a preferred solution of the video streaming method based on target detection and visual annotation described in the present invention, the target annotation information is encoded and placed in the RTP packet header, and the urban basic environment video is compressed and encoded, and the RTP packet header and the encoded video are synchronously pushed to the client through the RTP protocol, which means that the optimized target annotation information is converted into a string format and embedded in the RTP packet header file after Base64 encoding, and the encoding parameters are set, the urban basic environment video frames are compressed and encoded using an encoder to form an encoded video, and the RTP transmission parameters are configured to synchronously push the encoded video and the RTP packet header file to the client.

[0018] As a preferred solution of the video streaming method based on target detection and visual annotation described in the present invention, wherein: the client decodes the encoded video and target annotation information, and annotates the target annotation information in the video frame in real time, which means that the client uses a decoder to decode the encoded video to obtain the city basic environment video, and extracts the target annotation information from the RTP packet header file for decoding, and converts the decoded character string into the target annotation information, aligns the city basic environment video and the target annotation information according to the timestamp information of the target annotation information, and draws a boundary box and a label on the city basic environment video frame in real time according to the target annotation information for synchronous display.

[0019] As a preferred solution of the video streaming method based on target detection and visual annotation described in the present invention, wherein: the client stores the decoded video and target annotation information, which means that the client stores the decoded video and target annotation information in a database, and implements security protection measures for the stored data. The database backs up the stored data in the cloud, regularly performs integrity checks on the stored data and the backup data, and stores the detection results synchronously.

[0020] Another object of the present invention is to provide a video streaming system based on target detection and visual annotation, which includes a video acquisition module for acquiring urban basic environment videos and urban specific environment videos, and preprocessing the acquired videos;

[0021] The target detection module is used to train the YOLOv5 model using the urban characteristic environment video, and detect the target in the urban basic environment video based on the trained YOLOv5 model;

[0022] The target annotation module is used to generate and optimize target annotation information based on the target detection output of the YOLOv5 model;

[0023] The encoding and transmission module is used to encode the basic urban environment video and target annotation information and transmit them through the RTP protocol;

[0024] The decoding and annotation module is used to decode the encoded video and target annotation information and perform time synchronization, and annotate the target annotation information in the basic urban environment video in real time;

[0025] The data storage module is used to store data in the database and implement security protection measures.

[0026] A computer device comprises: a memory and a processor; the memory stores a computer program, and the processor implements the steps of the above-mentioned video streaming method based on target detection and visual annotation when executing the computer program.

[0027] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned video streaming method based on target detection and visual annotation.

[0028] The beneficial effects of the present invention are as follows: the present invention collects urban basic environment videos and urban specific environment videos and uses the urban specific environment videos to train the YOLOv5 model, performs target detection on the urban basic environment videos based on the trained YOLOv5 model to generate target annotation information, embeds the target annotation information into an RTP packet header file and transmits it synchronously with the video encoding, and annotates the target annotation information in the video after decoding, thereby improving the accuracy and efficiency of video target detection and greatly improving the frame rate performance of the video playback after annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0030] Figure 1 The figure is a flowchart of a video streaming method based on target detection and visual annotation.

[0031] Figure 2 A flowchart for annotating video streaming.

[0032] Figure 3 This is a structural diagram of the video streaming system based on target detection and visual annotation. DETAILED DESCRIPTION

[0033] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0034] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0035] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.

[0036] Example 1

[0037] Reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, and which provides a video streaming method based on target detection and visual annotation. The video streaming method based on target detection and visual annotation includes:

[0038] S1, use video surveillance cameras to collect basic urban environment videos and perform preprocessing;

[0039] Specifically, using a video surveillance camera to collect and preprocess the city's basic environment videos refers to randomly selecting a city's basic environment to shoot videos using a high-definition surveillance camera, adjusting the frame rate of the collected city's basic environment videos to 30fps, and using a Gaussian filter to perform denoising, converting the denoised video frames into RGB format, and adjusting the size to 640*640 pixels.

[0040] Collecting video data through high-definition surveillance cameras can provide high-resolution and high-quality image information, laying the foundation for subsequent video analysis. The frame rate is adjusted to 30fps to ensure the smoothness and real-time nature of the video, which is convenient for real-time monitoring and target detection. The denoising process of the Gaussian filter effectively reduces the noise in the video and improves the clarity and stability of the video image. The conversion of RGB format and the unification of image size standardize the image data, which is convenient for subsequent image processing and analysis. The formatting and optimization of video data improves the standardization of image data, which is convenient for subsequent processing and analysis. The conversion of RGB format ensures the integrity and accuracy of image color information, providing a reliable data basis for subsequent target detection. The size adjustment unifies the image size, reduces the complexity of image processing of different sizes, and improves the efficiency and consistency of image processing.

[0041] S2. Obtain a city-specific environment video to train a YOLOv5 model, perform target detection in the city's basic environment video based on the YOLOv5 model, generate visual target annotations, and optimize them;

[0042] Specifically, obtain a city-specific environment video to train the YOLOv5 model. Performing city basic environment video target detection based on the YOLOv5 model means selecting a city-specific environment video, preprocessing the city-specific environment video, and annotating the target objects in the specific environment to generate training data. The training data is enhanced by rotation, scaling, and flipping. The training data is input into the YOLOv5 model, and the loss function and Adam optimizer are defined to iteratively train the model parameters until the loss is minimized and the iteration is stopped to obtain the trained YOLOv5 model. The YOLOv5 model divides the input video image into S*S grid cells. Each grid cell predicts B target bounding box parameters, target category probability, and target bounding box confidence P. The YOLOv5 model includes three parts: Backbone, Neck, and Head:

[0043] Backbone uses the CSPDarknet53 structure, which contains multiple convolutional layers, a normalization layer, and an activation function layer. Backbone is responsible for extracting image features. The CSPDarknet53 structure consists of multiple ResNet residual blocks, each of which contains multiple convolutional layers and a batch normalization layer.

[0044] Neck uses the PANet structure, which includes an upsampling layer, a convolution layer, and a feature fusion layer. Neck is used to fuse features of different scales. The PANet structure consists of an upsampling layer, a convolution layer, and a feature fusion layer. The upsampling layer is used to upsample the low-resolution feature map to a high resolution and fuse it with the high-resolution feature map.

[0045] Head uses YOLO Layer, which contains multiple convolutional layers. Head generates the final prediction result. YOLO Layer contains multiple convolutional layers to generate predictions for each grid cell. Specifically, each grid cell predicts B target bounding boxes, and each target bounding box contains 4 target bounding box parameters (x, y, w, h), 1 confidence and target category probability;

[0046] The preprocessed urban basic environment video is input into the Backbone part to extract video image features, the extracted video image features are input into the Neck part for feature fusion, and the fused features are input into the Head part to output B target bounding box parameters, target category probability and target bounding box confidence P, and the target label b is determined according to the target category probability.

[0047] By shooting and preprocessing videos of specific urban environments, the diversity and quality of training data are guaranteed. Data enhancement technology further increases the number of training samples and improves the generalization and robustness of the model. The Adam optimizer is used for model training to accelerate the convergence of the model and reduce the training time. The YOLOv5 model finally obtained has the characteristics of high precision and efficiency, and can accurately detect targets in complex urban environments. The Backbone part of the CSPDarknet53 structure can efficiently extract image features and improve the detection accuracy of the model. The Neck part of the PANet structure enhances the model's detection ability for targets of different sizes through multi-scale feature fusion. The Head part of the YOLO Layer generates accurate target bounding boxes and category prediction results by processing multiple convolutional layers in parallel, which improves the speed and accuracy of target detection. Through the target detection process of the YOLOv5 model, efficient and accurate detection of target objects in basic urban environment videos can be achieved. The process of feature extraction, feature fusion and final prediction result generation is closely connected, which improves the efficiency and accuracy of target detection. Based on the detection results of the YOLOv5 model, visual annotations can be generated in real time, providing strong technical support for urban management and monitoring.

[0048] Further, generating and optimizing visual target annotations refers to extracting target bounding box coordinates (x, y, w, h) and target label b from the target bounding box parameters and target categories output by the YOLOv5 model, where x and y are the coordinates of the center point of the target bounding box, w and h are the width and height of the target bounding box, generating target annotations based on the target label b and the target bounding box, and attaching timestamp information to each target annotation;

[0049] Calculate the coordinates of the upper left corner and lower right corner of the target bounding box based on the coordinates of the center point and the width and height of the target bounding box:

[0050]

[0051]

[0052] in and is the coordinate of the upper left corner of the target bounding box, and is the coordinate of the lower right corner of the target bounding box;

[0053] Calculate the intersection-over-union ratio of adjacent object bounding boxes:

[0054]

[0055] Where IoU is the intersection over union ratio, Y and Z are adjacent target bounding boxes, is the intersection area of ​​Y and Z, is the area of ​​the union of Y and Z, where the width and height of the intersection area are:

[0056]

[0057]

[0058] in and The width and height of the intersection area are used to calculate the area of ​​the intersection area. , and calculate the area of ​​the union region as:

[0059]

[0060] in is the area of ​​the target bounding box Y, is the area of ​​the target bounding box Z;

[0061] The intersection-over-union ratio threshold is set. When the intersection-over-union ratio of adjacent target bounding boxes is greater than the intersection-over-union ratio threshold, the confidence P of the adjacent target bounding boxes is compared, the target bounding boxes with low confidence and the corresponding target annotations are removed, and all target bounding boxes are traversed to complete the target annotation optimization.

[0062] By generating visual target annotations, the position and category of the target object in the video frame can be intuitively displayed, which is convenient for monitoring and analysis. Adding timestamp information ensures that the target annotation is synchronized with the video content, which improves the timeliness and traceability of the data. The calculation of intersection over union and the optimization of the target bounding box reduce redundant annotations and improve the accuracy and reliability of target detection. By calculating IoU and optimizing the bounding box, the redundant bounding boxes with large overlap can be effectively removed to improve the accuracy of the detection results. The optimized target bounding box can more accurately reflect the position and size of the actual target object, reduce the occurrence of false positives and false negatives, and the confidence comparison and removal of redundant bounding boxes make the target detection results more reliable, which improves the stability and reliability of the system. The addition of timestamp information gives the target annotation a time dimension, which can accurately reflect the appearance time and position of the target object in the video. Through timestamp information, the target annotation can be accurately synchronized with the video frame, which is convenient for subsequent analysis and tracking. Timestamp information also provides basic data support for multi-target tracking and behavior analysis, improving the intelligence and accuracy of the video surveillance system.

[0063] S3, encode the target marking information into the RTP packet header, compress and encode the basic urban environment video, and synchronously push the RTP packet header and the encoded video to the client through the RTP protocol;

[0064] Specifically, the target annotation information is encoded and placed in the RTP packet header, and the basic urban environment video is compressed and encoded, and the RTP packet header and the encoded video are synchronously pushed to the client through the RTP protocol. This means converting the optimized target annotation information into a string format and embedding it into the RTP packet header file after using Base64 encoding, setting encoding parameters, using an encoder to compress and encode the basic urban environment video frames to form an encoded video, and configuring the RTP transmission parameters to synchronously push the encoded video and the RTP packet header file to the client.

[0065] Through stringification and Base64 encoding, the target annotation information is converted into a format that is easy to transmit and parse, ensuring the integrity and reliability of the data during network transmission. It is embedded in the RTP packet header file to achieve synchronous transmission of the target annotation information and video data, improving the system's integration and real-time performance. Through video compression encoding, the amount of video data is significantly reduced, the transmission efficiency is improved, and the network bandwidth occupancy is reduced. The efficient compression algorithm of the H.264 and H.265 encoding formats can minimize the size of video files while ensuring video quality, improve transmission and storage efficiency, and configure the RTP transmission parameters to ensure the correctness of the target address and transmission channel of the video stream transmission, avoid data loss and transmission delays, and use the RTP protocol to push the stream synchronously, realizing the synchronous transmission of video data and annotation information, ensuring that the client can receive the complete video stream and target annotation information in real time, and improving the real-time performance and reliability of the system.

[0066] S4, the client decodes the encoded video and the target annotation information, and annotates the target annotation information in the video frame in real time;

[0067] Specifically, the client decodes the encoded video and target annotation information, and annotates the target annotation information in the video frame in real time. The client uses a decoder to decode the encoded video to obtain the city basic environment video, extracts the target annotation information from the RTP packet header file for decoding, and converts the decoded character string into the target annotation information, aligns the city basic environment video and the target annotation information according to the timestamp information of the target annotation information, and draws a boundary box and a label on the city basic environment video frame in real time according to the target annotation information for synchronous display.

[0068] The video and target annotation information are decoded by the decoder to ensure the integrity and accuracy of the video and annotation data. The decoded target annotation information can accurately reflect the position and category of the target object in the video frame, providing basic data support for subsequent synchronous display. Through timestamp alignment, the synchronization of target annotation information and video content is ensured, improving the timeliness and accuracy of data. The real-time drawing of bounding boxes and labels enables the target objects to be intuitively displayed in the video frame, enhancing the visualization effect of video surveillance, and providing users with clear and accurate target detection information. By drawing bounding boxes and labels in real time, the synchronous display of target annotation information and video content is realized, improving the real-time and visualization effect of the video surveillance system. Users can intuitively view the target objects and their location information in the video, enhancing the effectiveness of video surveillance and user experience.

[0069] S5, the client stores the decoded video and target annotation information;

[0070] Specifically, the client stores the decoded video and target annotation information, which means that the client stores the decoded video and target annotation information in a database, and implements security protection measures on the stored data. The database backs up the stored data in the cloud, regularly performs integrity checks on the stored data and backup data, and stores the test results synchronously.

[0071] By storing the decoded video and target annotation information in the database, orderly management and efficient query of the data are ensured. The efficient storage and retrieval functions provided by the database system can meet the storage needs of a large amount of video data and annotation information. The reasonable data storage structure design improves the efficiency of data storage and query, and facilitates subsequent data analysis and processing. Through security protection measures such as data encryption, access control and data backup, the security and reliability of stored data are improved. Data encryption prevents data leakage and tampering, access control ensures the security of data access, and data backup provides data recovery guarantees, which comprehensively improves the security of data storage and management. Through cloud backup and integrity detection, the security and reliability of stored data are ensured. Cloud backup provides off-site storage and disaster recovery guarantees for data, avoiding the risk of data loss. Integrity detection improves data reliability and ensures that data has not been tampered with or damaged during storage and transmission. The synchronous storage of detection results improves the transparency and traceability of data management.

[0072] Example 2

[0073] Reference Figure 3 , which is the second embodiment of the present invention, and which is different from the previous embodiment, provides a video streaming system based on target detection and visual annotation, which includes:

[0074] The video acquisition module is used to acquire the basic urban environment video and the specific urban environment video, and pre-process the acquired video;

[0075] The target detection module is used to train the YOLOv5 model using the urban characteristic environment video, and detect the target in the urban basic environment video based on the trained YOLOv5 model;

[0076] The target annotation module is used to generate and optimize target annotation information based on the target detection output of the YOLOv5 model;

[0077] The encoding and transmission module is used to encode the basic urban environment video and target annotation information and transmit them through the RTP protocol;

[0078] The decoding and annotation module is used to decode the encoded video and target annotation information and perform time synchronization, and annotate the target annotation information in the basic urban environment video in real time;

[0079] The data storage module is used to store data in the database and implement security protection measures.

[0080] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0081] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0082] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0083] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit with a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit with a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0084] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A video streaming method based on target detection and visual annotation, characterized in that: include, Use video surveillance cameras to collect and pre-process basic urban environment videos; Obtain city-specific environment videos to train the YOLOv5 model, perform target detection in city basic environment videos based on the YOLOv5 model, generate visual target annotations, and perform optimization; The target marking information is encoded into the RTP packet header, and the basic urban environment video is compressed and encoded. The RTP packet header and the encoded video are synchronously pushed to the client through the RTP protocol; The client decodes the encoded video and target annotation information, and annotates the target annotation information in the video frame in real time; The client stores the decoded video and target annotation information.

2. The video streaming method based on target detection and visual annotation as claimed in claim 1 is characterized by: The use of video surveillance cameras to collect urban basic environment videos and perform preprocessing refers to randomly selecting urban basic environments and using high-definition surveillance cameras to shoot videos, adjusting the frame rate of the collected urban basic environment videos to 30fps, and using a Gaussian filter for denoising, converting the denoised video frames into RGB format, and adjusting the size to 640*640 pixels.

3. The video streaming method based on target detection and visual annotation as claimed in claim 2 is characterized by: The method of obtaining a city-specific environment video to train a YOLOv5 model and performing city basic environment video target detection based on the YOLOv5 model refers to selecting a city-specific environment video, preprocessing the city-specific environment video, and annotating the target objects in the specific environment to generate training data, enhancing the training data by rotation, scaling, and flipping, inputting the training data into the YOLOv5 model, defining a loss function and an Adam optimizer to iteratively train the model parameters until the loss is minimized, and then stopping the iteration to obtain a trained YOLOv5 model. The YOLOv5 model divides the input video image into S*S grid units, and each grid unit predicts B target bounding box parameters, target category probability, and target bounding box confidence P. The YOLOv5 model includes three parts: Backbone, Neck, and Head: Backbone uses the CSPDarknet53 structure, which includes multiple convolutional layers, a normalization layer, and an activation function layer; Neck uses the PANet structure, which includes upsampling layers, convolutional layers, and feature fusion layers; Head uses YOLO Layer, which contains multiple convolutional layers; The preprocessed urban basic environment video is input into the Backbone part to extract video image features, the extracted video image features are input into the Neck part for feature fusion, and the fused features are input into the Head part to output B target bounding box parameters, target category probability and target bounding box confidence P, and the target label b is determined according to the target category probability.

4. The video streaming method based on target detection and visual annotation as claimed in claim 3 is characterized by: Generating and optimizing the visual target annotations refers to extracting the target bounding box coordinates (x, y, w, h) and the target label b from the target bounding box parameters and target categories output by the YOLOv5 model, where x and y are the coordinates of the center point of the target bounding box, w and h are the width and height of the target bounding box, generating the target annotation according to the target label b and the target bounding box, and attaching timestamp information to each target annotation; Calculate the coordinates of the upper left corner and lower right corner of the target bounding box according to the coordinates of the center point and the width and height of the target bounding box, and calculate the intersection and union ratio of adjacent target bounding boxes: , Where IoU is the intersection over union ratio, Y and Z are adjacent target bounding boxes, is the intersection area of ​​Y and Z, is the area of ​​the union of Y and Z; The intersection-over-union ratio threshold is set. When the intersection-over-union ratio of adjacent target bounding boxes is greater than the intersection-over-union ratio threshold, the confidence P of the adjacent target bounding boxes is compared, the target bounding boxes with low confidence and the corresponding target annotations are removed, and all target bounding boxes are traversed to complete the target annotation optimization.

5. The video streaming method based on target detection and visual annotation as claimed in claim 4 is characterized by: The target annotation information is encoded and placed in the RTP packet header, and the city's basic environment video is compressed and encoded, and the RTP packet header and the encoded video are synchronously pushed to the client through the RTP protocol. This means converting the optimized target annotation information into a string format and embedding it into the RTP packet header file after using Base64 encoding, setting encoding parameters, using an encoder to compress and encode the city's basic environment video frames to form an encoded video, and configuring the RTP transmission parameters to synchronously push the encoded video and the RTP packet header file to the client.

6. The video streaming method based on target detection and visual annotation as claimed in claim 5 is characterized by: The client decodes the encoded video and target annotation information, and annotates the target annotation information in the video frame in real time. The client uses a decoder to decode the encoded video to obtain the city basic environment video, extracts the target annotation information from the RTP packet header file for decoding, and converts the decoded character string into the target annotation information, aligns the city basic environment video and the target annotation information according to the timestamp information of the target annotation information, and draws a boundary box and a label on the city basic environment video frame in real time according to the target annotation information for synchronous display.

7. The video streaming method based on target detection and visual annotation according to claim 6 is characterized in that: The client storing the decoded video and target annotation information means that the client stores the decoded video and target annotation information in a database, and implements security protection measures on the stored data. The database backs up the stored data in the cloud, regularly performs integrity checks on the stored data and backup data, and synchronously stores the test results.

8. A video streaming system according to any one of claims 1 to 7, characterized in that: include, The video acquisition module is used to acquire the basic urban environment video and the specific urban environment video, and pre-process the acquired video; The target detection module is used to train the YOLOv5 model using the urban characteristic environment video, and detect the target in the urban basic environment video based on the trained YOLOv5 model; The target annotation module is used to generate and optimize target annotation information based on the target detection output of the YOLOv5 model; The encoding and transmission module is used to encode the basic urban environment video and target annotation information and transmit them through the RTP protocol; The decoding and annotation module is used to decode the encoded video and target annotation information and perform time synchronization, and annotate the target annotation information in the basic urban environment video in real time; The data storage module is used to store data in the database and implement security protection measures.

9. A computer device comprising: Memory and processor; The memory stores a computer program, characterized in that: when the processor executes the computer program, the steps of the video streaming method based on target detection and visual annotation are implemented as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the video streaming method based on target detection and visual annotation are implemented as described in any one of claims 1 to 7.

Citation Information

Cited By

  • Video target labeling method and device, storage medium and computer program product

    CN121531193A