A Pipeline-Based Parallel Video Structured Inference Method and System

Abstract

This invention discloses a pipeline-based parallel video structured inference method and system, belonging to the field of artificial intelligence visual analysis technology. The technical problem it addresses is the low efficiency of video structured inference due to serial chained parsing, which cannot meet the parallel processing requirements of pipeline parsing. The method includes the following steps: splitting the pipeline into a main pipeline and branch pipelines; performing pedestrian and vehicle detection analysis on the acquired video stream through the main pipeline to obtain pedestrian and vehicle images; performing attribute analysis and feature extraction on pedestrians through the pedestrian detection branch pipeline, and performing attribute analysis and feature extraction on vehicles through the vehicle detection branch pipeline; and summarizing the pedestrian and vehicle analysis results through the main pipeline.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence visual analysis technology, specifically to a pipeline-based parallel video structured reasoning method and system. Background Technology

[0002] With the rapid development of artificial intelligence and the country's emphasis on the security field, intelligent vision technology, as a core technology in the security field, has also developed rapidly, and video structuring is a key research direction in the field of intelligent vision. Video structuring refers to the process of establishing a big data platform for video structure based on the characteristics of people, vehicles, objects, colors, numbers, and other attributes presented in video footage through reasoning and analysis.

[0003] Video structured inference parsing requires extracting eligible pedestrians and vehicles from video footage and sending the results to lower-level services. However, existing frameworks do not provide elements with the property of parallel merging pipeline results; therefore, video structured inference can only parse each model sequentially to obtain the corresponding information. This sequential chaining parsing leads to low parsing efficiency.

[0004] Serial chain parsing leads to low efficiency in video structured inference parsing and cannot meet the parallel processing requirements of pipeline parsing, which is a technical problem that needs to be solved. Summary of the Invention

[0005] The technical objective of this invention is to address the above shortcomings by providing a pipeline-based parallel video structured inference method and system, thereby solving the problem that serial chain parsing leads to low efficiency in video structured inference parsing and fails to meet the requirements of parallel processing in pipeline parsing.

[0006] In a first aspect, the pipeline-based parallel video structured inference method of the present invention includes the following steps:

[0007] The Pipeline is split into a main Pipeline and branch Pipelines. There are multiple main Pipelines that run in parallel. The branch Pipelines include a pedestrian detection branch Pipeline and a vehicle detection branch Pipeline.

[0008] Configure and initialize the main pipeline, which is used to take the acquired video stream as input, decode and divide the video stream into frames to obtain multiple frames of images, and take pedestrians and vehicles as targets, perform target detection and target tracking and capture on each frame of images, and output pedestrian and vehicle inference results, as well as pedestrian and vehicle images;

[0009] Configure and initialize the pedestrian detection pipeline and the vehicle detection pipeline. The pedestrian detection pipeline is used to perform attribute analysis and feature extraction on pedestrian images as input to obtain pedestrian analysis results. The vehicle detection pipeline is used to perform attribute analysis and feature extraction on multiple vehicles as input to obtain vehicle analysis results. It is also used to return the pedestrian analysis results and vehicle analysis results to the main pipeline. The main pipeline is used to aggregate the pedestrian analysis results and vehicle analysis results into a final result and return it to the next-level service.

[0010] The main pipeline is used to detect and analyze pedestrians and vehicles in the acquired video stream, resulting in pedestrian and vehicle images.

[0011] Using pedestrian images as input, the pedestrian detection branch Pipeline performs attribute analysis and feature extraction on pedestrians to obtain pedestrian analysis results. Similarly, using vehicle images as input, the vehicle detection branch Pipeline performs attribute analysis and feature extraction on vehicles to obtain vehicle analysis results.

[0012] The main pipeline aggregates the pedestrian and vehicle inference results, as well as the pedestrian analysis results and vehicle analysis results, to obtain the final result, which is then returned to the next-level service.

[0013] Preferably, the main pipeline is configured with a pedestrian and vehicle detection model, which is a neural network model built based on deep learning. It is used to perform reasoning and classification on pedestrians and vehicles with images as input, to obtain target names and target location coordinates, and to obtain pedestrian images and vehicle images. The targets include pedestrians and vehicles.

[0014] The pedestrian detection branch Pipeline is configured with a pedestrian detection model, which is a neural network model built based on deep learning. It is used to perform attribute analysis and feature extraction on pedestrians with pedestrian images as input, and obtain pedestrian analysis results.

[0015] The vehicle detection branch Pipeline is equipped with a vehicle detection model, which is a neural network model built based on deep learning. It is used to perform attribute analysis and feature extraction on vehicle images as input to obtain vehicle analysis results.

[0016] The number of main pipelines that can be parallelized is calculated using the server's video memory size and the size of the models in the main pipeline and branch pipelines. The formula for calculating the number of main pipelines is as follows:

[0017]

[0018] Among them, X m Indicates the size of the server's video memory, X d This indicates the size of the models in the main pipeline and branch pipelines, where the models include a pedestrian and vehicle detection model, a pedestrian detection model, and a vehicle detection model; num is the integer result obtained by rounding down.

[0019] Preferably, the main pipeline includes the following modules:

[0020] The video stream input module is used to open the stream by means of video monitoring points and bitstream type, and to obtain the RTSP stream of the video source;

[0021] The video decoding and frame segmentation module interacts with the video stream input module to segment the video stream into frames and obtain multiple frames of images.

[0022] The detection and reasoning module interacts with the video decoding and frame segmentation module and is configured with a pedestrian and vehicle detection model. The pedestrian and vehicle detection model is a neural network model built based on deep learning. It is used to perform inference and classification of pedestrians and vehicles on the input image through the pedestrian and vehicle detection model, obtain the target name and target location coordinates, and obtain pedestrian images and vehicle images. The targets include pedestrians and vehicles.

[0023] The tracking module is used to track a target in an area using the IOU calculation method. The tracking rule is to track and capture the target when it first appears in the detection area or when it reappears in the area after being lost. The target includes pedestrians and vehicles.

[0024] The return result module interacts with the detection and inference module. It is used to perform image matting on pedestrian and vehicle images, send the matted pedestrian images to the pedestrian detection branch Pipeline, and send the matted vehicle images to the vehicle detection branch Pipeline. It is also used to obtain pedestrian and vehicle analysis results, summarize the pedestrian and vehicle inference results as well as the pedestrian and vehicle analysis results into a final result, and return it to the next-level service.

[0025] Preferably, the pedestrian detection branch Pipeline includes:

[0026] Image source input module, which is used to obtain pedestrian images from the main pipeline;

[0027] The pedestrian attribute and feature extraction module interacts with the image source input module and is configured with a pedestrian detection model. The pedestrian detection model is a neural network model built based on deep learning, which is used to perform attribute analysis and feature extraction on pedestrian images through the pedestrian detection module to obtain pedestrian analysis results.

[0028] The result return module interacts with the pedestrian attribute and feature extraction module to return the pedestrian analysis results to the main pipeline.

[0029] Preferably, the pedestrian detection model is used to detect pedestrian attributes such as gender, age, whether they are wearing masks, and whether they are wearing glasses, to obtain pedestrian feature values ​​and thus obtain pedestrian analysis results.

[0030] Preferably, the vehicle detection branch pipeline is configured with the following modules:

[0031] Image source input module, which is used to obtain vehicle images from the main pipeline;

[0032] The decoding preprocessing module is used to perform decoding preprocessing on vehicle images. By performing image cropping and normalization on the vehicle images, it outputs vehicle images that meet a unified standard.

[0033] The vehicle attribute analysis and feature extraction module is equipped with a vehicle analysis model, which is a neural network model built based on deep learning. The vehicle analysis model is used to perform vehicle attribute analysis and extraction on vehicle images and output vehicle analysis results.

[0034] The return result module is used to return the vehicle analysis results to the main pipeline.

[0035] Preferably, the vehicle analysis model is used to identify vehicle brand attributes and license plate attributes, obtain corresponding feature values, and obtain vehicle analysis results. Vehicle information is obtained by identifying the vehicle, including vehicle color, items in front of the vehicle, whether the driver is making a phone call, whether the driver is wearing a seat belt, whether the passenger is wearing a seat belt, vehicle brand, vehicle model, vehicle year, and vehicle type. License plate information is obtained by identifying the license plate, including the license plate number and license plate location coordinates.

[0036] Secondly, the Pipeline-based parallel video structured inference system of the present invention is characterized by comprising:

[0037] The Pipeline splitting module is used to split a Pipeline into a main Pipeline and branch Pipelines, wherein the branch Pipelines include a pedestrian detection branch Pipeline and a vehicle detection branch Pipeline.

[0038] The branch pipeline configuration module is used to configure and initialize the main pipeline, pedestrian detection branch pipeline, and vehicle detection branch pipeline. Multiple main pipelines run in parallel. They take the acquired video stream as input, decode and segment the video stream to obtain multiple frames of images, and perform target detection, tracking, and capture on each frame, outputting pedestrian and vehicle inference results, as well as pedestrian and vehicle images. The pedestrian and vehicle inference results include target names and target location coordinates. The targets include pedestrians and vehicles. The pedestrian detection branch pipeline takes pedestrian images as input, performs attribute analysis and feature extraction on pedestrians, and obtains pedestrian analysis results. The vehicle detection branch pipeline takes vehicle images as input, performs attribute analysis and feature extraction on multiple vehicles, and obtains vehicle analysis results. These results are then returned to the main pipeline. The main pipeline aggregates the pedestrian and vehicle inference results and the pedestrian and vehicle analysis results into a final result and returns it to the next-level service.

[0039] The inference module is used to detect and analyze pedestrians and vehicles in the acquired video stream through the main pipeline, obtaining pedestrian and vehicle images; it is used to perform attribute analysis and feature extraction on pedestrians through the pedestrian detection branch pipeline, taking pedestrian images as input, to obtain pedestrian analysis results; it is used to perform attribute analysis and feature extraction on vehicles through the vehicle detection branch pipeline, taking vehicle images as input, to obtain vehicle analysis results; and it is used to summarize the pedestrian and vehicle inference results, as well as the pedestrian and vehicle analysis results, through the main pipeline to obtain the final result, and return the final result to the next-level service.

[0040] Preferably, the main pipeline includes the following modules:

[0041] The video stream input module is used to open the stream by means of the video monitoring point and the bitstream type, and to obtain the RTSP stream of the video source;

[0042] The video decoding and frame segmentation module interacts with the video stream input module to segment the video stream into frames and obtain multiple frames of images.

[0043] The detection and reasoning module interacts with the video decoding and frame segmentation module and is equipped with a pedestrian and vehicle detection model. The pedestrian and vehicle detection model is a neural network model built based on deep learning. It is used to perform inference and classification of pedestrians and vehicles on the input image through the pedestrian and vehicle detection model to obtain pedestrian and vehicle inference results as well as pedestrian images and vehicle images.

[0044] The tracking module is used to track a target in an area using the IOU calculation method. The tracking rule is to track and capture the target when it first appears in the detection area or when it reappears in the area after being lost. The target includes pedestrians and vehicles.

[0045] The return result module interacts with the detection and inference module. It is used to perform image matting on pedestrian and vehicle images, send the matted pedestrian images to the pedestrian detection branch Pipeline, and send the matted vehicle images to the vehicle detection branch Pipeline. It is also used to obtain pedestrian and vehicle analysis results, summarize the pedestrian and vehicle analysis results into a final result, and return it to the next-level service.

[0046] The pedestrian detection branch Pipeline includes:

[0047] Image source input module, which is used to obtain pedestrian images from the main pipeline;

[0048] The pedestrian attribute and feature extraction module interacts with the image source input module and is configured with a pedestrian detection model. The pedestrian detection model is a neural network model built based on deep learning, which is used to perform attribute analysis and feature extraction on pedestrian images through the pedestrian detection module to obtain pedestrian analysis results.

[0049] The result return module interacts with the pedestrian attribute and feature extraction module to return the pedestrian analysis results to the main pipeline.

[0050] The pedestrian detection model is used to detect pedestrians' gender, age, whether they are wearing masks, and whether they are wearing glasses, to obtain pedestrian feature values ​​and thus obtain pedestrian analysis results.

[0051] The vehicle detection branch pipeline is configured with the following modules:

[0052] Image source input module, which is used to obtain vehicle images from the main pipeline;

[0053] The decoding preprocessing module is used to perform decoding preprocessing on vehicle images. By performing image cropping and normalization on the vehicle images, it outputs vehicle images that meet a unified standard.

[0054] The vehicle attribute analysis and feature extraction module is equipped with a vehicle analysis model, which is a neural network model built based on deep learning. The vehicle analysis model is used to perform vehicle attribute analysis and extraction on vehicle images and output vehicle analysis results.

[0055] The return result module is used to return the vehicle analysis results back to the main pipeline;

[0056] The vehicle analysis model is used to identify vehicle brand attributes and license plate attributes, obtain corresponding feature values, and obtain vehicle analysis results. Vehicle information is obtained by identifying the vehicle, including vehicle color, items in front of the vehicle, whether the driver is making a phone call, whether the driver is wearing a seat belt, whether the passenger is wearing a seat belt, vehicle brand, vehicle model, vehicle year, and vehicle type. License plate information is obtained by identifying the license plate, including the license plate number and license plate location coordinates.

[0057] Preferably, the Pipeline splitting module is used to calculate the number of main pipelines that can be parallelized based on the size of the server's video memory and the size of the models in the main pipeline and branch pipelines. The formula for calculating the number of main pipelines is as follows:

[0058]

[0059] Among them, X m Indicates the size of the server's video memory, X d This indicates the size of the models in the main pipeline and branch pipelines, where the models include a pedestrian and vehicle detection model, a pedestrian detection model, and a vehicle detection model; num is the integer result obtained by rounding down.

[0060] The pipeline-based parallel video structured inference method and system of the present invention have the following advantages:

[0061] 1. By splitting the pipeline, pedestrian and vehicle detection are processed in parallel. The main pipeline calls the inference results of the branch pipelines and returns the final result to the lower-level service. This parallel processing improves parsing efficiency. By splitting the pipeline, different branch pipelines can be flexibly combined and expanded, which improves the flexibility of the algorithm service and facilitates parallel orchestration.

[0062] 2. The number of main pipelines that can run in parallel is calculated based on the server's GPU memory size and the size of the deep learning trained model. Multiple main pipelines run in parallel, which improves the efficiency of the algorithm. The number of main pipelines is also accurately calculated, providing a suitable main pipeline and avoiding unnecessary waste of resources.

[0063] 3. Initializing only one pedestrian pipeline branch and one vehicle pipeline branch reduces the GPU resources required for initializing the pipeline and loading the model, thus saving resources significantly. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] The invention will be further described below with reference to the accompanying drawings.

[0066] Figure 1 This is a block diagram illustrating the working principle of the Pipeline-based parallel video structured inference method in Example 1. Detailed Implementation

[0067] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0068] This invention provides a pipeline-based parallel video structured inference method and system to solve the technical problem that serial chain parsing leads to low efficiency in video structured inference parsing and cannot meet the parallel processing requirements of pipeline parsing.

[0069] Example 1:

[0070] The present invention provides a pipeline-based parallel video structured inference method, comprising the following steps:

[0071] S100. The Pipeline is split into a main Pipeline and branch Pipelines. There are multiple main Pipelines that run in parallel. The branch Pipelines include a pedestrian detection branch Pipeline and a vehicle detection branch Pipeline.

[0072] S200. Configure and initialize the main pipeline. The main pipeline is used to take the acquired video stream as input, decode and divide the video stream into frames to obtain multiple frames of images, and take pedestrians and vehicles as targets to perform target detection and target tracking and capture on each frame of images, and output pedestrian images and vehicle images.

[0073] Configure and initialize the pedestrian detection pipeline and the vehicle detection pipeline. The pedestrian detection pipeline is used to perform attribute analysis and feature extraction on pedestrian images as input to obtain pedestrian analysis results. The vehicle detection pipeline is used to perform attribute analysis and feature extraction on multiple vehicles as input to obtain vehicle analysis results. It is also used to return the pedestrian analysis results and vehicle analysis results to the main pipeline. The main pipeline is used to aggregate the pedestrian analysis results and vehicle analysis results into a final result and return it to the next-level service.

[0074] S300: The main pipeline performs pedestrian and vehicle detection and analysis on the acquired video stream to obtain pedestrian and vehicle images;

[0075] S400. Taking pedestrian images as input, the pedestrian detection branch Pipeline performs attribute analysis and feature extraction on pedestrians to obtain pedestrian analysis results. Taking vehicle images as input, the vehicle detection branch Pipeline performs attribute analysis and feature extraction on vehicles to obtain vehicle analysis results.

[0076] S500 summarizes the pedestrian and vehicle analysis results through the main pipeline to obtain the final result, and returns the final result to the next level service.

[0077] In this embodiment, the main pipeline is configured with a pedestrian and vehicle detection model, which is a neural network model built based on deep learning. This model takes images as input and performs reasoning and classification on pedestrians and vehicles to obtain pedestrian and vehicle images. The pedestrian detection branch pipeline is configured with a pedestrian detection model, which is also a neural network model built based on deep learning. This model takes pedestrian images as input and performs attribute analysis and feature extraction on pedestrians to obtain pedestrian analysis results. Similarly, the vehicle detection branch pipeline is configured with a vehicle detection model, which is also a neural network model built based on deep learning. This model takes vehicle images as input and performs attribute analysis and feature extraction on vehicles to obtain vehicle analysis results.

[0078] In step S100, the pipeline is split into different pipeline branches for parallel processing of pedestrians and vehicles. There are multiple main pipelines that are processed in parallel, and one pedestrian detection branch pipeline and one vehicle detection branch pipeline. The number of main pipelines needs to be determined during the splitting process.

[0079] In this embodiment, the video memory size X provided by the server is... m And the size X of the model trained by deep learning d Therefore, the number of parallel main pipelines can be calculated as follows:

[0080]

[0081] `num` is the integer result rounded down. The number of parallel main pipelines is initialized based on the calculation result. Each main pipeline inherits from a `bin` container that manages multiple elements, providing a bus for message transmission. When there is no data in the pipeline, each element is in a waiting state until data appears, at which point it enters a working state. Simultaneously with the initialization of the main pipelines, a pedestrian pipeline branch and a vehicle pipeline branch are also initialized. These branches process pedestrian and vehicle targets in parallel.

[0082] The main pipeline consists of the following modules: video stream input module, video decoding and framing module, detection and inference module, tracking module, and result return module.

[0083] The video stream input module is used to open the stream based on the video monitoring point and the bitstream type, obtaining the RTSP stream of the video source. The video bitstream segmentation module uses probes to implement hardware decoding, passing the traditional buffer from one element to another via the hardware buffer ID, for segmenting the video stream into multiple frames.

[0084] The detection and inference module interacts with the video decoding and frame segmentation module. It is equipped with a pedestrian and vehicle detection model, which is a neural network model built based on deep learning. The pedestrian and vehicle detection model is used to infer and classify pedestrians and vehicles in the input image, obtain the target name and target location coordinates. The targets include pedestrians and vehicles, and obtain pedestrian images and vehicle images.

[0085] The tracking module is used to track a target in an area using the IOU calculation method. The tracking rule is to track and capture the target when it first appears in the detection area or when it reappears in the area after being lost. The target includes pedestrians and vehicles.

[0086] The return result module interacts with the detection and inference module to perform image cutout processing on pedestrian and vehicle images to reduce their size. It then sends the cutout pedestrian images to the pedestrian detection branch Pipeline and the cutout vehicle images to the vehicle detection branch Pipeline. The module also retrieves pedestrian and vehicle analysis results, summarizes the pedestrian and vehicle inference results as well as the pedestrian and vehicle analysis results into a final result, and returns it to the next-level service.

[0087] The pedestrian detection pipeline includes an image source input module, a pedestrian attribute and feature extraction module, and a result return module. The image source input module retrieves pedestrian images from the main pipeline. The pedestrian attribute and feature extraction module interacts with the image source input module and is configured with a pedestrian detection model. This model is a neural network model built based on deep learning, used to perform attribute analysis and feature extraction on the pedestrian images to obtain pedestrian analysis results. The result return module interacts with the pedestrian attribute and feature extraction module to return the pedestrian analysis results to the main pipeline. The pedestrian detection model detects pedestrian attributes such as gender, age, whether they are wearing masks, and whether they are wearing glasses, obtaining the pedestrian's feature values ​​and thus the pedestrian analysis results.

[0088] This pedestrian detection pipeline acts as an image processing pipeline, synchronously returning detection data. The image source input module is a small image obtained by cutting out pedestrians detected by the main pipeline, which is then preprocessed and loaded into the pipeline's data source element. The pedestrian attribute and feature extraction module mainly detects attributes such as gender, age, whether wearing a mask, and whether wearing glasses, obtaining the pedestrian's feature values. The result return module mainly returns the detected pedestrian attribute and feature value detection results from the cache. This process only processes specific image targets, simplifying the processing.

[0089] The vehicle detection branch pipeline is configured with the following modules: an image source input module, a decoding preprocessing module, a vehicle attribute analysis and feature extraction module, and a return result module. The image source input module is used to obtain vehicle images from the main pipeline; the decoding preprocessing module is used to decode and preprocess the vehicle images, and output vehicle images that meet a unified standard by performing image cropping and normalization; the vehicle attribute analysis and feature extraction module is configured with a vehicle analysis model, which is a neural network model built based on deep learning, used to perform vehicle attribute analysis and extraction on the vehicle images, and output the vehicle analysis results; the return result module is used to return the vehicle analysis results to the main pipeline.

[0090] The vehicle detection pipeline's input source is small images of vehicles detected by the main pipeline, obtained through image matting. These images are preprocessed, cropped, and normalized to a standard size before being loaded into the data source elements of the vehicle detection pipeline. The vehicle attribute analysis and feature extraction module involves multi-model processing, treating vehicle brand and attribute recognition as primary inference and license plate detection and recognition as secondary inference. The vehicle brand / attribute recognition model acquires vehicle and license plate information. Vehicle information includes vehicle color, objects in front of the vehicle, whether the driver is using a phone, whether the driver and passenger are wearing seatbelts, vehicle brand, model, year, and type. License plate information includes license plate number and location coordinates. License plate detection and recognition filter model data by reading the configuration parameters `gie-unique-id` and `class_ids` to skip elements in the pipeline. During the license plate detection process, the detected license plate is cut out and then pasted onto the original image. This process utilizes the small vehicle image sent to the vehicle branch pipeline, thus avoiding the previous problem of image size limitations for cutting out and pasting license plates from multiple vehicles in a large image.

[0091] In this embodiment, the elements in the main pipeline and branch pipeline work together to decode the structured video frame by frame to obtain the image to be parsed, and then perform pedestrian / vehicle reasoning detection and tracking processing. For targets that meet the conditions, pedestrian attributes (such as age, gender, accessories (whether wearing glasses, masks)) are extracted, and then vehicle attributes, vehicle color, license plate recognition and other features are extracted.

[0092] In practical applications, the main pipeline asynchronously returns video detection results and calls a synchronous image detection task within the returned results module. The main pipeline sends detected pedestrian thumbnails as input to the pedestrian detection pipeline branch and detected vehicle thumbnails as input to the vehicle detection pipeline branch. The pedestrian detection branch synchronously returns pedestrian attributes and features through image inference. The vehicle detection branch, after inference, synchronously returns information such as vehicle brand, vehicle attributes, and the coordinates of detected license plates to the image detection task.

[0093] This embodiment implements parallel pipeline processing of video streams by splitting the pipeline into different pipeline branches for parallel processing of pedestrians and vehicles. Since current frameworks do not provide elements capable of parallel pipeline processing of pedestrians and vehicles, previous pipeline-based video parsing methods could only achieve serial parsing of different models by skipping elements. The main process involved first decoding the structured video frame by frame to obtain images to be parsed, then performing pedestrian / vehicle inference detection and tracking, extracting pedestrian attributes (such as age, gender, and accessories (whether wearing glasses or masks)) for targets that meet the criteria, and then extracting features such as vehicle attributes, vehicle color, and license plate recognition. Finally, the processing of each module was placed in the same pipeline, where images were transferred and parsed using the same memory block. This invention simplifies the pipeline organization structure by splitting the pipeline, placing pedestrian attribute detection and vehicle attribute detection in separate branch pipelines. Finally, the main pipeline achieves parallel pipeline processing by calling the detection results of the branch pipelines.

[0094] This method simplifies the structured video detection architecture. Pedestrian and vehicle detection models only need to be created once, reducing GPU resource consumption during model loading. By reading parameters from the configuration file, model categories and data that do not need to be detected can be filtered, which facilitates orchestration.

[0095] Example 2:

[0096] This invention relates to a pipeline-based parallel video structured inference system, which includes a pipeline splitting module, a branch pipeline configuration module, and an inference module.

[0097] The Pipeline splitting module is used to split a Pipeline into a main Pipeline and branch Pipelines. The branch Pipelines include pedestrian detection branch Pipelines and vehicle detection branch Pipelines.

[0098] The branch pipeline configuration module is used to configure and initialize the main pipeline, pedestrian detection branch pipeline, and vehicle detection branch pipeline. Multiple main pipelines run in parallel. They take the acquired video stream as input, decode and segment the video stream to obtain multiple frames of images, and perform target detection, tracking, and capture on each frame, outputting pedestrian and vehicle inference results, as well as pedestrian and vehicle images. The pedestrian and vehicle inference results include target names and target location coordinates; targets include both pedestrians and vehicles. The pedestrian detection branch pipeline takes pedestrian images as input, performs attribute analysis and feature extraction on pedestrians, and obtains pedestrian analysis results. The vehicle detection branch pipeline takes vehicle images as input, performs attribute analysis and feature extraction on multiple vehicles, and obtains vehicle analysis results. These results are then returned to the main pipeline. The main pipeline aggregates the pedestrian and vehicle inference results and the pedestrian and vehicle analysis results into a final result, which is then returned to the next-level service.

[0099] The inference module is used to detect and analyze pedestrians and vehicles in the acquired video stream through the main pipeline, obtaining pedestrian and vehicle images; it is used to perform attribute analysis and feature extraction on pedestrians through the pedestrian detection branch pipeline, taking pedestrian images as input, and to perform attribute analysis and feature extraction on vehicles through the vehicle detection branch pipeline, obtaining vehicle analysis results; and it is used to summarize the pedestrian and vehicle inference results, as well as the pedestrian and vehicle analysis results, through the main pipeline to obtain the final result, and return the final result to the next-level service.

[0100] In this embodiment, the main pipeline includes the following modules: video stream input module, video decoding and framing module, detection and inference module, tracking module, and result return module.

[0101] The video stream input module is used to open the stream based on the video monitoring point and the bitstream type, obtaining the RTSP stream of the video source. The video bitstream segmentation module uses probes to implement hardware decoding, passing the traditional buffer from one element to another via the hardware buffer ID, for segmenting the video stream into multiple frames.

[0102] The detection and inference module interacts with the video decoding and frame segmentation module. It is equipped with a pedestrian and vehicle detection model, which is a neural network model built based on deep learning. The pedestrian and vehicle detection model is used to infer and classify pedestrians and vehicles in the input image, obtain the target name and target location coordinates. The targets include pedestrians and vehicles, and obtain pedestrian images and vehicle images.

[0103] The tracking module is used to track a target in an area using the IOU calculation method. The tracking rule is to track and capture the target when it first appears in the detection area or when it reappears in the area after being lost. The target includes pedestrians and vehicles.

[0104] The return result module interacts with the detection and inference module to perform image matting on pedestrian and vehicle images. It sends the matted pedestrian images to the pedestrian detection branch Pipeline and the matted vehicle images to the vehicle detection branch Pipeline. It also retrieves pedestrian and vehicle analysis results, summarizes the pedestrian and vehicle inference results as well as the pedestrian and vehicle analysis results into a final result, and returns it to the next-level service.

[0105] The pedestrian detection pipeline includes an image source input module, a pedestrian attribute and feature extraction module, and a result return module. The image source input module retrieves pedestrian images from the main pipeline. The pedestrian attribute and feature extraction module interacts with the image source input module and is configured with a pedestrian detection model. This model is a neural network model built based on deep learning, used to perform attribute analysis and feature extraction on the pedestrian images to obtain pedestrian analysis results. The result return module interacts with the pedestrian attribute and feature extraction module to return the pedestrian analysis results to the main pipeline. The pedestrian detection model detects pedestrian attributes such as gender, age, whether they are wearing masks, and whether they are wearing glasses, obtaining the pedestrian's feature values ​​and thus the pedestrian analysis results.

[0106] This pedestrian detection pipeline acts as an image processing pipeline, synchronously returning detection data. The image source input module is a small image obtained by cutting out pedestrians detected by the main pipeline, which is then preprocessed and loaded into the pipeline's data source element. The pedestrian attribute and feature extraction module mainly detects attributes such as gender, age, whether wearing a mask, and whether wearing glasses, obtaining the pedestrian's feature values. The result return module mainly returns the detected pedestrian attribute and feature value detection results from the cache. This process only processes specific image targets, simplifying the processing.

[0107] The vehicle detection branch pipeline is configured with the following modules: an image source input module, a decoding preprocessing module, a vehicle attribute analysis and feature extraction module, and a return result module. The image source input module is used to obtain vehicle images from the main pipeline; the decoding preprocessing module is used to decode and preprocess the vehicle images, and output vehicle images that meet a unified standard by performing image cropping and normalization; the vehicle attribute analysis and feature extraction module is configured with a vehicle analysis model, which is a neural network model built based on deep learning, used to perform vehicle attribute analysis and extraction on the vehicle images, and output the vehicle analysis results; the return result module is used to return the vehicle analysis results to the main pipeline.

[0108] The vehicle detection pipeline's input source is small images of vehicles detected by the main pipeline, obtained through image matting. These images are preprocessed, cropped, and normalized to a standard size before being loaded into the data source elements of the vehicle detection pipeline. The vehicle attribute analysis and feature extraction module involves multi-model processing, treating vehicle brand and attribute recognition as primary inference and license plate detection and recognition as secondary inference. The vehicle brand / attribute recognition model acquires vehicle and license plate information. Vehicle information includes vehicle color, objects in front of the vehicle, whether the driver is using a mobile phone, whether the driver and passenger are wearing seatbelts, vehicle brand, vehicle model, vehicle year, and vehicle type. License plate information includes license plate number and location coordinates. License plate detection and recognition filter model data by reading the configuration parameters `gie-unique-id` and `class_ids`, skipping elements in the pipeline. During the license plate detection process, the detected license plate is cut out and then pasted onto the original image. This process utilizes the small vehicle image sent to the vehicle branch pipeline, thus avoiding the previous problem of image size limitations for cutting out and pasting license plates from multiple vehicles in a large image.

[0109] This embodiment can execute the method disclosed in Embodiment 1. The elements in the main pipeline and the branch pipeline cooperate to decode the structured video frame by frame to obtain the image to be parsed. Then, pedestrian / vehicle reasoning detection and tracking processing are performed. For targets that meet the conditions, pedestrian attributes (such as age, gender, accessories (whether wearing glasses or masks)) are extracted, and then vehicle attributes, vehicle color, license plate recognition and other features are extracted.

[0110] The present invention has been shown and described in detail above with reference to the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above embodiments, those skilled in the art will know that more embodiments of the present invention can be obtained by combining the means in the different embodiments described above, and these embodiments are also within the protection scope of the present invention.

Claims

1. A pipeline-based parallel video structured inference method, characterized in that... Includes the following steps: The Pipeline is split into a main Pipeline and branch Pipelines. There are multiple main Pipelines that run in parallel. The branch Pipelines include a pedestrian detection branch Pipeline and a vehicle detection branch Pipeline. Configure and initialize the main pipeline, which is used to take the acquired video stream as input, decode and divide the video stream into frames to obtain multiple frames of images, and take pedestrians and vehicles as targets, perform target detection and target tracking and capture on each frame of images, and output pedestrian and vehicle inference results, as well as pedestrian and vehicle images; Configure and initialize the pedestrian detection pipeline and the vehicle detection pipeline. The pedestrian detection pipeline is used to perform attribute analysis and feature extraction on pedestrian images as input to obtain pedestrian analysis results. The vehicle detection pipeline is used to perform attribute analysis and feature extraction on vehicle images as input to obtain vehicle analysis results. It is also used to return the pedestrian analysis results and vehicle analysis results to the main pipeline. The main pipeline is used to aggregate the pedestrian analysis results and vehicle analysis results into a final result and return it to the next-level service. The main pipeline is used to detect and analyze pedestrians and vehicles in the acquired video stream, resulting in pedestrian and vehicle images. Using pedestrian images as input, the pedestrian detection branch Pipeline performs attribute analysis and feature extraction on pedestrians to obtain pedestrian analysis results. Similarly, using vehicle images as input, the vehicle detection branch Pipeline performs attribute analysis and feature extraction on vehicles to obtain vehicle analysis results. The main pipeline aggregates the pedestrian and vehicle inference results, as well as the pedestrian analysis results and vehicle analysis results, to obtain the final result, which is then returned to the next-level service.

2. The pipeline-based parallel video structured inference method according to claim 1, characterized in that... The main pipeline is configured with a pedestrian and vehicle detection model, which is a neural network model built based on deep learning. It is used to infer and classify pedestrians and vehicles with images as input, obtain target names and target location coordinates, and obtain pedestrian images and vehicle images. The targets include pedestrians and vehicles. The pedestrian detection branch Pipeline is configured with a pedestrian detection model, which is a neural network model built based on deep learning. It is used to perform attribute analysis and feature extraction on pedestrians with pedestrian images as input, and obtain pedestrian analysis results. The vehicle detection branch Pipeline is equipped with a vehicle detection model, which is a neural network model built based on deep learning. It is used to perform attribute analysis and feature extraction on vehicle images as input to obtain vehicle analysis results. The number of main pipelines that can be parallelized is calculated using the server's video memory size and the size of the models in the main pipeline and branch pipelines. The formula for calculating the number of main pipelines is as follows: Among them, X m Indicates the size of the server's video memory, X d This indicates the size of the models in the main pipeline and branch pipelines, where the models include a pedestrian and vehicle detection model, a pedestrian detection model, and a vehicle detection model; num is the integer result obtained by rounding down.

3. The pipeline-based parallel video structured inference method according to claim 1, characterized in that... The main pipeline includes the following modules: The video stream input module is used to open the stream by means of video monitoring points and bitstream type, and to obtain the RTSP stream of the video source; A video decoding and frame-segmentation module interacts with the video stream input module to segment the video stream into frames, thereby obtaining multiple frames of images. The detection and reasoning module interacts with the video decoding and frame segmentation module and is configured with a pedestrian and vehicle detection model. The pedestrian and vehicle detection model is a neural network model built based on deep learning. It is used to perform inference and classification of pedestrians and vehicles on the input image through the pedestrian and vehicle detection model, obtain the target name and target location coordinates, and obtain pedestrian images and vehicle images. The targets include pedestrians and vehicles. The tracking module is used to track a target in an area using the IOU calculation method. The tracking rule is to track and capture the target when it first appears in the detection area or when it reappears in the area after being lost. The target includes pedestrians and vehicles. The return result module interacts with the detection inference module and is used to perform image cutout processing on pedestrian and vehicle images. The cutout pedestrian images are sent to the pedestrian detection branch Pipeline, and the cutout vehicle images are sent to the vehicle detection branch Pipeline. It is used to obtain pedestrian analysis results and vehicle analysis results, summarize the pedestrian and vehicle inference results as well as the pedestrian analysis results and vehicle analysis results into the final result and return it to the next level service.

4. The pipeline-based parallel video structured inference method according to claim 1, characterized in that... The pedestrian detection branch Pipeline includes: Image source input module, which is used to obtain pedestrian images from the main pipeline; The pedestrian attribute and feature extraction module interacts with the image source input module and is configured with a pedestrian detection model. The pedestrian detection model is a neural network model built based on deep learning, which is used to perform attribute analysis and feature extraction on pedestrian images through the pedestrian detection module to obtain pedestrian analysis results. The result return module interacts with the pedestrian attribute and feature extraction module to return the pedestrian analysis results to the main pipeline.

5. The pipeline-based parallel video structured inference method according to any one of claims 1-4, characterized in that... The pedestrian detection branch Pipeline is used to detect pedestrian attributes such as gender, age, whether they are wearing masks, and whether they are wearing glasses, to obtain pedestrian feature values ​​and thus obtain pedestrian analysis results.

6. The pipeline-based parallel video structured inference method according to claim 1, characterized in that... The vehicle detection branch pipeline is configured with the following modules: Image source input module, which is used to obtain vehicle images from the main pipeline; The decoding preprocessing module is used to perform decoding preprocessing on vehicle images. By performing image cropping and normalization on the vehicle images, it outputs vehicle images that meet a unified standard. The vehicle attribute analysis and feature extraction module is equipped with a vehicle detection model, which is a neural network model built based on deep learning. The vehicle detection model is used to perform vehicle attribute analysis and extraction on vehicle images and output vehicle analysis results. The return result module is used to return the vehicle analysis results to the main pipeline.

7. The pipeline-based parallel video structured inference method according to claim 1, 2, 3, 4 or 6, characterized in that... The vehicle detection branch Pipeline is used to identify vehicle brand attributes and license plate attributes, obtain corresponding feature values, and obtain vehicle analysis results. Vehicle information is obtained by identifying the vehicle, including vehicle color, items in front of the vehicle, whether the driver is making a phone call, whether the driver is wearing a seat belt, whether the passenger is wearing a seat belt, vehicle brand, vehicle model, vehicle year, and vehicle type. License plate information is obtained by identifying the license plate, including the license plate number and license plate location coordinates.

8. A pipeline-based parallel video structured inference system, characterized in that... include: The Pipeline splitting module is used to split a Pipeline into a main Pipeline and branch Pipelines, wherein the branch Pipelines include a pedestrian detection branch Pipeline and a vehicle detection branch Pipeline. The branch pipeline configuration module is used to configure and initialize the main pipeline, the pedestrian detection branch pipeline, and the vehicle detection branch pipeline. There are multiple main pipelines that run in parallel. They are used to take the acquired video stream as input, decode and segment the video stream to obtain multiple frames of images, and perform target detection and target tracking and capture on each frame of image with pedestrians and vehicles as targets. The output is pedestrian and vehicle inference results, as well as pedestrian and vehicle images. The pedestrian and vehicle inference results include target names and target location coordinates. The targets include pedestrians and vehicles. The pedestrian detection branch Pipeline is used to perform attribute analysis and feature extraction on pedestrian images as input, and obtain pedestrian analysis results. The vehicle detection branch Pipeline is used to perform attribute analysis and feature extraction on multiple vehicles using vehicle images as input to obtain vehicle analysis results. It is also used to return pedestrian and vehicle analysis results to the main Pipeline. The main Pipeline is used to aggregate the pedestrian and vehicle inference results, as well as the pedestrian and vehicle analysis results, into a final result and return it to the next-level service. The inference module is used to detect and analyze pedestrians and vehicles in the acquired video stream through the main pipeline to obtain pedestrian images and vehicle images. This tool is used to take pedestrian images as input, perform attribute analysis and feature extraction on pedestrians through the pedestrian detection branch Pipeline, and obtain pedestrian analysis results. It is also used to take vehicle images as input, perform attribute analysis and feature extraction on vehicles through the vehicle detection branch Pipeline, and obtain vehicle analysis results. It is used to summarize the pedestrian and vehicle inference results, as well as the pedestrian analysis results and vehicle analysis results through the main pipeline, to obtain the final result, and then return the final result to the lower-level service.

9. The pipeline-based parallel video structured inference system according to claim 8, characterized in that... The main pipeline includes the following modules: The video stream input module is used to open the stream by means of video monitoring points and bitstream type, and to obtain the RTSP stream of the video source; A video decoding and frame-segmentation module interacts with the video stream input module to segment the video stream into frames, thereby obtaining multiple frames of images. The detection and reasoning module interacts with the video decoding and frame segmentation module and is equipped with a pedestrian and vehicle detection model. The pedestrian and vehicle detection model is a neural network model built based on deep learning. It is used to perform inference and classification of pedestrians and vehicles on the input image through the pedestrian and vehicle detection model to obtain pedestrian and vehicle inference results as well as pedestrian images and vehicle images. The tracking module is used to track a target in an area using the IOU calculation method. The tracking rule is to track and capture the target when it first appears in the detection area or when it reappears in the area after being lost. The target includes pedestrians and vehicles. The return result module interacts with the detection inference module and is used to perform image cutout processing on pedestrian and vehicle images. The cutout pedestrian images are sent to the pedestrian detection branch Pipeline, and the cutout vehicle images are sent to the vehicle detection branch Pipeline. Used to obtain pedestrian and vehicle analysis results, summarize the pedestrian and vehicle analysis results into a final result and return it to the next level service; The pedestrian detection branch Pipeline includes: Image source input module, which is used to obtain pedestrian images from the main pipeline; The pedestrian attribute and feature extraction module interacts with the image source input module and is configured with a pedestrian detection model. The pedestrian detection model is a neural network model built based on deep learning, which is used to perform attribute analysis and feature extraction on pedestrian images through the pedestrian detection module to obtain pedestrian analysis results. The result return module interacts with the pedestrian attribute and feature extraction module to return the pedestrian analysis results to the main pipeline. The pedestrian detection model is used to detect pedestrians' gender, age, whether they are wearing masks, and whether they are wearing glasses, to obtain pedestrian feature values ​​and thus obtain pedestrian analysis results. The vehicle detection branch pipeline is configured with the following modules: Image source input module, which is used to obtain vehicle images from the main pipeline; The decoding preprocessing module is used to perform decoding preprocessing on vehicle images. By performing image cropping and normalization on the vehicle images, it outputs vehicle images that meet a unified standard. The vehicle attribute analysis and feature extraction module is equipped with a vehicle detection model, which is a neural network model built based on deep learning. The vehicle detection model is used to perform vehicle attribute analysis and extraction on vehicle images and output vehicle analysis results. The return result module is used to return the vehicle analysis results back to the main pipeline; The vehicle detection model is used to identify vehicle brand attributes and license plate attributes, obtain corresponding feature values, and obtain vehicle analysis results. Vehicle information is obtained by identifying the vehicle, including vehicle color, items in front of the vehicle, whether the driver is making a phone call, whether the driver is wearing a seat belt, whether the passenger is wearing a seat belt, vehicle brand, vehicle model, vehicle year, and vehicle type. License plate information is obtained by identifying the license plate, including the license plate number and license plate location coordinates.

10. The pipeline-based parallel video structured inference system according to claim 8 or 9, characterized in that... The Pipeline splitting module is used to calculate the number of main pipelines that can be parallelized based on the server's video memory size and the size of the models in the main pipeline and branch pipelines. The formula for calculating the number of main pipelines is as follows: Among them, X m Indicates the size of the server's video memory, X d This indicates the size of the models in the main pipeline and branch pipelines, where the models include a pedestrian and vehicle detection model, a pedestrian detection model, and a vehicle detection model; num is the integer result obtained by rounding down.

Images