Structured target detection method, device, equipment and storage medium

Abstract

The embodiments of the present application disclose a structured target detection method, apparatus, device and storage medium, the method comprising: obtaining a batch of video frame images to be processed; performing target detection and tracking on at least two targets in the batch of video frame images to obtain detection and tracking results for each of the targets; determining an association relationship between the at least two targets based on the detection and tracking results of the at least two targets; determining a target snapshot image of the corresponding target according to a preset frame selection strategy based on the detection and tracking result of each of the targets; the frame selection strategy is used to constrain the number of target snapshot images of the same target; and determining the structured targets of the batch of video frame images based on the association relationship between the at least two targets and the target snapshot image of each of the targets.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and is related to but not limited to structured target detection methods, devices, equipment and storage media. Background Art

[0002] Video structured analysis is a common application of computer vision and deep learning technologies in video surveillance analysis. It generally includes detection and tracking of various human-centric targets and analysis of their characteristic attributes. Currently, common video structured analysis technologies suffer from a limited number of target types, simple functions, and slow performance when dealing with a large number of targets. Summary of the Invention

[0003] Embodiments of the present application provide a structured target detection method, apparatus, device, and storage medium.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a structured object detection method, comprising:

[0006] Obtain a batch of video frame images to be processed;

[0007] Performing target detection and tracking on at least two targets in the batch of video frame images to obtain detection and tracking results for each target;

[0008] Determining an association relationship between the at least two targets based on detection and tracking results of the at least two targets;

[0009] Based on the detection and tracking results of each target, determining a target snapshot image corresponding to the target according to a preset frame selection strategy; the frame selection strategy is used to restrict the number of target snapshot images of the same target;

[0010] Based on the association relationship between the at least two objects and the object snapshot image of each object, the structured objects of the batch of video frame images are determined.

[0011] In some possible implementations, obtaining a batch of video frame images to be processed includes: obtaining at least two video streams captured by a camera module; decoding one of the at least two video streams frame by frame through at least two threads to obtain a current frame image in the corresponding video stream; and using the current frame image in the at least two video streams as the batch of video frame images.

[0012] In this way, multiple videos are decoded into a frame image simultaneously in multiple threads to form a batch of video frame images, which makes it easier to simultaneously process the analysis problem of full target structure under multiple video streams and reduce the problem of fewer target types in related video structuring technologies.

[0013] In some possible implementations, the detection and tracking results include a detection frame, a tracking identifier, and a tracking trajectory of the target. The target detection and tracking are performed on at least two targets in the batch of video frame images to obtain a detection and tracking result for each target, including: performing target detection on at least two targets in the current frame image in the batch of video frame images to obtain a detection frame for each target; and tracking the corresponding target based on the detection frame of each target in the batch of video frame images to obtain a tracking identifier and a tracking trajectory for the corresponding target.

[0014] In this way, multiple targets on the current frame image are first detected to obtain the detection frame of each target, and then the corresponding target is further tracked based on the detection frame of the target, thereby ensuring the detection accuracy of a single target.

[0015] In some possible embodiments, the at least two targets include a key target and a non-key target; the key target is used to characterize the identity of the object to be detected; the target detection is performed on the at least two targets of the current frame image in the batch of video frame images to obtain a detection frame of each target, including: performing target detection on the key target of the current frame image in the batch of video frame images to obtain a detection frame of the key target; in response to the current frame image being a key frame image, performing target detection on the non-key target in the current frame image to obtain an initial detection frame of the non-key target; the key frame image is sampled according to a preset sampling interval number; in response to the current frame image being a non-key frame image, single target tracking is performed on the non-key target in the current frame image to obtain an updated detection frame of the non-key target.

[0016] In this way, for non-key targets, the combination of key frame detection and non-key frame prediction can prevent the use of model detection in every frame, improve the efficiency of target detection, and thus improve the overall performance.

[0017] In some possible implementations, each current frame image in the batch of video frame images carries a timestamp; in response to the current frame image being the non-key frame image, single-target tracking is performed on the non-key target in the current frame image to obtain an updated detection frame of the non-key target, including: initializing a single-target tracker based on the initial detection frame of the non-key target; the initial detection frame is detected in a key frame image whose timestamp is earlier than the current frame image; and predicting the position of the non-key target in the current frame image through the initialized single-target tracker to obtain an updated detection frame of the non-key target.

[0018] In this way, the initial detection frame of the non-key target detected in the key frame is used to initialize the single target tracker, and the updated detection frame of the non-key target in the non-key frame is predicted by the single target tracker, thereby taking advantage of the advantages of fast single target tracking and high prediction accuracy to improve the overall performance.

[0019] In some possible implementations, tracking the corresponding target based on the detection frame of each target in the batch of video frame images to obtain the tracking identifier and tracking trajectory of the corresponding target includes: inputting the detection frames of all the targets in the batch of video frame images into a multi-target tracker to obtain the tracking identifier of each target; and determining the tracking trajectory of the corresponding target based on the detection frames belonging to the same target and the tracking identifier of the corresponding target in one video stream.

[0020] In this way, by outputting the tracking identification of each target and determining the tracking trajectory through the multi-target tracker, the isolated target frames detected in each frame can be strung together and processed as targets with the same tracking identification, which facilitates the reduction of data processing volume during subsequent target association and frame selection analysis.

[0021] In some possible implementations, the at least two targets include the following: a face target, a human target, a motor vehicle target, a non-motor vehicle target, and a rider target; determining the association relationship between the at least two targets based on the detection and tracking results of the at least two targets includes: determining a human face matching relationship belonging to the same person based on the positional relationship between the detection frame of the face target and the detection frame of the human target; determining a person getting on and off the vehicle relationship between the human target and the motor vehicle target based on the positional relationship between the detection frame of the human target and the detection frame of the motor vehicle target; determining a riding relationship or a cart pushing relationship between the rider target and the non-motor vehicle based on the positions of the respective detection frames of the rider target and the non-motor vehicle target, and the respective tracking trajectories of the rider target and the non-motor vehicle target.

[0022] In this way, the detection frame of each target is used to match and associate the face target and human target belonging to the same person, and to associate the human target and motor vehicle target that have the relationship of getting on and off the vehicle; at the same time, the detection frame and tracking trajectory of each target are combined to associate the human target and non-motor vehicle target, so that the video stream can be finally parsed and stored as a structured target.

[0023] In some possible embodiments, the detection and tracking results also include a quality score of the target, and the detection and tracking results based on each of the targets are used to determine the target snapshot image of the corresponding target according to a preset frame selection strategy, including: in response to the quality score of the first target in the current frame image satisfying a first quality threshold, determining a candidate snapshot image based on the detection frame of the first target; the first target is any one of the at least two targets; storing the candidate snapshot image in the snapshot cache pool of the first target; in response to the snapshot cache pool of the first target reaching the configured frame selection output condition, determining that the snapshot image stored in the snapshot cache pool is the target snapshot image of the corresponding first target.

[0024] In this way, for each target, we first evaluate whether the quality detection score detected in the current frame image meets the first quality threshold, i.e., the frame selection threshold, and then determine the candidate snapshot images of the target and store them in the snapshot cache pool. Finally, according to the configured frame selection output conditions, the best quality target snapshot images stored in the snapshot cache pool are screened out, thereby saving storage resources while improving the accuracy of subsequent feature extraction.

[0025] In some possible implementations, storing the candidate captured image in the capture cache pool of the first target includes: when the number of cached images in the capture cache pool of the first target does not reach a cache threshold, directly storing the candidate captured image in the capture cache pool of the first target.

[0026] In this way, by comparing the number of images cached in the snapshot buffer pool with the cache threshold, it is determined that the target snapshot buffer pool is not full and then the candidate snapshot images are stored in the snapshot buffer pool, thereby reducing the loss of candidate snapshot images of the target.

[0027] In some possible implementations, storing the candidate snapshot image in the snapshot cache pool of the first target includes: when the number of cached images in the snapshot cache pool of the first target reaches the cache threshold, deleting the first snapshot image in the snapshot cache pool; the quality score of the first snapshot image is lower than the quality score of the candidate snapshot image; and storing the candidate snapshot image in the snapshot cache pool of the first target.

[0028] In this way, by comparing the number of images cached in the snapshot cache pool with the cache threshold, when it is determined that the target's snapshot cache pool is full, the snapshot images with poor detection quality in the snapshot cache pool are first deleted, and then the candidate snapshot images are stored in the snapshot cache pool, reducing the overflow of the target's snapshot cache pool, which leads to the loss of snapshot images with better quality.

[0029] In some possible implementations, the method further includes: in response to the snapshot buffer pool of the first target meeting the frame selection output condition, clearing the snapshot buffer pool of the first target.

[0030] In this way, when the capture buffer pool of the first target reaches the frame selection output condition, the frame selection process ends, and the capture buffer pool is cleared to facilitate the normal frame selection process of the next batch of video frames.

[0031] In some possible embodiments, the configured frame selection output conditions include at least one of the following: a quick frame selection condition, a periodic frame selection condition, a maximum tracking time frame selection condition, a quality threshold frame selection condition and a tracking end frame selection condition; the quick frame selection condition is used to characterize that the time length of continuous tracking of the first target meets the first time length threshold; the periodic frame selection condition is used to characterize that the time interval for tracking the first target meets the interval threshold; the maximum tracking time frame selection condition is used to characterize that the cumulative time length for tracking the first target reaches the second time length threshold; the second time length threshold is greater than or equal to the first time length threshold; the quality threshold frame selection condition is used to characterize that there is an image with a quality score that meets the second quality threshold in the capture cache pool of the first target; the second quality threshold is greater than the first quality threshold; the tracking end selection condition is used to characterize the full tracking of the first target until the end of the acquired video stream.

[0032] In this way, the frame selection output conditions for each target are pre-configured as needed, and multiple frame selection conditions are used to constrain the number of target capture images of the same target used for analysis, thereby improving the quality of target screening, reducing redundant calculations, saving storage resources and improving performance.

[0033] In some possible implementations, determining the structured targets of the batch of video frame images based on the association relationship between the at least two targets and the target captured image of each target includes: performing feature extraction and / or attribute extraction on the target captured image of each target to obtain the parsing result of the corresponding target; associating the parsing results of the at least two targets based on the association relationship between the at least two targets; and determining the structured targets of the batch of video frame images based on the parsing results of the at least two targets after association.

[0034] In this way, the captured target images after frame selection are analyzed, and the analysis results are integrated based on the correlation between different targets as a batch of structured targets of video frame images, which improves the richness of the structured analysis items of the main core targets of concern in the video stream scene.

[0035] In some possible implementations, the target is a facial target, and the number of the target captured images of the facial target is greater than or equal to 2; the feature extraction and / or attribute extraction are performed on the target captured image of each of the targets to obtain the analysis result of the corresponding target, including: feature extraction is performed on at least two target captured images of the facial target respectively to obtain at least two feature vectors; the at least two feature vectors are fused to obtain the fusion feature of the facial target; the attribute extraction is performed on the target captured image with the highest quality detection score among the at least two target captured images to obtain the attributes of the facial target; the fusion feature of the facial target and the attributes of the facial target are used as the analysis result of the facial target.

[0036] In this way, by extracting features from multiple captured images of human faces and performing feature fusion, the accuracy of the fused features is relatively high; at the same time, attribute extraction is performed on the best quality captured image to improve the overall detection performance.

[0037] In some possible implementations, the method is executed in a parallel pipeline manner through functional modules in at least two threads; wherein, the parallel pipeline manner represents that between each two dependent functional modules, the result data output by the previous functional module is used as raw material data for the next functional module through a cache queue.

[0038] In this way, a multi-threaded design is adopted to execute the structured target detection method with multiple functional modules. There is a cache queue between each two dependent modules, so that the entire system can be parallelized in a pipeline manner to maintain high throughput and high performance.

[0039] In a second aspect, an embodiment of the present application provides a structured target detection device, comprising an acquisition module, a detection and tracking module, an association and matching module, a frame selection module, and a determination module, wherein:

[0040] The acquisition module is used to acquire a batch of video frame images to be processed;

[0041] The detection and tracking module is used to perform target detection and tracking on at least two targets in the batch of video frame images to obtain detection and tracking results for each target;

[0042] The association matching module is configured to determine an association relationship between the at least two targets based on the detection and tracking results of the at least two targets;

[0043] The frame selection module is used to determine the target captured image of the corresponding target according to a preset frame selection strategy based on the detection and tracking results of each target; the frame selection strategy is used to restrict the number of the target captured images of the same target;

[0044] The determination module is configured to determine the structured targets of the batch of video frame images based on the association relationship between the at least two targets and the target snapshot image of each target.

[0045] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps in the above-mentioned structured target detection method are implemented.

[0046] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the above-mentioned structured target detection method when executed by a processor.

[0047] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0048] In an embodiment of the present application, by performing main process steps such as detection, tracking, association, frame selection and analysis on multiple targets in a batch of video frame images, the problem of full analysis of the main core targets in the fully structured analysis scenario of the video stream is solved. At the same time, the structured analysis items of the core targets are enriched, and the performance of the full-target structured video analysis is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. Those skilled in the art can also derive other drawings based on these drawings without inventive work, among which:

[0050] Figure 1 A schematic diagram of a flow chart of a structured target detection method provided in an embodiment of the present application;

[0051] Figure 2 A schematic diagram of a flow chart of a structured target detection method provided in an embodiment of the present application;

[0052] Figure 3A A schematic diagram of a flow chart of a structured target detection method provided in an embodiment of the present application;

[0053] Figure 3BA logical flow chart of the target detection and tracking process provided in an embodiment of the present application;

[0054] Figure 4 A schematic diagram of a flow chart of a structured target detection method provided in an embodiment of the present application;

[0055] Figure 5 A schematic diagram of a flow chart of a structured target detection method provided in an embodiment of the present application;

[0056] Figure 6 A schematic diagram of the structure of a structured target detection device provided in an embodiment of the present application;

[0057] Figure 7 A hardware entity diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. The following examples are used to illustrate the present application, but are not intended to limit the scope of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0059] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0060] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with the preset order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0061] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the embodiments of the present application belong. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art, and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0062] The solution provided in the embodiments of the present application relates to the field of deep learning technology. To facilitate understanding of the solution in the embodiments of the present application, a brief description of the terms involved in the related technology is first provided:

[0063] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also encompasses the study of the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0064] Artificial intelligence technology is a comprehensive discipline covering a wide range of fields, encompassing both hardware and software technologies. Basic AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning. The embodiments of this application involve machine learning technology.

[0065] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0066] Video structured analysis is a common application of computer vision and deep learning technologies in video surveillance analysis. It generally includes functions such as detection, tracking, and feature attribute analysis of various human-centric targets. The term "structured" is used in contrast to unstructured data. It primarily refers to detecting and continuously tracking targets of interest within video images consisting of only large-scale tiled pixels (unstructured data), extracting relationships between these targets, and performing in-depth attribute and feature extraction on each target. This data is ultimately stored in a database in a structured format for subsequent retrieval and clustering.

[0067] Common video structuring technologies in related technologies have problems such as a small number of target types, simple functions, or slow speed when there are many targets, or they cover a wide range of types but have poor performance.

[0068] The embodiment of the present application provides a structured target detection method, which is applied to a server or electronic device. The electronic device includes but is not limited to a mobile phone, a laptop computer, a tablet computer and a handheld Internet access device, a multimedia device, a streaming media device, a mobile Internet device, a wearable device or other types of devices. The function implemented by the method can be implemented by calling a program code by a processor in the electronic device. Of course, the program code can be stored in a computer storage medium. It can be seen that the electronic device includes at least a processor and a storage medium. The processor can be used to process the structured target detection process, and the memory can be used to store the data required and the data generated during the structured target detection process.

[0069] Figure 1 A flow chart of a structured target detection method provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the method comprises at least the following steps:

[0070] Step S110, obtaining a batch of video frame images to be processed;

[0071] Here, the batch of video frame images includes at least two video frame images.

[0072] In some implementations, the video frame image may be an image acquisition device provided on the electronic device, such as a video frame image in a video stream acquired in real time by a camera module; in other implementations, the video frame image may be an image transmitted to the electronic device by other devices through instant messaging for target detection; in still other implementations, the electronic device may respond to a task processing instruction, call a locally stored video file, and obtain a video frame image therefrom, and the embodiments of the present application do not impose any restrictions on this.

[0073] Step S120, performing target detection and tracking on at least two targets in the batch of video frame images to obtain detection and tracking results for each target;

[0074] Here, the at least two objects are important objects or objects of interest in the field of video structured analysis, such as human-centered face objects, human bodies, motor vehicle objects, non-motor vehicle objects, and rider objects. In implementation, various objects of interest are detected and continuously tracked in each frame of a batch of video frames.

[0075] The process of object detection refers to the process of detecting and classifying the locations of multiple objects within a video frame using a deep learning algorithm model. The target detection and tracking results can include the target detection box (i.e., target box), target type, tracking identifier, tracking trajectory, etc.

[0076] In some possible implementations, before performing target detection on video frame images, a batch of acquired video frame images may be normalized, i.e., the mean is reduced and the variance is divided to eliminate the average characteristics of the video frame images and retain the difference characteristics of the video frame images.

[0077] Step S130, determining an association relationship between the at least two targets based on the detection and tracking results of the at least two targets;

[0078] Here, after obtaining the detection and tracking results of each target in the same video frame image, the association relationship between the face target and the human body target belonging to the same person can be identified based on the target's detection frame and tracking identifier; the two association relationships of a person getting on a vehicle and a person getting off a vehicle can also be identified based on the relationship between the detection frames of the two targets; and the association relationship of a person pushing (riding) a non-motor vehicle being stationary or moving forward can also be identified based on the target frame position and tracking trajectory.

[0079] Step S140, based on the detection and tracking results of each target, determining a target captured image corresponding to the target according to a preset frame selection strategy;

[0080] Here, after tracking all detected targets in a batch of video frame images and determining the association relationship, the detection quality of the targets can be evaluated, and then the video frames with quality scores that meet the requirements can be screened out for analysis.

[0081] The frame selection strategy is used to constrain the number of target snapshot images of the same target; wherein, frame selection refers to sorting the quality scores of the target snapshot images of a certain tracking target on each frame during the entire tracking trajectory, and selecting the target on the frame with the best quality for output.

[0082] Among them, the tracking target refers to a target carrying a tracking mark, which exists continuously in a series of video frame images, and the target capture image is a regional image captured from the video frame image based on the detected target detection frame, including the target information of the tracking target on a certain frame image (including the target detection frame, target thumbnail and other detection information), which is used for subsequent feature extraction, attribute analysis, etc.

[0083] It should be noted that the quality score is calculated for each target separately. There may be different calculation methods for different targets, but ultimately each target needs to calculate a quality score of 0 to 1 to express the quality of the target. This is used to sort the target images by quality score and select the best quality target snapshot images for feature and attribute analysis.

[0084] Step S150 : determining structured objects of the batch of video frame images based on the association relationship between the at least two objects and the captured object image of each object.

[0085] Here, feature extraction and attribute analysis are performed on the target captured image of each target output in the selected frame, and combined with the correlation relationship between different targets, the analysis results are integrated and output as a batch of structured targets of video frame images.

[0086] After the features and attributes of the target captured images of all targets in the selected frame output are extracted and analyzed, all targets are integrated and output together, and then each frame image of each video stream is looped and executed until all videos are processed.

[0087] It should be noted that the process described in steps S110 to S150 of the embodiment of the present application is a single processing process of a batch of video frame images. During actual operation, the process in steps S110 to S150 is executed in a loop, and the latest batch of video frame images are continuously obtained for processing until the processing of all video frames is completed or the user issues a stop command.

[0088] In an embodiment of the present application, by performing main process steps such as detection, tracking, association, frame selection and analysis on multiple targets in a batch of video frame images, the problem of full analysis of the main core targets in the fully structured analysis scenario of the video stream is solved. At the same time, the structured analysis items of the core targets are enriched, and the performance of the full-target structured video analysis is improved.

[0089] In some possible implementations, the above steps S110 to S150 are executed in a parallel pipeline manner through functional modules in at least two threads; wherein, the parallel pipeline manner represents that between each two dependent functional modules, the result data output by the previous functional module is used as the raw material data of the next functional module through a cache queue.

[0090] For example, the fully structured parsing process for video streams is broken down into distinct modules by function. Each module completes a separate functional step, primarily including detection, tracking, frame selection, matching and association, attribute extraction, license plate extraction, and feature extraction. Modules are grouped according to granularity and executed in separate threads, enabling parallel pipeline execution to improve performance. Furthermore, based on the dependencies between modules, cache queues of a certain size are established between them, allowing the entire system to operate in parallel, maintaining high throughput and performance.

[0091] In this way, a multi-threaded design is adopted to execute the structured target detection method with multiple functional modules. There is a cache queue between each two dependent modules, so that the entire system can be parallelized in a pipeline manner to maintain high throughput and high performance.

[0092] In some possible embodiments, the detection and tracking result includes a detection frame, a tracking mark, and a tracking trajectory of the target. Figure 2 A flow chart of a structured target detection method provided in an embodiment of the present application is shown as follows: Figure 2 As shown, the method comprises at least the following steps:

[0093] Step S210, obtaining at least two video streams captured by the camera module;

[0094] Step S220: decoding one of the at least two video streams frame by frame using at least two threads to obtain a current frame image in the corresponding video stream;

[0095] Here, a video decoder is used in each thread to decode one video stream or video file frame by frame. For example, thread A decodes the first video stream frame by frame, and thread B decodes the second video stream frame by frame. In this way, multiple threads can be used to process multiple video streams at the same time, and the current frame image in each video stream can be obtained at the same time.

[0096] It should be noted that the decoding algorithm can use soft decoding or hardware decoding: soft decoding generally refers to pure central processing unit software decoding, which is slow; while hardware decoding refers to the use of specific hardware for video decoding, which is fast. The embodiment of the present application does not limit the decoding method. After being decoded by the decoder, the video stream generally becomes a color frame image in pixel format (YUV) or red, green and blue format (RGB). Since the image memory of the grayscale part of the YUV format image is continuous, it can be directly taken out and used. For deep learning models that only need grayscale images, the processing efficiency is higher. RGB format images need to be converted into grayscale images in some deep learning models, and the efficiency may be relatively low.

[0097] The video frame images in the video stream are generally continuous frames with millisecond timestamp information. If not, you need to manually add a millisecond-level continuously increasing timestamp field to the video frame image for subsequent multi-target tracking and frame selection.

[0098] Step S230, using the current frame image in the at least two video streams as the batch of video frame images;

[0099] Here, multiple video streams are decoded simultaneously in multiple threads to obtain current frame images, forming a batch, i.e., a group of video frame images.

[0100] Step S240, performing target detection on at least two targets in the current frame image among the batch of video frame images to obtain a detection frame for each target;

[0101] Here, the target detection box is the target box. Target detection can be performed on a frame of image using a single-classification detection model or a multi-classification detection model.

[0102] In some possible implementations, a single-category detection model is used to detect each target type in turn to obtain a detection frame for the target. For example, for a face, which is the only target that can confirm a person's identity, a single-category detection model is used to detect the face in each video frame to ensure accurate detection of the face.

[0103] In another possible implementation, a multi-classification detection model is used to simultaneously detect multiple targets, generating detection frames for each target within a single frame. For example, for non-key targets such as humans, motor vehicles, non-motor vehicles, and riders, the multi-classification detection model is used to detect targets within video frames selected at a specific sampling interval. This reduces the time-consuming nature of model detection and speeds up the process.

[0104] Step S250, tracking the corresponding target based on the detection frame of each target in the batch of video frame images to obtain a tracking identifier and a tracking trajectory of the corresponding target;

[0105] Here, the target tracking identifier is output by the multi-target tracker. Usually, the same target in the upper and lower frames has the same and unique tracking identifier, which can be used for subsequent target association or frame selection analysis.

[0106] During implementation, we first detect multiple targets on the current frame image to obtain the detection frame of each target, and then further track the corresponding target based on the target detection frame. Through tracking identification, the isolated target frames detected in each frame are strung together to form a tracking trajectory, thereby ensuring the detection accuracy of a single target.

[0107] Step S260: determining an association relationship between the at least two targets based on the detection and tracking results of the at least two targets;

[0108] Here, the detection and tracking result includes the detection frame, tracking mark and tracking trajectory of the target.

[0109] The association relationship between the at least two targets may include a human face matching relationship, a person getting on or off a vehicle relationship, a person riding a vehicle relationship, or a person pushing a vehicle relationship, etc. The association relationship identification between various targets may be performed simultaneously or separately without affecting each other.

[0110] Among them, the human-face matching relationship mainly refers to the association between the face target and the human target belonging to the same person; the relationship between people getting on and off the vehicle refers to the association between the human target and the motor vehicle target, which are two types of association relationships: people getting on the vehicle and people getting off the vehicle; the relationship between people riding a bicycle and people pushing a bicycle both refer to the association between the human target and the non-motor vehicle target, where the person is pushing the non-motor vehicle to be stationary or moving forward, etc.

[0111] The above-mentioned association relationship can be determined through the following process: based on the positional relationship between the detection frame of the face target and the detection frame of the human target, the human face matching relationship belonging to the same person is determined; based on the positional relationship between the detection frame of the human target and the detection frame of the motor vehicle target, the relationship between the human target and the motor vehicle target in terms of getting on and off the vehicle is determined; based on the position of the respective detection frames of the rider target and the non-motor vehicle target, and the respective tracking trajectories of the rider target and the non-motor vehicle target, the relationship between the rider target and the non-motor vehicle in terms of riding or pushing the vehicle is determined. In this way, the detection frames of each target are used to match and associate the face target and the human target belonging to the same person, and to associate the human target and the motor vehicle target with the relationship of getting on and off the vehicle; at the same time, the detection frames and tracking trajectories of each target are combined to associate the human target and the non-motor vehicle target, so that the video stream can be parsed and stored as a structured target.

[0112] Step S270, based on the detection and tracking results of each target, determining a target captured image corresponding to the target according to a preset frame selection strategy;

[0113] Here, the frame selection strategy is used to constrain the number of the target captured images of the same target.

[0114] Step S280 : determining the structured objects of the batch of video frame images based on the association relationship between the at least two objects and the captured object image of each object.

[0115] In an embodiment of the present application, first, a frame of image is decoded simultaneously in multiple threads for multiple channels of video to form a batch of video frame images, and then target detection is performed on a batch of video frame images at the same time to obtain the detection frame of each target, and then the corresponding target is tracked based on the detection frame of the target, thereby ensuring the detection accuracy of a single target, improving the performance of the fully structured parsing system, and enriching the target types of the fully structured parsing.

[0116] In some embodiments, the at least two targets include a key target and a non-key target; the key target is used to characterize the identity of the object to be detected. Figure 1 , Figure 3A A flow chart of a structured target detection method provided in an embodiment of the present application is shown as follows: Figure 3A As shown, the above step S120 of "performing target detection and tracking on at least two targets in the batch of video frame images to obtain detection and tracking results for each target" is implemented by the following steps:

[0117] Step S310, performing target detection on the key target of the current frame image in the batch of video frame images to obtain a detection frame of the key target;

[0118] Here, a key target is an object in the current frame that uniquely identifies the subject being detected. For most everyday scenarios, such as behavior recognition and crowd counting, which primarily involve various human-centric target detection applications, faces are key targets, uniquely confirming a person's identity.

[0119] In the implementation, a single-classification detection model is used to detect the face target for each frame in the video stream, and the detection frame of the face target in each frame image, namely the face frame, is obtained to ensure the accuracy of the face target.

[0120] Step S320: In response to the current frame image being a key frame image, performing target detection on the non-key target in the current frame image to obtain an initial detection frame of the non-key target;

[0121] Here, the key frame image is sampled according to a preset sampling interval. In some embodiments, a key frame can be determined by taking the frame number (frame_index) modulo the frame sampling interval number (block_num) (i.e., frame_index%block_num) to see whether the result is equal to zero. If the result is equal to zero, it is a key frame, and if the result is not equal to zero, it is a non-key frame.

[0122] The non-key targets are other targets in the video frame except the key targets, such as human targets related to personnel, motor vehicle targets, non-motor vehicle targets, and rider targets.

[0123] In keyframe images, a multi-classification detection model is used to simultaneously detect various non-key targets, generating initial detection frames for each non-key target. Multi-classification models used for these target types are typically larger and slower than face detection models. Using this model for every frame is time-consuming, so simultaneous multi-target detection is performed only when the current frame is a keyframe.

[0124] Step S330: In response to the current frame image being a non-key frame image, single target tracking is performed on the non-key target in the current frame image to obtain an updated detection frame of the non-key target;

[0125] Here, each current frame in the batch of video frames carries a timestamp. After detecting the initial detection frames of non-key targets in the key frames in the previous step, the initial detection frames of each non-key target are used to predict the updated detection frames of each non-key target in subsequent non-key frames. This prevents the need to use the model for detection of non-key targets in every frame, improving target detection efficiency and overall performance.

[0126] In some embodiments, a single target tracker is initialized based on an initial detection frame of the non-key target; the initial detection frame is detected in a key frame image whose timestamp is earlier than the current frame image; and the position of the non-key target in the current frame image is predicted by the initialized single target tracker to obtain an updated detection frame of the non-key target.

[0127] In this way, the initial detection frame of the non-key target detected in the key frame is used to initialize the single target tracker, and the updated detection frame of the non-key target in the non-key frame is predicted by the single target tracker, thereby taking advantage of the advantages of fast single target tracking and high prediction accuracy to improve the overall performance.

[0128] Step S340: inputting the detection frames of all the targets in the batch of video frame images into a multi-target tracker to obtain a tracking identifier for each target;

[0129] Here, for all current frame images in a batch of video frame images, after the key target detection in the above-mentioned step S310, and the detection and prediction of non-key targets in steps 320 to S330, the positions and types of all targets on the video frame images have been determined, and the multi-target tracker module can be used to perform multi-target tracking to obtain the tracking identifier of each target frame.

[0130] Step S350 : determining a tracking trajectory of the corresponding target based on the detection frames belonging to the same target in one video stream and the tracking identifier of the corresponding target.

[0131] Here, the detection frames with the same tracking identifier detected in each current frame image are strung together and processed as the tracking trajectory of the target to facilitate subsequent frame selection analysis.

[0132] It is worth noting that the reason why the above steps S320 to S330 adopt the key frame detection and non-key frame prediction method for non-key targets is because the multi-classification detection models used for human targets, motor vehicle targets, non-motor vehicle targets and rider targets are generally larger and slower than the face detection model. If each frame is detected with the model, it will take a lot of time. The key frame detection and non-key frame prediction method takes advantage of the fast speed of single target tracking and improves the overall performance. The sampling interval number of key frames in single target tracking is generally 4 to 8 frames. If the sampling interval number is too small, the speed-up effect cannot be achieved. If the sampling interval number is too large, the single target prediction accuracy will decrease, and the predicted target detection frame will have a large deviation.

[0133] In the implementation, after key target detection is performed on each frame to obtain the key target detection frame, the current video frame is determined to be either a key frame or a non-key frame. The key frame detection is combined with the non-key frame prediction to obtain the non-key target detection frame. Figure 3B FIG. 1 is a logic flow chart of the target detection and tracking process provided by an embodiment of the present application, and the process includes the following steps:

[0134] Step S311, detecting a detection frame of a key target;

[0135] Step S312, determining whether the current frame image is a key frame image;

[0136] If the determination result is yes, execute step S313; if the determination result is no, execute step S315.

[0137] Step S313, detecting an initial detection frame of a non-key target;

[0138] Step S314, initializing a single target tracker;

[0139] Here, the single object tracker (SOT) refers to predicting the updated detection frames of each target in the next few frames based on the initial detection frames of each target in the key frame image. The process of initializing the single object tracker is to use the positions of each detection frame in the key frame image as the initial positions required for the single object tracker to track and predict the new position of the target.

[0140] Step S315, predicting an updated detection frame of the non-key target through a single target tracker;

[0141] Here, the same single object tracker is used to track different non-key targets in the current frame. The single object tracker can track and predict the positions of multiple different target boxes at once. The single object tracker is specifically optimized for tracking these non-key targets.

[0142] In step S316, the detection frames of all detected targets are input into a multi-target tracker, and each target in the current frame image is tracked separately.

[0143] Here, the multi-object tracker (MOT) aims to associate the detection boxes belonging to the same target in the previous and next frames and assign a unique tracking identifier. The main process is to input the detection boxes of all targets in the current frame and output the tracking identifier corresponding to the detection box of each target. The same target in the previous and next frames has the same, unique tracking identifier.

[0144] In an embodiment of the present application, the initial detection frame of the non-key target detected in the key frame is used to initialize the single target tracker, and the updated detection frame of the non-key target in the non-key frame is predicted by the single target tracker, thereby taking advantage of the advantages of fast single target tracking and high prediction accuracy to improve overall performance. At the same time, in this way, by outputting the tracking identifier of each target and determining the tracking trajectory through the multi-target tracker, the isolated target frames detected in each frame can be strung together and processed as targets with the same tracking identifier, which is convenient for reducing the amount of data processing during subsequent target association and frame selection analysis.

[0145] In some embodiments, the detection and tracking result also includes the quality score of the target. Figure 1 , Figure 4 A flow chart of a structured target detection method provided in an embodiment of the present application is shown as follows: Figure 4 As shown, the above step S140 "determining the target captured image of the corresponding target according to the preset frame selection strategy based on the detection and tracking results of each target" can be implemented by the following steps:

[0146] Step S410 , in response to a quality score of a first target in the current frame image meeting a first quality threshold, determining a candidate captured image based on a detection frame of the first target;

[0147] Here, the first target is any one of the at least two targets, and the first quality threshold represents the minimum quality score used as a frame selection condition, which is generally an empirical value, such as 0.3.

[0148] In some embodiments, if the quality score of the detected first target is not lower than a first quality threshold, an image of a region where the detection frame of the first target is located is captured from the current frame image as a candidate captured image.

[0149] In other embodiments, if the quality score of the detected first target is lower than a first quality threshold, the detection result of the first target is discarded.

[0150] Step S420, storing the candidate captured image in the captured buffer pool of the first target;

[0151] Here, it is first determined whether the snapshot buffer pool of the first target is full, and based on the determination result, it is determined whether the candidate snapshot image is directly stored in the snapshot buffer pool or whether the candidate snapshot image replaces the cached snapshot image with the lowest quality.

[0152] In some embodiments, if the number of images cached in the first target's snapshot cache pool does not reach a cache threshold, the candidate snapshot image is directly stored in the first target's snapshot cache pool. This effectively reduces the loss of candidate snapshot images of the target by comparing the number of images cached in the snapshot cache pool with the cache threshold and determining that the target's snapshot cache pool is not full before storing the candidate snapshot image in the snapshot cache pool.

[0153] In other embodiments, when the number of images cached in the first target's snapshot cache pool reaches the cache threshold, the first snapshot image in the snapshot cache pool is deleted; the quality score of the first snapshot image is lower than the quality score of the candidate snapshot image; and the candidate snapshot image is stored in the first target's snapshot cache pool. In this way, by comparing the number of images cached in the snapshot cache pool with the cache threshold, if it is determined that the target's snapshot cache pool is full, the snapshot images with poor detection quality in the snapshot cache pool are first deleted, and then the candidate snapshot images are stored in the snapshot cache pool, effectively reducing the possibility of the target's snapshot cache pool overflowing and causing the loss of high-quality snapshot images.

[0154] It is worth noting that since the type of each target and the time it enters and exits the screen are different, the timing of frame selection and output for each target is also different. For facial targets, facial features are more important for identity comparison. If only one snapshot is taken during the entire facial target tracking process for subsequent feature extraction, the feature accuracy may not be optimal. Through practice, it has been found that each facial tracking target caches 3 to 5 best snapshots during the frame selection process for subsequent feature extraction and feature fusion, and the accuracy of the fused features is relatively high. For human bodies, vehicles, non-motor vehicles and rider targets, since their features are not as important as facial features and cannot be effectively fused, the number of cached snapshots for these targets during the frame selection process is generally set to only 1, which not only ensures performance but also saves storage resources occupied by the cache.

[0155] Step S430, in response to the snapshot buffer pool of the first target meeting the configured frame selection output condition, determining that the snapshot image stored in the snapshot buffer pool is the target snapshot image corresponding to the first target;

[0156] Here, the configured frame selection output conditions include at least one of the following: rapid frame selection, periodic frame selection, maximum tracking time frame selection, quality threshold frame selection, and tracking end frame selection. By using a variety of flexible frame selection output conditions, information redundancy is reduced, improving target screening efficiency and quality.

[0157] The fast frame selection condition indicates that the duration of continuous tracking of the first target meets a first duration threshold. For example, a first duration threshold is set, and frame selection output is triggered once the target is continuously tracked for longer than this duration. This is typically used in scenarios where you want to select a frame for analysis shortly after a target enters the frame.

[0158] The periodic frame selection condition indicates that the time interval for tracking the first target meets the interval threshold. For example, frames are selected at regular intervals from the time the target enters the frame and is tracked for subsequent analysis. This is typically used when the target's status is constantly changing and you want to keep track of its characteristics. Shorter intervals increase the frequency of analysis and increase the time required.

[0159] The maximum tracking time frame selection condition is used to indicate that the cumulative time length of tracking the first target reaches a second time length threshold; the second time length threshold is greater than or equal to the first time length threshold. For example, when the target tracking time reaches the second time length threshold, frame selection output is generally used for target selection output that resides in the video screen for a long time (such as a vehicle parked in the screen). The difference from fast frame selection is that after the maximum tracking time is reached, no frame selection is performed after the target frame selection output, saving computing and storage resources.

[0160] The quality threshold frame selection condition is used to indicate that there are images in the snapshot buffer pool of the first target whose quality scores meet a second quality threshold; this second quality threshold is greater than the first quality threshold. For example, when the quality score of the target's cached snapshot reaches the second quality threshold, a frame is selected for output, and subsequent frame selection may be discontinued. Compared to maximum tracking time frame selection, using whether the quality score meets a higher threshold to determine the selection condition is more effective and practical in certain scenarios, saving computational effort while ensuring high-quality target selection.

[0161] The tracking end selection condition specifies that the first target will be tracked until the end of the captured video stream. In other words, after target tracking is complete, the best-quality captured image from the entire tracking trajectory is selected and output. This is typically the default strategy because it aligns with intuition. For general targets, selecting the best captured image from the entire tracking process is the optimal strategy.

[0162] Step S440 : in response to the snapshot buffer pool of the first target meeting the frame selection output condition, clearing the snapshot buffer pool of the first target.

[0163] Here, when the capture buffer pool of the first target meets the frame selection output condition, the frame selection process ends, and the capture buffer pool is cleared to facilitate the normal frame selection process of the next batch of video frames.

[0164] In the embodiment of the present application, for each target, the quality detection score detected in the current frame image is first evaluated to see if it meets the first quality threshold, i.e., the frame selection threshold. Then, candidate snapshot images of the target are determined and stored in the snapshot buffer pool. Finally, the best-quality target snapshot images stored in the snapshot buffer pool are filtered according to the configured frame selection output conditions, thereby saving storage resources while improving the accuracy of subsequent feature extraction. Multiple frame selection strategies are used simultaneously, and the frame selection logic for each target and each scene is configured as needed to reduce the duplication of computing and storage resources.

[0165] based on Figure 1 , Figure 5 A flow chart of a structured target detection method provided in an embodiment of the present application is shown as follows: Figure 5 As shown, the above step S150 of "determining the structured targets of the batch of video frame images based on the association relationship between the at least two targets and the target captured image of each target" includes the following steps:

[0166] Step S510, performing feature extraction and / or attribute extraction on the captured image of each target to obtain an analysis result of the corresponding target;

[0167] Here, for targets related to people, feature extraction is performed through the feature extraction model to obtain feature vectors of a certain dimension, and attribute extraction is performed through the attribute extraction model to obtain attributes such as face attributes, body attributes, and clothing accessories attributes; for other unimportant targets such as motor vehicle targets, only attribute extraction is performed through the attribute extraction model to obtain attributes such as color, type, license plate position and content.

[0168] During implementation, different types of targets are analyzed separately. For facial targets, feature extraction is performed on at least two captured images of the target face, yielding at least two feature vectors. These at least two feature vectors are then fused to yield a fused feature of the target face. Attribute extraction is performed on the target image with the highest quality detection score among the at least two captured images to yield the attributes of the target face. The fused features and attributes of the target face are then used as the parsing result for the target face. By extracting and fusing features from multiple captured images of target faces, the accuracy of the fused features is relatively high. Attribute extraction is also performed on the highest-quality captured image, improving overall detection performance.

[0169] For human targets, a frame is typically selected and a captured image of the target is output. A deep learning model is then used to directly extract human features. A human attribute model is then used to extract attributes from the human body. The features and attributes of the human body are then used as the parsing result for the human target. For motor vehicle targets, a vehicle attribute model is first used to extract attributes. Vehicle attributes, such as license plate location and content, are then used as the parsing result for the motor vehicle target. For non-motor vehicle targets, an attribute model is used to extract non-motor vehicle attributes, which are then used as the parsing result for the non-motor vehicle target.

[0170] Step S520, associating the parsing results of the at least two targets based on the association relationship between the at least two targets;

[0171] Here, associating the parsing results of each target through the association relationship between different targets can greatly reduce the amount of data analysis. For example, for the rider target, since the rider's main information comes from the human body, the attributes of the human target associated with the rider target are used as the attributes of the rider target.

[0172] Step S530 : determining the structured objects of the batch of video frame images based on the analysis results of the at least two associated objects.

[0173] Here, after the features and attributes of the target captured images of all types of selected frames are extracted and analyzed, the parsing results of all targets are integrated together and output as structured targets of a batch of video frame images.

[0174] In an embodiment of the present application, the captured target image after frame selection is analyzed and the analysis results are integrated based on the correlation between different targets as a batch of structured targets of video frame images, thereby improving the richness of the structured analysis items of the main core targets of concern in the video stream scene.

[0175] The structured target detection method provided in the embodiments of the present application is used in at least the following scenarios: efficiently detecting, tracking, and associating the main types of targets in smart city scenarios, and selecting and identifying targets by configuring different frame selection strategies. By storing the targets whose attributes and features have been analyzed in a database, when a specific target needs to be retrieved later, the target image can be directly input, and after extracting the attributes and features, all information about the target that has ever appeared can be searched in the database. Based on the association information between targets, other target information associated with the target can also be found, thereby mining more abundant information.

[0176] Based on the foregoing embodiments, an embodiment of the present application further provides a structured target detection device, which includes the modules included, and the sub-modules and units included in each module, and can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0177] Figure 6 A schematic diagram of the structure of a structured target detection device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the apparatus 600 includes an acquisition module 610, a detection and tracking module 620, an association and matching module 630, a frame selection module 640, and a determination module 650, wherein:

[0178] The acquisition module 610 is used to acquire a batch of video frame images to be processed;

[0179] The detection and tracking module 620 is configured to perform target detection and tracking on at least two targets in the batch of video frame images to obtain detection and tracking results for each target;

[0180] The association matching module 630 is configured to determine an association relationship between the at least two targets based on the detection and tracking results of the at least two targets;

[0181] The frame selection module 640 is configured to determine a target captured image of the corresponding target according to a preset frame selection strategy based on the detection and tracking results of each target; the frame selection strategy is configured to restrict the number of target captured images of the same target;

[0182] The determining module 650 is configured to determine the structured objects in the batch of video frame images based on the association relationship between the at least two objects and the captured object image of each object.

[0183] In some possible embodiments, the acquisition module 610 includes: an acquisition submodule, used to acquire at least two video streams captured by the camera module; a decoding submodule, used to decode one of the at least two video streams frame by frame through at least two threads to obtain the current frame image in the corresponding video stream; a first determination submodule, used to use the current frame image in the at least two video streams as the batch of video frame images.

[0184] In some possible embodiments, the detection and tracking results include a detection frame, a tracking identifier, and a tracking trajectory of the target, and the detection and tracking module 620 includes: a detection submodule, used to perform target detection on at least two targets in the current frame image in the batch of video frame images, and obtain a detection frame for each of the targets; and a tracking submodule, used to track the corresponding targets based on the detection frame of each of the targets in the batch of video frame images, and obtain a tracking identifier and a tracking trajectory for the corresponding targets.

[0185] In some possible embodiments, the at least two targets include key targets and non-key targets; the key targets are used to characterize the identity of the object to be detected; the detection submodule includes: a first detection unit, used to perform target detection on the key target in the current frame image in the batch of video frame images, and obtain a detection frame of the key target; a second detection unit, used to perform target detection on the non-key target in the current frame image in response to the current frame image being a key frame image, and obtain an initial detection frame of the non-key target; the key frame image is sampled according to a preset sampling interval number; a tracking unit, used to perform single target tracking on the non-key target in the current frame image in response to the current frame image being a non-key frame image, and obtain an updated detection frame of the non-key target.

[0186] In some possible embodiments, each current frame image in the batch of video frame images carries a timestamp; the tracking unit includes: an initialization subunit, used to initialize the single target tracker based on the initial detection frame of the non-key target; the initial detection frame is detected in the key frame image before the timestamp is earlier than the current frame image; a prediction subunit, used to predict the position of the non-key target in the current frame image through the initialized single target tracker to obtain an updated detection frame of the non-key target.

[0187] In some possible embodiments, the tracking submodule includes: a first determination unit, used to input the detection frames of all the targets in the batch of video frame images into a multi-target tracker to obtain a tracking identifier of each target; and a second determination unit, used to determine the tracking trajectory of the corresponding target based on the detection frames belonging to the same target in one video stream and the tracking identifier of the corresponding target.

[0188] In some possible embodiments, the at least two targets include the following: a face target, a human target, a motor vehicle target, a non-motor vehicle target and a rider target; the association matching module 630 includes: a second determination submodule, for determining the human face matching relationship belonging to the same person based on the positional relationship between the detection frame of the face target and the detection frame of the human target; a third determination submodule, for determining the relationship between the human target and the motor vehicle target for getting on and off the vehicle based on the positional relationship between the detection frame of the human target and the detection frame of the motor vehicle target; a fourth determination submodule, for determining the riding relationship or the pushing relationship between the rider target and the non-motor vehicle based on the positions of the respective detection frames of the rider target and the non-motor vehicle target, and the respective tracking trajectories of the rider target and the non-motor vehicle target.

[0189] In some possible embodiments, the detection and tracking result also includes the quality score of the target, and the frame selection module 640 includes: a fifth determination submodule, used to determine the candidate captured image based on the detection frame of the first target in response to the quality score of the first target in the current frame image satisfying the first quality threshold; the first target is any one of the at least two targets; a storage submodule, used to store the candidate captured image in the capture cache pool of the first target; and a sixth determination submodule, used to determine that the captured image stored in the capture cache pool is the target captured image corresponding to the first target in response to the capture cache pool of the first target reaching the configured frame selection output condition.

[0190] In some possible embodiments, the storage submodule is further configured to directly store the candidate captured image in the capture cache pool of the first target when the number of images cached in the capture cache pool of the first target does not reach a cache threshold.

[0191] In some possible embodiments, the storage submodule includes: a deletion unit, used to delete the first captured image in the capture cache pool when the number of images cached in the capture cache pool of the first target reaches the cache threshold; the quality score of the first captured image is lower than the quality score of the candidate captured image; and a storage unit, used to store the candidate captured image in the capture cache pool of the first target.

[0192] In some possible embodiments, the frame selection module 640 further includes a clearing submodule configured to clear the snapshot buffer pool of the first target in response to the snapshot buffer pool of the first target meeting the frame selection output condition.

[0193] In some possible embodiments, the configured frame selection output conditions include at least one of the following: a quick frame selection condition, a periodic frame selection condition, a maximum tracking time frame selection condition, a quality threshold frame selection condition and a tracking end frame selection condition; the quick frame selection condition is used to characterize that the time length of continuous tracking of the first target meets the first time length threshold; the periodic frame selection condition is used to characterize that the time interval for tracking the first target meets the interval threshold; the maximum tracking time frame selection condition is used to characterize that the cumulative time length for tracking the first target reaches the second time length threshold; the second time length threshold is greater than or equal to the first time length threshold; the quality threshold frame selection condition is used to characterize that there is an image with a quality score that meets the second quality threshold in the capture cache pool of the first target; the second quality threshold is greater than the first quality threshold; the tracking end selection condition is used to characterize the full tracking of the first target until the end of the captured video stream.

[0194] In some possible embodiments, the determination module 650 includes an extraction submodule for performing feature extraction and / or attribute extraction on the target captured image of each of the targets to obtain the analysis results of the corresponding targets; an association submodule for associating the analysis results of the at least two targets based on the association relationship between the at least two targets; and a seventh determination submodule for determining the structured targets of the batch of video frame images based on the analysis results of the at least two targets after association.

[0195] In some possible embodiments, the target is a face target, and the number of the target captured images of the face target is greater than or equal to 2; the extraction submodule includes: a feature extraction unit, which performs feature extraction on at least two of the target captured images of the face target respectively to obtain at least two feature vectors; a fusion unit, which is used to fuse the at least two feature vectors to obtain a fusion feature of the face target; an attribute extraction unit, which is used to perform attribute extraction on the target captured image with the highest quality detection score among the at least two target captured images to obtain the attributes of the face target; a third determination unit, which is used to use the fusion feature of the face target and the attributes of the face target as the analysis result of the face target.

[0196] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of this application, please refer to the description of the method embodiment of this application for understanding.

[0197] It should be noted that, in the embodiment of the present application, if the above-mentioned structured target detection method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a smart phone with a camera, a tablet computer, etc.) to execute all or part of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any preset hardware and software combination.

[0198] Correspondingly, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the structured target detection method described in any of the above embodiments. Correspondingly, an embodiment of the present application also provides a chip, which includes a programmable logic circuit and / or program instructions, which, when running, is used to implement the steps of the structured target detection method described in any of the above embodiments. Correspondingly, an embodiment of the present application also provides a computer program product, which, when executed by a processor of an electronic device, is used to implement the steps of the structured target detection method described in any of the above embodiments.

[0199] Based on the same technical concept, an embodiment of the present application provides an electronic device for implementing the structured target detection method described in the above method embodiment. Figure 7 A hardware entity diagram of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the electronic device 700 includes a memory 710 and a processor 720, and the memory 710 stores a computer program that can be run on the processor 720. When the processor 720 executes the program, the steps in any structured target detection method described in the embodiments of the present application are implemented.

[0200] The memory 710 is configured to store instructions and applications executable by the processor 720, and can also cache data to be processed or processed by the processor 720 and various modules in the electronic device (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (RAM).

[0201] When the processor 720 executes the program, the steps of any of the above structured object detection methods are implemented. The processor 720 generally controls the overall operation of the electronic device 700.

[0202] The processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It is understood that the electronic device that implements the functions of the processor may also be other electronic devices, which are not specifically limited in the embodiments of the present application.

[0203] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); it can also be various electronic devices that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0204] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0205] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these preset features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0206] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0207] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms. The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment scheme of this application.

[0208] In addition, the functional units in the various embodiments of the present application can all be integrated into a single processing unit, or each unit can be independently used as a unit, or two or more units can be integrated into a single unit; the above-mentioned integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units. Alternatively, if the above-mentioned integrated units of the present application are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling the automatic test line of the device to execute all or part of the methods described in the various embodiments of the present application. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks. The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined to obtain new method embodiments if they do not conflict. The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined to obtain new method embodiments or device embodiments if they do not conflict. The above is merely an embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A structured target detection method, characterized in that: The method comprises: Obtain a batch of video frame images to be processed; Performing target detection and tracking on at least two targets in the batch of video frame images to obtain detection and tracking results for each target; the detection and tracking results include a detection frame, a tracking identifier, and a tracking trajectory of the target; the at least two targets include a key target and a non-key target; the key target is a human face target used to represent the identity of the object to be detected; Determining an association relationship between the at least two targets based on detection and tracking results of the at least two targets; Based on the detection and tracking results of each target, determining a target snapshot image corresponding to the target according to a preset frame selection strategy; the frame selection strategy is used to restrict the number of target snapshot images of the same target; Determining structured objects of the batch of video frame images based on the association relationship between the at least two objects and the captured object image of each object; The performing target detection and tracking on at least two targets in the batch of video frame images to obtain a detection and tracking result for each target includes: Performing target detection on the key target in the current frame image among the batch of video frame images to obtain a detection frame of the key target; In response to the current frame image being a key frame image, target detection is performed on the non-key target in the current frame image to obtain an initial detection frame of the non-key target; the key frame image is sampled according to a preset sampling interval number; in response to the current frame image being a non-key frame image, single target tracking is performed on the non-key target in the current frame image to obtain an updated detection frame of the non-key target; Based on the detection frame of each target in the batch of video frame images, the corresponding target is tracked to obtain a tracking identifier and a tracking trajectory of the corresponding target.

2. The method according to claim 1, wherein The step of obtaining a batch of video frame images to be processed includes: Obtain at least two video streams captured by the camera module; Decoding one of the at least two video streams frame by frame using at least two threads to obtain a current frame image in the corresponding video stream; The current frame images in the at least two video streams are used as the batch of video frame images.

3. The method according to claim 1, wherein Each current frame image in the batch of video frame images carries a timestamp; In response to the current frame image being the non-key frame image, performing single target tracking on the non-key target in the current frame image to obtain an updated detection frame of the non-key target includes: Initializing a single target tracker based on an initial detection frame of the non-key target; the initial detection frame is detected in a key frame image whose timestamp is earlier than that of the current frame image; The initialized single target tracker is used to predict the position of the non-key target in the current frame image to obtain an updated detection frame of the non-key target.

4. The method according to claim 1, wherein Tracking the corresponding target based on the detection frame of each target in the batch of video frame images to obtain the tracking identifier and tracking trajectory of the corresponding target includes: Inputting the detection frames of all the targets in the batch of video frame images into a multi-target tracker to obtain a tracking identifier for each target; Based on the detection frames belonging to the same target in one video stream and the tracking identifier of the corresponding target, a tracking trajectory of the corresponding target is determined.

5. The method according to any one of claims 1 to 4, characterized in that The at least two targets include the following: a human face target, a human body target, a motor vehicle target, a non-motor vehicle target and a rider target; The determining the association relationship between the at least two targets based on the detection and tracking results of the at least two targets includes: Determining a human face matching relationship belonging to the same person based on a positional relationship between the detection frame of the face target and the detection frame of the human target; Determining a person getting on and off the vehicle relationship between the human target and the motor vehicle target based on a positional relationship between the detection frame of the human target and the detection frame of the motor vehicle target; Based on the positions of the detection frames of the rider target and the non-motor vehicle target, and the tracking trajectories of the rider target and the non-motor vehicle target, a riding relationship or a pushing relationship between the rider target and the non-motor vehicle target is determined.

6. The method according to any one of claims 1 to 4, characterized in that The detection and tracking result also includes a quality score of the target. The method of determining a target captured image of the corresponding target according to a preset frame selection strategy based on the detection and tracking result of each target includes: In response to the quality score of a first target in the current frame image satisfying a first quality threshold, determining a candidate captured image based on a detection frame of the first target; the first target being any one of the at least two targets; Storing the candidate captured image in the captured buffer pool of the first target; In response to the snapshot buffer pool of the first target meeting the configured frame selection output condition, the snapshot image stored in the snapshot buffer pool is determined to be the target snapshot image corresponding to the first target.

7. The method according to claim 6, wherein The storing the candidate captured image into the captured buffer pool of the first target includes: When the number of images cached in the snapshot cache pool of the first target does not reach a cache threshold, the candidate snapshot image is directly stored in the snapshot cache pool of the first target.

8. The method according to claim 7, wherein The storing the candidate captured image into the captured buffer pool of the first target includes: When the number of images cached in the snapshot cache pool of the first target reaches the cache threshold, deleting the first snapshot image in the snapshot cache pool; and the quality score of the first snapshot image is lower than the quality score of the candidate snapshot image; The candidate captured image is stored in the captured buffer pool of the first target.

9. The method according to claim 7 or 8, wherein The method further comprises: In response to the snapshot buffer pool of the first target meeting the frame selection output condition, the snapshot buffer pool of the first target is cleared.

10. The method according to claim 6, wherein The configured frame selection output condition includes at least one of the following: a fast frame selection condition, a periodic frame selection condition, a maximum tracking time frame selection condition, a quality threshold frame selection condition, and a tracking end frame selection condition; The fast frame selection condition is used to indicate that the time length of continuous tracking of the first target meets a first time length threshold; The periodic frame selection condition is used to indicate that the time interval for tracking the first target meets an interval threshold; The maximum tracking time frame selection condition is used to indicate that the cumulative time length of tracking the first target reaches a second time length threshold; The second duration threshold is greater than or equal to the first duration threshold; The quality threshold frame selection condition is used to indicate that there is an image with a quality score that meets a second quality threshold in the snapshot buffer pool of the first target; and the second quality threshold is greater than the first quality threshold; The tracking end frame selection condition is used to represent the entire tracking of the first target until the acquisition of the video stream ends.

11. The method according to any one of claims 1 to 4, characterized in that The determining of the structured targets of the batch of video frame images based on the association relationship between the at least two targets and the target captured image of each target includes: Performing feature extraction and / or attribute extraction on the captured image of each target to obtain an analysis result of the corresponding target; Associating the parsing results of the at least two targets based on the association relationship between the at least two targets; Based on the parsing results of the at least two associated targets, the structured targets of the batch of video frame images are determined.

12. The method according to claim 11, wherein The target is a face target, and the number of the target captured images of the face target is greater than or equal to 2; The performing of feature extraction and / or attribute extraction on the captured target image of each target to obtain a parsing result of the corresponding target includes: Performing feature extraction on at least two captured images of the face target to obtain at least two feature vectors; fusing the at least two feature vectors to obtain a fused feature of the face target; Extracting attributes of the target captured image with the highest quality detection score among the at least two target captured images to obtain attributes of the face target; The fusion features of the face target and the attributes of the face target are used as the parsing result of the face target.

13. The method according to any one of claims 1 to 4, characterized in that The method is executed in a parallel pipeline manner by functional modules in at least two threads; The parallel pipeline method represents that between each two dependent functional modules, the result data output by the previous functional module is used as the raw material data of the next functional module through a cache queue.

14. A structured target detection device, characterized in that: The device includes an acquisition module, a detection and tracking module, an association and matching module, a frame selection module, and a determination module, wherein: The acquisition module is used to acquire a batch of video frame images to be processed; The detection and tracking module is configured to perform target detection and tracking on at least two targets in the batch of video frame images to obtain detection and tracking results for each target; the detection and tracking results include a detection frame, a tracking identifier, and a tracking trajectory of the target; the at least two targets include a key target and a non-key target; the key target is a human face target used to represent the identity of the object to be detected; The association matching module is configured to determine an association relationship between the at least two targets based on the detection and tracking results of the at least two targets; The frame selection module is used to determine the target captured image of the corresponding target according to a preset frame selection strategy based on the detection and tracking results of each target; the frame selection strategy is used to restrict the number of the target captured images of the same target; The determining module is configured to determine the structured targets of the batch of video frame images based on the association relationship between the at least two targets and the target captured image of each target; The detection and tracking module is also used to perform target detection on the key target in the current frame image in the batch of video frame images to obtain the detection frame of the key target; in response to the current frame image being a key frame image, perform target detection on the non-key target in the current frame image to obtain the initial detection frame of the non-key target; the key frame image is sampled according to a preset sampling interval number; in response to the current frame image being a non-key frame image, perform single target tracking on the non-key target in the current frame image to obtain an updated detection frame of the non-key target; based on the detection frame of each target in the batch of video frame images, track the corresponding target to obtain the tracking identifier and tracking trajectory of the corresponding target.

15. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 13 are implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Images