A video target detection method and electronic device
By allocating neural network processor cores to each video channel and setting core affinity, configuring independent network graph instances and frame sequence management, the load balancing and timing consistency issues in multi-core video object detection are resolved, and latency is reduced.
Patent Information
- Application Number
- CN202611604930.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-16
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, multi-core parallel computing is difficult to achieve load balancing and temporal consistency of output results in video object detection, resulting in some cores being idle while others are backed up, and old frames overwriting new frames, which continuously increases latency.
Each video channel is assigned a neural network processor core, core affinity is set and an independent network graph instance is configured. By sharing a bounded frame queue and managing frame sequence numbers, the inference thread is ensured to run on an independent core, video frames are processed in parallel, and results are output according to frame sequence number.
It achieves load balancing in multi-core parallel computing, reduces end-to-end latency from video frame acquisition to detection result output, and ensures timing consistency of result output.
Smart Images

Figure CN123119221A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, specifically to a video target detection method and electronic device. Background Technology
[0002] With the widespread adoption of Neural Processing Units (NPUs) in edge computing devices, the use of NPUs to perform object detection on camera video streams has been widely applied in scenarios such as intelligent monitoring, industrial inspection, and traffic analysis. Cameras typically output images continuously at a fixed frame rate, while the recognition of a single target requires sequential image preprocessing, network inference, and post-processing. The processing power of a single core is often difficult to match the input frame rate of the camera.
[0003] In related technologies, processing power is usually improved by having multiple inference threads execute inference in parallel: multiple inference threads share the same network graph instance; or multiple network graph instances are created without specifying the core on which they run; or images are polled and assigned to different cores according to frame sequence number, and frames to be inferred are stored in a queue, and the results are output directly according to the order in which inference is completed.
[0004] However, multiple inference threads cannot concurrently share the same network graph instance, and multiple network graph instances, without core binding, will default to using the same core, making it difficult to practically utilize the parallel computing capabilities of multi-cores. When polling by fixed frame sequence number, due to the different processing times of different images, it is easy to have an unbalanced load where some cores are idle while others still have backlogs. Furthermore, the completion order of multi-core inference is not consistent with the frame acquisition order. Directly outputting in the completion order will cause earlier acquired frames to overwrite later acquired frames, and if old frames are not discarded, the old frames in the queue will continue to accumulate, which will continuously increase latency. The common limitation of the above methods is that they fail to balance load balancing among multiple cores and the timing consistency of output results under the condition that network graph instances are configured independently by core. Summary of the Invention
[0005] This application provides a video target detection method and electronic device to at least partially solve the above-mentioned technical problems.
[0006] To achieve the above objectives, according to a first aspect of this application, a video object detection method is provided, comprising: Allocate at least one neural network processor core to each video channel; For each assigned neural network processor core, a core affinity is set and a corresponding network graph instance is configured, and an inference thread corresponding to the network graph instance is created. The inference thread is configured with the core affinity corresponding to the neural network processor core during runtime, and the network graph instance is used to perform network inference on the image. Acquire video frames and write the video frames into the shared bounded frame queue of the corresponding video channel, wherein each video frame is configured with a frame sequence number and an acquisition timestamp; The inference thread and its corresponding neural network processor core process the video frames in the shared bounded frame queue to obtain detection results. The detection result is associated with the frame number corresponding to the detection result and then written into a bounded result set managed by the frame number; The candidate result with the largest frame number is determined from the bounded result set, and the candidate result is output when the frame number of the candidate result is greater than the most recently output frame number.
[0007] In some embodiments, the process of processing the video frames in the shared bounded frame queue through the inference thread and its corresponding neural network processor core to obtain detection results includes: The inference thread obtains the video frame from the shared bounded frame queue, wherein when at least one video channel is allocated multiple neural network processor cores, the multiple inference threads corresponding to that video channel compete to obtain the video frame at the head of the queue from the shared bounded frame queue.
[0008] In some embodiments, the video target detection method further includes: When the shared bounded frame queue is full, the video frame at the head of the shared bounded frame queue is deleted, and a new video frame is added to the tail of the shared bounded frame queue.
[0009] In some embodiments, before processing the video frames in the shared bounded frame queue through the inference thread and its corresponding neural network processor core to obtain the detection result, the method further includes: Determine whether the bounded result set satisfies the preset congestion condition; When the congestion condition is met, the processing of the video frame is skipped; The congestion condition is determined based on at least one of the following: the number of results in the bounded result set, the waiting time of the oldest result, the current end-to-end delay, and the result output rate.
[0010] In some embodiments, the process of processing the video frames in the shared bounded frame queue through the inference thread and its corresponding neural network processor core to obtain detection results includes: The inference thread scales the video frames proportionally to their original aspect ratio and converts the color order of the scaled images to match the color order of the network graph instance to obtain the network input tensor. The network input tensor is sent to the corresponding neural network processor core, so that the neural network processor core can perform network inference on the network input tensor through the network graph instance to generate the detection result.
[0011] In some embodiments, the video target detection method further includes: Through the inference thread, based on the scaling ratio and preset fill amount, the coordinates of the detection box in the detection result are inversely transformed to obtain the coordinates of the detection box in the coordinate system of the video frame. The preset fill amount includes: the fill value when the network input tensor is filled with an image based on the network graph instance.
[0012] In some embodiments, allocating at least one neural network processor core to each video channel includes: Each video channel is allocated a non-overlapping set of neural network processor cores, wherein, when the number of available neural network processor cores is divisible by the number of video channels, the number of neural network processor cores allocated to each video channel is the same and their numbers are consecutive; or The number of neural network processor cores is configured for each video channel according to at least one of the following: video resolution, object detection network complexity, channel priority, historical processing time, and input frame rate, either statically or dynamically.
[0013] In some embodiments, setting core affinity and creating a corresponding network graph instance includes: Set the neural network processor core identifier corresponding to the network graph instance to be created; Create a pre-compiled network graph corresponding to the neural network processor core; Verify whether the pre-compiled network graph can run on the neural network processor core; The pre-compiled network graph that passes the verification is used as a network graph instance and saved to the network graph set of the corresponding video channel.
[0014] In some embodiments, the capacity of the bounded result set is determined based on the number of neural network processor cores allocated to the corresponding video channel; the step of associating the detection result with the frame number corresponding to the video frame and writing it into the bounded result set managed by the frame number further includes: When the number of results in the bounded result set exceeds the capacity, delete the result with the smallest frame sequence number in the bounded result set; and / or When the frame number corresponding to the video frame is less than or equal to the most recently output frame number, the detection result is rejected from being written into the bounded result set.
[0015] In a second aspect, an electronic device is provided, comprising: at least one processing device; and a memory communicatively connected to the at least one processing device; wherein the memory stores instructions executable by the at least one processing device to enable the at least one processing device to implement the method described above.
[0016] This application embodiment configures a corresponding network graph instance for each allocated neural network processor core and sets core affinity for the inference thread corresponding to the same core, so that multiple inference threads run on independently bound cores, avoiding multiple network graph instances from using the same core in a concentrated manner; by sharing a bounded frame queue and identifying video frames with frame sequence number and acquisition timestamp, the parallel frame acquisition and the management of new and old identifiers on the input side are unified; by managing the detection results by frame sequence number and using the sequence number of the most recently output frame for monotonic output control, the out-of-order results generated by parallel inference will not overwrite the new ones with the old ones, thereby reducing the end-to-end latency from video frame acquisition to detection result output while utilizing the parallel computing power of multiple cores.
[0017] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a flowchart illustrating a video target detection method provided in one embodiment of this application; Figure 2 This is a schematic diagram of the process for setting core affinity and creating a corresponding network graph instance provided in one embodiment of this application; Figure 3This is a flowchart illustrating a video target detection method provided in another embodiment of this application; Figure 4 This is a schematic diagram illustrating the competition among inference threads to acquire video frames provided in one embodiment of this application; Figure 5 This is a schematic diagram of the detection result processing and output in one embodiment of this application; Figure 6 This is a flowchart illustrating a video target detection method provided in another embodiment of this application; Figure 7 This is a schematic diagram of in-figure edge patching and inverse coordinate transformation provided in one embodiment of this application; Figure 8 This is a structural block diagram of an electronic device provided in one embodiment of this application. Detailed Implementation
[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.
[0021] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.
[0022] Reference Figure 1 and Figure 3 As shown, according to a first aspect of this application, this application provides a video object detection method, comprising the following steps: Step 100: Allocate at least one neural network processor core to each video channel. Specifically, the method of this application embodiment can be applied to edge computing devices with multiple neural network processor (NPU) cores for real-time target detection of video streams from one or more cameras. One video channel corresponds to one video stream. In this application embodiment, at least one neural network processor core is allocated to each video channel according to the number of video channels and the number of available neural network processor cores, thereby dividing the available core resources among multiple video channels. The method of this application embodiment is executed by threads running on the central processing unit of the edge computing device, with each neural network processor core serving as the invoked computing power carrier.
[0023] Step 200: For each assigned neural network processor core, set core affinity and configure the corresponding network graph instance, and create an inference thread corresponding to the network graph instance. The inference thread is configured with the core affinity corresponding to the neural network processor core during runtime, and the network graph instance is used to perform network inference on the image. Specifically, network graph instances (i.e. Figure 3 and Figure 4 Independent networks in Figure 1 -M) refers to an executable computation graph object built or loaded based on an object detection network, which internally includes the object detection network for performing network inference on images; core affinity refers to the configuration relationship that assigns a network graph instance or execution thread to a specific neural network processor core. This step, for each assigned neural network processor core, first sets the core affinity of that core, and then configures the corresponding network graph instance on that core, so that each neural network processor core corresponds to one network graph instance for its use.
[0024] In this embodiment, after the inference thread is created, its core affinity is set again to correspond to the same neural network processor core during runtime, so that the inference thread and the network graph instance it uses run on the same core. That is, the core affinity in this embodiment is set twice: the first time it acts on the network graph instance, so that the network graph instance is created on the specified core; the second time it acts on the inference thread, so that the inference thread runs on the same core, thereby realizing the correspondence between the network graph instance and the neural network processor core and the network graph instance.
[0025] Therefore, the embodiments of this application establish a correspondence between video channel, inference thread, network graph instance, and neural network processor core: one video channel corresponds to one or more neural network processor cores, one neural network processor core corresponds to one network graph instance, and one network graph instance corresponds to one inference thread.
[0026] Step 300: Acquire video frames and write the video frames into the shared bounded frame queue of the corresponding video channel, wherein each video frame is configured with a frame sequence number and an acquisition timestamp. Specifically, a video frame refers to the acquired image data and its identification information. A video frame may include a frame number, an acquisition timestamp, and image data. The image data can be recorded in the form of image data references, thus passing the references in the queue without having to copy the entire image each time. The frame number monotonically increases according to the acquisition order of the video frames, and the acquisition timestamp indicates the time when the video frame was captured or acquired. Both can be used to characterize the newness or oldness of the video frame.
[0027] Optionally, "shared" in the shared bounded frame queue means that the queue is used by all inference threads of the video channel, and "bounded" means that the queue has an upper limit on its capacity; once the upper limit is reached, video frames in the queue can be discarded according to preset rules. After acquiring a video frame, the input format, resolution, and continuous storage status can be checked first, then a frame sequence number can be generated, a timestamp can be obtained, and the video frame can be written into the queue.
[0028] Step 400: The video frames in the shared bounded frame queue are processed by the inference thread and its corresponding neural network processor core to obtain the detection result. Specifically, the video frame processing described in this application embodiment may include three stages: preprocessing, network inference, and post-processing. For example, preprocessing converts the video frame into the input format required by the network; network inference can be performed by an inference thread on the corresponding neural network processor core through its corresponding network graph instance; and post-processing is used to filter and decode the output of the network inference. The inference thread processes the video frame through the network graph instance on the corresponding neural network processor core and obtains the detection result, which may include information such as the location, category, and confidence level of the detection box. Since the network graph instances on each neural network processor core are independent, multiple inference threads can perform the above processing simultaneously.
[0029] Step 500: Associate the detection result with the frame number corresponding to the detection result and write it into the bounded result set managed by the frame number. Specifically, each video channel has a bounded result set managed by frame number. Each detection result is associated with the frame number of the video frame that generated the detection result before being written into this set. Since multiple inference threads execute in parallel, the order in which the inference threads complete their processing may not be consistent with the order in which the video frames are acquired. Therefore, the order in which the detection results are written into the bounded result set is uncertain.
[0030] Step 600: Determine the candidate result with the largest frame number from the bounded result set, and output the candidate result when the frame number of the candidate result is greater than the most recently output frame number. Specifically, this embodiment maintains the most recently output frame sequence number. This state variable represents the frame sequence number corresponding to the most recently output detection result. Its initial value can be set to a value less than any valid frame sequence number, for example, 0 when the frame sequence number starts to increment from 1. When it is necessary to update the display or send results to downstream modules, the detection result with the largest frame sequence number is selected from the bounded result set as a candidate result, and the frame sequence number of the candidate result is compared with the most recently output frame sequence number: when the frame sequence number of the candidate result is greater than the most recently output frame sequence number, the candidate result is output and the most recently output frame sequence number is updated to the frame sequence number of the candidate result; when the frame sequence number of the candidate result is less than or equal to the most recently output frame sequence number, the candidate result belongs to the lagging historical result and is no longer output. Since the frame sequence number monotonically increases according to the acquisition order, the frame sequence number of the output result also monotonically advances, and there will be no situation where the output content rolls back.
[0031] This application embodiment configures a corresponding network graph instance for each allocated neural network processor core and sets core affinity for the inference thread corresponding to the same core, so that multiple inference threads run on independently bound cores, avoiding multiple network graph instances from using the same core in a concentrated manner; by sharing a bounded frame queue and identifying video frames with frame sequence number and acquisition timestamp, the parallel frame acquisition and the management of new and old identifiers on the input side are unified; by managing the detection results by frame sequence number and using the sequence number of the most recently output frame for monotonic output control, the out-of-order results generated by parallel inference will not overwrite the new ones with the old ones, thereby reducing the end-to-end latency from video frame acquisition to detection result output while utilizing the parallel computing power of multiple cores.
[0032] Reference Figure 4 As shown, in an optional embodiment of this application, the step of processing the video frames in the shared bounded frame queue through the inference thread and its corresponding neural network processor core to obtain the detection result includes: The inference thread obtains the video frame from the shared bounded frame queue, wherein when at least one video channel is allocated multiple neural network processor cores, the multiple inference threads corresponding to that video channel compete to obtain the video frame at the head of the queue from the shared bounded frame queue. Specifically, multiple inference threads on the same video channel share a shared bounded frame queue for that video channel. Each inference thread is not assigned a fixed frame according to its frame number, but rather competes to obtain the first video frame in the queue through mutual exclusion and conditional notification mechanisms. Mutual exclusion ensures that only one inference thread can retrieve the first video frame at a time, while the conditional notification mechanism wakes up waiting inference threads when the queue changes from empty to non-empty or from full to non-full, preventing inference threads from idling and consuming processor resources.
[0033] Optionally, any inference thread can retrieve the next video frame from the shared bounded frame queue after completing the processing of the current video frame, without needing to pre-specify that a particular frame must be processed by a specific neural network processor core. Therefore, faster processing cores can retrieve more video frames in the same amount of time, while slower processing cores will not block other cores, avoiding the localized backlogs that occur in fixed-polling methods where each core corresponds to its own queue.
[0034] It is understandable that when a video channel is allocated only one neural network processor core, the video channel has only one inference thread. This inference thread also obtains the video frame at the head of the shared bounded frame queue. That is, the above frame retrieval method is applicable to each video channel and is not based on the premise that the video channel is allocated multiple neural network processor cores.
[0035] This application embodiment enables multiple inference threads of the same video channel to compete for the first video frame from a shared bounded frame queue, allowing inference tasks to be dynamically allocated according to the actual processing progress of each core. This reduces core idleness and local backlog caused by fixed polling frame dispatch, and improves the overall utilization of multiple neural network processor cores.
[0036] Reference Figure 6 As shown, in an optional embodiment of this application, the method further includes: When the shared bounded frame queue is full, the video frame at the head of the shared bounded frame queue is deleted, and a new video frame is added to the tail of the shared bounded frame queue. Specifically, the capacity of the shared bounded frame queue is determined based on the number of neural network processor cores allocated to its corresponding video channel. In one embodiment, if the number of neural network processor cores allocated to a certain video channel is M, then the capacity Q of the shared bounded frame queue for that video channel is... f The maximum value can be max(3, 3M), which is the larger of 3 and 3M. Therefore, the queue can simultaneously hold several video frames for each parallel inference thread without accumulating too many outdated frames due to excessive capacity.
[0037] Optionally, when a new video frame arrives and the shared bounded frame queue is full, the oldest video frame at the head of the queue is first deleted, and then the new video frame is added to the tail of the queue, ensuring that the queue always contains newer video frames; the deleted video frame at the head of the queue is no longer processed by any inference thread. The queue capacity and its ratio to the number of cores can be adjusted based on the device's available memory, single-frame processing time, and allowed end-to-end latency.
[0038] It is understandable that the above-mentioned method for determining the queue capacity and the method for handling when the queue is full also apply to video channels that are allocated only one neural network processor core. The queue management method for this type of video channel is the same, and will not be repeated here.
[0039] This application embodiment suppresses the continuous accumulation of old frames in the queue by prioritizing the removal of the oldest video frame at the head of the queue when the queue is full. This ensures that limited parallel computing power is always used to process newer video frames, thereby limiting the continuous growth of end-to-end latency.
[0040] Reference Figure 6 As shown, in an optional embodiment of this application, before processing the video frames in the shared bounded frame queue through the inference thread and its corresponding neural network processor core to obtain the detection result, the method further includes: Determine whether the bounded result set meets a preset congestion condition; when it is determined that the congestion condition is met, skip the processing of the video frame; wherein the congestion condition is determined based on at least one of the number of results in the bounded result set, the waiting time of the oldest result, the current end-to-end delay, and the result output speed. Specifically, after the inference thread obtains a video frame from the shared bounded frame queue, but before processing the video frame through the network graph instance on the neural network processor core corresponding to the inference thread, it determines whether the bounded result set of the video channel meets a preset congestion condition. The congestion condition characterizes the backlog on the output side. For example, it is determined that the congestion condition is met when the number of results in the bounded result set exceeds a preset number threshold, or when the waiting time of the oldest result exceeds a preset time threshold. The current end-to-end latency or the output speed of the results can also be considered simultaneously. When the congestion condition is met, it means that there are already many unoutputted results backlogged on the output side. At this time, the processing of the video frame is skipped, and the number of inference skips is recorded. This number can be used for runtime statistics and scheduling parameter adjustments.
[0041] In this process, skipped video frames have already been removed from the shared bounded frame queue by the inference thread, so they will not be put back into the queue, nor will they generate detection results or be written into the bounded result set. The inference thread then returns to the shared bounded frame queue to continue competing for a new head video frame. Skipping some video frames will cause gaps in the frame sequence numbers of the output detection results, but the output method of this embodiment does not require the frame sequence numbers of the detection results to be consecutive, so the two do not conflict.
[0042] Optionally, the congestion assessment and skipping process described above also apply to video channels that are allocated only one neural network processor core and have only one inference thread.
[0043] This application embodiment reduces ineffective neural network processor computation by skipping video frames that have lost their timeliness before performing network inference, compared to discarding detection results after network inference is completed. This allows limited computing power to be prioritized for updating video frames.
[0044] Reference Figure 7 As shown, in an optional embodiment of this application, the step of processing the video frames in the shared bounded frame queue through the inference thread and its corresponding neural network processor core to obtain the detection result includes: The inference thread scales the video frames proportionally to their original aspect ratio and converts the color order of the scaled images to match the color order of the network graph instance to obtain the network input tensor. Specifically, in one embodiment, the camera outputs a 1920×1080 three-channel, eight-bit unsigned video frame arranged in blue-green-red order. The inference thread scales the image contained in this video frame proportionally to 640×360, maintaining the aspect ratio of the image before and after scaling. Then, the color order is converted to red-green-blue order, and the continuously stored 640×360 red-green-blue data is copied to the network input tensor, thereby converting the video frame into a network input tensor. Here, the color order matched by the network graph instance refers to the color channel arrangement order of the input data required by the network graph instance during inference.
[0045] In this embodiment, from the acquisition of the detection result output from the video frame, the objects processed are, in sequence, the video frame, the network input tensor, the filled square input image, and the detection result; what is sent to the core of the neural network processor is the network input tensor, not the entire video frame.
[0046] The network input tensor is sent to the corresponding neural network processor core, so that the neural network processor core can perform network inference on the network input tensor through the network graph instance to generate the detection result. Optionally, in addition to the object detection network, the network graph instance may also include nodes for processing the input image, such as padding nodes for padding the input tensor and normalization nodes for performing normalization. In one embodiment, the object detection network may employ the YOLOv8s object detection network, whose output is a 1×84×8400 half-precision floating-point tensor, where each candidate box includes 4 bounding box parameters and 80 class scores.
[0047] Optionally, post-processing after network inference is performed by the inference thread on the central processing unit side: the inference thread directly reads the half-precision floating-point output data, determines the highest category score and category identifier for each candidate box, decodes the center coordinates, width, and height only for candidate boxes whose scores reach the score threshold, and then performs non-maximum suppression (NMS) according to the category. To reduce the overhead of redundant memory allocation, each inference thread reuses its own local output buffer, score buffer, and category buffer. These buffers are exclusively used by the inference thread and are reused when processing multiple frames consecutively.
[0048] This application embodiment clarifies the processing boundary between the central processing unit and the neural network processor by converting video frames into network input tensors before sending them to the corresponding neural network processor core. It also reuses the local cache within the inference thread, reducing the additional overhead caused by repeatedly allocating memory frame by frame.
[0049] In an optional embodiment of this application, the method further includes: Through the inference thread, based on the scaling ratio and the preset fill amount, the coordinates of the detection box in the detection result are subjected to inverse coordinate transformation to obtain the coordinates of the detection box in the coordinate system of the video frame; wherein, the preset fill amount includes: the fill value when the network input tensor is filled with image based on the network graph instance. Specifically, when the network graph instance processes the network input tensor internally, it may change the relative position of the image content within the container. Therefore, the coordinates of the detection box output by the object detection network are located in the coordinate system of the filled square image, and need to be restored to the coordinate system of the video frame through inverse coordinate transformation. Let the width of the video frame be W, the height be H, and the target size of the object detection network be L. Then the scaling ratio and the preset padding amount in both directions can be determined by the following formula: scale = min(L / W, L / H); pad left = (LW × scale) / 2; pad top = (LH × scale) / 2; Where scale represents the scaling ratio, pad left pad indicates the preset fill amount in the horizontal direction. top This represents the preset fill amount in the vertical direction, and "min" indicates that the minimum value is taken during the calculation. For the detection result coordinates (x, y), first subtract the preset fill amount in the corresponding direction, and then divide by the scaling factor to obtain the coordinates of the detection box in the coordinate system of the video frame.
[0050] In some embodiments, the detection box coordinates can also be limited to the effective range of the video frame. For example, if the detection box coordinates exceed the effective range of the video frame, they can be corrected to the boundary value of the effective range, i.e., the detection box is cropped at the image boundary.
[0051] It is understood that placing the padding operation within the network graph instance allows the central processing unit (CPU) to transmit only 640×360 valid image data to the network input tensor, without transmitting a pre-generated 640×640 padded image from the host. In one embodiment, the network graph instance pads 140 pixels on each side outside the width direction of the 640×360 image, with a padding value of 114, forming a 640×640 input image. Compared to transmitting a 640×640 three-channel image, the theoretical data volume of transmitting a 640×360 three-channel image is reduced by 43.75%. The above dimensions, padding amount, and padding value are only one embodiment. For images with other aspect ratios, they can be scaled to a non-square input size according to the original aspect ratio, and then padded by the network graph instance to the size required by the object detection network.
[0052] This application embodiment performs an inverse coordinate transformation on the detection box coordinates based on the scaling ratio and a preset padding amount, so that the detection box can be correctly restored to the coordinate system of the video frame. Furthermore, by setting the padding inside the network graph instance, the amount of invalid data transmitted from the host to the neural network processor is reduced.
[0053] In one optional embodiment of this application, the allocation of at least one neural network processor core to each video channel includes the following two types: In the first form, a set of non-overlapping neural network processor cores is allocated to each of the video channels, wherein the number of available neural network processor cores is divisible by the number of video channels, and the number of neural network processor cores allocated to each video channel is the same and consecutively numbered. Specifically, let the total number of available neural network processor cores be K and the number of video channels be C. When K is divisible by C, each video channel receives the same number of consecutively numbered cores, M = K / C. The c-th video channel receives M cores starting from the c×M-th core. Non-overlapping means that the same neural network processor core will not be allocated to more than two video channels, thus creating independent computational power allocation among the video channels.
[0054] The second approach involves configuring the number of neural network processor cores for each video channel according to a static or dynamic configuration method, based on at least one of the following: video resolution, object detection network complexity, channel priority, historical processing time, and input frame rate. Optionally, static configuration refers to determining and maintaining the number of neural network processor cores for each video channel at system startup; dynamic configuration refers to adjusting the number of neural network processor cores for each video channel during system operation based on changes in the aforementioned factors, while ensuring that the core sets for each video channel do not overlap. For example, video channels with higher video resolution or higher channel priority can be configured with more neural network processor cores; video channels with lower input frame rates or lower object detection network complexity can be configured with fewer neural network processor cores.
[0055] This application embodiment assigns non-overlapping core sets to each video channel and uses a partitioning method with the same number and consecutive numbers when the number of cores is divisible by the number of channels. This simplifies and makes the mapping relationship between cores and video channels easy to maintain. At the same time, it meets the different computing power requirements of different video channels through static or dynamic configuration.
[0056] Reference Figure 2 As shown, in an optional embodiment of this application, setting core affinity and creating a corresponding network graph instance includes: Step 201: Set the neural network processor core identifier corresponding to the network graph instance to be created. Specifically, since network graph instances need to be created on the neural network processor cores on which they will run, the core identifier corresponding to the network graph instance is set before the network graph instance is created, so that the subsequent creation operation is directed to the specified neural network processor core, thereby setting core affinity.
[0057] Step 202: Create a pre-compiled network graph corresponding to the neural network processor core. In this embodiment, a precompiled network graph refers to a computation graph that has been pre-compiled and can be directly loaded and run on the neural network processor core. Precompiled network graphs created on different neural network processor cores are independent of each other, thus ensuring that each inference thread has its own dedicated network graph instance, and multiple threads do not need to concurrently access the same network graph.
[0058] Step 203: Verify whether the pre-compiled network graph can run on the neural network processor core.
[0059] Optionally, the above verification may include checks on operator support, memory usage, and input / output size matching; pre-compiled network graphs that fail the verification will not be used, thereby ensuring that the network graph instance finally configured on the core can complete inference normally and avoid loading failure or inference interruption during the runtime phase.
[0060] Step 204: The pre-compiled network graph that has passed verification is used as the network graph instance and saved to the network graph set of the corresponding video channel. Specifically, each video channel maintains a network graph set to store the network graph instances corresponding to each neural network processor core of that video channel. Each network graph instance in the network graph set corresponds one-to-one with each inference thread, meaning that each network graph instance is used by only one inference thread. Correspondingly, the core affinity set by the inference thread during runtime is consistent with the core to which the network graph instance belongs.
[0061] This application embodiment sets a core identifier, creates a pre-compiled network graph for the core, and verifies the running capability of the pre-compiled network graph, so that each allocated neural network processor core obtains a network graph instance that can run normally on it, providing a basis for parallel inference of multiple inference threads.
[0062] In an optional embodiment of this application, the capacity of the bounded result set is determined based on the number of neural network processor cores allocated to the corresponding video channel; the step of associating the detection result with the frame number corresponding to the video frame and writing it into the bounded result set managed by the frame number further includes: When the number of results in the bounded result set exceeds the capacity, delete the result with the smallest frame sequence number in the bounded result set. Specifically, in one embodiment, if the number of neural network processor cores allocated to a certain video channel is M, then the capacity Qr of the bounded result set of that video channel can be max(2, 2M), that is, the larger value between 2 and 2M. When the number of results in the bounded result set exceeds the above capacity, the result with the smallest frame number is deleted, so that the bounded result set retains relatively new detection results, avoiding the result set from continuously expanding with the increase of running time. The result data may at least include the frame number, acquisition timestamp, inference completion timestamp, and the detected image or structured detection result.
[0063] When the frame number corresponding to the video frame is less than or equal to the most recently output frame number, the detection result is rejected from being written into the bounded result set. Specifically, before writing the detection result, the inference thread compares the frame number of the video frame corresponding to the detection result with the most recently output frame number. If the frame number is less than or equal to the most recently output frame number, it means that the video frame corresponding to the detection result is earlier than the already output result, and is therefore a late result. Thus, it is rejected from being written into the bounded result set. This method can reduce invalid writes and storage usage after results have already been output.
[0064] It is understood that only the result of deleting the frame with the smallest sequence number when the capacity is exceeded can be executed, or only the result of rejecting late writes can be executed, or both can be executed simultaneously. This application does not limit this.
[0065] In one specific embodiment, refer to Figure 5 As shown, frames F20, F21, and F22 are acquired in the order F20, F21, F22. However, after processing on three different neural network processor cores, the detection results arrive at the output side in the order F21, F22, F20. When F21 arrives, it is output, and the sequence number of the most recently output frame is updated to 21. When F22 arrives next, it is output, and the sequence number of the most recently output frame is updated to 22. The last to arrive, F20, is not output because its frame sequence number is less than the most recently output frame number. Based on this, if the above-mentioned write rejection method is adopted, late results with frame sequence numbers less than or equal to 22 are intercepted during the write phase and will not occupy the storage location of the result set.
[0066] This application embodiment deletes the result with the smallest frame sequence number when the result set exceeds its capacity and refuses to write late detection results, so that the bounded result set always retains the newest detection results. This limits the storage overhead of the result set and avoids late old results from participating in subsequent output scheduling.
[0067] In an optional embodiment of this application, a timestamp can be used instead of a frame number to determine the relationship between the new and old detection results; alternatively, only results with frame numbers less than the most recently output frame number can be deleted; or in scenarios where local graphical display is not performed, such as network transmission or alarm output, the latest structured detection results can be directly sent to the downstream module without local display.
[0068] This application embodiment controls the freshness of output results through multiple stages: in the acquisition stage, the oldest unprocessed video frame is discarded when the shared bounded frame queue is full; in the inference admission stage, low-timeliness video frames that have not yet started processing are skipped when the output side meets the congestion condition; in the result writing stage, the detection result with the smallest frame number is deleted when the bounded result set exceeds its capacity; in the output stage, the candidate result with the largest frame number is selected, and the candidate result with a frame number greater than the most recently output frame number is output, and historical results with frame numbers less than or equal to the most recently output frame number are no longer retained. Each of the above stages prioritizes retaining the most up-to-date video frames or detection results, allowing the limited computing and output capabilities of the neural network processor to reflect the latest on-site state. Compared to the first-in-first-out processing method that does not discard video frames, this application embodiment allows sacrificing some complete processing rate to limit queue backlog, making it suitable for target detection scenarios where real-time performance is prioritized over frame-by-frame completeness.
[0069] In some embodiments, when the video input ends or a stop command is received, the resources occupied by each network graph instance, the shared bounded frame queue, and the bounded result set can be released and the processing can be terminated; before that, each inference thread continues to acquire and process video frames in a loop as described above.
[0070] Optionally, in multi-camera scenarios, each video channel has a separate acquisition thread, a shared bounded frame queue, a core group, a network graph set, and a bounded result set. The latest detection results of each video channel can be displayed in a grid layout. The system can also statistically analyze information such as the number of acquired frames, the number of processed frames, the number of displayed frames, the number of frames dropped at each stage, the processing frame rate, the display frame rate, the frame queue waiting time, the result waiting time, and the end-to-end latency from acquisition to display. The above statistical functions are used to verify and adjust the number of cores, queue capacity, and eviction conditions, and are not necessary restrictions for implementing the multi-core inference scheduling of this application.
[0071] In a specific example, an edge computing device has three neural network processor cores connected to a camera that outputs 1920×1080 images at 25 frames per second. On this hardware platform and the YOLOv8s object detection network, the system processing frame rate for a single core is approximately 10 frames per second, and for two cores it is approximately 20 frames per second, neither of which can match the camera's input frame rate. After the video channel is allocated to three cores, and each core is configured with a dedicated network graph instance and bound to the corresponding core, the system processing frame rate is approximately 24 frames per second, which basically meets the real-time detection requirements of the camera. In this example, the capacity Qf of the shared bounded frame queue is max(3,3M), and the capacity Qr of the bounded result set is max(2,2M). The inference thread scales the image to 640×360 and converts the color order from blue-green-red to red-green-blue before copying it to the network input tensor. The network graph instance is filled to 640×640 and normalized before being sent to the object detection network. Then, the detection box coordinates are inversely transformed according to the relationships given in Equations 1 to 3 above to obtain the detection box coordinates in the coordinate system of the video frame.
[0072] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.
[0073] The embodiments, implementation methods, and related technical features of this application can be combined and substituted for each other without conflict.
[0074] According to a second aspect of this application, an electronic device is provided, comprising at least one processing device; and a memory communicatively connected to the at least one processing device; wherein the memory stores instructions executable by the at least one processing device, the instructions being executed by the at least one processing device to enable the at least one processing device to implement the method described in the above-described method embodiments and have all the beneficial effects of the above-described method, which will not be elaborated further in this application.
[0075] like Figure 8As shown, the electronic device includes a processing unit 901 and a memory. The processing unit 901 may include one or more combinations of devices with data processing capabilities, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a tensor processing unit (TPU), and a field-programmable gate array (FPGA). The memory may include one or more combinations of a read-only memory (ROM) 902, a random access memory (RAM) 903, a storage portion 908, and a removable medium 911. The processing unit 901 executes various appropriate actions and processes according to instructions stored in the memory to implement the methods described above. In some embodiments, the instructions are stored in the read-only memory 902 as a program, or loaded from the storage portion 908 into the random access memory 903 as a program and then executed. In some embodiments, the acceleration unit such as the graphics processor or neural processing unit in the processing unit 901 may also be configured with video memory as its local memory. The video memory may be implemented as a high-bandwidth memory or a graphics double data rate memory, or it may be partitioned from a portion of the storage space in the random access memory 903. Such video memory is also a specific implementation of the memory described in this application.
[0076] Processing device 901, read-only memory 902, and random access memory 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904. In some embodiments, the following components are connected to I / O interface 905: input section 906 including keyboard, mouse, etc.; output section 907 including cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; storage section 908 including hard disk, etc.; and communication section 909 including network interface card, such as LAN card, modem, etc. Communication section 909 performs communication processing via a network such as the Internet. Drive 910 is also connected to I / O interface 905 as needed. Removable media 911, such as disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 910 as needed so that computer programs read from there can be installed into storage section 908 as needed.
[0077] The above are merely preferred embodiments of this application and are not intended to limit this application in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of this application without departing from the scope of the technical solution of this application shall still fall within the scope of the technical solution of this application.
Claims
1. A video target detection method, characterized in that, include: Allocate at least one neural network processor core to each video channel; For each assigned neural network processor core, a core affinity is set and a corresponding network graph instance is configured, and an inference thread corresponding to the network graph instance is created. The inference thread is configured with the core affinity corresponding to the neural network processor core during runtime, and the network graph instance is used to perform network inference on the image. Acquire video frames and write the video frames into the shared bounded frame queue of the corresponding video channel, wherein each video frame is configured with a frame sequence number and an acquisition timestamp; The inference thread and its corresponding neural network processor core process the video frames in the shared bounded frame queue to obtain detection results. The detection result is associated with the frame number corresponding to the detection result and then written into a bounded result set managed by the frame number; The candidate result with the largest frame number is determined from the bounded result set, and the candidate result is output when the frame number of the candidate result is greater than the most recently output frame number.
2. The method according to claim 1, characterized in that, The process of processing the video frames in the shared bounded frame queue through the inference thread and its corresponding neural network processor core to obtain detection results includes: The inference thread obtains the video frame from the shared bounded frame queue, wherein when at least one video channel is allocated multiple neural network processor cores, the multiple inference threads corresponding to that video channel compete to obtain the video frame at the head of the queue from the shared bounded frame queue.
3. The method according to claim 2, characterized in that, The method further includes: When the shared bounded frame queue is full, the video frame at the head of the shared bounded frame queue is deleted, and a new video frame is added to the tail of the shared bounded frame queue.
4. The method according to claim 2, characterized in that, Before processing the video frames in the shared bounded frame queue through the inference thread and its corresponding neural network processor core to obtain the detection result, the method further includes: Determine whether the bounded result set satisfies the preset congestion condition; When the congestion condition is met, the processing of the video frame is skipped; The congestion condition is determined based on at least one of the following: the number of results in the bounded result set, the waiting time of the oldest result, the current end-to-end delay, and the result output rate.
5. The method according to claim 1, characterized in that, The process of processing the video frames in the shared bounded frame queue through the inference thread and its corresponding neural network processor core to obtain detection results includes: The inference thread scales the video frames proportionally to their original aspect ratio and converts the color order of the scaled images to match the color order of the network graph instance to obtain the network input tensor. The network input tensor is sent to the corresponding neural network processor core, so that the neural network processor core can perform network inference on the network input tensor through the network graph instance to generate the detection result.
6. The method according to claim 5, characterized in that, The method further includes: Through the inference thread, based on the scaling ratio and preset fill amount, the coordinates of the detection box in the detection result are inversely transformed to obtain the coordinates of the detection box in the coordinate system of the video frame. The preset fill amount includes: the fill value when the network input tensor is filled with an image based on the network graph instance.
7. The method according to any one of claims 1 to 5, characterized in that, The method of allocating at least one neural network processor core to each video channel includes: Each video channel is allocated a non-overlapping set of neural network processor cores, wherein, when the number of available neural network processor cores is divisible by the number of video channels, the number of neural network processor cores allocated to each video channel is the same and their numbers are consecutive; or The number of neural network processor cores is configured for each video channel according to at least one of the following: video resolution, object detection network complexity, channel priority, historical processing time, and input frame rate, either statically or dynamically.
8. The method according to any one of claims 1 to 5, characterized in that, The process of setting core affinity and creating corresponding network graph instances includes: Set the neural network processor core identifier corresponding to the network graph instance to be created; Create a pre-compiled network graph corresponding to the neural network processor core; Verify whether the pre-compiled network graph can run on the neural network processor core; The pre-compiled network graph that passes the verification is used as a network graph instance and saved to the network graph set of the corresponding video channel.
9. The method according to any one of claims 1 to 5, characterized in that, The capacity of the bounded result set is determined based on the number of neural network processor cores allocated to the corresponding video channel. The step of associating the detection result with the frame number corresponding to the video frame and writing it into a bounded result set managed by the frame number further includes: When the number of results in the bounded result set exceeds the capacity, delete the result with the smallest frame sequence number in the bounded result set; and / or When the frame number corresponding to the video frame is less than or equal to the most recently output frame number, the detection result is rejected from being written into the bounded result set.
10. An electronic device, characterized in that, include: At least one processing device; And a memory communicatively connected to the at least one processing device; wherein the memory stores instructions executable by the at least one processing device to enable the at least one processing device to implement the method as described in any one of claims 1 to 9.
