Video stream analysis processing method based on edge computing and electronic device

Abstract

The application provides a video stream analysis processing method based on edge computing, which is applied to an edge computing device. The method comprises the following steps: acquiring a video stream from a video source accessed by the edge computing device, and decoding the acquired video stream; storing the decoded video stream into a shared storage space; wherein the shared storage space is a storage space shared by video analysis tasks created on the edge computing device; in response to a video stream acquisition request sent by any target video analysis task created on the edge computing device, distributing the decoded target video stream stored in the shared storage space to the target video analysis task, and calling task execution hardware carried by the edge computing device to execute the target video analysis task, so as to analyze and process the target video stream.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of video processing, and more particularly to a video stream analysis and processing method and electronic device based on edge computing. Background Technology

[0002] With the rapid development of artificial intelligence and Internet of Things technologies, the demand for video analytics applications in retail security, industrial inspection, smart cities, and other fields is growing. To reduce network bandwidth consumption and cloud processing latency, the industry generally adopts edge computing, which offloads video analytics tasks to edge computing devices deployed near the data source.

[0003] However, current mainstream video analytics systems based on edge computing devices face technical bottlenecks that severely limit their ability and efficiency in processing large-scale video streams. Summary of the Invention

[0004] In view of the above, one or more embodiments of this specification provide the following technical solutions: According to a first aspect of one or more embodiments of this specification, a video stream analysis and processing method based on edge computing is provided, applied to an edge computing device, the method comprising: The video stream is obtained from the video source accessed by the edge computing device, and the obtained video stream is decoded. The decoded video stream is stored in a shared storage space; wherein, the shared storage space is the storage space shared by the video analysis tasks created on the edge computing device; In response to a video stream acquisition request sent by any target video analysis task created on the edge computing device, the decoded target video stream stored in the shared storage space is distributed to the target video analysis task, and the task execution hardware on the edge computing device is invoked to execute the target video analysis task to analyze and process the target video stream.

[0005] Optionally, obtaining a video stream from a video source accessed by the edge computing device includes: Obtain task information corresponding to the video stream analysis task sent to the edge computing device; wherein, the task information includes a specified video source; The video stream is obtained from the designated video source accessed by the edge computing device.

[0006] Decoding the acquired video stream includes: The edge computing device is equipped with decoding hardware to decode the acquired video stream.

[0007] Optionally, the edge computing device is equipped with dedicated video decoding hardware or a general-purpose processor with video decoding capabilities; The edge computing device is equipped with decoding hardware to decode the acquired video stream, including: The edge computing device utilizes dedicated video decoding hardware or a general-purpose processor with video decoding capabilities to decode the acquired video stream.

[0008] Optionally, the decoded video stream stored in the shared storage space can be pushed to the video analysis task in a publish-subscribe mode; In response to a video stream acquisition request sent by any target video analysis task created on the edge computing device, the decoded target video stream stored in the shared storage space is distributed to the target video analysis task, including: Receive subscription requests for the target video stream sent by any target video analytics task created on the edge computing device; When the decoded target video stream is stored in the shared storage space, a target video analysis task that has subscribed to the target video stream is obtained, and the decoded target video stream stored in the shared storage space is published to the target video analysis task.

[0009] Optionally, the video analysis task includes a pipeline task composed of multiple subtasks arranged in sequence; the subtasks include subtasks that analyze video streams based on video analysis models. The task execution hardware on the edge computing device is invoked to execute the target video analysis task to analyze and process the target video stream, including: From the video analysis model library managed by the edge computing device, determine the video analysis models that match the task requirements of the multiple sub-tasks, and create inference sub-tasks for the multiple sub-tasks to analyze and process the target video stream based on the video analysis models. The inference subtasks created for the multiple subtasks are combined into an inference task to analyze the target video stream according to the execution order of the multiple subtasks. The task execution hardware on the edge computing device is invoked to execute each inference subtask contained in the inference task in the order of execution, and to analyze and process the target video stream.

[0010] Optionally, the edge computing device runs a thread pool for performing inference tasks; wherein the thread pool contains multiple threads; The task execution hardware on the edge computing device is invoked to sequentially execute each inference subtask included in the inference task according to the execution order, and to analyze and process the target video stream, including: The inference task is added to a preset task processing queue, so that multiple threads in the thread pool concurrently read the inference task from the task processing queue and call the task execution hardware on the edge computing device to execute each inference subtask contained in the inference task asynchronously in the order of execution, so as to analyze and process the target video stream.

[0011] Optionally, the edge computing device is equipped with dedicated inference hardware or a general-purpose processor with inference computing capabilities; The task execution hardware on the edge computing device is invoked to execute the various inference subtasks included in the inference task asynchronously and sequentially according to the execution order, to analyze and process the target video stream, including: The edge computing device utilizes dedicated inference hardware or a general-purpose processor with inference computing capabilities to asynchronously execute each inference subtask contained in the inference task, according to the execution order, and analyzes and processes the target video stream.

[0012] Optionally, the video analysis models in the video analysis model library include machine learning models and / or LLM; the video analysis model library supports dynamic updates of the included video analysis models.

[0013] According to a second aspect of one or more embodiments of this specification, an electronic device is also provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor performs the executable instructions to implement the steps of the method as described in any of the first aspects above.

[0014] According to a third aspect of one or more embodiments of this specification, a computer program product is also provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any of the first aspects above.

[0015] In the above embodiments, on the one hand, by adopting a video stream sharing mechanism, the same video stream is decoded only once, and the decoded video stream is stored in a shared storage space for reuse by multiple video analysis tasks, which can fundamentally eliminate the waste of resources caused by repeatedly decoding the same video stream.

[0016] On the other hand, by offloading computationally intensive tasks such as video analytics from the CPU and entrusting them to task execution hardware on edge computing devices, CPU computing resources can be fully freed up, reducing CPU resource utilization. Moreover, since the task execution hardware is no longer limited by the runtime environment of the programming language used by the upper-layer application when performing such tasks, the throughput and real-time performance when processing video streams can be improved, achieving true multi-core parallelism and hardware-level acceleration. This enables concurrent video analysis of more video streams under the same hardware conditions and significantly reduces the processing latency of video stream analysis. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the architecture of a video stream analysis and processing system provided in an exemplary embodiment; Figure 2 This is a flowchart of a video stream analysis and processing method based on edge computing, provided in an exemplary embodiment; Figure 3 This is a schematic diagram of a video stream processing framework based on an edge computing device, provided in an exemplary embodiment. Figure 4 This is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment; Figure 5 This is a block diagram of a video stream analysis and processing apparatus based on edge computing, provided in an exemplary embodiment. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0019] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0020] With the deep integration of artificial intelligence (AI) and Internet of Things (IoT) technologies, video analytics applications have been widely applied to many fields such as retail security, industrial inspection, and smart cities.

[0021] For example, in retail scenarios such as supermarkets and convenience stores, in order to achieve accurate customer flow analysis, digital shelf management, and abnormal behavior monitoring, a large number of surveillance cameras need to be installed in the store.

[0022] In the traditional cloud processing model, all raw video streams need to be transmitted to the cloud via a wide area network for video decoding and analysis.

[0023] However, this model faces enormous technical challenges when dealing with hundreds or even thousands of high-definition video streams within shopping malls: On the one hand, the uplink transmission of massive amounts of video data consumes extremely expensive network bandwidth resources; On the other hand, the round-trip time (RTT) of cloud processing is difficult to meet the requirements of video analytics scenarios that have strict requirements for low latency.

[0024] Therefore, the industry generally adopts the edge computing model, which moves video analytics tasks from the cloud to edge computing devices deployed near the data source.

[0025] For example, in retail scenarios such as supermarkets and convenience stores, these edge computing devices can be edge computing boxes or industrial computers deployed inside supermarkets and convenience stores, closer to the cameras.

[0026] This edge computing model enables real-time processing of video streams locally on edge computing devices, which not only significantly reduces network bandwidth costs but also achieves millisecond-level response speeds, thereby supporting the implementation of complex business scenarios such as smart retail.

[0027] However, despite the inherent architectural advantages of edge computing, existing video analytics systems still face significant technical bottlenecks in practical applications when handling large-scale, high-concurrency video streams. These bottlenecks are mainly reflected in the following aspects: On the one hand, in the existing architecture, each video analytics application typically builds its own end-to-end processing flow, from streaming and decoding to inference. When multiple analysis tasks are performed on the same video source (such as a camera at the entrance of a shopping mall), the system will perform repetitive hardware decoding operations multiple times, resulting in redundant consumption of CPU and GPU resources.

[0028] Furthermore, since a large amount of application-layer business logic is developed using high-level programming languages ​​such as Python, the system is limited by the concurrency restriction mechanisms of its runtime environment (such as the Global Interpreter Lock, GIL), making it difficult to fully utilize the parallel computing performance of multi-core processors. As a result, the overall processing speed in high-concurrency scenarios is limited by single-core performance, resulting in low throughput.

[0029] Second, the system architecture is highly coupled and difficult to adapt to the hardware ecosystem. In current solutions, business processing logic (such as passenger flow statistics) is typically deeply coupled with specific AI models and underlying hardware platforms. This tight coupling results in extremely low model reusability across different tasks and a lack of system architecture flexibility. Simultaneously, edge hardware environments are highly heterogeneous, with significant architectural differences between hardware from different vendors (such as various GPUs and NPU acceleration units). Due to the lack of a unified hardware abstraction layer and resource orchestration mechanism, the system exhibits poor adaptability to heterogeneous hardware environments, making it difficult to achieve unified task scheduling and performance optimization across different devices.

[0030] It is evident that current mainstream video analytics systems based on edge computing devices lack an execution framework capable of enabling hardware and software collaboration, data reuse, and dynamic resource orchestration when dealing with the analysis needs of massive video streams at the edge. This makes it difficult to effectively utilize heterogeneous computing resources and execute tasks with high concurrency while ensuring real-time performance.

[0031] [qf1][qf2] On the one hand, in the existing technical architecture, each independent video analytics application needs to independently establish a complete processing chain from video streaming and decoding to inference in order to complete its own analysis tasks. When performing multiple analysis tasks on the same video source (such as a camera at the entrance of a shopping mall), the system usually needs to repeatedly execute computationally intensive tasks such as decoding, resulting in a serious waste of computing resources such as CPU and GPU, and failing to fully utilize the hardware performance of edge computing devices.

[0032] Furthermore, many video analytics applications utilize high-level languages ​​like Python for their application-layer business logic. To ensure memory safety and reduce implementation complexity, the runtime environments of these programming languages ​​may limit the CPU's multi-threaded concurrency capabilities. For example, Python's runtime environment employs a mechanism called the GIL (Global Interpreter Lock) to restrict the CPU's multi-threaded concurrency.

[0033] This can lead to the processing speed being limited to the processing performance of a single CPU when analyzing and processing multiple video streams concurrently. The system cannot fully utilize the parallel computing power of multi-core processors, resulting in the overall processing speed being limited by single-core performance in high-concurrency scenarios, with low throughput and low overall processing efficiency.

[0034] On the other hand, existing solutions typically introduce video analytics models to analyze video streams and bind video stream analysis functions (such as passenger flow statistics) to specific AI models required to perform these functions, thus achieving tight coupling between business logic and model inference. This deeply coupled architecture can lead to extremely low model reusability and poor system flexibility. For example, an AI model already bound to a certain video analytics service may temporarily be unable to be invoked by other video analytics services.

[0035] Furthermore, edge computing hardware environments are typically highly heterogeneous. Hardware from different manufacturers (such as various GPUs and NPU acceleration units) exhibit significant architectural differences, resulting in vastly different decoding and inference capabilities across edge computing devices. Existing solutions often lack a unified hardware abstraction layer, making it difficult to adapt to hardware from different manufacturers, limiting system scalability, and hindering unified task scheduling and performance optimization across different devices. Under the combined effect of these problems, when edge computing-based video analytics systems need to process large-scale concurrent video streams, response latency increases significantly, failing to meet the stringent low-latency requirements of applications such as security monitoring and real-time quality inspection. Therefore, current mainstream edge computing-based video analytics systems lack an execution framework capable of achieving hardware-software collaboration, data reuse, and dynamic resource orchestration when dealing with the analysis needs of massive video streams at the edge. This makes it difficult to effectively utilize heterogeneous computing resources and execute tasks with high concurrency while ensuring real-time performance.

[0036] Based on this, this specification proposes a video stream analysis and processing framework based on edge computing.

[0037] In this analysis and processing framework, on the one hand, a sharing mechanism is proposed that allows multiple video analysis tasks to share the decoded video stream.

[0038] Under this sharing mechanism, only one decoding operation needs to be performed on the same video stream (such as decoding operation based on decoding hardware), and the decoded video stream is stored in the shared storage space, which can be reused by multiple video analysis tasks, fundamentally eliminating the waste of resources caused by repeatedly decoding the same video stream.

[0039] On the other hand, a mechanism is proposed to decouple computationally intensive tasks, including video analytics, from the CPU.

[0040] Under this mechanism, computationally intensive tasks such as video analysis can be offloaded to the task execution hardware on edge computing devices, which can not only fully release CPU computing resources but also reduce CPU resource utilization. For example, the CPU utilization of edge computing devices can be reduced from 100% full load in traditional solutions to below 30%, thereby freeing up computing resources for more complex analysis tasks.

[0041] Moreover, since these task execution hardware are no longer limited by the runtime environment of the programming language used by the upper-layer application when performing such tasks, they can improve the throughput and real-time performance when processing video streams, achieve true multi-core parallelism and hardware-level acceleration, thereby enabling concurrent video analysis of more video streams under the same hardware conditions and significantly reducing the processing latency of video stream analysis.

[0042] For example, it can increase the maximum number of concurrent video analysis channels supported by a single edge computing device from the traditional 10 channels to more than 30 channels.

[0043] Please see Figure 1 , Figure 1 This is a schematic diagram of the architecture of a video stream analysis and processing system based on edge computing, as shown in this specification.

[0044] like Figure 1 As shown, the system can be applied to retail scenarios and may include a video source 110, an edge computing device 120, a network 130, a task management platform (server) 140, and a management terminal 150.

[0045] In retail settings such as supermarkets and convenience stores, video source 110 can specifically include several surveillance cameras deployed in key locations within physical stores (such as store entrances, cash registers, and shelf areas). These cameras are responsible for collecting raw video data from within the store in real time and using it as the video source for video analysis tasks.

[0046] Edge computing device 120 is the core processing node of the system. It can typically be deployed inside physical stores such as supermarkets and convenience stores, within a local area network that is physically closer to the video source 110. This device can take the form of an edge computing box or an industrial computer, integrating a CPU, GPU, or dedicated AI acceleration unit.

[0047] During operation, the edge computing device 120 is responsible for receiving video analysis tasks (such as passenger flow statistics and queue detection) issued by the task management platform 140, and directly accessing the video stream of the video source 110 for localized hardware decoding and parallel inference analysis, thereby avoiding the bandwidth pressure caused by uploading massive amounts of raw video to the cloud.

[0048] The task management platform (server) 140 can be a server or server cluster deployed in the cloud or a central data center. Its main responsibility is to perform global task scheduling and device management. For example, the administrator can use the task management platform 140 to issue specific customer flow analysis tasks and other task information to the edge computing devices 120 deployed in various stores.

[0049] Edge computing device 120 and task management platform 140 can interact via network 130. This network can be based on mobile communication networks (such as 4G / 5G), wired broadband, etc. Since the core video stream analysis process is completed locally on edge computing device 120, network 130 only needs to transmit a small number of task instructions and structured analysis results (such as passenger flow data), significantly reducing the bandwidth consumption of the backbone network.

[0050] The management terminal 150 can be a PC, mobile phone, or tablet used by supermarket managers. Supermarket managers can remotely view real-time customer flow analysis reports or early warning information reported by edge computing devices through the client program running on the management terminal, thereby assisting stores in making digital business decisions.

[0051] The technical solution of this specification will be described in detail below with reference to the accompanying drawings.

[0052] Please see Figure 2 , Figure 2 This specification illustrates a video stream analysis and processing method based on edge computing, the method comprising the following execution steps: Step 202: Obtain a video stream from the video source accessed by the edge computing device, and decode the obtained video stream; The entity executing the above method can be an edge computing device. For example, it could be... Figure 1 The illustrated edge computing device 120 is part of the architecture of a video stream analysis and processing system applied in a retail scenario. Specifically, the aforementioned edge computing device may include dedicated hardware devices deployed near the data generation source, possessing certain computing, storage, and network communication capabilities.

[0053] It should be noted that the specific form of the aforementioned edge computing devices is not specifically limited in this specification.

[0054] For example, in practical applications, typical forms of the aforementioned edge computing devices may include, but are not limited to: edge computing boxes integrating CPUs, GPUs, and dedicated artificial intelligence acceleration units (such as neural network engines), industrial computers, high-performance microcomputers, smart cameras integrating computing modules, and so on. These devices share the common characteristic of being able to process video streams in real time locally, at the source of the video stream (such as at the camera), without needing to upload all the raw video data to the cloud for processing.

[0055] When using edge computing devices to process video streams, the video stream to be processed can be obtained from the video source connected to the edge computing device.

[0056] The specific method for obtaining the video stream to be processed from the video source accessed by the edge computing device is not specifically limited in this specification.

[0057] In some embodiments, the video stream analysis task issued by the task management platform can specify a video source corresponding to the video stream to be processed. For example, the video source can be a camera deployed offline.

[0058] In this scenario, the edge computing device can obtain task information corresponding to the video stream analysis task issued by the task management platform, parse the task information to obtain the content contained in the task information, and further obtain the video stream to be analyzed from the specified video source contained in the task information.

[0059] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating a video stream processing framework based on an edge computing device, as shown in this specification.

[0060] like Figure 3 As shown, in some embodiments, a video analytics application can be implemented on an edge computing device. The task management platform can send task information corresponding to the video stream analysis task to the video analytics application running on the edge computing device through a secure channel (such as HTTPS).

[0061] The task information mentioned above can typically be a structured data description (such as JSON format), which may specifically include a specified video source.

[0062] For example, the task information mentioned above may specifically include: the task identifier of the video analytics task to be created, the access addresses of one or more video sources that the task needs to analyze, the analysis type of the task, the scheduling plan, and so on. After parsing this task information, the video analytics application can determine which data sources need to obtain video streams.

[0063] The method used to obtain the video stream to be analyzed from the specified video source contained in the task information will not be specifically limited in this specification.

[0064] like Figure 3 As shown, in some embodiments, a video streaming module can also be implemented on the edge computing device. The aforementioned video analysis application can use this video streaming module to actively obtain the video stream to be analyzed from the video source using a pull method. Specifically, this video streaming module can be implemented using a source component; for example, FFmpeg can be used to implement the video streaming module.

[0065] Once the edge computing device obtains the video stream to be analyzed from the specified video source, this video stream is usually the original video stream that has been encoded. The edge computing device can also decode the original video stream.

[0066] It should be noted that for computationally intensive tasks such as video decoding, the decoding can be separated from the CPU and no longer performed in software, but instead executed by the decoding hardware on edge computing devices.

[0067] For example, computationally intensive tasks such as video decoding can no longer be performed by... Figure 3 Instead of being executed by the aforementioned video analytics application, the video analytics application executes the task by calling the underlying decoding hardware on the edge computing device.

[0068] Please continue reading Figure 3 In some embodiments, after the video analysis application obtains the video stream to be analyzed from the specified video source through the video streaming module, it can further perform decoding operations on the original video stream.

[0069] The decoding operation of the original video stream can be performed using either software decoding or hardware decoding.

[0070] In some embodiments, after the video analysis application obtains the video stream to be analyzed from the specified video source through the video streaming module, it can further call the decoding hardware on the edge computing device to perform hardware decoding on the original video stream.

[0071] For example, the acquired video stream to be analyzed can be submitted as a call parameter to the decoding hardware on the edge computing device, initiating a call to the decoding hardware, which will then perform hardware decoding on the original video stream.

[0072] In this context, "decoding hardware" is a broad concept, referring to any physical unit capable of performing video decoding calculations.

[0073] In some embodiments, the decoding hardware on the edge computing device may specifically include dedicated inference hardware or a general-purpose processor with inference computing capabilities.

[0074] Dedicated video decoding hardware refers to fixed-function circuits or processing cores within a chip specifically designed for video encoding and decoding algorithms. For example, dedicated inference hardware could be a videotoolbox. Videotoolbox is a media processing engine that can be integrated into iOS device hardware, allowing developers to directly access hardware codecs and utilize the hardware resources of iOS devices to achieve efficient video encoding and decoding operations. A general-purpose processor with video decoding capabilities can refer to a GPU.

[0075] In this scenario, the edge computing device can utilize its dedicated video decoding hardware or a general-purpose processor with video decoding capabilities to decode the acquired video stream. The specific implementation details of decoding the video stream using the hardware are not elaborated upon in this specification.

[0076] By utilizing the decoding hardware on edge computing devices to perform hardware decoding of the original video stream, computationally intensive tasks such as video decoding can be offloaded from the CPU and executed by the decoding hardware on the edge computing devices. This can fully free up CPU computing resources and reduce CPU resource utilization. Moreover, since the decoding hardware is no longer limited by the runtime environment of the programming language used by the upper-layer application when performing such tasks, it can improve the throughput and real-time performance when processing video streams, achieving true multi-core parallelism and hardware-level acceleration.

[0077] Step 204: Store the decoded video stream to a shared storage space; wherein, the shared storage space is the storage space shared by the video analysis task created on the edge computing device; Once the edge computing device has decoded the acquired video stream by calling the decoding hardware on the edge computing device, the decoded video stream can be stored in a shared storage space.

[0078] Specifically, this shared storage space could be a shared storage space for all video analytics tasks created on an edge computing device.

[0079] For example, this shared storage space could specifically be a storage area allocated within the memory or cache of the edge computing device, which can be shared and visible to all video analytics tasks running on the edge computing device. In practical applications, this storage area could be a shared memory area provided by the operating system of the edge computing device, or it could be a global buffer accessible to all video analytics tasks.

[0080] In this way, the same video stream can be decoded only once using the decoding hardware, and the decoded video stream can be stored in a shared storage space for reuse by multiple video analysis tasks, thereby fundamentally eliminating the waste of resources caused by repeatedly decoding the same video stream.

[0081] For example, when multiple different video analytics tasks run on an edge computing device, and these tasks need to analyze video streams from the same camera, there's no need for each task to independently decode the video stream repeatedly. The decoding module only needs to decode the video stream once and place it into this shared storage space, making it available for all relevant video analytics tasks to read.

[0082] It should be noted that the method of storing the decoded video stream in the aforementioned shared storage space is not specifically limited in this specification.

[0083] For example, in some embodiments, the decoded video stream can be persistently stored in the shared storage space. In other embodiments, the decoded video stream can also be temporarily cached in the shared storage space. If the cached decoded video stream is not used by the video analysis task within a preset time period, the cached decoded video stream can be cleared from the shared storage space.

[0084] Step 206: In response to a video stream acquisition request sent by any target video analysis task created on the edge computing device, the decoded target video stream stored in the shared storage space is distributed to the target video analysis task, and the task execution hardware on the edge computing device is invoked to execute the target video analysis task to analyze and process the target video stream.

[0085] Multiple video analytics tasks can be pre-created on the edge computing device. These video analytics tasks can acquire the video stream to be analyzed by sending a video stream acquisition request to the edge computing device.

[0086] When an edge computing device receives a video stream acquisition request from any target video analysis task created on the edge computing device, it can respond to the video stream acquisition request and distribute the decoded target video stream required by the target video analysis task stored in the shared storage space to the target video analysis task.

[0087] In some embodiments, these video analytics tasks can acquire the video stream to be analyzed by sending a video stream acquisition request to the edge computing device. Upon receiving the video stream acquisition request from the target video analytics task, the edge computing device can, in response to the request, locate the target video stream required by the target video analytics task in the shared storage space, and then distribute the located target video analytics task to it.

[0088] Of course, if the target video stream required for the target video analysis task has not yet been found in the shared storage space, the target video stream may not have been decoded or has not been fully decoded. The target video stream can be searched for in the shared storage space after it has been decoded and successfully stored in the shared storage space.

[0089] It should be noted that the distribution method of distributing the decoded target video stream stored in the shared storage space to the target video analysis task is not specifically limited in this specification.

[0090] In some embodiments, the decoded video stream stored in the shared storage space can be pushed to specific video analysis tasks in a publish-subscribe model.

[0091] In this scenario, when acquiring the target video stream to be analyzed, the aforementioned target video analysis task can send a subscription request for the target video stream to the edge computing device. Upon receiving the subscription request, the edge computing device, after successfully storing the decoded target video stream in the shared storage space, can then acquire the video analysis tasks that have subscribed to the target video stream and distribute it to the various video analysis tasks that have subscribed to it. At this point, the video analysis tasks subscribed to the target video stream can include the aforementioned target video analysis task, or they can include other video analysis tasks.

[0092] Please continue reading Figure 3 In some embodiments, a distributor can be implemented on an edge computing device, and this distributor maintains the shared storage space. When a new video analytics task is created, it can send a subscription request to the distributor, declaring which video stream(s) it needs to subscribe to. The distributor can maintain a mapping between video streams and subscribers. Whenever the decoding hardware stores a decoded video stream in the shared storage space and notifies the distributor, the distributor can proactively push references or copies pointing to that decoded video stream to all video analytics tasks that have subscribed to that video stream, based on the mapping relationship. This publish-subscribe model decouples producers and consumers, resulting in high distribution efficiency.

[0093] After the decoded target video stream stored in the shared storage space is distributed to the target video analysis task, the edge computing device can execute the target video analysis task to analyze and process the target video stream to be analyzed.

[0094] It should be noted that computationally intensive tasks such as video analytics can also be offloaded from the CPU and executed by edge computing devices instead of software-based execution. Similarly, inference tasks can also be performed by hardware mounted on edge computing devices. Figure 3 Instead of being executed by the video analytics application shown above, the video analytics application executes the task by calling the underlying task execution hardware on the edge computing device.

[0095] In this scenario, after the edge computing device distributes the decoded target video stream stored in the shared storage space to the target video analysis task, it can invoke the onboard task execution hardware to execute the target video analysis task to analyze and process the target video stream.

[0096] Here, the aforementioned task execution hardware is also a broad concept, referring to the physical unit capable of performing calculations related to video analysis tasks.

[0097] In some embodiments, taking the video analysis task described above as an example of a task that requires inference calculations based on a video analysis model, the hardware for executing the task may specifically include any form of physical hardware with inference calculation capabilities related to model inference.

[0098] For example, the task hardware on an edge computing device may specifically include dedicated inference hardware or a general-purpose processor with inference computing capabilities. The dedicated inference hardware may include acceleration hardware optimized for specific inference computations (such as matrix operations) of video analytics models (such as neural network models).

[0099] In some embodiments, taking the video analysis model as an example of a neural network-based model, the dedicated inference hardware can specifically be a neural network engine integrated into the hardware.

[0100] For example, in one instance, the dedicated inference hardware mentioned above could be CoreML. CoreML is a machine learning inference framework deeply integrated into the iOS hardware architecture, allowing developers to easily access heterogeneous computing resources within the iOS hardware to achieve efficient model inference operations.

[0101] The general-purpose processor with inference computing capabilities mainly refers to GPUs. The powerful parallel floating-point computing capabilities of GPUs also make them very suitable for performing deep learning inference.

[0102] This design allows for flexible adaptation to different types of hardware resources on edge computing devices. Whether the device has dedicated AI acceleration hardware or only a high-performance GPU, it can effectively utilize its computing power to perform video analysis tasks.

[0103] It should be noted that the specific types of the aforementioned video analysis tasks are not further limited in this specification.

[0104] Please continue reading Figure 3 In some embodiments, the video analytics task described above may include multiple subtasks executed in sequence; specifically, a video analytics task created on an edge computing device may include a pipeline task consisting of multiple subtasks executed in sequence.

[0105] For example, taking the above-mentioned pipeline task as a task of "people counting" based on the video stream collected by cameras deployed in the mall, the pipeline task may be composed of three sub-tasks in sequence: "human body detection", "head counting" and "trajectory tracking".

[0106] Please continue reading Figure 3 In some embodiments, the subtasks included in the above pipeline task may specifically be subtasks that analyze video streams based on video analysis models.

[0107] For example, taking the above-mentioned pipeline task as a task of "people counting" based on video streams collected by cameras deployed in shopping malls, the pipeline task may consist of three sub-tasks in sequence: "human detection based on human detection model", "head counting based on head calculation model" and "trajectory tracking based on trajectory tracking model".

[0108] In this scenario, the aforementioned edge computing device can determine video analysis models from the managed video analysis model library that match the task requirements of each of the multiple sub-tasks, and create inference sub-tasks for each of the multiple sub-tasks based on the video analysis models that match the task requirements of each sub-task, to analyze and process the target video stream. Then, the inference sub-tasks created for each of the multiple sub-tasks can be combined into an inference task for analyzing the target video stream according to the execution order of the multiple sub-tasks.

[0109] For example, such as Figure 3 As shown, the reasoning task can specifically be a chain of reasoning tasks based on a video analysis model, consisting of multiple atomized reasoning subtasks.

[0110] In this way, the atomic models managed in the video analytics model library can be separated from specific video analytics tasks. The system can dynamically orchestrate and combine different atomic models to form a processing pipeline according to task requirements. This not only greatly improves the reusability of models, but also allows the development and deployment of new tasks to be quickly implemented through configuration without starting from scratch, thus enhancing the maintainability and scalability of the system.

[0111] Once an inference task is generated, the task execution hardware on the edge computing device can be invoked to execute the various inference subtasks contained in the inference task in the order of execution, and to analyze and process the target video stream.

[0112] When using the task execution hardware on the edge computing device to execute the inference task, a specific selection strategy can be employed to choose suitable hardware for the inference task. This specification does not impose specific restrictions on the selection strategy used to choose suitable hardware for the inference task.

[0113] For example, in the implementation of the application, you can choose dedicated inference hardware (such as CoreML) to perform the inference task, or you can choose a general-purpose processor with inference computing capabilities (such as a GPU) to perform the inference task.

[0114] In this way, the video analysis model can be separated from the underlying task execution hardware. The inference task can focus only on what model is needed, while the hardware abstraction layer decides which hardware to call in the backend to execute the inference task. This allows the above solution to be adapted to edge computing devices from different manufacturers, enhancing the scalability of the system.

[0115] Please continue reading Figure 3 In some embodiments, a model orchestration engine can be implemented on the edge computing device. Specifically, this model orchestration engine can serve as a hardware abstraction layer, responsible for orchestrating specific inference tasks for the various pipeline tasks created on the edge computing device, and selecting appropriate hardware from the task execution hardware on the edge computing device to execute the inference task.

[0116] like Figure 3 As shown, during the execution phase of the inference task, the model orchestration engine can determine video analysis models from the video analysis model library that match the task requirements of the multiple sub-tasks included in the target video analysis task, and create inference sub-tasks for each of the multiple sub-tasks based on the video analysis models that match the task requirements of each sub-task to analyze and process the target video stream; then, the inference sub-tasks created for each of the multiple sub-tasks can be orchestrated into an inference task to analyze the target video stream according to the execution order of the multiple sub-tasks.

[0117] Furthermore, the model orchestration engine can select the task execution hardware to perform the inference task from the task execution hardware on the edge computing device, and send the orchestrated inference task to the task execution hardware. The task execution hardware will then execute each inference subtask contained in the inference task in the order of execution to analyze and process the target video stream.

[0118] For example, taking the above-mentioned pipeline task as an example of "people counting" based on video streams collected by cameras deployed in shopping malls, this pipeline task can be composed of three sub-tasks in sequence: "human body detection", "head counting" and "trajectory tracking".

[0119] In this scenario, the edge computing device can sequentially determine from the managed video analytics model library a human detection model for performing human detection tasks, a head counting model for performing head counting tasks, and a trajectory tracking model for performing trajectory tracking tasks.

[0120] Then, a first inference subtask based on a human detection model for human detection, a second inference subtask based on a head counting model for head counting, and a third single inference subtask based on a trajectory tracking model for trajectory tracking can be created sequentially. The first, second, and third inference subtasks are then combined in sequence into a complete inference task, a video analysis model matching the task requirements of these multiple subtasks is created, and inference subtasks based on the video analysis model matching the task requirements of each subtask are created for each of these multiple subtasks to analyze and process the target video stream.

[0121] Furthermore, the inference subtasks created separately for these multiple subtasks can be combined into an inference task to analyze the target video stream according to the execution order of these multiple subtasks.

[0122] Finally, the task execution hardware on the edge computing device can be invoked to execute each inference subtask contained in the inference task in the order of execution, and to analyze and process the target video stream.

[0123] In some embodiments, the aforementioned video analysis model library may specifically be a repository that centrally stores and manages various video analysis models, which may include machine learning models (such as neural network models) or LLM (Large Language Model).

[0124] For example, these machine learning models can be specialized machine learning models that have been trained to perform specific video analysis subtasks. The aforementioned LLM can be a multimodal LLM with image processing capabilities.

[0125] In some embodiments, the video analytics model library described above may also support dynamic updates to the included video analytics models. For example, upgrading existing models or replacing / adding new models.

[0126] In this way, not only can the aforementioned video analysis model library adapt to different analysis needs, but it can also avoid using a fixed model for the same analysis task.

[0127] To improve the overall throughput and response speed of video stream analysis, the inference process corresponding to the above inference task can be executed concurrently in an asynchronous manner.

[0128] Please continue reading Figure 3 In some embodiments, after the model orchestration engine distributes the orchestrated inference task to the selected task execution hardware, the task execution hardware can specifically execute the inference task concurrently in an asynchronous manner.

[0129] It should be noted that during the asynchronous concurrent execution of the above inference tasks by the task execution hardware, any type of asynchronous task execution framework or component can be used, and no special restrictions will be imposed in this specification.

[0130] In some embodiments, a thread pool for executing inference tasks can be created on the edge computing device described above, and an asynchronous task processing queue can be implemented for the threads in the thread pool; wherein, the thread pool contains multiple threads. These threads are responsible for executing the subtasks contained in the aforementioned inference task.

[0131] In this scenario, during the execution phase of the inference task, the task execution hardware can first add the inference task to the asynchronous task processing queue, so that multiple threads in the thread pool can concurrently read the inference task from the task processing queue.

[0132] For example, in practical applications, each thread in the thread pool can be in a circular waiting state, continuously reading inference tasks from the head of the task processing queue. This task processing queue can be thread-safe, ensuring that multiple threads do not compete for the same inference task. For instance, a locking mechanism can be introduced into the task processing queue to transform "concurrent task reading" into "sequential task retrieval".

[0133] In this way, since multiple threads can read inference tasks from the task processing queue at the same time, different threads can process different inference tasks simultaneously, which makes full use of the task execution hardware on the edge computing device and greatly improves the system's concurrent processing capability.

[0134] Once any thread in the thread pool reads an inference task from the task processing queue, it can further invoke the task execution hardware on the edge computing device to execute each inference subtask contained in the inference task asynchronously in the order of execution.

[0135] For example, the hardware execution hardware selected by the hardware abstraction layer for the inference task can be invoked to execute the various inference subtasks contained in the inference task asynchronously in the order of execution. Once the underlying task execution hardware completes the inference calculation corresponding to the above inference task, the asynchronous task callback mechanism can be used to return the calculation result to the original pipeline task corresponding to the inference task in the form of a callback.

[0136] For example, in practical applications, the inference task described above may specifically include a callback address of a callback function. After the task execution hardware completes the inference calculation corresponding to the inference task, it can call the callback function based on the callback address to return the calculation result to the corresponding pipeline task.

[0137] By employing asynchronous execution to perform the aforementioned inference task, the original pipeline task that submitted the inference task no longer needs to wait for the calculation result of the inference task. Instead, the underlying task execution hardware actively returns the result to the original pipeline task using the asynchronous task callback mechanism, thereby improving the execution efficiency of the inference task and the system response speed.

[0138] Figure 4 This is a schematic structural diagram of an electronic device provided in an exemplary embodiment.

[0139] like Figure 4As shown, device 400 mainly consists of a communication interface 402, a user interface 404, a processor 406, and a data storage 408. These components are interconnected and communicate with each other via a system bus, network, or other connection mechanism 410. The communication interface 402 enables device 400 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 402 may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface 402 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface 402 can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 402 may also include multiple physical communication interfaces, such as Wi-Fi, Bluetooth, and wide-area wireless interfaces.

[0140] User interface 404 includes receiving user input and providing output to the user. Therefore, user interface 404 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 404 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 404 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, device 400 may support remote access from other devices via communication interface 402 or another physical interface (not shown). User interface 404 may be configured to receive user input, the position and movement of which may be indicated by an indicator or cursor described herein. User interface 404 may also be configured as a display device for rendering or displaying text fragments.

[0141] Processor 406 may contain one or more general-purpose processors and / or special-purpose processors.

[0142] Data storage 408 may include one or more volatile and / or non-volatile storage components and may be integrated wholly or partially with processor 406. Data storage 408 may include removable and non-removable components.

[0143] Processor 406 is capable of executing program instructions 418 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 408 to perform the various functions described herein. Data storage 408 may comprise a non-transitory computer-readable medium on which program instructions are stored, which, when executed by device 400, enable device 400 to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Processor 406 executing program instructions 418 may result in processor 406 using data 412.

[0144] For example, program instructions 418 may include an operating system 422 (e.g., an operating system kernel, device drivers, and / or other modules) installed on device 400 and one or more applications 420 (e.g., a browser, social application, or game application). Similarly, data 412 may include operating system data 416 and application data 414. Operating system data 416 is primarily accessible to the operating system 422, while application data 414 is primarily accessible to one or more applications 420. Application data 414 may reside in a file system visible or hidden from the user of device 400.

[0145] Application 420 can communicate with operating system 422 through one or more application programming interfaces (APIs). These APIs help application 420 read and / or write application data 414, transmit or receive information via communication interface 402, receive or display information on user interface 404, etc.

[0146] In some terminology, application 420 may be simply referred to as "app". Furthermore, application 420 can be downloaded to device 400 through one or more online app stores or app markets. However, applications can also be installed on device 400 in other ways, such as through a web browser or a physical interface on device 400 (e.g., a USB port).

[0147] Please refer to Figure 5 This specification also proposes a video stream analysis and processing device based on edge computing, which can be applied to applications such as... Figure 4 The device shown is used to implement the technical solution of this specification. The device may include: The acquisition module 501 acquires a video stream from the video source accessed by the edge computing device and decodes the acquired video stream. Storage module 502 stores the decoded video stream to a shared storage space; wherein, the shared storage space is a storage space shared by video analysis tasks created on the edge computing device; The execution module 503, in response to a video stream acquisition request sent by any target video analysis task created on the edge computing device, distributes the decoded target video stream stored in the shared storage space to the target video analysis task, and calls the task execution hardware on the edge computing device to execute the target video analysis task to analyze and process the target video stream.

[0148] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more of these specifications, the functions of each module or unit can be implemented in the same or different software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0149] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.

[0150] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0151] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0152] What those skilled in the art will understand is: In this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded.

[0153] In this specification, “a,” “an,” and “the” do not specifically refer to the singular, but may also include the plural.

[0154] In this specification, ordinal numbers such as "first," "second," etc., do not necessarily indicate order; they are often used to distinguish between objects. For example, "first server" and "second server" usually refer to two servers. To differentiate between these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0155] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0156] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0157] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0158] Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is only one of many possible execution orders and does not represent the only execution order. Therefore, when the claims involve method steps, any changes or adjustments to the order of such steps, or the parallelism between steps, are also within the scope of protection of the claims.

Claims

1. A video stream analysis and processing method based on edge computing, applied to an edge computing device; the method includes: The video stream is obtained from the video source accessed by the edge computing device, and the obtained video stream is decoded. The decoded video stream is stored in a shared storage space; wherein, the shared storage space is the storage space shared by the video analysis tasks created on the edge computing device; In response to a video stream acquisition request sent by any target video analysis task created on the edge computing device, the decoded target video stream stored in the shared storage space is distributed to the target video analysis task, and the task execution hardware on the edge computing device is invoked to execute the target video analysis task to analyze and process the target video stream.

2. The method as described in claim 1, wherein obtaining a video stream from a video source accessed by the edge computing device comprises: Obtain task information corresponding to the video stream analysis task sent to the edge computing device; wherein, the task information includes a specified video source; Obtain a video stream from the designated video source accessed by the edge computing device; Decoding the acquired video stream includes: The edge computing device is equipped with decoding hardware to decode the acquired video stream.

3. The method as described in claim 2, wherein the edge computing device is equipped with dedicated video decoding hardware or a general-purpose processor with video decoding function; The edge computing device is equipped with decoding hardware to decode the acquired video stream, including: The edge computing device utilizes dedicated video decoding hardware or a general-purpose processor with video decoding capabilities to decode the acquired video stream.

4. The method as described in claim 3, wherein the decoded video stream stored in the shared storage space supports being pushed to the video analysis task in a publish-subscribe mode; In response to a video stream acquisition request sent by any target video analysis task created on the edge computing device, the decoded target video stream stored in the shared storage space is distributed to the target video analysis task, including: Receive subscription requests for the target video stream sent by any target video analytics task created on the edge computing device; When the decoded target video stream is stored in the shared storage space, a target video analysis task that has subscribed to the target video stream is obtained, and the decoded target video stream stored in the shared storage space is published to the target video analysis task.

5. The method as described in claim 4, wherein the video analysis task comprises a pipeline task consisting of multiple subtasks arranged in sequence; the subtasks include a subtask for analyzing a video stream based on a video analysis model; The task execution hardware on the edge computing device is invoked to execute the target video analysis task to analyze and process the target video stream, including: From the video analysis model library managed by the edge computing device, determine the video analysis models that match the task requirements of the multiple sub-tasks, and create inference sub-tasks for the multiple sub-tasks to analyze and process the target video stream based on the video analysis models. The inference subtasks created for the multiple subtasks are combined into an inference task to analyze the target video stream according to the execution order of the multiple subtasks. The task execution hardware on the edge computing device is invoked to execute each inference subtask contained in the inference task in the order of execution, and to analyze and process the target video stream.

6. The method of claim 5, wherein a thread pool for performing inference tasks runs on the edge computing device; wherein, The thread pool contains multiple threads; The task execution hardware on the edge computing device is invoked to sequentially execute each inference subtask included in the inference task according to the execution order, and to analyze and process the target video stream, including: The inference task is added to a preset task processing queue, so that multiple threads in the thread pool concurrently read the inference task from the task processing queue and call the task execution hardware on the edge computing device to execute each inference subtask contained in the inference task asynchronously in the order of execution, so as to analyze and process the target video stream.

7. The method as described in claim 6, wherein the edge computing device is equipped with dedicated inference hardware or a general-purpose processor with inference computing capabilities; The task execution hardware on the edge computing device is invoked to execute the various inference subtasks included in the inference task asynchronously and sequentially according to the execution order, to analyze and process the target video stream, including: The edge computing device utilizes dedicated inference hardware or a general-purpose processor with inference computing capabilities to asynchronously execute each inference subtask contained in the inference task, according to the execution order, and analyzes and processes the target video stream.

8. The method as described in claim 5, wherein the video analysis models in the video analysis model library include machine learning models and / or LLM; and the video analysis model library supports dynamic updates to the included video analysis models.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor implements the steps of the method as described in any one of claims 1-8 by running the executable instructions.

10. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-8.

Images