Video stitching method and system and related equipment

Abstract

The invention provides a video stitching method, system and related equipment, which are applied to the technical field of data processing, and the video stitching method comprises the following steps: obtaining multiple paths of video streams input by monitoring equipment, and decoding the multiple paths of video streams to obtain multiple paths of decoded video data of the multiple paths of video streams; performing feature extraction on the multi-channel decoded video data through a neural network processor to obtain multi-channel video feature information of the multi-channel video streams; performing feature matching on the multi-channel video feature information through a neural network processor to obtain a multi-channel feature matching result of the multi-channel video streams; when the multi-channel video feature information is subjected to feature matching to obtain a multi-channel feature matching result of the multi-channel video stream, removing mismatching points in the multi-channel feature matching result in parallel; and through a neural network processor, aligning the multiple paths of video frames according to the effective feature matching points in the multiple paths of feature matching results, and generating target video stitching data. The video stitching efficiency and the resource utilization rate can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a video splicing method, system, and related equipment. Background Art

[0002] As the demand for panoramic perception in intelligent surveillance systems grows, multi-camera video stitching technology becomes key. Multi-camera video stitching technology generates seamless, wide-viewing angle or panoramic videos by fusing video streams captured by multiple cameras.

[0003] Existing video stitching technology uses a centralized server-side processing architecture, relying on high-performance CPUs or GPUs for feature matching and fusion. However, this technology has significant drawbacks: the transmission of multiple channels of high-definition video data to the server results in response delays, making it difficult to meet real-time requirements; the increase in video channels and resolution exponentially increases the computational load and sharply degrades energy efficiency. In view of this, edge computing has become a mainstream trend, reducing latency by offloading tasks to edge devices. However, the limited computing power of existing edge devices makes it difficult to complete the low-latency fusion and simultaneous high-complexity tasks of multi-channel video in real time. This often requires compromising performance by reducing resolution or frame rate, resulting in reduced detection accuracy. Currently, there is no efficient solution for video stitching.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0005] The present disclosure provides a video stitching method, system and related equipment, which greatly reduces the energy consumption of the entire device, improves video stitching efficiency and resource utilization, and further reduces the dependence on the CPU and GPU. It better supports the need to output frame data to storage media outside the GPU in actual application scenarios, and can meet more application scenarios.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0007] According to one aspect of the present disclosure, a video splicing method is provided, which is applied to a terminal device, wherein the terminal device includes a central processing unit and a neural network processor, and the video stream decoding method is executed by the central processing unit. The video stream splicing method includes: obtaining multiple video streams input by a monitoring device, and decoding the multiple video streams to obtain multiple decoded video data of the multiple video streams; performing feature extraction on the multiple decoded video data by the neural network processor to obtain multiple video feature information of the multiple video streams; performing feature matching on the multiple video feature information by the neural network processor to obtain multiple feature matching results of the multiple video streams; when performing feature matching on the multiple video feature information to obtain multiple feature matching results of the multiple video streams, erroneous matching points in the multiple feature matching results are eliminated in parallel; and aligning multiple video frames by the neural network processor according to valid feature matching points in the multiple feature matching results to generate target video splicing data, wherein the valid feature matching points represent correct matching points in the multiple feature matching results excluding the erroneous matching points.

[0008] In some embodiments, the terminal device includes a graphics processor and a memory, and the decoding processing of the multiple video streams to obtain target decoded video data of the multiple video streams includes: creating multiple decoding threads in a thread pool corresponding to the graphics processor, and allocating video memory space to each decoding thread; allocating a decoding thread in a one-to-one correspondence among the multiple decoding threads for each video stream in the multiple video streams; writing the data of the target video stream into the video memory space corresponding to the target decoding thread through video memory pass-through technology, wherein the target video stream is a video stream in the multiple video streams corresponding to the target decoding thread; the graphics processor performs parallel decoding processing on the multiple video streams in the video memory space of the graphics processor to generate multiple decoded video data; and outputs the multiple decoded video data to the memory through the graphics processor.

[0009] In some embodiments, before decoding the multiple video streams to obtain multiple decoded video data of the multiple video streams, the method further includes: obtaining timing information and resource requirement information of each video stream in the multiple video streams; determining timing synchronization requirement information based on the timing information of each video stream; allocating dynamic priority weights to each video stream based on the timing synchronization requirement information and resource requirement information; and determining the input order of each video stream into the memory based on the dynamic priority weights of each video stream.

[0010] In some embodiments, the neural network processor is deployed with a lightweight feature extraction model, and the neural network processor is used to perform feature extraction on the multi-channel decoded video data to obtain multi-channel video feature information of the multi-channel video streams, including: performing feature extraction on the multi-channel decoded video data through the lightweight feature extraction model on the neural network processor to obtain multi-channel video feature information of the multi-channel video streams, wherein the lightweight feature extraction model is obtained by lightweight processing of a trained neural network model.

[0011] In some embodiments, the neural network processor has a data stream scheduler, and the neural network processor is used to perform feature matching on the multi-channel video feature information to obtain multi-channel feature matching results of the multi-channel video streams, including: creating multiple subtasks in the neural network processor based on the multi-channel video feature information using the data stream scheduler; allocating a respective computing unit to each of the multiple subtasks; using the computing unit corresponding to each subtask as an execution unit, reading and writing the multi-channel video feature information in the shared memory in parallel to perform feature matching to obtain feature matching results corresponding to each subtask, and storing the feature matching results in the shared memory; when performing feature matching on the multi-channel video feature information to obtain multi-channel feature matching results of the multi-channel video streams, incorrect matching points in the multi-channel feature matching results are eliminated in parallel, including: when performing feature matching on the multi-channel video feature information in the shared memory in parallel to obtain feature matching results corresponding to each subtask, based on a multimodal incorrect matching elimination strategy, incorrect matching points in the feature matching results are eliminated in parallel by reading and writing the feature matching results in the shared memory.

[0012] In some embodiments, aligning multiple video frames according to valid feature matching points in the multiple feature matching results to generate target video stitching data includes: calculating the transformation parameters between the multiple video frames through a geometric transformation model according to the valid feature matching points in the multiple feature matching results; and fusing the multiple video frames according to the transformation parameters between the multiple video frames to generate target video stitching data.

[0013] In some embodiments, the method further includes: obtaining resource occupancy information of the neural network processor and matching resource requirement information and matching error threshold of the multi-channel decoded video data; determining feature matching parameters and geometric transformation model parameters based on the resource occupancy information of the neural network processor and matching resource requirement information and matching error threshold of the multi-channel decoded video data.

[0014] In some embodiments, the method further includes: obtaining load information of the image processor; adjusting the operating voltage and operating frequency of the image processor according to the load information of the image processor; and / or obtaining load information of the neural network processor; adjusting the operating voltage and operating frequency of the neural network processor according to the load information of the neural network processor.

[0015] According to another aspect of the present disclosure, a video splicing system is also provided, wherein the terminal device includes a central processing unit and a neural network processor, including: the central processing unit is configured to obtain multiple video streams input by a monitoring device, and decode the multiple video streams to obtain multiple decoded video data of the multiple video streams; the central processing unit is configured to perform feature extraction on the multiple decoded video data through the neural network processor to obtain multiple video feature information of the multiple video streams; the central processing unit is configured to perform feature matching on the multiple video feature information through the neural network processor to obtain multiple feature matching results of the multiple video streams; the central processing unit is configured to, when performing feature matching on the multiple video feature information to obtain multiple feature matching results of the multiple video streams, simultaneously eliminate erroneous matching points in the multiple feature matching results; the central processing unit is configured to align multiple video frames according to valid feature matching points in the multiple feature matching results through the neural network processor to generate target video splicing data, wherein the valid feature matching points represent correct matching points in the multiple feature matching results after eliminating the erroneous matching points.

[0016] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-described video stitching methods by executing the executable instructions.

[0017] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the video stitching method described above is implemented.

[0018] According to another aspect of the present disclosure, a computer program product is further provided, including: a computer program or instructions, wherein when the computer program or instructions are executed by a processor, any one of the above-mentioned video stitching methods is implemented.

[0019] The video splicing method, system and related equipment provided in the embodiments of the present disclosure are applied to a terminal device, which includes a central processing unit and a neural network processor. The video stream decoding method is executed by the central processing unit, and the video stream splicing method includes: obtaining multiple video streams input by a monitoring device, and decoding the multiple video streams to obtain multiple decoded video data of the multiple video streams; using the neural network processor, performing feature extraction on the multiple decoded video data to obtain multiple video feature information of the multiple video streams; using the neural network processor, performing feature matching on the multiple video feature information to obtain multiple feature matching results of the multiple video streams; when performing feature matching on the multiple video feature information to obtain multiple feature matching results of the multiple video streams, erroneous matching points in the multiple feature matching results are eliminated in parallel; using the neural network processor, aligning the multiple video frames according to the valid feature matching points in the multiple feature matching results to generate target video splicing data, wherein the valid feature matching points represent the correct matching points in the multiple feature matching results after eliminating the erroneous matching points. In this way, the neural network processor replaces the CPU or GPU to process computationally intensive tasks (such as feature matching), reducing the long-term high-load operation of the CPU or GPU and reducing overall power consumption; at the same time, the CPU eliminates incorrect matching points in parallel, accelerates the stitching process, and improves video stitching efficiency and resource utilization.

[0020] Furthermore, the dependence on the CPU and GPU is reduced, and the actual application scenarios that require outputting frame data to storage media outside the GPU are better supported, which can meet more application scenarios.

[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0023] Figure 1 A schematic diagram showing an application system architecture of a video splicing method according to an embodiment of the present disclosure is shown;

[0024] Figure 2 A flow chart of a video splicing method according to an embodiment of the present disclosure is shown;

[0025] Figure 3 A flow chart of a method for performing feature matching in an embodiment of the present disclosure is shown;

[0026] Figure 4A flow chart of a method for adjusting calculation accuracy according to an embodiment of the present disclosure is shown;

[0027] Figure 5 A flowchart of a decoding method according to an embodiment of the present disclosure is shown;

[0028] Figure 6 A flow chart of a method for controlling power consumption according to an embodiment of the present disclosure is shown;

[0029] Figure 7 A flow chart of another video splicing method according to an embodiment of the present disclosure is shown;

[0030] Figure 8 A schematic diagram of a video splicing system according to an embodiment of the present disclosure is shown;

[0031] Figure 9 A structural block diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0032] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0033] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor systems and / or microcontroller systems.

[0034] For ease of understanding, before introducing the embodiments of the present disclosure, several terms involved in the embodiments of the present disclosure are explained as follows:

[0035] The Central Processing Unit (CPU) is the core component of a computer, responsible for executing instructions and processing data. It controls the operation of the entire system by reading instructions, decoding them, and performing calculations.

[0036] A graphics processing unit (GPU) is an electronic chip designed for parallel computing, used to accelerate graphics rendering. Compared to a central processing unit (CPU), a GPU has more simple computing cores and can handle a large number of simple tasks simultaneously, making it suitable for batch operations on data such as image pixels and 3D model vertices. Because of its expertise in parallel computing, GPUs are widely used in fields such as deep learning and scientific computing.

[0037] A neural processing unit (NPU) is a chip designed specifically to accelerate artificial intelligence (AI), particularly neural network computing. Unlike general-purpose CPUs or GPUs, NPUs optimize core operations like matrix multiplication and convolution through hardware circuitry, significantly improving the efficiency of AI tasks.

[0038] An embedded system is a specialized computer system that tightly integrates hardware and software, embedded within a specific device to perform a desired control or data processing function. It is typically designed to perform one or more specific tasks. These systems typically include a microprocessor, memory, input / output devices, and are often embedded within other devices, such as smartphones, cars, and smart home appliances.

[0039] Heterogeneous computing refers to the technology that combines different types of computing units (such as CPUs, GPUs, and NPUs) to collaboratively complete computing tasks. Specifically, different types of computing units utilize different instruction set architectures (ISAs) and execute in parallel to improve overall system performance.

[0040] Edge computing is a distributed computing paradigm that moves data processing and storage from central servers to edge devices close to the data source. Its core goals are to reduce data transmission latency, lower bandwidth consumption, and improve real-time responsiveness.

[0041] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0042] Figure 1 FIG. 1 shows an exemplary application system architecture diagram to which the video splicing method according to the embodiment of the present disclosure can be applied. Figure 1 As shown, the system architecture may include a terminal device 101 , a network 102 and a monitoring device 103 .

[0043] The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103 , and can be a wired network or a wireless network.

[0044] Optionally, the above-mentioned wireless network or wired network uses standard communication technologies and / or protocols. The network is typically the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of a virtual private network). In some embodiments, technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged over the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPSec) can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0045] The terminal device 101 is integrated with a video splicing system and may include a central processing unit 1011, a graphics processing unit 1012, a neural network processor 1013, and a memory 1014. In the embodiment of the present disclosure, the terminal device 101 is mainly used to obtain multiple video streams input by the monitoring device through the central processing unit 1011, and decode the multiple video streams to obtain multiple decoded video data of the multiple video streams; perform feature extraction on the multiple decoded video data through the neural network processor to obtain multiple video feature information of the multiple video streams; perform feature matching on the multiple video feature information through the neural network processor to obtain multiple feature matching results of the multiple video streams; when performing feature matching on the multiple video feature information to obtain multiple feature matching results of the multiple video streams, erroneous matching points in the multiple feature matching results are eliminated in parallel; and align the multiple video frames through the neural network processor according to valid feature matching points in the multiple feature matching results to generate target video splicing data, wherein the valid feature matching points represent correct matching points in the multiple feature matching results after eliminating the erroneous matching points.

[0046] In this embodiment, terminal device 101 achieves optimally coordinated technology allocation by integrating multiple hardware components, including a CPU, NPU, and GPU. This disclosed embodiment utilizes a RISC-based CPU as the scheduling core, dynamically allocating tasks to the GPU for image processing and the NPU for neural network computing based on hardware characteristics and task requirements. This achieves efficient resource utilization and optimal task coordination, thereby improving overall system performance and energy efficiency.

[0047] In addition, if Figure 1 As shown, the memory in the terminal device 102 is used to store data, such as monitoring video data, decoded video data or spliced ​​video data, so that it can be viewed later when needed. The memory can include at least one of system memory, shared memory and disk.

[0048] System memory is a synonym for RAM, short for random-access memory. It's the primary area where the system temporarily stores program instructions and data. Each location in RAM is identified by a number called a memory address. Any data stored in RAM is lost when the terminal device system is shut down. System memory consists of physical memory and virtual memory. When physical memory is insufficient during operation, the system creates virtual memory, which is typically 1.5 to 3 times the size of physical memory.

[0049] A disk is a memory device that uses magnetic recording technology to store data. Disks are the primary storage medium for computers, capable of storing large amounts of binary data and maintaining data integrity even after power outages. Early computers used floppy disks, while today's hard disks are commonly used. The disk referred to in this disclosure can be either a floppy disk or a hard disk.

[0050] Shared memory refers to a large amount of memory in a terminal device that can be accessed by different central processing units (CPUs). Since the CPU needs to access the disk quickly, the disk must be cached. Once any cached data is updated, other processors may also need to access it, so the shared memory needs to be updated immediately. Otherwise, different processors may use different data.

[0051] In some embodiments, a unified memory architecture is used to enable CPU / NPU / GPU shared data buffers, reducing the number of cross-device data copies and effectively lowering memory usage. This design maximizes hardware resource utilization and ensures manageable end-to-end processing latency.

[0052] The terminal device 101 can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, wearable devices, augmented reality devices, virtual reality devices, etc.

[0053] Optionally, the client of the application installed in different terminal devices 101 is the same, or the client of the same type of application based on different operating systems. Based on the different terminal platforms, the specific form of the client of the application can also be different, for example, the application client can be a mobile phone client, a PC client, etc.

[0054] The monitoring device 103 is used to shoot monitoring video streams and perform video splicing, and transmit them to the terminal device 101 for video splicing. Each monitoring device can correspond to one video stream, and the monitoring device 103 can be a monitoring camera.

[0055] Those skilled in the art will know that Figure 1 The number of terminal devices, networks, and monitoring devices in the embodiment is merely illustrative, and any number of terminal devices, networks, and monitoring devices may be provided according to actual needs. This embodiment of the present disclosure does not limit this.

[0056] Under the above system architecture, an embodiment of the present disclosure provides a video splicing method, which can be executed by any electronic device with computing and processing capabilities.

[0057] Figure 2 A flow chart of a video splicing method according to an embodiment of the present disclosure is shown as follows: Figure 2 As shown, the video stitching method provided in the embodiment of the present disclosure includes the following steps:

[0058] S202: Acquire multiple video streams input by the monitoring device, and decode the multiple video streams to obtain multiple decoded video data of the multiple video streams.

[0059] In this embodiment, the monitoring device may refer to Figure 1 The monitoring device 103 shown in the figure can monitor and shoot video images. Specifically, the monitoring device can be a monitoring camera, wherein the monitoring device can be one monitoring device or multiple monitoring devices, each monitoring device corresponding to the transmission of one video stream, and the monitoring device can directly transmit the captured video to the terminal device in the form of a video file in the form of a video stream.

[0060] In some examples, the terminal device can establish a connection with the monitoring device through a wire or wireless, etc., to receive the video stream captured by the monitoring device. The terminal device can obtain the video stream from the monitoring device through an Rtsp protocol, a GB / T 28181 protocol, or a third-party software development kit (SDK), etc.

[0061] In some embodiments, the video images captured by the monitoring device can be encapsulated into different formats to form a video stream transmitted to the terminal device, for example, a multimedia container (MKV) format or a flash video (FLV) format. It can be understood that other video formats such as an mp4 format, an rmvb format, etc. can also be used, which are not limited herein.

[0062] It should be noted that obtaining the video stream data input by the monitoring device can include: obtaining the original video stream input by the monitoring device, and obtaining the video stream data input by the monitoring device after the original video stream is parsed and processed. At this time, the video stream data input by the monitoring device is stored in the form of a data packet.

[0063] The decoding process restores the video stream data input by the monitoring device into a playable original frame sequence. The multi-path decoding video data is a set of independent video data obtained after each path of video stream is decoded. Each path of data contains corresponding frame sequence, resolution, frame rate, etc. It should be noted that the decoded video stream data is usually stored in the form of image frames.

[0064] S204, obtaining multi-path video feature information of the multi-path video stream by performing feature extraction on the multi-path decoding video data through the neural network processor.

[0065] In this embodiment, the neural network processor (NPU) is a special chip specially designed for accelerating artificial intelligence. Unlike general-purpose CPUs, NPU hardwires matrix operations, convolution, and other high-frequency operations in neural networks, greatly improving the energy efficiency and real-time performance of image recognition. In this embodiment, the terminal device includes an NPU. Since the NPU has higher energy efficiency in neural network inference, in the present embodiment, the video stream is transmitted to the NPU, which processes the feature extraction of the video stream. This can greatly reduce the burden on the CPU and GPU and improve the work efficiency of feature extraction.

[0066] It should be noted that the NPU is a separate hardware module specially designed for neural network calculation, which has a special computing unit (such as a matrix multiplication engine), on-chip cache and a customized instruction set. Its architecture is optimized for parallel matrix operations, and can directly process data streams without CPU, achieving efficient computation task execution. It can be understood that although the NPU has hardware independence, its task starting and resource allocation still accept the scheduling instructions of the CPU. Specifically, the CPU on the terminal device is responsible for compiling the neural network model, loading the weight data, and issuing the calculation task to the NPU through a special interface. For example: the CPU transmits the model file and input data into the NPU memory; the NPU receives the instruction and executes the calculation autonomously without real-time intervention of the CPU. That is, the neural network processor in this embodiment extracts features from the multi-channel decoded video data, which is essentially a manifestation of the cooperative architecture with the CPU.

[0067] In some embodiments, the neural network processor is deployed with a lightweight feature extraction model, and the multi-channel decoded video data is extracted by the neural network processor to obtain multi-channel video feature information of the multi-channel video stream, including: extracting features from the multi-channel decoded video data by the lightweight feature extraction model on the neural network processor to obtain multi-channel video feature information of the multi-channel video stream, wherein the lightweight feature extraction model is obtained by lightweight processing of the trained neural network model.

[0068] In this embodiment, the traditional feature extraction uses the Scale-Invariant Feature Transform (SIFT) algorithm. However, as a classic feature extraction algorithm, the SIFT algorithm has the limitations of high computational complexity and insufficient real-time performance. Therefore, the present embodiment proposes to deploy a lightweight feature extraction model on a neural network processor (NPU) to replace the SIFT algorithm. The lightweight feature extraction model has the advantages of small parameter quantity and low computational complexity, and is suitable for resource-constrained devices. At the same time, with the high parallel computing capability of the NPU, such models can be efficiently executed to complete the feature extraction task.

[0069] In some embodiments, the deployment of a lightweight feature extraction model on a neural network processor (NPU) can be achieved through the following steps: first, the models of other frameworks are converted to the Open Neural Network Exchange (ONNX) format to achieve cross-platform compatibility and facilitate subsequent optimization and deployment. Then, the amount of computation in the feature matching process is reduced by model pruning and model quantization. Among them, model pruning refers to reducing the number of model parameters, reducing computational complexity, and improving reasoning speed by removing redundant neurons or layers. Model quantization is the conversion of model parameters from high precision to low precision, for example, reducing the precision from 0.2245 to 0.2 to accelerate reasoning. In actual application, the minimum number of bits of accuracy can be set according to actual conditions to reduce memory usage and computational overhead. Finally, the optimized feature extraction model is integrated into the NPU, and the feature extraction task is efficiently performed through the lightweight feature extraction model.

[0070] In this embodiment, a lightweight feature extraction model is deployed on the NPU, and the NPU is used to accelerate the calculation process of feature extraction, replacing the traditional SIFT algorithm, which can improve the processing speed and reduce the computing overhead.

[0071] S206, performing feature matching on the multi-channel video feature information through a neural network processor to obtain multi-channel feature matching results of the multi-channel video streams.

[0072] In this embodiment, multi-channel video feature information refers to key features extracted from multiple video streams. Feature matching compares the feature data of different video streams to identify similarities. The NPU leverages parallel computing capabilities to rapidly complete large-scale feature comparisons, significantly improving efficiency. The multi-channel feature matching results, representing the feature correlation information between the video streams output by the NPU, are used for splicing multi-channel surveillance.

[0073] Specifically, the neural network processor performs similarity calculations on feature points extracted from multiple video streams, matching identical or similar content across different video streams. This similarity calculation compares feature descriptors across different video streams using a distance metric (such as Euclidean distance or cosine similarity). Smaller / larger values ​​(depending on the metric) indicate higher similarity. Furthermore, a threshold is set, and if exceeded, feature points are considered matched, thereby linking identical or similar content.

[0074] S208 , when performing feature matching on the feature information of multiple channels of video to obtain the multiple channel feature matching results of the multiple channel video streams, simultaneously eliminate the mismatching points in the multiple channel feature matching results.

[0075] In the embodiment, the terminal device utilizes the parallel computing capability of the neural network processing unit to process the feature matching task of the multi-path video stream. The CPU in the terminal device utilizes an algorithm (such as RANSAC or geometric consistency verification) to detect and remove the incorrect matching points in each path of the video stream in parallel, thereby improving the matching accuracy. The NPU and the CPU can directly interact data through shared memory, and this cooperative mode can give full play to the advantages of the NPU in parallel computing, and combine the ability of the CPU in logical control, thereby avoiding the performance bottleneck caused by frequent data exchange between the GPU and the CPU in the traditional scheme, and improving the overall processing efficiency.

[0076] In S210, the multi-path video frames are aligned according to the effective feature matching points in the multi-path feature matching result by the neural network processor, and target video stitching data is generated. The effective feature matching points represent the correct matching points in the multi-path feature matching result after the incorrect matching points are removed.

[0077] In some embodiments, the multi-path video frames are aligned according to the effective feature matching points in the multi-path feature matching result to generate the target video stitching data, specifically including: calculating the transformation parameters between the multi-path video frames by a geometric transformation model according to the effective feature matching points in the multi-path feature matching result; and performing multi-path video frame fusion according to the transformation parameters between the multi-path video frames to generate the target video stitching data.

[0078] In the embodiment, the NPU utilizes the effective feature matching points in the multi-path feature matching result to calculate the transformation parameters between the multi-path video frames by matrix decomposition through the geometric transformation model. The geometric transformation model is used to describe the spatial mapping relationship between the video frames, and the selection of the geometric transformation model depends on the scene characteristics. The geometric transformation model provided by the present disclosure can include: homography, affine transformation, projective transformation, and Euclidean transformation. It can be understood that the above-mentioned geometric transformation models are only examples, and other geometric transformation models can also be applied in actual application process, and the present disclosure does not limit this.

[0079] Exemplarily, the homography matrix is ​​used as an example for detailed explanation. After establishing the effective matching feature points of multiple frames of video through the lightweight feature matching model, it is necessary to use these effective matching feature points to calculate the homography matrix as the inter-frame transformation parameter. The homography matrix H is a 3×3 projection transformation matrix that can describe the perspective mapping relationship between two images in a planar scene (including rotation, translation, scaling and projection distortion). Since the number of effective matching feature point pairs provided by the lightweight matching model is usually far more than 4 pairs, the core of the solution process is to find an optimal H matrix that can best fit the vast majority of correct matching points.

[0080] In some embodiments, the graphics processing unit (GPU) of the terminal device fuses the video frames based on these transformation parameters to generate the final target video stitching data to ensure visual continuity and geometric consistency. Specifically, the geometric differences of the images can be aligned through a multi-band fusion algorithm, and temporal anti-aliasing can be used to eliminate stitching boundary artifacts. After optimizing the graphics processing unit (GPU) access mode, the memory bandwidth utilization rate reaches 95%, supporting 100FPS continuous rendering output. In this way, in the image fusion stage, the multi-band fusion algorithm is used to eliminate stitching artifacts and optimize the image stitching effect. Combined with memory bandwidth optimization and efficient rendering, high frame rate rendering output can be achieved.

[0081] In this embodiment, a neural network processor is used to replace the CPU or GPU to process computationally intensive tasks (such as feature matching), thereby reducing the long-term high-load operation of the CPU or GPU and reducing overall power consumption; at the same time, the CPU eliminates mismatching points in parallel, accelerates the stitching process, and improves video stitching efficiency and resource utilization.

[0082] In some embodiments, considering that heterogeneous solutions rely on closed-source hardware and are difficult to deeply optimize for open architectures, the terminal device in this disclosure implements dynamic resource scheduling through a RISC architecture CPU, allocates GPU and NPU loads in real time, and avoids resource contention; at the same time, the terminal device optimizes overall energy efficiency and reduces energy consumption of multi-channel video processing through memory sharing and low-power instruction set design.

[0083] It should be noted that there are differences in the architectures of different CPUs. According to the instruction set type, CPUs can be divided into two categories: reduced instruction set computing (RISC) and complex instruction set computing (CISC). The instruction set architecture determines the core processing mechanism of the CPU: CISC usually adopts a closed-source architecture, while RISC adopts an open-source architecture. The RISC architecture is particularly suitable for heterogeneous computing scenarios. Dynamic resource scheduling is achieved through the RISC architecture CPU, and the GPU and NPU loads are allocated in real time, thereby effectively avoiding resource contention problems. In view of this, the embodiment of the present disclosure collaborates with the RISC architecture CPU, the graphics processor, and the neural network processor to complete efficient decoding, resource scheduling, and video splicing of multiple video streams.

[0084] In some embodiments, Figure 3 A flow chart of a method for feature matching provided in an embodiment of the present disclosure, combined with Figure 3 As shown, the neural network processor has a data stream scheduler, and through the neural network processor, feature matching is performed on multi-channel video feature information to obtain multi-channel feature matching results of the multi-channel video streams, including:

[0085] S302, creating multiple subtasks in the neural network processor using a data stream scheduler based on multi-channel video feature information.

[0086] In this embodiment, multi-channel video feature information is often stored and processed in matrix form. The NPU uses a data flow scheduler to automatically identify matrix operation steps and split them into parallelizable subtasks (such as local feature matching and global geometry verification). Simultaneously, the data flow scheduler orchestrates the data flow based on subtask dependencies and generates an execution flow chart.

[0087] S304: Allocate a computing unit to each of the multiple subtasks.

[0088] In this embodiment, the data flow scheduler analyzes the sequence and resource requirements of each subtask based on the execution flow chart, and automatically matches each subtask to a suitable computing unit to achieve efficient resource allocation.

[0089] S306 , using the computing units corresponding to the respective subtasks as execution units, reading and writing the multi-channel video feature information in the shared memory in parallel to perform feature matching to obtain feature matching results corresponding to the respective subtasks, and storing the feature matching results in the shared memory.

[0090] In this embodiment, the NPU uses hardware acceleration units to perform feature matching operations in parallel. Each computing unit independently processes the assigned video stream feature data, applies a similarity algorithm to calculate the matching score, and efficiently writes the generated matching results into shared memory for direct call by subsequent tasks such as target tracking or behavior analysis. In this way, the computing speed is significantly improved through parallel processing, and the efficiency of video splicing is improved. In some embodiments, the NPU relies on hardware acceleration units to parallelize the key matrix operations in feature matching, and combines a dedicated data stream scheduler to shorten the feature matching time to 5ms / frame.

[0091] In this embodiment, the NPU solves the computational bottleneck of video feature matching through parallelization, and the NPU hardware acceleration unit directly matches the matrix operation requirements. In the feature matching stage, it is necessary to calculate the similarity between feature points, which usually involves a large number of matrix operations. The NPU identifies these key matrix operations and divides them into multiple subtasks that can be processed in parallel. The NPU hardware acceleration unit performs parallel processing on these subtasks. Through multi-core parallel computing, each subtask is executed simultaneously on an independent computing unit, thereby greatly improving computing efficiency.

[0092] S308, when performing feature matching on the multi-channel video feature information in the shared memory to obtain the feature matching results corresponding to each subtask, based on the multimodal false matching elimination strategy, the false matching points in the feature matching results are eliminated in parallel by reading and writing the feature matching results in the shared memory.

[0093] In this embodiment, the NPU is responsible for accelerating the key matrix operations in feature matching and significantly shortening the feature matching time through parallel processing. At the same time, while the NPU is performing feature matching in parallel, the CPU synchronously monitors and optimizes the matching results in real time based on dynamic thresholds or other false matching point elimination methods to eliminate false matching points. There is a data interaction and coordination mechanism between the NPU and the CPU to achieve the macroscopic synchronization of feature matching and false matching point elimination. The NPU provides powerful computing power to support feature matching, while the CPU uses its powerful logic control and optimization capabilities to ensure the accuracy and efficiency of the matching results.

[0094] It should be noted that the execution order of the CPU and NPU is still sequential. First, the NPU executes the first feature matching task, the second feature matching task, and the third feature matching task in sequence. Then the CPU also needs to execute the first false matching point elimination task, the second false matching point elimination task, and the third false matching point elimination task. However, within a period of time, it can be understood that when the NPU executes the second feature matching task, the CPU can execute the first false matching point elimination task. The data operated by the first false matching point elimination task is the feature matching result of the first feature matching task. Similarly, the tasks are not completely synchronized, but are processed at the same time, that is, the CPU eliminates false matching points in the feature matching results in parallel.

[0095] In some embodiments, the multimodal mismatch elimination strategy may include: RANSAC iterative optimization based on dynamic threshold: controlling the iterative process of the RANSAC algorithm by dynamically adjusting the threshold to eliminate mismatch points; cross-validation method: ensuring the stability and accuracy of the matching by multiple verifications of the matching feature points, thereby effectively eliminating unstable mismatch points; geometric constraint method: using the geometric relationship between images (such as perspective transformation, affine transformation, etc.) to impose constraints on the matching feature points, and eliminating mismatch points that do not meet the geometric constraint conditions; distance threshold method: setting a distance threshold between matching feature points, and eliminating mismatch points whose distance exceeds the threshold. By utilizing one or more of the above mismatch elimination strategies, the accuracy of feature matching is improved.

[0096] In this embodiment, feature matching is optimized through a collaborative architecture between the NPU and CPU. The NPU inherits the feature points detected in the previous step, accelerating parallel matrix operations and reducing feature matching time by 5ms per frame. Simultaneously, the CPU uses dynamic thresholding to control RANSAC iterations to eliminate mismatched points. Furthermore, shared memory avoids the bottleneck of frequent data exchange between different hardware, significantly improving overall processing efficiency.

[0097] In some embodiments, considering that in actual applications, high precision is often the optimal solution for tasks in each link of the video stitching process, high precision means that there will be "computational waste" or "computational surplus" in the process of video stitching. However, for edge devices, the computing power of edge devices is limited, so it is necessary to balance accuracy and efficiency. Therefore, in order to further improve the efficiency of video stitching, dynamic precision switching technology can be used to flexibly switch between different precision modes according to actual operating requirements. Among them, dynamic precision switching technology means that during the video stitching process, the system can automatically adjust the processing accuracy (such as switching from high precision to low precision) according to real-time computing requirements.

[0098] Figure 4 A flow chart of a method for adjusting calculation accuracy provided by an embodiment of the present disclosure. Figure 4 As shown, the video stitching method provided by the embodiment of the present disclosure also includes:

[0099] S402, obtaining resource occupancy information of the neural network processor and matching resource requirement information and matching error threshold of the multi-channel decoded video data.

[0100] In this embodiment, the neural network processor resource occupancy information refers to the usage of the current computing resources (such as computing power, memory, and bandwidth) of the NPU, reflecting its load status. The matching resource requirement information of multi-channel decoded video data refers to the hardware resources required for matching multi-channel decoded videos. The matching resource requirement information can be obtained by evaluating parameters such as the resolution and frame rate of the multi-channel decoded video data. The matching resource requirement information directly affects the computing pressure of the NPU. The matching error threshold refers to the deviation range between the allowed resource demand and the actual supply. Exceeding the threshold may cause decoding jamming or performance degradation.

[0101] S404 , determining feature matching parameters and geometric transformation model parameters according to the resource occupancy information of the neural network processor, the matching resource requirement information of the multi-channel decoded video data, and the matching error threshold.

[0102] In this embodiment, during the feature matching phase, if NPU resources are sufficient and high computational accuracy is required, a high-precision mode can be selected to ensure feature matching accuracy. If NPU resources are limited or computational speed is required, a lower-precision mode can be selected to speed up matching. Specifically, key features (such as corners and edges) of adjacent video frames are first extracted, and corresponding feature points are matched using high- and low-precision modes to determine the motion relationship between frames.

[0103] In the matrix decomposition stage, the NPU also selects the appropriate precision mode based on the demand assessment results to balance the calculation accuracy and efficiency. Specifically, based on the matched feature points, a transformation matrix (such as a homography matrix) is constructed, and the optimal parameters are solved through matrix decomposition (such as SVD), and the inter-frame transformation relationship (such as rotation and translation parameters) is output for subsequent image alignment and fusion. The output of the matrix decomposition is a parameter matrix that describes the geometric transformation between frames, which is directly used for image correction and synthesis during splicing. Exemplarily, after selecting the precision mode, the system achieves precision switching by adjusting the internal parameters of the NPU (neural processing unit) and CPU. In high-precision mode, the system can increase the number of iterations in the calculation process, improve the data quantization accuracy, etc. to ensure the accuracy of the calculation results; in low-precision mode, the number of iterations can be reduced, the data quantization accuracy can be reduced, etc. to speed up the calculation.

[0104] In this embodiment, by monitoring the task load in real time (such as inter-frame differences and computing power usage), flexible switching between precision levels is achieved—using high precision to maintain results under high load and switching to low precision to save resources under low load, thereby balancing speed and quality. In this way, through dynamic precision switching technology, the calculation precision can be flexibly adjusted during feature matching and matrix decomposition, significantly improving computational efficiency. At the same time, extremely high precision is maintained to ensure the accuracy of the calculation results.

[0105] In some embodiments, the terminal device includes a graphics processor, Figure 5 A flowchart of a decoding method provided by an embodiment of the present disclosure, combined with Figure 5 As shown, the decoding process of multiple video streams provided in this embodiment to obtain target decoded video data of the multiple video streams may include the following steps:

[0106] S502: Create multiple decoding threads in a thread pool corresponding to the graphics processor, and allocate video memory space to each decoding thread.

[0107] In some embodiments, before the GPU decodes multiple video streams to obtain multiple decoded video data of the multiple video streams, the method also includes: the CPU obtains timing information and resource requirement information of each video stream in the multiple video streams; the CPU determines timing synchronization requirement information based on the timing information of each video stream; the CPU assigns dynamic priority weights to each video stream based on the timing synchronization requirement information and resource requirement information; the CPU determines the input order of each video stream into the graphics processor based on the dynamic priority weights of each video stream.

[0108] In this embodiment, the CPU serves as the scheduling core and coordinates the timing alignment of multiple input video streams through a dynamic priority allocation strategy to avoid input blocking caused by resource contention.

[0109] S504 , allocating a decoding thread from the multiple decoding threads to each of the multiple video streams.

[0110] S506 , writing the data of the target video stream into the video memory space corresponding to the target decoding thread through the video memory pass-through technology, wherein the target video stream is a video stream corresponding to the target decoding thread among the multiple video streams.

[0111] In this embodiment, memory pass-through technology refers to bypassing the CPU and system memory, allowing the GPU to directly access the memory or memory of other GPUs or devices (such as storage, network card), reducing data transmission latency. This embodiment bypasses the CPU memory copy through memory pass-through technology, reduces data transmission latency, and improves decoding efficiency. It should be noted that the target video stream can be stored in the shared memory of the terminal device, which can be accessed by the CPU, NPU, and GPU to achieve efficient data sharing.

[0112] S508 , the graphics processor performs parallel decoding processing on the multiple video streams in the graphics memory space of the graphics processor to generate multiple decoded video data.

[0113] In this embodiment, the GPU hardware decoder is based on a dedicated video codec unit and performs parallel decoding in low-latency mode. The GPU hardware decoder is the portion of the graphics processing unit dedicated to video decoding. The dedicated video codec unit is its core hardware foundation, optimized for video codec algorithms and capable of efficiently processing video data. The GPU can process multiple video frames or data blocks simultaneously, enabling parallel decoding and significantly increasing decoding speed. Low-latency mode is designed to meet the needs of real-time applications by minimizing the time from receiving video data to completing decoded output.

[0114] In this embodiment, through hardware-level collaborative design, a stable, low-jitter video frame sequence input is provided for subsequent processing.

[0115] In some embodiments, the CPU acts as a global controller, automatically adjusting the operating voltage and frequency of the NPU and GPU based on the processor load using dynamic voltage and frequency scaling (DVFS). This allows for precise control of power consumption, ensuring that the system maintains a stable, low power consumption level under varying loads, and achieving energy savings. DVFS is a commonly used power management technology that allows the processor to dynamically adjust its operating voltage and frequency based on the current workload. Under light loads, power consumption can be reduced by lowering the voltage and frequency; under heavy loads, the voltage and frequency are increased to ensure processing performance. As a global controller, the CPU continuously monitors system load conditions. This includes key metrics such as the task queue lengths and compute unit utilization of the NPU and GPU. By analyzing these metrics, the CPU can assess the system's current load status and future load trends in real time. Based on the system load monitoring results, the CPU formulates voltage and frequency adjustment strategies for the NPU and GPU.

[0116] Figure 6 A flow chart of a method for controlling power consumption provided by an embodiment of the present disclosure. Figure 6 As shown, the video stitching method provided by the embodiment of the present disclosure further includes the following steps:

[0117] S602: Obtain image processor load information.

[0118] In this embodiment, the CPU acts as a global controller and continuously monitors the load information of the graphics processor, such as the task queue length of the GPU, the computing unit utilization rate, and other indicators.

[0119] S604: Adjust the operating voltage and operating frequency of the image processor according to the load information of the image processor.

[0120] In this embodiment, the CPU analyzes the GPU load information to assess the system's current load status and future load trends in real time. Based on the CPU load monitoring results, the CPU formulates a voltage and frequency adjustment strategy for the GPU.

[0121] For example, if the CPU detects a long GPU task queue and high compute unit utilization, it determines that the GPU is under heavy load and will increase the GPU's operating voltage and frequency to enhance its computing power. Conversely, if the GPU's task queue is short and compute unit utilization is low, the CPU will reduce the GPU's operating voltage and frequency to reduce unnecessary power consumption.

[0122] For example, suppose that at a certain moment, the CPU detects that the GPU's task queue suddenly increases and the computing unit utilization rises rapidly, indicating that the GPU is about to face heavy load. At this time, the CPU increases the GPU's operating voltage from 1.0V to 1.2V and the operating frequency from 800MHz to 1.0GHz according to the preset voltage and frequency adjustment strategy. In this way, the GPU can operate at a higher voltage and frequency to cope with the upcoming heavy-load tasks. On the contrary, if the CPU detects that the GPU's task queue gradually decreases and the computing unit utilization decreases, it indicates that the GPU is currently in a light-load state. At this time, the CPU reduces the GPU's operating voltage from 1.1V to 0.9V and the operating frequency from 600MHz to 400MHz. In this way, the GPU can reduce power consumption to save energy while ensuring basic performance.

[0123] and / or,

[0124] S606, obtaining load information of the neural network processor.

[0125] In this embodiment, the CPU acts as a global controller and continuously monitors the load information of the neural network processor, such as the task queue length of the NPU, the computing unit utilization rate and other indicators.

[0126] S608, adjusting the operating voltage and operating frequency of the neural network processor according to the load information of the neural network processor.

[0127] In this embodiment, the CPU analyzes the load information of the neural network processor to evaluate the current load status and future load trends of the system in real time. Based on the CPU load monitoring results, the CPU formulates the voltage and frequency adjustment strategy of the NPU.

[0128] For example, if the CPU detects a long NPU task queue and high computing unit utilization, it determines that the NPU is under heavy load and will increase the NPU's operating voltage and frequency to enhance its computing power. Conversely, if the NPU's task queue is short and computing unit utilization is low, the CPU will reduce the NPU's operating voltage and frequency to reduce unnecessary power consumption.

[0129] For example, suppose that at a certain moment, the CPU detects that the NPU's task queue suddenly increases and the computing unit utilization rate rises rapidly, indicating that the NPU is about to face a heavy load. At this time, the CPU increases the NPU's operating voltage from 1.1V to 1.3V and the operating frequency from 1.0GHz to 1.2GHz according to the preset voltage and frequency adjustment strategy. In this way, the NPU can run at a higher voltage and frequency to cope with the upcoming heavy-load tasks. On the contrary, if the CPU detects that the NPU's task queue gradually decreases and the computing unit utilization rate decreases, it indicates that the NPU is currently in a light-load state. At this time, the CPU reduces the NPU's operating voltage from 1.3V to 1.0V and reduces the operating frequency from 800MHz to 600MHz. In this way, the NPU can reduce power consumption to save energy while ensuring basic performance.

[0130] This embodiment implements power consumption control through dynamic voltage and frequency adjustment technology, ensuring that the system achieves energy efficiency management and low-latency control while ensuring efficient processing.

[0131] Figure 7 A specific video splicing method flow chart provided in the embodiment of the present disclosure, combined with Figure 7 As shown, the video stitching method provided by the embodiment of the present disclosure may include the following steps:

[0132] In S701, the RISC-based CPU schedules multiple video stream inputs and calls the GPU hardware decoder to complete low-latency decoding of the video streams.

[0133] At S702 , the NPU uses computing power to extract video frame feature points and descriptors through a lightweight neural network.

[0134] S703: The NPU and the CPU collaborate to complete the homography matrix calculation.

[0135] At S704, the GPU performs image geometry correspondence and multi-band fusion through a multi-threaded rendering pipeline to achieve pixel-level seamless stitching.

[0136] S705, the GPU completes real-time rendering output, while the RISC-based CPU dynamically adjusts hardware energy efficiency, ensuring that the overall system power consumption is within the specified power consumption limit.

[0137] Based on the same inventive concept, the present disclosure also provides a video splicing system, as described in the following embodiments. Since the principles of the system embodiment to solve the problem are similar to those of the above method embodiment, the implementation of the system embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0138] Figure 8 A schematic diagram of a video splicing system according to an embodiment of the present disclosure is shown in FIG. Figure 8 As shown, the system includes: a central processing unit 81 and a neural network processor 82, the central processing unit 81 is configured to obtain multiple video streams input by the monitoring device, and decode the multiple video streams to obtain multiple decoded video data of the multiple video streams; the central processing unit 81 is configured to perform feature extraction on the multiple decoded video data through the neural network processor to obtain multiple video feature information of the multiple video streams; the central processing unit 81 is configured to perform feature matching on the multiple video feature information through the neural network processor to obtain multiple feature matching results of the multiple video streams; the central processing unit 81 is configured to, when performing feature matching on the multiple video feature information to obtain multiple feature matching results of the multiple video streams, simultaneously eliminate erroneous matching points in the multiple feature matching results; the central processing unit 81 is configured to align multiple video frames according to valid feature matching points in the multiple feature matching results through the neural network processor to generate target video splicing data, wherein the valid feature matching points represent correct matching points in the multiple feature matching results after eliminating the erroneous matching points.

[0139] In some embodiments, including a graphics processor 83, the central processing unit 81 is configured to: create multiple decoding threads in a thread pool corresponding to the graphics processor, and allocate video memory space for each decoding thread; allocate a decoding thread one-to-one among the multiple decoding threads for each video stream in the multiple video streams; write the data of the target video stream into the video memory space corresponding to the target decoding thread through the video memory pass-through technology, wherein the target video stream is a video stream in the multiple video streams corresponding to the target decoding thread; the graphics processor performs parallel decoding processing on the multiple video streams in the video memory space of the graphics processor to generate multiple decoded video data.

[0140] In some embodiments, the central processing unit 81 is further configured to: obtain the timing information and resource requirement information of each video stream in the multiple video streams; determine the timing synchronization requirement information based on the timing information of each video stream; assign a dynamic priority weight to each video stream based on the timing synchronization requirement information and resource requirement information; and determine the input order of each video stream into the graphics processor based on the dynamic priority weight of each video stream.

[0141] In some embodiments, the neural network processor is deployed with a lightweight feature extraction model, and the central processing unit 81 is configured to: perform feature extraction on the multi-channel decoded video data through the lightweight feature extraction model on the neural network processor to obtain multi-channel video feature information of the multi-channel video streams, wherein the lightweight feature extraction model is obtained by lightweight processing of the trained neural network model.

[0142] In some embodiments, the neural network processor has a data stream scheduler, and the central processing unit 81 is configured to: create multiple subtasks in the neural network processor based on the multi-channel video feature information using the data stream scheduler; assign a respective computing unit to each of the multiple subtasks; use the computing unit corresponding to each subtask as an execution unit, read and write the multi-channel video feature information in the shared memory in parallel to perform feature matching to obtain the feature matching results corresponding to each subtask, and store the feature matching results in the shared memory; when reading and writing the multi-channel video feature information in the shared memory in parallel to perform feature matching to obtain the feature matching results corresponding to each subtask, based on the multimodal mismatch elimination strategy, by reading and writing the feature matching results in the shared memory, the mismatch points in the feature matching results are eliminated in parallel.

[0143] In some embodiments, the central processor 81 is configured to: calculate the transformation parameters between the multiple video frames through a geometric transformation model based on the effective feature matching points in the multiple feature matching results; and perform multi-channel video frame fusion based on the transformation parameters between the multiple video frames to generate target video splicing data.

[0144] In some embodiments, the central processor 81 is further configured to: obtain resource occupancy information of the neural network processor and matching resource requirement information and matching error threshold of the multi-channel decoded video data; determine feature matching parameters and geometric transformation model parameters based on the resource occupancy information of the neural network processor and matching resource requirement information and matching error threshold of the multi-channel decoded video data.

[0145] In some embodiments, the central processor 81 is further configured to: obtain load information of the image processor; adjust the operating voltage and operating frequency of the image processor according to the load information of the image processor; and / or obtain load information of the neural network processor; adjust the operating voltage and operating frequency of the neural network processor according to the load information of the neural network processor.

[0146] It should be noted that the examples and application scenarios implemented by the modules in the above system embodiment are the same as those implemented by the corresponding steps in the method embodiment, but are not limited to the contents disclosed in the above method embodiment. It should be noted that the above modules, as part of the system, can be executed in a computer system, such as a set of computer-executable instructions.

[0147] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation that combines hardware and software aspects, which may be collectively referred to herein as a "circuit," "module," or "system."

[0148] Based on the same inventive concept, an embodiment of the present disclosure further provides an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any of the above-described video stitching methods by executing the executable instructions. Since the principles for solving the problem in this electronic device embodiment are similar to those in the above-described method embodiment, the implementation of this electronic device embodiment can refer to the implementation of the above-described method embodiment, and any repetitions will not be repeated.

[0149] Refer to the following Figure 9 1 and 2 to describe the electronic device 900 according to this embodiment of the present disclosure. Figure 9 The electronic device 900 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0150] like Figure 9 As shown, electronic device 900 is implemented as a general-purpose computing device. Components of electronic device 900 may include, but are not limited to, at least one processing unit 910, at least one storage unit 920, and a bus 930 connecting various system components (including storage unit 920 and processing unit 910).

[0151] The storage unit stores program code, which can be executed by the processing unit 910, so that the processing unit 910 performs the steps described in the "Exemplary Method" section of the present disclosure according to various exemplary embodiments. For example, the processing unit 910 can perform the following steps of the above-mentioned method embodiment: obtaining multiple video streams input by the monitoring device, and decoding the multiple video streams to obtain multiple decoded video data of the multiple video streams; performing feature extraction on the multiple decoded video data using a neural network processor to obtain multiple video feature information of the multiple video streams; performing feature matching on the multiple video feature information using the neural network processor to obtain multiple feature matching results of the multiple video streams; when performing feature matching on the multiple video feature information to obtain multiple feature matching results of the multiple video streams, simultaneously eliminating incorrect matching points in the multiple feature matching results; and aligning the multiple video frames using the neural network processor according to valid feature matching points in the multiple feature matching results to generate target video splicing data, wherein the valid feature matching points represent correct matching points in the multiple feature matching results after eliminating incorrect matching points.

[0152] The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 9201 and / or a cache memory unit 9202 , and may further include a read-only memory unit (ROM) 9203 .

[0153] The storage unit 920 may also include a program / utility 9204 having a set (at least one) of program modules 9205, such program modules 9205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0154] Bus 930 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0155] The electronic device 900 can also communicate with one or more external devices 940 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 900, and / or any device that enables the electronic device 900 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 950. Furthermore, the electronic device 900 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 960. As shown, the network adapter 960 communicates with other modules of the electronic device 900 via a bus 930. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 900, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0156] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal system, or a network side, etc.) to execute the method according to the embodiments of the present disclosure.

[0157] Based on the same inventive concept, embodiments of the present disclosure also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the above-described video stitching methods. Because the principles underlying the problem solved by this computer-readable storage medium embodiment are similar to those of the above-described method embodiment, the implementation of this computer-readable storage medium embodiment can be referenced to the implementation of the above-described method embodiment, and any repetitions will not be repeated.

[0158] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0159] In the present disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, system, or device.

[0160] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0161] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a standalone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0162] Based on the same inventive concept, the present disclosure also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the video stitching method of any one of the above-mentioned method embodiments. Since the principles for solving the problem in this computer program product embodiment are similar to those in the above-mentioned method embodiment, the implementation of this computer program product embodiment can refer to the implementation of the above-mentioned method embodiment, and the repeated parts will not be repeated here.

[0163] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0164] Furthermore, although the steps of the method of the present disclosure are described in a particular order in the accompanying drawings, this does not require or imply that the steps must be performed in this particular order, or that all steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0165] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network side, etc.) to execute the method according to the embodiments of the present disclosure.

[0166] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

Claims

1. A video splicing method, characterized in that: Applied to a terminal device, the terminal device includes a central processing unit and a neural network processor, the video stream decoding method is executed by the central processing unit, and the video stream splicing method includes: Acquire multiple video streams input by a monitoring device, and decode the multiple video streams to obtain multiple decoded video data of the multiple video streams; Performing feature extraction on the multi-channel decoded video data to obtain multi-channel video feature information of the multi-channel video streams through the neural network processor; Performing feature matching on the multi-channel video feature information to obtain a multi-channel feature matching result of the multi-channel video streams by the neural network processor; When performing feature matching on the multi-channel video feature information to obtain multi-channel feature matching results of the multi-channel video streams, simultaneously eliminating incorrect matching points in the multi-channel feature matching results; Through the neural network processor, multiple video frames are aligned according to the valid feature matching points in the multiple feature matching results to generate target video splicing data, wherein the valid feature matching points represent the correct matching points in the multiple feature matching results after eliminating the incorrect matching points.

2. The video splicing method according to claim 1, wherein: The terminal device includes a graphics processor, and the decoding process for the multiple video streams to obtain target decoded video data of the multiple video streams includes: Creating multiple decoding threads in a thread pool corresponding to the graphics processor, and allocating video memory space to each decoding thread; Allocating a decoding thread from the plurality of decoding threads to each of the multiple video streams; Writing the data of the target video stream into the video memory space corresponding to the target decoding thread through the video memory pass-through technology, wherein the target video stream is a video stream corresponding to the target decoding thread in the multiple video streams; The graphics processor performs parallel decoding processing on multiple video streams in the video memory space of the graphics processor to generate multiple decoded video data.

3. The video splicing method according to claim 2, characterized in that: Before decoding the multiple video streams to obtain multiple decoded video data of the multiple video streams, the method further includes: Obtaining timing information and resource requirement information of each video stream in the multiple video streams; Determining timing synchronization requirement information according to the timing information of each video stream; Allocating a dynamic priority weight to each video stream according to the timing synchronization requirement information and resource requirement information; An input order of the video streams into the graphics processor is determined according to the dynamic priority weights of the video streams.

4. The video splicing method according to claim 1, wherein: The neural network processor is deployed with a lightweight feature extraction model. The neural network processor is used to extract features from the multi-channel decoded video data to obtain multi-channel video feature information of the multi-channel video streams, including: The multi-channel decoded video data is subjected to feature extraction through a lightweight feature extraction model on the neural network processor to obtain multi-channel video feature information of the multi-channel video streams, wherein the lightweight feature extraction model is obtained by lightweight processing of a trained neural network model.

5. The video splicing method according to claim 1, wherein: The neural network processor has a data stream scheduler, and performing feature matching on the multi-channel video feature information by the neural network processor to obtain a multi-channel feature matching result of the multi-channel video stream includes: Based on the multi-channel video feature information, the data stream scheduler is used to create a plurality of subtasks in the neural network processor; Allocating a respective computing unit to each of the plurality of subtasks; Using the computing units corresponding to the subtasks as execution units, the computing units are read and written in parallel in the shared memory to perform feature matching on the multi-channel video feature information to obtain feature matching results corresponding to the subtasks, and the feature matching results are stored in the shared memory; When performing feature matching on the multi-channel video feature information to obtain multi-channel feature matching results of the multi-channel video streams, simultaneously eliminating mismatching points in the multi-channel feature matching results, including: When performing feature matching on multiple video feature information in a shared memory in parallel to obtain feature matching results corresponding to each subtask, based on a multimodal mismatch elimination strategy, the feature matching results in the shared memory are read and written to eliminate mismatching points in the feature matching results in parallel.

6. The video splicing method according to claim 1, wherein: The step of aligning multiple video frames according to valid feature matching points in the multiple feature matching results to generate target video splicing data includes: Calculating the transformation parameters between the multiple video frames using a geometric transformation model based on the valid feature matching points in the multiple feature matching results; The multi-channel video frames are fused according to the transformation parameters between the multi-channel video frames to generate target video splicing data.

7. The video splicing method according to any one of claims 1 to 6, characterized in that: The method further comprises: Obtaining resource occupancy information of the neural network processor and matching resource requirement information and a matching error threshold of the multi-channel decoded video data; Feature matching parameters and geometric transformation model parameters are determined based on the resource occupancy information of the neural network processor, the matching resource requirement information of the multi-channel decoded video data, and the matching error threshold.

8. The video splicing method according to any one of claims 1 to 6, characterized in that: The method further comprises: Obtaining load information of the image processor; adjusting the operating voltage and operating frequency of the image processor according to load information of the image processor; and / or, Obtaining load information of the neural network processor; Adjust the operating voltage and operating frequency of the neural network processor according to the load information of the neural network processor.

9. A video splicing system, characterized in that: include: CPU and neural network processor, A central processing unit is configured to obtain multiple video streams input by a monitoring device and decode the multiple video streams to obtain multiple decoded video data of the multiple video streams; A central processing unit is configured to perform feature extraction on the multiple decoded video data through the neural network processor to obtain multiple video feature information of the multiple video streams; A central processing unit is configured to perform feature matching on the multiple video feature information through the neural network processor to obtain a multiple feature matching result of the multiple video streams; The central processing unit is configured to, when performing feature matching on the multi-channel video feature information to obtain the multi-channel feature matching results of the multi-channel video streams, simultaneously eliminate the mismatched points in the multi-channel feature matching results; The central processing unit is configured to align multiple video frames through the neural network processor according to the valid feature matching points in the multiple feature matching results to generate target video splicing data, wherein the valid feature matching points represent the correct matching points in the multiple feature matching results after eliminating the incorrect matching points.

10. An electronic device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to execute the video stitching method according to any one of claims 1 to 8 by executing the executable instructions.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video stitching method according to any one of claims 1 to 8 is implemented.

12. A computer program product comprising: A computer program or instruction, characterized in that when the computer program or instruction is executed by a processor, it implements the video stitching method described in any one of claims 1 to 8.

Images