Video processing device and method, electronic equipment and storage medium

Abstract

The invention provides a video processing device and method, electronic equipment and a storage medium. The device comprises an image splicing carrier, a plurality of loading units and an output module, the image splicing carrier comprises a continuous physical memory for providing corresponding logic storage areas for video frames in the video streams of the multiple channels, and the logic storage areas are in one-to-one correspondence with the channels; each loading unit is configured to load pixel data of a video frame of a first channel into a logic storage area, corresponding to the first channel, on the image splicing carrier in response to the received video frame in the video stream of the first channel; the first channel is a channel corresponding to the loading unit in the plurality of channels; and the output module is configured to read the pixel data from the image splicing carrier according to the memory address sequence to obtain a spliced image. The device can improve the image processing efficiency.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of video processing, and in particular, to a video processing apparatus, method, electronic device and storage medium. BACKGROUND

[0002] In a scenario of video processing based on an edge device, the edge device is an intermediate layer connecting a video collection device and a cloud, and is responsible for local processing of video data collected by the video collection device. However, in the process of processing multiple videos, the central processing unit (CPU) of the edge device needs to participate in each processing process of each video from input to output, resulting in high CPU load and low processing efficiency of multiple videos. SUMMARY

[0003] The present disclosure provides a video processing apparatus, method, electronic device and storage medium to improve video processing efficiency.

[0004] According to a first aspect of an embodiment of the present disclosure, a video processing apparatus is provided, which comprises an image splicing carrier, a plurality of loading units and an output module; the image splicing carrier comprises continuous physical memory for providing corresponding logical storage areas for video frames in a video stream of a plurality of channels, and the logical storage areas correspond one-to-one to the channels; each loading unit is configured to, in response to receiving a video frame in a first channel of the video stream, load pixel data of the video frame of the first channel into a logical storage area corresponding to the first channel on the image splicing carrier; the first channel is one of the channels corresponding to the loading unit in the plurality of channels; and the output module is configured to read the pixel data from the image splicing carrier in the order of memory addresses to obtain a spliced image.

[0005] In some embodiments, the loading unit is further configured to, in response to receiving first identification information corresponding to the video frame of the first channel, convert the first identification information into second identification information corresponding to the image splicing carrier, and cache the second identification information; wherein the first identification information comprises position information of a target image region in the video frame; and the second identification information comprises position information of pixel data of the target image region in the video frame in the image splicing carrier; and the video processing apparatus further comprises an annotation unit; the annotation unit is configured to perform annotation processing on the pixel data of the target image region in the image splicing carrier based on the second identification information of the video frames of the plurality of channels.

[0006] In some embodiments, the video processing apparatus further comprises a detection unit connected to the plurality of loading units; the detection unit is configured to detect at least one video frame of at least one channel, to generate first identification information corresponding to each video frame of the at least one video frame, and to send each video frame and the corresponding first identification information to the loading unit corresponding to the channel to which each video frame belongs.

[0007] In some embodiments, the detection unit is configured to pre-process the received video frames to obtain tensors corresponding to the video frames, to obtain a tensor set corresponding to a preset number of video frames based on the tensors corresponding to the video frames each time the preset number of video frames is received, and to detect the preset number of video frames based on the tensor set to generate first identification information corresponding to each video frame of the preset number of video frames.

[0008] In some embodiments, the video processing apparatus further comprises a decoding unit corresponding to the video stream of each channel; the decoding unit is connected to the loading unit or the detection unit; and each decoding unit is configured to decode compressed encoded frames in the video stream of the corresponding channel into video frames in a preset format, and to send the video frames to the loading unit or the detection unit.

[0009] In some embodiments, the output module comprises a scheduling unit, a video encoding unit, and a sending unit; the scheduling unit is configured to send, in a first output mode, a memory address of the image splicing carrier to the video encoding unit; the video encoding unit is configured to read pixel data from the image splicing carrier in the order of the memory address to obtain a spliced image, and to perform encoding processing on the spliced image; and the sending unit is configured to send the encoded spliced image to a receiving end through a network.

[0010] In some embodiments, the output module comprises a scheduling unit and a display control unit; the scheduling unit is configured to, in a second output mode, establish an association between the image splicing carrier and the superimposition plane in the display control unit; the superimposition plane associated with the image splicing carrier is configured with a memory address of the image splicing carrier; and the display control unit is configured to read pixel data from the image splicing carrier in the order of the memory address based on the superimposition plane associated with the image splicing carrier to obtain a spliced image, and to send the spliced image to a display end through an output interface.

[0011] In some embodiments, each loading unit is configured to, in response to receiving a video frame in the video stream of the corresponding channel, load pixel data of the video frame into a logical storage area corresponding to the channel to which the video frame belongs in the image splicing carrier based on image processing parameters in a first state; the image processing parameters in the first state include position information of the logical storage area corresponding to the channel to which the video frame belongs.

[0012] In some embodiments, the video processing apparatus described above further comprises a parameter maintenance module; the parameter maintenance module is configured to: in response to a modification instruction, modify the image processing parameter in the second state; and in a case where the modification of the image processing parameter in the second state is completed, perform state conversion on the image processing parameter in the second state and the image processing parameter in the first state.

[0013] According to a second aspect of the embodiments of the present disclosure, a video processing method is provided, which comprises: in response to receiving a video frame in a video stream of at least one channel, loading pixel data of the video frame into a logical storage area corresponding to the channel to which the video frame belongs on an image stitching carrier; the image stitching carrier comprises continuous physical memory for providing corresponding logical storage areas for video frames in video streams of multiple channels; the logical storage area corresponds to the channel one by one; and reading the pixel data from the image stitching carrier in the order of memory addresses to obtain a stitched image.

[0014] In some embodiments, the method described above further comprises: in response to receiving first identification information corresponding to a video frame in a video stream of at least one channel, converting the first identification information into second identification information corresponding to the image stitching carrier; wherein the first identification information comprises position information of a target image region in the video frame; the second identification information comprises position information of pixel data of the target image region in the image stitching carrier; caching the second identification information; and based on the second identification information of the video frames of the multiple channels, performing annotation processing on the pixel data of the target image region in the image stitching carrier.

[0015] In some embodiments, the method described above further comprises: detecting at least one video frame of at least one channel to generate first identification information corresponding to each video frame of the at least one video frame.

[0016] In some embodiments, the reading of the pixel data from the image stitching carrier in the order of memory addresses to obtain the stitched image comprises: in a first output mode, reading the pixel data from the image stitching carrier in the order of memory addresses to obtain the stitched image; the method described above further comprises: performing encoding processing on the stitched image; and sending the stitched image after the encoding processing to a receiving end through a network.

[0017] In some embodiments, the reading of the pixel data from the image stitching carrier in the order of memory addresses to obtain the stitched image comprises: in a second output mode, establishing an association relationship between the image stitching carrier and a superimposition plane; the superimposition plane associated with the image stitching carrier comprises a memory address of the image stitching carrier; based on a second output instruction and the superimposition plane associated with the image stitching carrier, reading the pixel data from the image stitching carrier in the order of memory addresses to obtain the stitched image; and the method described above further comprises: sending the stitched image to a display end through an output interface.

[0018] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing computer programs or instructions; wherein the processor executes the computer programs or instructions to implement the steps in any of the video processing methods in the first aspect.

[0019] According to a fourth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, which stores computer programs or instructions, when the computer programs or instructions in the storage medium are executed by a processor, the steps in any of the video processing methods in the first aspect are implemented.

[0020] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, which comprises computer programs or instructions, when the computer programs or instructions are executed by a processor, the steps in any of the video processing methods in the first aspect are implemented.

[0021] The technical solutions provided by the embodiments of the present disclosure can include the following beneficial effects:

[0022] The video processing device provided by the embodiments of the present disclosure comprises: an image splicing carrier, a plurality of loading units and an output module; the image splicing carrier comprises continuous physical memory for providing corresponding logical storage areas for video frames in video streams of a plurality of channels, and the logical storage areas correspond one-to-one to the channels; each loading unit is configured to load pixel data of a video frame of a first channel into a logical storage area corresponding to the first channel on the image splicing carrier in response to receiving the video frame in the video stream of the first channel; the first channel is one of the channels corresponding to the loading unit in the plurality of channels; and the output module is configured to read the pixel data from the image splicing carrier in the order of memory addresses to obtain a spliced image. The video processing device can realize splicing processing of video frames of a plurality of channels through loading and reading of pixel data, without performing image splicing operations at the software level by the CPU, so that the participation of the CPU in the image processing process can be reduced, thereby reducing the influence of the performance bottleneck of the CPU on the image processing and improving the image processing efficiency.

[0023] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure.

[0025] Figure 1 A structural schematic diagram of a video processing device is provided for the embodiments of the present disclosure;

[0026] Figure 2 A schematic diagram of an image splicing carrier provided by an embodiment of the present disclosure is shown in FIG. 1.

[0027] Figure 3 A structural schematic diagram of another video processing apparatus provided by an embodiment of the present disclosure is shown in FIG. 2.

[0028] Figure 4 An example diagram of spliced images provided by an embodiment of the present disclosure is shown in FIG. 3.

[0029] Figure 5 A structural schematic diagram of a video processing apparatus provided by an embodiment of the present disclosure is shown in FIG. 4.

[0030] Figure 6 A flowchart of a video processing method provided by an embodiment of the present disclosure is shown in FIG. 5.

[0031] Figure 7 A structural block diagram of an electronic device provided by an embodiment of the present disclosure is shown in FIG. 6. DETAILED DESCRIPTION

[0032] The exemplary embodiments will be described in detail herein with reference to the accompanying drawings. The following description is with reference to the drawings, in which like numerals indicate like elements, unless otherwise described in detail. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of apparatuses consistent with some aspects of the present disclosure as detailed in the appended claims.

[0033] In a scenario of video processing based on an edge device, the edge device is an intermediate layer connecting a video capture apparatus and a cloud, and is responsible for local processing of video data captured by the video capture apparatus. However, in the process of processing multiple videos, the central processing unit (CPU) of the edge device needs to deeply participate in each processing process of each video from input to output, resulting in high CPU load and low processing efficiency of multiple videos.

[0034] To this end, an embodiment of the present disclosure provides a video processing apparatus, which, in the process of processing multiple videos, implements processing of the multiple videos through corresponding processing modules, reduces the process of CPU participating in video processing to reduce CPU load, and further improves the processing efficiency of multiple videos.

[0035] The video processing apparatus can be located in an electronic device. Exemplarily, the electronic device can be a mobile terminal and a fixed terminal. The mobile terminal can include a smartphone, a tablet computer, a notebook computer, a digital camera, a wearable electronic device, and the like. The fixed terminal can include a desktop computer, an all-in-one computer, and the like.

[0036] Figure 1 A structural schematic diagram of a video processing apparatus is provided for the embodiments of the present disclosure. As shown in the figure, the video processing apparatus 100 comprises an image splicing carrier 11, a plurality of loading units 12 and an output module 13. Figure 1

[0037] The image splicing carrier 11 comprises a continuous physical memory for providing corresponding logical storage areas for video frames in video streams of a plurality of channels, and the logical storage areas correspond to the channels one by one.

[0038] In some embodiments, the image splicing carrier 11 is located in a shared memory for direct access by other modules in the video processing apparatus, such as the plurality of loading units 12 and the output module 13. It can be understood that, since the plurality of loading units 12 and the output module 13 can directly access the image splicing carrier 11, the plurality of loading units 12 and the output module 13 can directly obtain data in the image splicing carrier 11 for use, without copying the data in the image splicing carrier 11 to the private memories or caches of the modules themselves for use.

[0039] Figure 2 A schematic diagram of an image splicing carrier is provided for the embodiments of the present disclosure. As shown in the figure, the continuous physical memory in the image splicing carrier 11 comprises a plurality of logical storage areas, such as the logical storage area 110 to the logical storage area 113 in the figure. Figure 2 Figure 2 Each logical storage area comprises at least one address segment in the continuous physical memory, for example, the logical storage area 111 comprises the address segment 1111, the address segment 1112 and the address segment 1113. Each address segment comprises a plurality of continuous storage units, such as a storage unit 1111a in the address segment 1111, and the storage units are arranged in an increasing order of memory addresses in a basic unit of bytes, and collectively constitute a storage space of the address segment. In an example, if a video frame adopts an RGB888 pixel format (each pixel occupies 3 bytes), then every 3 continuous storage units in the address segment can store red, green and blue components of a pixel in the video frame, respectively.

[0040] As shown in the figure, the address segments included in each logical storage area do not overlap with each other, and there is no address conflict among the logical storage areas. Moreover, the sum of the address segments included in all the logical storage areas covers all memory addresses from the start address to the end address of the continuous physical memory, and there is no unallocated storage space in the continuous physical memory in the image splicing carrier 11. Figure 2

[0041] ​​​In some embodiments, the electronic device obtains video streams captured by a plurality of image capturing devices, wherein a video stream from one image capturing device is a video stream of one channel. Each video stream of a channel is composed of video frames, and each video frame includes pixel data and a channel identifier of the channel to which the video frame belongs.

[0042] In some embodiments, the amount of pixel data included in a video frame is related to the resolution of the video frame. For example, when the resolution of a video frame is 1920x1080, the total amount of pixel data included in the video frame is 1920x1080 pixels; when the resolution of a video frame is 1280x720, the total amount of pixel data included in the video frame is 1280x720 pixels.

[0043] Each loading unit 12 is configured to, in response to receiving a video frame in a video stream of a first channel, load pixel data of the video frame of the first channel into a logical storage area corresponding to the first channel on the image stitching carrier, wherein the first channel is one of the plurality of channels corresponding to the loading unit.

[0044] In some embodiments, each video stream of a channel corresponds to one loading unit 12. The number of loading units 12 is the same as the number of channels of the video streams.

[0045] In one example, there are two video streams of channels: a video stream of channel 1 and a video stream of channel 2, wherein the video stream of channel 1 corresponds to loading unit a, and the video stream of channel 2 corresponds to loading unit b. In this example, loading unit a is configured to, in response to receiving a video frame in the video stream of channel 1, load pixel data of the video frame of channel 1 into a logical storage area 1 corresponding to channel 1 on the image stitching carrier; and loading unit b is configured to, in response to receiving a video frame in the video stream of channel 2, load pixel data of the video frame of channel 2 into a logical storage area 2 corresponding to channel 2 on the image stitching carrier.

[0046] In some embodiments, each loading unit 12 is configured to, in response to receiving a video frame in the video stream of the first channel, load pixel data of the video frame of the first channel into a logical storage area corresponding to the first channel on the image stitching carrier by invoking a hardware overlay engine, wherein the first channel is one of the channels corresponding to the loading unit. In this embodiment, the video frame received by the loading unit 12 is stored in the input buffer of the loading unit 12; the hardware overlay engine is used to request the same number of DMA channels as the number of channels of the video stream from a DMA controller in advance, and to load the pixel data of the video frame into the logical storage area through the DMA channels. The hardware overlay engine is, for example, an Overlay Engine or a Hardware Composer. The DMA channel refers to a hardware channel in the DMA controller for independently completing a set of data transmissions. Each DMA channel corresponds to a loading unit 12, and the source address of the data transmission by the DMA channel corresponding to the loading unit 12 is determined based on the address of the input buffer of the loading unit 12, and the destination address is determined based on the position information (including the starting address and the address segment offset) of the logical storage area corresponding to the channel to which the video stream of the loading unit 12 belongs.

[0047] In some embodiments, each loading unit 12 is configured to, in response to receiving a video frame in the video stream of the first channel, load pixel data of the video frame into a first address segment, wherein the first address segment is an address segment included in the logical storage area corresponding to the first channel, and the first address segment can be one continuous address segment or a plurality of discontinuous address segments.

[0048] In some embodiments, each loading unit 12 is configured to, in the order of rows of the video frame, sequentially write each row of pixel data in the pixel data of the video frame into the first address segment, or, in the order of columns of the video frame, sequentially write each column of pixel data in the pixel data of the video frame into the first address segment. In the case where the first address segment is a plurality of discontinuous address segments, a row of pixel data or a column of pixel data is written into one address segment.

[0049] In some examples, the resolution of the video frame of the first channel is WxH, i.e., the pixel data in the video frame is W columns and H rows; the logical storage area corresponding to the first channel includes H address segments, and the length of each address segment is Wxthe number of bytes of a single pixel. Each loading unit 12 is configured to, when loading the pixel data of the video frame into the logical storage area, write the first row of pixels into the first address segment, the second row of pixels into the second address segment, and so on, in the order of the first row to the Hth row of the video frame.

[0050] Based on the foregoingFigure 2 Further to the example, if the pixel data in the video frame of the first channel is 5 columns and 3 rows, the logical storage area corresponding to the first channel is Figure 2 the logical storage area 111 shown in FIG. 1. When loading the pixel data of the video frame into the logical storage area 111, the loading unit writes the pixel data of the first row into the address segment 1111, the pixel data of the second row into the address segment 1112, and the pixel data of the third row into the address segment 1113 in the order of the first row, the second row, and the third row of the video frame.

[0051] It can be understood that the loading units load the pixel data of the video frames of the corresponding channels into the logical storage areas corresponding to the channels on the image splicing carrier respectively. When the logical storage areas in the image splicing carrier are filled, the pixel data of the video frames of the multiple channels can be merged into the pixel data of one video frame, so as to realize the effect of splicing the video frames of the multiple channels into one video frame.

[0052] The output module 13 is configured to read the pixel data from the image splicing carrier in the order of memory addresses to obtain a spliced image.

[0053] In some embodiments, the output module 13 is configured to read the pixel data from the image splicing carrier in the order of increasing memory addresses to obtain a spliced image. The order of increasing memory addresses indicates the order from the start address to the end address of the continuous physical memory in the image splicing carrier.

[0054] In some other embodiments, the output module 13 is configured to read the pixel data from the image splicing carrier in the order of decreasing memory addresses to obtain a spliced image. The order of increasing memory addresses indicates the order from the end address to the start address of the continuous physical memory in the image splicing carrier.

[0055] It can be understood that the output module reads the pixel data from the image splicing carrier in the order of memory addresses, and can read the pixel data of each row or each column in the spliced image, so as to obtain the spliced image.

[0056] In some embodiments, the output module 13 is configured to read the pixel data from the image splicing carrier in the order of memory addresses based on a first preset frequency to obtain a spliced image.

[0057] In one example, the first preset frequency can be determined based on the acquisition frequency of video frames from multiple channels. For instance, if the acquisition frequencies of video frames from multiple channels are the same, the first preset frequency is equal to the acquisition frequency of the video frames; if the acquisition frequencies of video frames from at least two channels are different, the first preset frequency is the acquisition frequency of the lowest video frame, such as one at 30 frames per second (fps) and the other at 60 fps, then the first preset frequency is 30 fps.

[0058] In another example, the first preset frequency can also be determined based on the display frequency of the pre-configured stitched images.

[0059] In some embodiments, to achieve the effect of stitching video frames from multiple channels into a single video frame, another method can be used: the CPU in the electronic device schedules multiple CPU threads to stitch video frames from multiple channels. For example, each CPU thread copies video frames from the corresponding channel's video stream into a buffer, and then performs image stitching operations on the video frames in the buffer. However, multiple CPU threads cannot write video frames into the buffer simultaneously, otherwise it will cause a write conflict. Only the CPU thread that acquires a software lock (such as a mutex) can write video frames into the buffer. After writing video frames into the buffer, the CPU thread that acquired the software lock releases the software lock; CPU threads that have not acquired the software lock wait for the software lock to be released before acquiring the software lock to write video frames into the buffer.

[0060] The process of stitching video frames using multiple CPU threads not only requires copying the video frames but also necessitates the introduction of a locking mechanism to ensure that the data copying process does not conflict. This results in a time-consuming and inefficient image processing process with high CPU overhead.

[0061] In the video processing apparatus provided in the foregoing embodiments of this disclosure, each loading unit loads the pixel data of the video frame of the corresponding channel into the logical storage area on the image stitching carrier corresponding to the channel to which the video frame belongs. Since the logical storage areas corresponding to each loading unit are different and physically isolated, multiple loading units can perform the loading operation of the pixel data of the video frame in parallel, thereby improving image processing efficiency. Furthermore, through the loading operation of the pixel data of the video frame by each loading unit, the pixel data of the video frames of multiple channels can be merged into the pixel data of one video frame on the image stitching carrier. The output module reads the pixel data from the image stitching carrier in the order of memory address and can read the pixel data of the merged video frame to obtain the stitched image. That is, the video processing apparatus can realize the stitching processing of video frames of multiple channels by loading and reading pixel data, without the need for the CPU to perform image stitching operations at the software level. This reduces the CPU's involvement in the image processing process, thereby reducing the impact of the CPU's performance bottleneck on image processing and further improving image processing efficiency.

[0062] In some embodiments, each of the above loading units is configured to: in response to receiving a video frame from the video stream of the corresponding channel, load the pixel data of the video frame into the logical storage area on the image stitching carrier corresponding to the channel to which the video frame belongs, based on the image processing parameters in the first state.

[0063] In some implementations, the image processing parameters include location information of the logical storage area corresponding to the channel to which the video frame belongs. These image processing parameters can be represented by arrays, where each array includes the location information of the logical storage area corresponding to one channel.

[0064] In one example, the location information of the logical storage area in the image processing parameters includes: the location information of the pixel area corresponding to the video frame to be written in the stitched image, such as the horizontal and vertical coordinate offsets of the top-left pixel in this pixel area compared to the top-left pixel in the video frame to be written, as well as the width and height of the pixel area. It is understood that the coordinates of the top-left pixel in a video frame are typically (0, 0), therefore, this location information can be represented by the pixel coordinates of the top-left pixel in the pixel area corresponding to the video frame in the stitched image.

[0065] In another example, the location information of the logical storage region in the image processing parameters includes: the starting address of the logical storage region and the address segment offset, which represents the byte difference between the starting addresses of adjacent address segments within the logical storage region. It is understood that if the logical storage region contains only one address segment, the address segment offset can be 0.

[0066] In some implementations, each of the above loading units is configured to: in response to receiving a video frame from the video stream of the corresponding channel, obtain the starting address and address segment offset of the logical storage area corresponding to the channel to which the video frame belongs from the image processing parameters in the first state; determine the starting address of the address segment to be written for each row of pixel data in the video frame based on the starting address of the logical storage area, the address segment offset, and the row index of each row of pixels in the video frame; and sequentially write each row of pixel data in the video frame into the corresponding address segment according to the row order of the video frame and based on the determined starting address of the address segment to be written for each row of pixel data.

[0067] In one example, the logical storage area corresponding to the channel to which the video frame belongs is as described above. Figure 2 In the case of the logical storage area 111 shown, the location information of the logical storage area includes the starting address of address segment 1111 and the address segment offset, which is the byte difference between the starting address of address segment 1111 and the starting address of address segment 1112.

[0068] In this example, the starting address of the address segment to be written for each row of pixel data is determined. For example: the row index of the first row of pixels in the video frame is 0, and the starting address of the address segment to be written for the first row of pixels is: starting address of address segment 1111 + 0 × address segment offset = starting address of address segment 1111; the row index of the second row of pixels in the video frame is 1, and the starting address of the address segment to be written for the second row of pixels is: starting address of address segment 1111 + 1 × address segment offset = starting address of address segment 1112; the row index of the third row of pixels in the video frame is 2, and the starting address of the address segment to be written for the third row of pixels is: starting address of address segment 1111 + 2 × address segment offset = starting address of address segment 1113.

[0069] In some embodiments, the image processing parameters further include image scaling information corresponding to the channel to which the video frame belongs. In this embodiment, each loading unit is configured to: in response to receiving a video frame from the video stream of the corresponding channel, scale the video frame based on the image scaling information; and load the pixel data of the scaled video frame into the logical storage area on the image stitching carrier corresponding to the channel to which the video frame belongs. The scaling method performed by the loading unit on the video frame may be, for example, one of the following: bilinear interpolation scaling, nearest neighbor interpolation scaling, or pixel sampling or padding based on a preset ratio.

[0070] In some examples, if the video frame size is large (e.g., greater than or equal to 1920×1080), the size of the stitched image after stitching multiple video frames will be even larger. For example, the size of the stitched image after stitching four 1920×1080 video frames will be 4×1920×1080, leading to increased output pressure on the stitched image. To reduce output pressure, the size of the stitched image carried by the image stitching carrier can be preset, and the image scaling parameters of each channel's video frames can be preset so that the stitched image after stitching multiple video frames is equal to the preset stitched image size.

[0071] In some embodiments, the image processing parameters in the first state used by the loading unit are stored in shared memory. The shared memory also stores image processing parameters in the second state. The first state indicates a readable state, and the second state indicates a writable state.

[0072] It is understandable that this disclosure stores two copies of image processing parameters in different states in shared memory. This ensures that reading and writing do not occur on the same copy of image processing parameters, thereby avoiding read and write conflicts of image processing parameters without introducing synchronization locks.

[0073] In some implementations, the image processing parameters include a status identifier. This status identifier includes a first identifier or a second identifier. When the status identifier is the first identifier, it indicates that the image processing parameter is in a first state; when the status identifier is the second identifier, it indicates that the image processing parameter is in a second state. In some examples, the first identifier is 0 and the second identifier is 1; in other examples, the first identifier is 1 and the second identifier is 2.

[0074] In this embodiment of the disclosure, each loading unit is configured to load the pixel data of the video frame into the logical storage area corresponding to the channel to which the video frame belongs on the image stitching carrier based on the image processing parameters in the first state. This not only accurately loads the pixel data of the video frame onto the image stitching carrier, but also avoids read / write conflicts of the image processing parameters.

[0075] In some embodiments, the video processing apparatus further includes a parameter maintenance module. This parameter maintenance module is configured to: modify image processing parameters in a second state in response to a modification instruction; and, upon completion of the modification of the image processing parameters in the second state, perform a state transition between the image processing parameters in the second state and the image processing parameters in the first state. The modification instruction is used to indicate modification of the image processing parameters, such as modifying the location information of the logical storage area corresponding to at least one channel.

[0076] In some embodiments, the above-described state transition between the image processing parameters in the second state and the image processing parameters in the first state includes: converting the image processing parameters in the second state to the image processing parameters in the first state by converting the second identifier in the image processing parameters in the second state to the first identifier, and converting the image processing parameters in the first state to the image processing parameters in the second state by converting the first identifier in the image processing parameters in the first state to the second identifier.

[0077] In this embodiment of the disclosure, the parameter maintenance module is configured to modify the image processing parameters in the second state, which can avoid read and write conflicts of the image processing parameters. Furthermore, when the image processing parameters in the second state are modified, the image processing parameters in the second state and the image processing parameters in the first state are switched to ensure that the image processing parameters used by the aforementioned loading unit are the updated image processing parameters, thereby improving the accuracy of the loading unit in loading pixel data.

[0078] In some embodiments, since the image stitching carrier may also be in a state of being read and written at the same time, in order to avoid read and write conflicts caused by simultaneous reading and writing of the image stitching carrier, two image stitching carriers in different states can be configured in the shared memory: an image stitching carrier in a first state and an image stitching carrier in a second state.

[0079] In this embodiment, each of the above loading units is configured to load the pixel data of the video frame in the first channel into the logical storage area corresponding to the first channel on the image stitching carrier in the second state in response to receiving a video frame in the video stream of the first channel, wherein the first channel is one of the multiple channels corresponding to the loading unit.

[0080] The aforementioned output module is configured to: read pixel data from the image stitching carrier in the first state in ascending order of memory address based on a first preset frequency, and obtain a stitched image.

[0081] The aforementioned parameter maintenance module is further configured to: perform state transitions between the image stitching carrier in the second state and the image stitching carrier in the first state based on a second preset frequency. The second preset frequency is the same as the first preset frequency but has a fixed phase difference, which is determined based on the reading time of the output module when reading pixel data from the image stitching carrier in the first state in ascending memory address order.

[0082] In this way, whenever the output module obtains a stitched image, the parameter maintenance module will perform a state transition between the image stitching carrier in the second state and the image stitching carrier in the first state. This ensures that the image stitching carrier in the first state before the state transition, i.e., the image stitching carrier in which all loaded image frames have been completely read, is written with new image frames after the state transition; and the image stitching carrier in the second state before the state transition, i.e., the image stitching carrier containing unread image frames, waits to be read after the state transition.

[0083] It should be noted that if the image stitching carrier in the second state is in the process of loading pixel data during the state transition, the state transition will not affect the ongoing pixel data loading. Furthermore, since the state transition of the image stitching carrier occurs when the output module has just acquired the stitched image, the image stitching carrier in the second state will need to wait for a period of time after transitioning to the first state before being read. During this period, the pixel data being loaded will also be completed, thus not affecting the next reading process of the output module.

[0084] In some embodiments, the loading unit is further configured to: in response to receiving first identification information corresponding to a video frame of the first channel, convert the first identification information into second identification information corresponding to an image stitching carrier, and cache the second identification information.

[0085] In some implementations, the first identification information includes the location information of at least one target image region within the video frame. In some examples, the target image region may be one of the following: a region containing a person, a region containing a vehicle, a region containing an animal, etc.

[0086] In some examples, the target image region can be represented by an irregular boundary line that fits the contour edge of an object. For example, if the target image region contains a vehicle, then the target image region can be represented by an irregular box formed by fitting the contour edge of the vehicle.

[0087] In other examples, the target image region can be represented by regular boundary lines surrounding the object, such as any of the following: a circular frame, an elliptical frame, a rectangular frame, etc.

[0088] In some implementations, the location information of the target image region in the video frame includes the pixel coordinates of each pixel on the boundary line of the target image region.

[0089] In some implementations, when the target image region is represented as a rectangular box, the position information of the target image region in the video frame is composed of the pixel coordinates (x1, y1) of the upper left corner of the target image region, the width (w1) and the height (h1) of the target image region. Here, x1 is an integer ranging from 0 to the video frame width - 1; y1 is an integer ranging from 0 to the video frame height - 1; w1 is an integer ranging from 1 to the video frame width; and h1 is an integer ranging from 1 to the video frame height.

[0090] In other embodiments, when the target image region is represented as a rectangular box, the position information of the target image region in the video frame is composed of the pixel coordinates (x1, y1) of the upper left corner and the pixel coordinates (x2, y2) of the lower right corner of the target image region. Here, x1 is an integer ranging from 0 to the video frame width - 1; y1 is an integer ranging from 0 to the video frame height - 1; x2 is an integer ranging from x1 (excluding) to the video frame width (including), and y2 is an integer ranging from y1 (excluding) to the video frame height (including).

[0091] In some other embodiments, when the target image region is represented as a rectangle or a circle, the position information of the target image region in the video frame includes the center and radius (or diameter) of the target image region.

[0092] In some implementations, the first identification information includes not only the location information of at least one target image region in the video frame, but also the target image region category of each target image region. This target image region category is used to indicate the object indicated by the target image region; for example, a first category indicates that the object indicated by the target image region is a person or animal, and a second category indicates that the object indicated by the target image region is a vehicle.

[0093] In some implementations, the second identification information includes the position information of pixel data of each target image region in the video frame in the image stitching carrier.

[0094] In other embodiments, the second identification information also includes the target image region category of each target image region in the video frame.

[0095] In some examples, the pixel data of the target image region is indicated as: pixel data representing the boundary lines of the target image region.

[0096] In some implementations, each loading unit is configured to: in response to receiving first identification information corresponding to a video frame of the first channel, convert the first identification information into second identification information corresponding to the image stitching carrier based on image processing parameters in a first state.

[0097] In some examples, the image processing parameters include location information of the logical storage area corresponding to the channel to which the video frame belongs. This location information includes the starting address and address range offset of the logical storage area. In this example, the second identification information, converted from the first identification information and corresponding to the image stitching carrier, includes the memory addresses of the pixel data of each target image region in the video frame within the image stitching carrier.

[0098] In other examples, the image processing parameters include the location information of the logical storage area corresponding to the channel to which the video frame belongs. This location information includes the horizontal coordinate offset Δx and vertical coordinate offset Δy of the top-left pixel, as well as the width and height of the pixel region. The first identification information includes the pixel coordinates (x1, y1) of the top-left corner of the target image region in the video frame, the width w1 of the target image region, and the height h1 of the target image region. In this example, the second identification information, converted from the first identification information and corresponding to the image stitching carrier, includes the pixel coordinates (x1', y1') of the top-left corner of the target image region in the stitched image, the width w1 of the target image region, and the height h1 of the target image region, where x1' = x1 + Δx, y1' = y1 + Δy.

[0099] In some other examples, the image processing parameters also include image scaling information corresponding to the channel to which the video frame belongs, such as the scaling factor s. The second identification information, converted from the first identification information and corresponding to the image stitching carrier, includes: the pixel coordinates (x11', y11') of the upper left corner of the target image region in the stitched image, the width w1' and height h1' of the target image region, where x11' = x1 × s + Δx, y11' = y1 × s + Δy, w1' = w1 × s, and h1' = h1 × s.

[0100] In this embodiment, the loading unit converts the first identification information into second identification information corresponding to the image stitching carrier, thereby converting the position of the target image region in the video frame into its position in the image stitching carrier. The second identification information of multiple channels of video frames is cached in a contiguous storage area, which may be located in the aforementioned shared memory or in other memory within the electronic device. The second identification information of the multiple channels of video frames can be stored in memory in the form of an array.

[0101] Figure 3 This is a schematic diagram of another video processing apparatus provided in an embodiment of the present disclosure.

[0102] like Figure 3As shown, the video processing apparatus 100 also includes an annotation unit 14. The annotation unit 14 is configured to: annotate the pixel data of the target image region in the image stitching carrier based on the second identification information of video frames from multiple channels.

[0103] It is understood that in this embodiment of the disclosure, only one instruction for annotation processing needs to be submitted to the system to uniformly annotate the target image region in video frames of multiple channels, instead of submitting an instruction for each video frame of each channel to perform separate annotation processing of the target image region. This reduces the number of instructions for annotation processing submitted to the system and thus improves the efficiency of annotation processing.

[0104] Furthermore, the unified annotation targets the stitched images in the image stitching carrier. When the resolution of the stitched image is lower than the sum of the resolutions of multiple video frames, this unified annotation process can reduce the number of pixels that need to be processed during annotation, thereby further improving the efficiency of annotation.

[0105] In some embodiments, the annotation unit 14 is configured to: annotate the pixel data of the target image region in the image stitching carrier based on a preset processing frequency and second identification information of video frames from multiple channels. The preset processing frequency is the same as the first preset frequency at which the output module reads the stitched image, but has a fixed phase difference. This fixed phase difference can be set according to actual application conditions, and is not specifically limited in this embodiment. Thus, the annotation unit can uniformly annotate the pixel data of the target image region in the image stitching carrier before the output module acquires the stitched image.

[0106] In some embodiments, the annotation unit 14 is configured to modify the color of the pixel data of the target image region in the image stitching carrier to the target color based on the second identification information of video frames from multiple channels. This allows the target image region to be identified using a color-coded frame.

[0107] In some examples, where the second identification information does not include the target image region category, the target color is a pre-defined color, such as any of the following: red, green, or blue.

[0108] In other examples, where the second identification information includes the target image region category, the target color is determined based on the target image region category. For example, the pixel data of the aforementioned first category of target image regions (corresponding to people or animals) is changed to red, and the pixel data of the aforementioned second category of target image regions (corresponding to vehicles) is changed to green. In this way, target image regions for different categories of objects can be identified with different colored boxes.

[0109] In some embodiments, the annotation unit 14 can be deployed in an NPU (Neural Processing Unit). In this way, the annotation unit 14 utilizes the NPU's two-dimensional image processing acceleration function, calls the NPU's dedicated drawing instructions, and submits a batch drawing instruction to uniformly annotate the target image regions in video frames of multiple channels, thereby reducing the overhead of drawing instruction submission and potential bus congestion, and achieving efficient drawing of detection boxes.

[0110] In this embodiment, the loading unit is configured to, in response to receiving first identification information corresponding to a video frame of the first channel, convert the first identification information into second identification information corresponding to the image stitching carrier, and cache the second identification information; the annotation unit is configured to annotate the pixel data of the target image region in the image stitching carrier based on the second identification information of video frames of multiple channels, and can uniformly annotate the target image region in video frames of multiple channels by submitting an annotation processing instruction to the system once, thereby reducing the number of annotation processing instructions submitted to the system and improving the efficiency of annotation processing.

[0111] In some embodiments, such as Figure 3 As shown, the video processing apparatus 100 further includes a detection unit 15 connected to a plurality of loading units 12. The detection unit 15 is configured to: detect at least one video frame of at least one channel, generate first identification information corresponding to each video frame in the at least one video frame; and send each video frame and the corresponding first identification information to the loading unit corresponding to the channel to which each video frame belongs.

[0112] In some embodiments, the above detection refers to image recognition processing. The detection unit 15 is configured to: perform image recognition processing on at least one video frame of at least one channel to determine a target image region in each of the at least one video frame; and, for each of the at least one video frame, generate first identification information corresponding to that video frame based on the target image region in each video frame.

[0113] In this embodiment, the detection unit is configured to detect at least one video frame of at least one channel, generate first identification information corresponding to each video frame in the at least one video frame, and then bind and send each video frame and the corresponding first identification information to the loading unit corresponding to the channel to which each video frame belongs. This can avoid the timing misalignment of the video frame and the first identification information, and also enable the loading unit to directly process the video frame and the first identification information of the corresponding channel without having to read the video frame and the first identification information from memory, thereby speeding up image processing efficiency.

[0114] In some embodiments, the detection unit 15 is configured to: preprocess the received video frames to obtain tensors corresponding to the video frames; for each preset number of video frames received, obtain a set of tensors corresponding to the preset number of video frames based on the tensors corresponding to the video frames; and detect the preset number of video frames based on the tensor set to generate first identification information corresponding to each video frame in the preset number of video frames.

[0115] In some implementations, preprocessing includes at least one of the following: image normalization, resizing, format conversion, and data augmentation. Image normalization normalizes pixel values ​​from the range [0, 255] to [0, 1] or [-1, 1] (e.g., subtracting the mean or dividing by the standard deviation) to avoid affecting detection performance due to differences in pixel value magnitudes. Resizing scales or crops video frames to the input size specified by the detection model, ensuring tensor dimension consistency. Format conversion converts the RGB / BGR channel formats of video frames to the channel order required by the detection model, and converts the data format of image frames to the data format required by the detection model. Data augmentation performs operations such as rotation or brightness adjustment on video frames.

[0116] In some implementations, the video frames received by the detection unit are received sequentially. The detection unit 15 is configured to: add the preprocessed video frames to the processing queue according to the order in which the video frames arrive at the detection unit; and when the number of preprocessed video frames in the processing queue reaches a preset number, retrieve the preset number of video frames from the processing queue and generate a corresponding tensor set.

[0117] In some embodiments, the detection unit 15 is configured to: input a tensor set into a detection model, detect a preset number of video frames through the detection model, and obtain the first identification information corresponding to each video frame in the preset number of video frames output by the detection model.

[0118] In some implementations, the preset number is set according to the structure of the detection model, and its value can be any positive integer. For example, the preset number can be set to one of the following: 1, 2, 4, or 8.

[0119] In some implementations, the detection model can be a deep learning-based model for object detection or image recognition.

[0120] In some implementations, the detection model is deployed in an NPU (Neural Processing Unit). This leverages the NPU's hardware acceleration capabilities for neural network computations (such as parallel processing of convolution operations and activation functions) to significantly improve the inference efficiency of batch tensors. Compared to CPU or GPU processing, this effectively reduces the detection time for multiple video frames while minimizing processor resource consumption. It ensures that the video processing device maintains real-time performance in high-channel, high-frame-rate scenarios, providing efficient support for the rapid processing of subsequent loading units.

[0121] In some implementations, the output of the detection model includes two parts: a video frame and a first identification information corresponding to the video frame. Both the video frame and the first identification information contain the channel identifier of the corresponding channel. Based on the channel identifier, the detection unit can send the video frame and the corresponding first identification information to the loading unit corresponding to the channel to which the video frame belongs.

[0122] In this embodiment, a preset number of video frames are preprocessed into tensors and formed into a set. Then, batch inference capability is used to complete the detection of multiple video frames at once. Compared with the method of processing each video frame one by one, the video processing device only needs to load one detection model, instead of loading a detection model for each video stream. In this embodiment, multiple video streams share the detection model through batch processing, and the video memory usage is constant (e.g., 500MB supports 16 video streams). This can effectively reduce the repetitive overhead of model loading and parameter calling, and improve the utilization rate of computing power.

[0123] In some embodiments, such as Figure 3 As shown, the video processing device 100 further includes a decoding unit 16 configured for each video stream channel.

[0124] In some embodiments, each decoding unit 16 is configured to decode the compressed encoded frames in the video stream of the corresponding channel into video frames of a preset format.

[0125] In some embodiments, the decoding unit 16 can decode the compressed encoded frames in the video stream of the corresponding channel into video frames of a preset format through a hardware video processing unit (VPU).

[0126] In some examples, the default video frame format is either YUV or RGB. YUV format video frames represent each pixel using a luminance component (Y) and chrominance components (U and V). The luminance component reflects the brightness of the image, while the chrominance component reflects color information. RGB format video frames represent each pixel using the three primary color components: red, green, and blue.

[0127] In some embodiments, the decoding unit 16 is connected to the aforementioned loading unit 12 or the aforementioned detection unit 15.

[0128] In some examples, where the video processing device 100 does not include the aforementioned detection unit 15, the decoding unit 16 is connected to the aforementioned loading unit 12.

[0129] In other examples, when the video processing apparatus 100 includes the aforementioned detection unit 15, the decoding unit 16 is connected to the aforementioned detection unit 15.

[0130] In some embodiments, each decoding unit 16 is further configured to: send video frames to the loading unit 12 when connected to the aforementioned loading unit 12, and send video frames to the detection unit 15 when connected to the aforementioned detection unit 15.

[0131] When the decoding unit 16 is connected to the aforementioned detection unit 15, the aforementioned detection unit 15 preprocesses the received video frames to obtain the tensors corresponding to the video frames. This process can be executed by the VPU or by the NPU coprocessor.

[0132] In this embodiment of the disclosure, the decoding unit corresponding to each channel's video stream is configured to decode the video stream of the corresponding channel into video frames. Each decoding unit can independently perform hardware decoding on the compressed video stream (such as H.264 or H.265 format) of its own channel and output standardized format video frames, avoiding decoding delay caused by multiple channels sharing a single decoder and significantly improving the parallel processing capability of multi-channel video streams. In addition, each decoding unit is also configured to send the video frames to the loading unit or the detection unit, which can directly transmit the decoded video frames to the downstream module, eliminating the intermediate data copying link and reducing data transmission delay.

[0133] In some embodiments, the output module described above can output the stitched image through at least one mode. This at least one mode includes a streaming media encoding mode and a direct display mode via a multimedia interface. The streaming media encoding mode refers to the mode of converting the stitched image into a media format compatible with network streaming for output. This streaming media encoding mode is, for example, Real-Time Messaging Protocol (RTMP) mode or HTTP Live Streaming (HLS) mode, suitable for scenarios involving sharing or remotely playing video over a network. The direct display mode via a multimedia interface refers to the output mode of directly displaying the stitched image on a display device (such as a screen) using a multimedia interface.

[0134] This disclosure describes the process of an output module splicing images using a streaming media encoding mode through the following embodiments.

[0135] In some embodiments, the output module described above includes: a scheduling unit, a sending unit, and a video encoding unit.

[0136] The scheduling unit is configured to send the memory address or handle of the image stitching carrier to the video encoding unit in the first output mode. The handle is an abstract identifier allocated to the image stitching carrier by the operating system or driver. The first output mode indicates the aforementioned streaming media encoding mode.

[0137] In some implementations, the scheduling unit is configured to, in a first output mode, acquire the memory address or handle of the image stitching carrier and send the memory address or handle of the image stitching carrier to the video encoding unit. In some examples, the memory address of the image stitching carrier may indicate an address range from the start address to the end address of the image stitching carrier.

[0138] In some implementations, the scheduling unit is further configured to send the memory address of an output buffer to the video encoding unit in a first output mode. This output buffer is used to store the stitched image encoded by the video encoding unit.

[0139] In some implementations, the scheduling unit is further configured to send a first output instruction to the video encoding unit in a first output mode. This first output instruction is an instruction used to trigger the video encoding unit to perform an operation.

[0140] In some examples, the scheduling unit is configured to, in response to the output mode being set to the first output mode, sequentially execute: obtain the memory address of the image stitching carrier, send the memory address of the image stitching carrier to the video encoding unit, send the memory address of the output buffer to the video encoding unit, and send the first output instruction to the video encoding unit.

[0141] In some implementations, the video encoding unit is a unit consisting of an encoder driver and a video encoder (VENC). The scheduling unit is configured to send the memory address of the image stitching carrier or the memory address of the output buffer to the encoder driver, which is configured to write the memory address of the image stitching carrier or the memory address of the output buffer into a first register of the video encoder, the first register being used to store the memory address. Furthermore, the scheduling unit is also configured to send a first output instruction to the encoder driver, which writes the first output instruction into a second register of the video encoder, the second register being used to store control instructions.

[0142] The video encoding unit is configured to read pixel data from the image stitching carrier in memory address order according to a first output instruction to obtain a stitched image, and to perform encoding processing on the stitched image.

[0143] In some implementations, encoding processing is used to convert the stitched image (uncompressed pixel data) into a compressed bitstream suitable for storage or network transmission. This encoding processing may, for example, involve compression using one of the following encoding formats: H.264, H.265, or H.266.

[0144] In some implementations, the video encoding unit is configured to store the encoded stitched image into the aforementioned output buffer.

[0145] The transmitting unit is configured to transmit the encoded stitched image to the receiving end via a network.

[0146] In some embodiments, the transmitting unit is configured to encapsulate the encoded stitched image into a target format and then transmit it to the receiving end. The target format may be, for example, FLV format or TS / M3U8 format.

[0147] In some implementations, the receiving end is a terminal device, server, or display system with network reception and video decoding capabilities. The receiving end is, for example, one of the following: mobile phone, tablet computer, laptop computer, smart TV, TV box, back-end management server of security monitoring system, etc.

[0148] In some implementations, the transmission over a network is performed, for example, via a wired network or a wireless network. Wired networks include, for example, fiber optic cables or Ethernet cables; wireless networks include, for example, Bluetooth, Wi-Fi networks, or mobile communication networks.

[0149] In this embodiment, the output module includes a scheduling unit, a video encoding unit, and a sending unit. The scheduling unit is configured to send the memory address of the image stitching carrier to the video encoding unit. The video encoding unit is configured to read pixel data from the image stitching carrier in memory address order according to a first output instruction to obtain a stitched image, and to encode the stitched image. The sending unit is configured to send the encoded stitched image to the receiving end via a network. In this embodiment, by using the Direct Memory Access (DMA) function of the video encoding unit to directly read pixel data from the image stitching carrier and encode the pixel data, the CPU's involvement in the pixel data encoding process can be reduced, thereby reducing the occupation of memory bandwidth and improving encoding efficiency.

[0150] This disclosure describes the process of outputting spliced ​​images through a multimedia interface direct display mode via the following embodiments.

[0151] In some embodiments, the output module includes a scheduling unit and a display control unit.

[0152] The scheduling unit is configured to establish an overlay plane associated with the image splicing carrier in the display control unit in the second output mode. The second output mode refers to the aforementioned direct display mode of the multimedia interface.

[0153] In some implementations, the overlay plane is a separate hardware layer used to directly carry the stitched image data and participate in hardware composite display. This overlay plane includes the memory address of the image stitching carrier.

[0154] In some implementations, the scheduling unit is configured to, in a second output mode, acquire the memory address or handle of the image stitching carrier and send overlay plane configuration parameters to the display control unit to establish an overlay plane associated with the image stitching carrier in the display control unit.

[0155] In some examples, the overlay plane configuration parameters include at least: an overlay plane identifier, a storage format for pixel data in the image stitching carrier, and a memory address or handle of the image stitching carrier. The overlay plane identifier is a unique identifier used to distinguish multiple overlay planes within the display control unit. In this example, the overlay plane associated with the image stitching carrier can be the overlay plane indicated by the overlay plane identifier in the overlay plane configuration parameters. The storage format for pixel data in the image stitching carrier is, for example, at least one of the following: pixel color representation (RGB or YUV), pixel depth, data arrangement, etc.; wherein, pixel depth is the number of bits occupied by each pixel, and the data arrangement is, for example, arranged in rows or in columns.

[0156] In other examples, the overlay plane configuration parameters may also include at least one of the following: overlay plane position, overlay plane size, and overlay plane transparency, wherein the overlay plane position indicates the position of the overlay plane in the screen coordinate system of the display terminal, usually with the top left corner of the screen as the origin, and is represented by (X-axis coordinate, Y-axis coordinate); the overlay plane size indicates the width and height of the overlay plane displayed on the screen of the display terminal, in pixels; and the overlay plane transparency indicates the degree of light transmission when the overlay plane is overlaid with other layers (such as the main plane, UI plane), usually expressed as an integer from 0 to 255 or a percentage from 0% to 100%.

[0157] In some implementations, the scheduling unit is further configured to send a second output command to the display control unit in the second output mode. This second output command is a command used to control the display control unit to perform operations.

[0158] In some examples, the scheduling unit is configured to, in response to the output mode being set to the second output mode, sequentially execute: obtain the memory address of the image stitching carrier, send the overlay plane configuration parameters to the display control unit, and send the second output command to the display control unit.

[0159] The display control unit is configured to read pixel data from the image stitching carrier in memory address order based on a second output command and an overlay plane associated with the image stitching carrier, obtain a stitched image, and send the stitched image to the display end through an output interface.

[0160] In some implementations, the output interface is, for example, an HDMI interface or a DisplayPort (DP).

[0161] In some implementations, the display control unit is also configured to send the stitched image to the display via an output interface after blending it with other layers.

[0162] In some implementations, the display control unit comprises at least one of a Direct Rendering Manager (DRM) driver and a Kernel Mode Setting (KMS) driver, a display controller, and an overlay plane. In this implementation, the scheduling unit sends overlay plane configuration parameters to the DRM driver and / or the KMS driver. The DRM driver and / or the KMS driver writes the overlay plane configuration parameters into the register of the overlay plane indicated by the overlay plane identifier to establish an association between the image stitching carrier and the overlay plane in the display control unit. In response to a second output instruction, the display controller reads pixel data from the image stitching carrier in memory address order based on the overlay plane associated with the image stitching carrier to obtain a stitched image. The display controller then writes the stitched image into the frame buffer of an output interface, such as an HDMI frame buffer. The display controller periodically accesses the frame buffer according to the screen refresh rate (e.g., 60Hz, reading one frame of the stitched image every 16.7ms), causing the stitched image to be displayed on the display.

[0163] In this embodiment of the disclosure, the display control unit directly reads the pixel data of the image stitching carrier in sequence based on the memory address of the image stitching carrier configured in the overlay plane. This eliminates the intermediate step of copying the stitched image from memory to the display buffer when the CPU controls the graphics synthesizer to output the image, thus avoiding the delay caused by data copying and improving the output efficiency of the stitched image.

[0164] The above embodiments provide a detailed description of the streaming media encoding mode and the direct display mode of the multimedia interface. In this embodiment, the image stitching carrier is located in shared memory. Each unit of the output module can directly access the shared memory to read pixel data from the image stitching carrier and obtain the stitched image. Thus, the output module can output the stitched image in parallel through multiple modes. Furthermore, when the output module needs to disable any output mode or switch output modes, only the hardware of the unit involved in the corresponding output mode can be configured. For example, when it is necessary to disable the output of the streaming media encoding mode, the video encoder can be disabled; when it is necessary to disable the output of the direct display mode of the multimedia interface, the display hardware path can be disabled through the display controller; when it is necessary to switch the direct display mode of the multimedia interface to the streaming media encoding mode, the video encoder can be enabled while disabling the display hardware path, without needing to perform pixel data migration or pixel data reprocessing to adapt to the corresponding output mode.

[0165] Figure 4 This is an example diagram of a stitched image provided in an embodiment of this disclosure. Figure 4 As shown, in a video processing scenario where the video processing device is used for road video surveillance, the stitched image 400 includes multiple road surveillance images, such as road surveillance images 401 to 406. The sizes of each road surveillance image can be the same or different. For example, the sizes of road surveillance image 401 and road surveillance image 402 are different. Road surveillance image 401 includes rectangular boxes 4011 and 4012 that identify the target image area.

[0166] Figure 5 This is a schematic diagram of the structure of a video processing apparatus provided in an embodiment of this disclosure. Figure 5 As shown, the video processing device 500 includes a video input module 501, a shared detection module 502, a hardware-accelerated image stitching module 503, a detection box centralized drawing module 504, and a multi-mode output module 505.

[0167] The video input module 501 includes the aforementioned multiple decoding units.

[0168] In some embodiments, the multi-channel video input module 501 is configured to receive compressed encoded frames from N video streams through N decoding units, and then decode the compressed encoded frames from the N video streams in parallel using a hardware decoder into video frames of a preset format. Each video frame includes a channel identifier of its respective channel.

[0169] In some embodiments, the multi-channel video input module 501 is used to send video frames to the input buffer of the shared detection module 502.

[0170] In some examples, the output buffer of the multi-channel video input module 501 and the input buffer of the shared detection module 502 can be the same buffer in shared memory. This reduces redundant copying of intermediate results between CPU memory and memory of different hardware units, thereby reducing memory bandwidth requirements and access latency.

[0171] The shared detection module 502 includes the aforementioned detection unit.

[0172] In some embodiments, the video frames arrive at the input buffer of the shared detection module 502 in a sequential order. The detection unit in the shared detection module 502 is used to add the preprocessed video frames to the processing queue according to the order of their arrival times. A detailed description of this preprocessing can be found in the foregoing embodiments, and will not be repeated here.

[0173] In some embodiments, the detection unit in the shared detection module 502 is used to retrieve the preset number of video frames from the queue to be processed when the number of preprocessed video frames in the queue to be processed reaches a preset number, generate a corresponding tensor set, input the tensor set into the detection model deployed on the NPU, and use the detection model to detect the preset number of video frames to obtain the first identification information corresponding to each video frame in the preset number of video frames output by the detection model.

[0174] In some embodiments, the detection unit in the shared detection module 502 is used to send the video frame and the first identification information corresponding to the video frame to the input buffer of the loading unit corresponding to the channel identification in the hardware accelerated image stitching module 503, according to the channel identifier of the channel to which the video frame belongs.

[0175] The hardware-accelerated image stitching module 503 includes the aforementioned multiple loading units and parameter maintenance module.

[0176] In some embodiments, the hardware-accelerated image stitching module 503 is used to manage and control the aforementioned image stitching carrier and the aforementioned image processing parameters throughout their entire lifecycle. For example, when the module starts up, the hardware-accelerated image stitching module 503 is used to request a contiguous block of physical memory from shared memory via a DMA memory allocation interface as the image stitching carrier. Furthermore, the hardware-accelerated image stitching module 503 is also used to modify the image processing parameters through a parameter maintenance module.

[0177] In some embodiments, the loading unit in the hardware-accelerated image stitching module 503 is used to obtain video frames from the input buffer and load the pixel data of the video frame into the logical storage area on the image stitching carrier corresponding to the channel to which the video frame belongs by calling the hardware overlay engine through the corresponding DMA channel.

[0178] In some embodiments, the loading unit in the hardware-accelerated image stitching module 503 is further configured to send the second identification information to the input buffer of the detection box set drawing module 504 when the first identification information of the video frame of the corresponding channel is converted into the second identification information.

[0179] The detection box drawing module 504 includes the aforementioned annotation unit.

[0180] In some embodiments, the annotation unit in the detection box drawing module 504 calls the dedicated drawing instructions of the NPU and submits them in a batch processing instruction to uniformly annotate the target image regions in video frames of multiple channels.

[0181] The multi-mode output module 505 is described in the output module of the previous embodiment, and will not be repeated here.

[0182] In this embodiment of the disclosure, the video processing device includes a video input module, a shared detection module, a hardware-accelerated image stitching module, a detection box centralized drawing module, and a multi-mode output module. It is a full-link hardware-accelerated video processing device built based on multiple dedicated hardware acceleration units in an edge system on a chip (SoC). This video processing device replaces the CPU in the video processing flow with hardware-level parallel processing. This not only reduces the impact of CPU performance bottlenecks on image processing but also achieves low latency (avoiding software serial scheduling and data migration delays to ensure real-time processing needs), low video memory usage (based on shared memory design and dynamic reuse of hardware parameters to avoid redundant caching of multiple modules), low bandwidth usage (dedicated hardware units support localized data transmission and burst transmission optimization, reducing bus bandwidth consumption), and low power consumption (the hardware acceleration unit only works when a task is triggered, significantly reducing energy consumption compared to full-load CPU operation, meeting the energy efficiency requirements of edge devices). Simultaneously, this video processing device supports parallel access and processing of multiple video streams (such as simultaneous decoding and splicing of 1 to 16 video streams). Combined with hardware-level detection accuracy optimization and lossless data processing across the entire link, it achieves high-precision image processing results. Ultimately, it improves image processing efficiency while ensuring high throughput (processing more frames of data per unit time) and high energy efficiency (outputting higher processing performance per watt of power consumption).

[0183] Figure 6 This is a schematic flowchart illustrating a video processing method provided in an embodiment of this disclosure. The video processing method is applied to an electronic device, which includes the aforementioned video processing apparatus. Figure 6 As shown, the video processing method includes steps 601 to 602.

[0184] In step 601, in response to receiving a video frame from the video stream of at least one channel, the pixel data of the video frame is loaded into the logical storage area on the image stitching carrier corresponding to the channel to which the video frame belongs.

[0185] In some embodiments, the image stitching carrier includes contiguous physical memory for providing corresponding logical storage areas for video frames in a video stream of multiple channels; the logical storage areas correspond one-to-one with the channels.

[0186] In some embodiments, the address segments included in each logical storage region do not overlap, and there are no address conflicts between logical storage regions. Furthermore, the sum of the address segments included in all logical storage regions covers all memory addresses from the start address to the end address of the contiguous physical memory, and there is no unallocated storage space in the contiguous physical memory of the image stitching carrier.

[0187] In some embodiments, the video stream for each channel consists of video frames, each video frame including pixel data and a channel identifier of its respective channel. When receiving video frames from multiple channels of video stream, the electronic device performs the step of loading the pixel data of the video frames into the logical storage area on the image stitching carrier corresponding to the channel to which the video frames belong in parallel, thereby improving the efficiency of pixel data loading.

[0188] In some embodiments, in response to receiving a video frame from a video stream of at least one channel, loading pixel data of the video frame into a logical storage area on the image stitching carrier corresponding to the channel to which the video frame belongs includes: in response to receiving a video frame from a video stream of at least one channel, loading pixel data of the video frame into a logical storage area on the image stitching carrier corresponding to the channel to which the video frame belongs, based on image processing parameters in a first state.

[0189] In some implementations, the image processing parameters include location information of the logical storage area corresponding to the channel to which the video frame belongs. These image processing parameters can be represented by arrays, where each array includes the location information of the logical storage area corresponding to one channel.

[0190] In one example, the location information of the logical storage area in the image processing parameters includes: the location information of the pixel area corresponding to the video frame to be written in the logical storage area within the stitched image. For example, this includes the horizontal and vertical coordinate offsets of the top-left pixel in this pixel area compared to the top-left pixel in the video frame to be written, as well as the width and height of the pixel area. It is understood that the coordinates of the top-left pixel in a typical video frame are (0, 0). Therefore, this location information can be represented not by coordinate offsets, but by the pixel coordinates of the top-left pixel in the corresponding pixel area of ​​the video frame within the stitched image.

[0191] In another example, the location information of the logical storage region in the image processing parameters includes: the starting address of the logical storage region and the address segment offset, which represents the byte difference between the starting addresses of adjacent address segments within the logical storage region. It is understood that if the logical storage region contains only one address segment, the address segment offset can be 0.

[0192] In some embodiments, in response to receiving a video frame from a video stream of at least one channel, loading the pixel data of the video frame into a logical storage area on an image stitching carrier corresponding to the channel to which the video frame belongs includes: in response to receiving a video frame from a video stream of a first channel, loading the pixel data of the video frame into a first address segment, wherein the first channel is one of a plurality of channels corresponding to the loading unit, and the first address segment is an address segment included in the logical storage area corresponding to the first channel, and the first address segment may be a continuous address segment or multiple non-contiguous address segments.

[0193] In some embodiments, in response to receiving a video frame from the video stream of the first channel, loading the pixel data of the video frame into a first address segment includes: in response to receiving a video frame from the video stream of the first channel, sequentially writing each row of pixel data from the video frame into the first address segment in row order. The row order includes row (horizontal) order or column (vertical) order; if the first address segment is multiple non-contiguous address segments, one row of pixel data is written into one address segment.

[0194] It is understandable that electronic devices load the pixel data of video frames into the logical storage area corresponding to the channel to which the video frame belongs on the image stitching carrier. Once the logical storage areas in the image stitching carrier are filled, the pixel data of video frames from multiple channels can be merged into the pixel data of one video frame, thereby achieving the effect of stitching video frames from multiple channels into one video frame.

[0195] In some embodiments, the image processing parameters further include image scaling information corresponding to the channel to which the video frame belongs. In response to receiving a video frame from at least one channel of the video stream, loading the pixel data of the video frame into a logical storage area on the image stitching carrier corresponding to the channel to which the video frame belongs includes: in response to receiving a video frame from the video stream of the corresponding channel, scaling the video frame based on the image scaling information; and loading the pixel data of the scaled video frame into the logical storage area on the image stitching carrier corresponding to the channel to which the video frame belongs. The scaling method may be, for example, one of the following: bilinear interpolation scaling, nearest neighbor interpolation scaling, or pixel sampling or filling based on a preset ratio.

[0196] In some implementations, the image processing parameters in the first state described above are stored in shared memory. This shared memory also stores image processing parameters in a second state. The first state indicates a readable state, and the second state indicates a writable state.

[0197] It is understandable that storing two copies of image processing parameters in different states in shared memory can ensure that reading and writing do not occur on the same copy of image processing parameters, thus avoiding read and write conflicts of image processing parameters without introducing synchronization locks.

[0198] In some implementations, image processing parameters in a first state can be represented by a first identifier, such as 1; image processing parameters in a second state can be represented by a second identifier, such as 0.

[0199] In step 602, pixel data is read from the image stitching carrier in memory address order to obtain the stitched image.

[0200] In some embodiments, reading pixel data from an image stitching carrier in memory address order to obtain a stitched image includes: reading pixel data from the image stitching carrier in ascending memory address order to obtain a stitched image. The ascending memory address order indicates the sequence from the start address to the end address of contiguous physical memory in the image stitching carrier.

[0201] In other embodiments, obtaining a stitched image by reading pixel data from the image stitching carrier in memory address order includes: reading pixel data from the image stitching carrier in descending memory address order. This ascending memory address order indicates the order from the end address to the start address of contiguous physical memory in the image stitching carrier.

[0202] It is understandable that by reading pixel data from the image stitching carrier in the order of memory addresses, the pixel data of each row or column in the stitched image can be read, thereby obtaining the stitched image.

[0203] In some embodiments, reading pixel data from an image stitching carrier in memory address order to obtain a stitched image includes: reading pixel data from the image stitching carrier in memory address order based on a first preset frequency to obtain a stitched image. The first preset frequency can be determined based on the acquisition frequency of video frames from multiple channels, or it can be determined based on a pre-configured display frequency of the stitched image. The first preset frequency is, for example, 30fps or 60fps.

[0204] In the video processing method provided in this embodiment, in response to receiving a video frame from at least one channel of video stream, the pixel data of the video frame is loaded into a logical storage area on an image stitching carrier corresponding to the channel to which the video frame belongs. The image stitching carrier includes contiguous physical memory for providing corresponding logical storage areas for video frames from multiple channels of video stream; each logical storage area corresponds one-to-one with a channel. Pixel data is read from the image stitching carrier in memory address order to obtain a stitched image. Thus, stitching processing of video frames from multiple channels can be achieved simply by loading and reading pixel data, eliminating the need for the CPU to perform software-level image stitching operations. This reduces the CPU's involvement in the image processing process, thereby reducing the impact of CPU performance bottlenecks on image processing and improving image processing efficiency.

[0205] In some embodiments, the video processing method further includes: modifying image processing parameters in a second state in response to a modification instruction; and, upon completion of the modification of the image processing parameters in the second state, performing a state transition between the image processing parameters in the second state and the image processing parameters in the first state. The modification instruction is used to indicate modification of the image processing parameters, such as modifying the location information of the logical storage area corresponding to at least one channel.

[0206] In some embodiments, the above-described state transition between image processing parameters in the second state and image processing parameters in the first state includes: converting image processing parameters in the second state to image processing parameters in the first state by converting a second identifier to a first identifier, and converting image processing parameters in the first state to image processing parameters in the second state by converting a first identifier to a second identifier.

[0207] In some embodiments, the video processing method further includes: in response to receiving compressed encoded frames from the video stream of at least one channel, decoding the compressed encoded frames from the video stream of the corresponding channel into video frames of a preset format. The preset format video frames are YUV format video frames or RGB format video frames.

[0208] In some embodiments, the video processing method described above further includes steps one to three.

[0209] In step one, in response to receiving first identification information corresponding to a video frame in the video stream of at least one channel, the first identification information is converted into second identification information corresponding to the image stitching carrier.

[0210] In some implementations, the first identification information includes the position information of the target image region within the video frame. The second identification information includes the position information of the pixel data of the target image region within the image stitching carrier.

[0211] In some implementations, the first identification information and the second identification information also include a target image region category for each target image region. The target image region category is used to indicate the object indicated by the target image region. For example, the first category indicates that the object indicated by the target image region is a person or animal, and the second category indicates that the object indicated by the target image region is a vehicle.

[0212] In step two, the second identifier information is cached.

[0213] In step three, the pixel data of the target image region in the image stitching carrier is annotated based on the second identification information of the video frames from multiple channels.

[0214] In this embodiment of the disclosure, only one instruction for annotation processing needs to be submitted to the system to uniformly annotate the target image region in video frames of multiple channels. It is not necessary to submit an instruction for each video frame of each channel to perform separate annotation processing of the target image region, thereby reducing the number of instructions for annotation processing submitted to the system and improving the efficiency of annotation processing.

[0215] Furthermore, the unified annotation targets the stitched images in the image stitching carrier. When the resolution of the stitched image is lower than the sum of the resolutions of multiple video frames, this unified annotation process can reduce the number of pixels that need to be processed during annotation, thereby further improving the efficiency of annotation.

[0216] In some embodiments, the annotation processing of pixel data of the target image region in the image stitching carrier based on the second identification information of video frames from multiple channels includes: annotating the pixel data of the target image region in the image stitching carrier based on a preset processing frequency and the second identification information of video frames from multiple channels. The preset processing frequency is the same as the first preset frequency for reading the stitched image but has a fixed phase difference. This fixed phase difference can be set according to the actual application and is not specifically limited in this embodiment. Thus, before acquiring the stitched image, the pixel data of the target image region in the image stitching carrier can be uniformly annotated.

[0217] In some embodiments, the second identification information based on multiple channels of video frames is used to annotate the pixel data of the target image region in the image stitching carrier. This includes: modifying the color of the pixel data of the target image region in the image stitching carrier to the target color based on the second identification information of multiple channels of video frames. In this way, the target image region can be identified using a color-coded frame.

[0218] In some examples, where the second identification information does not include the target image region category, the target color is a pre-defined color, such as any of the following: red, green, or blue.

[0219] In other examples, where the second identification information includes the target image region category, the target color is determined based on the target image region category. For example, the pixel data of the aforementioned first category of target image regions (corresponding to people or animals) is changed to red, and the pixel data of the aforementioned second category of target image regions (corresponding to vehicles) is changed to green. In this way, target image regions for different categories of objects can be identified with different colored boxes.

[0220] In some embodiments, the video processing method described above further includes: step four.

[0221] In step four, at least one video frame of at least one channel is detected to generate first identification information corresponding to each video frame in at least one video frame.

[0222] In some implementations, step four includes: performing image recognition processing on at least one video frame of at least one channel to determine a target image region in each of the at least one video frame; and generating first identification information corresponding to each video frame based on the target image region in each video frame.

[0223] In some embodiments, the aforementioned step 602, which involves reading pixel data from the image stitching carrier in memory address order to obtain a stitched image, includes: in a first output mode, reading pixel data from the image stitching carrier in memory address order to obtain a stitched image. The first output mode indicates a streaming media encoding mode. This streaming media encoding mode refers to a mode that converts the stitched image into a media format compatible with network streaming for output. This streaming media encoding mode is, for example, a Real-Time Messaging Protocol (RTP) mode or an HTTP live streaming mode, suitable for scenarios involving sharing or remotely playing video over a network.

[0224] In this embodiment, after obtaining the stitched image, the video processing method further includes: encoding the stitched image and sending the encoded stitched image to the receiving end via a network.

[0225] In some embodiments, the aforementioned step 602, which involves reading pixel data from the image stitching carrier in memory address order to obtain a stitched image, includes: establishing an association between the image stitching carrier and the overlay plane in a second output mode, wherein the overlay plane associated with the image stitching carrier includes the memory address of the image stitching carrier; the second output mode is indicated as a multimedia interface direct display mode. This multimedia interface direct display mode refers to an output mode in which the stitched image is directly displayed on a display device (such as a screen) via a multimedia interface.

[0226] In this embodiment, after obtaining the stitched image, the video processing method further includes sending the stitched image to the display end through an output interface.

[0227] For a detailed description of each step in the video processing method provided in this disclosure, please refer to the foregoing detailed description of the video processing apparatus; the video processing method will not be repeated here.

[0228] Figure 7 This is a structural block diagram of an electronic device provided in an embodiment of the present disclosure. For example, the electronic device 700 can be a terminal device such as a mobile phone, computer, digital broadcasting terminal, messaging device, tablet device, personal digital assistant, etc.; it can also be an edge device such as a gateway device, switch, router, etc.; or it can be a cloud device such as a server.

[0229] Reference Figure 7 The electronic device 700 may include one or more of the following components: processing component 702, memory 704, power supply component 706, multimedia component 708, audio component 710, input / output (I / O) interface 712, sensor component 714, and communication component 716.

[0230] Processing component 702 typically controls the overall operation of electronic device 700, such as operations associated with at least one of display, telephone call, data communication, camera operation, and recording operation. Processing component 702 may include one or more processors 720 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 702 may include one or more modules to facilitate interaction between processing component 702 and other components. For example, processing component 702 may include a multimedia module to facilitate interaction between multimedia component 708 and processing component 702.

[0231] Memory 704 is configured to store various types of data to support operation on electronic device 700. Examples of such data include at least one of the following: instructions for any application or method operating on electronic device 700, contact data, phonebook data, messages, pictures, and videos. Memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0232] Power supply component 706 provides power to various components of electronic device 700. Power supply component 706 may include at least one of the following: a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 700.

[0233] Multimedia component 708 includes a screen that provides an output interface between electronic device 700 and user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen may be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 708 includes a front-facing camera and / or a rear-facing camera. When electronic device 700 is in an operating mode, such as a shooting mode or video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0234] Audio component 710 is configured to output and / or input audio signals. For example, audio component 710 includes a microphone (MIC) configured to receive external audio signals when electronic device 700 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 704 or transmitted via communication component 716. In some embodiments, audio component 710 also includes a speaker for outputting audio signals.

[0235] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, and buttons. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0236] Sensor assembly 714 includes one or more sensors for providing state assessments of various aspects of electronic device 700. For example, sensor assembly 714 may detect the on / off state of electronic device 700, the relative positioning of components such as the display and keypad of electronic device 700, changes in position of electronic device 700 or one of its components, the presence or absence of user contact with electronic device 700, orientation or acceleration / deceleration of electronic device 700, and temperature changes of electronic device 700. Sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 714 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 714 may also include, but is not limited to, at least one of the following: an accelerometer, a gyroscope, a magnetometer, a pressure sensor, and a temperature sensor.

[0237] Communication component 716 is configured to facilitate wired or wireless communication between electronic device 700 and other devices. Electronic device 700 can access wireless networks based on communication standards, such as Wi-Fi, 4G, 5G, or combinations thereof. In one exemplary embodiment, communication component 716 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 716 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), and other technologies.

[0238] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0239] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including executable instructions or a computer program, which can be executed by a processor 720 of an electronic device 700 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0240] A non-transitory computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform any of the video processing methods described above in the embodiments of this disclosure.

[0241] This disclosure provides a computer program product comprising a computer program or executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer program or executable instructions from the computer-readable storage medium and executes the computer program or executable instructions, causing the computer device to perform any of the video processing methods described above in this disclosure.

[0242] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0243] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A video processing apparatus, characterized in that, The device includes: an image stitching carrier, multiple loading units, and an output module; the image stitching carrier includes continuous physical memory for providing corresponding logical storage areas for video frames in multiple channels of video streams, and the logical storage areas correspond one-to-one with the channels; Each of the loading units is configured to, in response to receiving a video frame from the video stream of the first channel, load the pixel data of the video frame of the first channel into the logical storage area on the image stitching carrier corresponding to the first channel; the first channel is one of the plurality of channels corresponding to the loading unit. The output module is configured to read the pixel data from the image stitching carrier in memory address order to obtain the stitched image.

2. The video processing apparatus according to claim 1, characterized in that, The loading unit is further configured to: in response to receiving first identification information corresponding to a video frame of the first channel, convert the first identification information into second identification information corresponding to the image stitching carrier, and cache the second identification information; wherein the first identification information includes the position information of the target image region in the video frame; and the second identification information includes the position information of the pixel data of the target image region in the video frame in the image stitching carrier. The device further includes: a labeling unit; The annotation unit is configured to annotate the pixel data of the target image region in the image stitching carrier based on the second identification information of video frames from multiple channels.

3. The video processing apparatus according to claim 2, characterized in that, The device further includes: a detection unit connected to the plurality of loading units; the detection unit is configured to: Detect at least one video frame of at least one channel and generate first identification information corresponding to each video frame in the at least one video frame; Each video frame and its corresponding first identifier information are sent to the loading unit corresponding to the channel to which each video frame belongs.

4. The video processing apparatus according to claim 3, characterized in that, The detection unit is configured as follows: The received video frames are preprocessed to obtain the tensors corresponding to the video frames; For each preset number of video frames received, a set of tensors corresponding to the preset number of video frames is obtained based on the tensors corresponding to the video frames. Based on the tensor set, the preset number of video frames are detected, and first identification information corresponding to each video frame in the preset number of video frames is generated.

5. The video processing apparatus according to claim 3, characterized in that, The device further includes: a decoding unit configured for each video stream channel; the decoding unit is connected to the loading unit or the detection unit; Each of the decoding units is configured to decode the compressed encoded frames in the video stream of the corresponding channel into video frames of a preset format, and to send the video frames to the loading unit or the detection unit.

6. The video processing apparatus according to claim 1, characterized in that, The output module includes: a scheduling unit, a video encoding unit, and a sending unit; The scheduling unit is configured to send the memory address of the image stitching carrier to the video encoding unit in the first output mode; The video encoding unit is configured to read the pixel data from the image stitching carrier in memory address order to obtain a stitched image, and to perform encoding processing on the stitched image; The transmitting unit is configured to transmit the encoded stitched image to the receiving end via a network.

7. The video processing apparatus according to claim 1, characterized in that, The output module includes a scheduling unit and a display control unit; The scheduling unit is configured to: in the second output mode, establish an association between the image stitching carrier and the overlay plane in the display control unit; the overlay plane associated with the image stitching carrier is configured with the memory address of the image stitching carrier; The display control unit is configured to read pixel data from the image stitching carrier in memory address order based on the overlay plane associated with the image stitching carrier to obtain a stitched image, and to send the stitched image to the display end through an output interface.

8. The video processing apparatus according to claim 1, characterized in that, Each of the loading units is configured as follows: In response to receiving a video frame from the video stream of the corresponding channel, the pixel data of the video frame is loaded into the logical storage area on the image stitching carrier corresponding to the channel to which the video frame belongs, based on the image processing parameters in the first state. The first state is used to indicate a readable state; the image processing parameters in the first state include the location information of the logical storage area corresponding to the channel to which the video frame belongs.

9. The video processing apparatus according to claim 8, characterized in that, The device further includes: a parameter maintenance module; the parameter maintenance module is configured to: In response to a modification command, the image processing parameters in the second state are modified; the second state is used to indicate a writable state. When the image processing parameters in the second state are modified, the image processing parameters in the second state and the image processing parameters in the first state are switched.

10. A video processing method, characterized in that, The method includes: In response to receiving a video frame from a video stream of at least one channel, the pixel data of the video frame is loaded into a logical storage area on an image stitching carrier corresponding to the channel to which the video frame belongs; the image stitching carrier includes contiguous physical memory for providing corresponding logical storage areas for video frames in video streams of multiple channels; the logical storage area corresponds one-to-one with the channel; The pixel data is read from the image stitching carrier in memory address order to obtain the stitched image.

11. The video processing method according to claim 10, characterized in that, The method further includes: In response to receiving first identification information corresponding to a video frame in a video stream from at least one channel, the first identification information is converted into second identification information corresponding to the image stitching carrier; wherein, the first identification information includes the position information of the target image region in the video frame; and the second identification information includes the position information of the pixel data of the target image region in the video frame in the image stitching carrier. Cache the second identification information; The pixel data of the target image region in the image stitching carrier is annotated based on the second identification information of video frames from multiple channels.

12. The video processing method according to claim 11, characterized in that, The method further includes: Detect at least one video frame of at least one channel and generate first identification information corresponding to each video frame in the at least one video frame.

13. The video processing method according to claim 12, characterized in that, The step of reading the pixel data from the image stitching carrier in memory address order to obtain the stitched image includes: In the first output mode, the pixel data is read from the image stitching carrier in memory address order to obtain the stitched image; The method further includes: The stitched image is encoded. The encoded and stitched image is sent to the receiving end via the network.

14. The video processing method according to claim 12, characterized in that, The step of reading the pixel data from the image stitching carrier in memory address order to obtain the stitched image includes: In the second output mode, the association between the image stitching carrier and the overlay plane is established; the overlay plane associated with the image stitching carrier includes the memory address of the image stitching carrier; Based on the second output instruction and the overlay plane associated with the image stitching carrier, the pixel data is read from the image stitching carrier in memory address order to obtain the stitched image; The method further includes: The stitched image is sent to the display device through the output interface.

15. An electronic device, characterized in that, include: processor; Memory used to store computer programs or instructions; The processor executes the computer program or instructions to implement the steps of the method according to any one of claims 10 to 14.

16. A computer-readable storage medium storing a computer program or instructions, characterized in that, When the computer program or instructions in the storage medium are executed by a processor, the steps of the method described in any one of claims 10 to 14 are implemented.

Images