Video image processing method, device and storage medium for unmanned aerial vehicle gimbal camera

Abstract

The present application proposes a video image processing method, device and storage medium of a drone gimbal camera, including: collecting multiple first YUV images; establishing RTSP protocol communication between a client and a gimbal camera; in response to obtaining a lens switching instruction sent by the client, obtaining a corresponding first YUV image according to the lens switching instruction; in response to the original resolution of the obtained first YUV image being different from the target resolution: scaling the first YUV image to a preset resolution to obtain a second YUV image, creating a first memory area based on the preset resolution, and using an IVE operation to move and save the second YUV image to the first memory area; creating a second memory area based on the target resolution, creating a blank YUV image of the target resolution and saving it to the second memory area, using an IVE operation to set the YUV value of the blank YUV image to a black pixel value to obtain a filled YUV image; using an IVE operation to move the second YUV image to cover the filled YUV image to obtain a first target YUV image.

Description

Technical Field

[0001] The present application relates to the technical field of video image processing, and in particular to a video image processing method, device and storage medium for a drone gimbal camera. Background Art

[0002] With the continuous development of drone technology, the gimbal camera carried by drones usually includes multiple lenses for shooting different video images of the same scene, such as zoom lens, fixed focus lens, thermal imaging lens, etc. The video images shot by the gimbal camera are finally transmitted back to the remote controller via the image transmission link. Users can switch between multiple lenses on the remote controller and watch different video images in real time.

[0003] However, due to the different aspect ratios of the video images shot by different lenses, it is impossible to fully adapt the display ratio of the remote control display screen. In this regard, the prior art generally has the following two solutions:

[0004] The video image is scaled to a certain extent and then forcibly stretched to fill the entire display screen. However, this solution causes the video image to be deformed, affecting the user's video viewing experience.

[0005] The video image is scaled to a certain extent, and then the black border area is filled in the width / height direction of the video image by calculating the size of the video image and the display position on the display screen, so that the video image fills the entire display screen. Although this solution can solve the problem of video image deformation, it still has the following problems: On the one hand, this solution requires complex calculations and resource scheduling through the processor, resulting in high latency and slow image output. Users will experience obvious freezes when switching lenses, so it is not suitable for real-time switching of multiple video images; on the other hand, this solution is implemented on the device side and can only be displayed in real time on the device side. It cannot save and transmit the processed images. Summary of the invention

[0006] In order to solve the above technical problems, the present application proposes a video image processing method, device and storage medium for a drone gimbal camera. Through low-latency video image processing, multiple video streams are kept at the same aspect ratio without affecting the user's lens switching viewing effect and experience.

[0007] According to the first aspect of the present application, a video image processing method of a drone gimbal camera is proposed, comprising:

[0008] Use multiple lenses of the drone gimbal camera to collect multiple first yuv images in real time;

[0009] Establish RTSP protocol communication between the client and the PTZ camera to obtain instructions from the client;

[0010] In response to obtaining a lens switching instruction sent by the client, obtaining the first YUV image corresponding to the lens switching instruction according to the lens switching instruction, wherein the lens switching instruction includes a lens ID;

[0011] In response to the acquired original resolution of the first YUV image being in different ratios to the target resolution, the following operations are performed:

[0012] scaling the first yuv image to a preset resolution smaller than the target resolution to obtain a second yuv image, creating a first memory area in a physical memory based on the preset resolution, and using an ive operation to move and save the second yuv image to the first memory area;

[0013] Based on the target resolution, a second memory area is created in the physical memory, a blank YUV image of the target resolution is created and saved in the second memory area, and a YUV value of the blank YUV image is set to a black pixel value by using an IVE operation to obtain a filled YUV image;

[0014] Use the IVE operation to transfer the second YUV image to cover the filling YUV image to obtain the first target YUV image.

[0015] Preferably, the method further comprises:

[0016] In response to the acquired original resolution of the first YUV image being in the same ratio as the target resolution, the following operations are performed:

[0017] The original resolution of the first YUV image is scaled to the target resolution to obtain a second target YUV image, a third memory area is created in the physical memory based on the target resolution, and the second target YUV image is moved and saved to the third memory area using an IVE operation.

[0018] Preferably, the lens switching instruction further includes a memory address and first coding information, and the method further includes:

[0019] Find the corresponding memory area through the memory address, and send the target YUV image in the memory area to the first VENC encoding module;

[0020] The first VENC encoding module encodes the target YUV image according to the first encoding information and stores it in the same local MP4 file;

[0021] Among them, for different target YUV images, the first VENC module uses the same parameters.

[0022] Preferably, the lens switching instruction further includes second coding information, and the method further includes:

[0023] Find the corresponding memory area through the memory address, copy the target yuv image in the memory area and send it to the second venc encoding module;

[0024] The second venc encoding module encodes the target yuv image according to the second encoding information and streams it to the client through the RTSP protocol;

[0025] Among them, for different target YUV images, the second VENC module uses the same parameters and pushes the streams to the client through the same RTSP channel.

[0026] Preferably, scaling the first YUV image to a preset resolution smaller than the target resolution comprises:

[0027] When the width or height of the first yuv image is in the same ratio as the width or height of the target resolution, the width or height of the first yuv image is scaled to be the same as the width or height of the target resolution.

[0028] Preferably, the first encoding information is in H264 encoding format, and the second encoding information is in H265 encoding format.

[0029] Preferably, hardware DMA is used to implement the ive operation.

[0030] Preferably, the multiple lenses include at least two or a combination of more than one of a zoom lens, a fixed-focus lens and a thermal imaging lens.

[0031] According to the second aspect of the present application, a video image processing device for a drone gimbal camera is proposed, comprising:

[0032] A collection module configured to collect multiple first YUV images in real time using multiple lenses of a drone gimbal camera;

[0033] A switching module is configured to establish RTSP protocol communication between the client and the PTZ camera, and obtain instructions from the client; in response to obtaining a lens switching instruction sent by the client, obtain the corresponding first YUV image according to the lens switching instruction, wherein the lens switching instruction includes a lens ID;

[0034] The processing module is configured to perform the following operations in response to the original resolution of the first YUV image being acquired and the target resolution being in different ratios:

[0035] scaling the first yuv image to a preset resolution smaller than the target resolution to obtain a second yuv image, creating a first memory area in a physical memory based on the preset resolution, and using an ive operation to move and save the second yuv image to the first memory area;

[0036] Based on the target resolution, a second memory area is created in the physical memory, a blank YUV image of the target resolution is created and saved in the second memory area, and a YUV value of the blank YUV image is set to a black pixel value by using an IVE operation to obtain a filled YUV image;

[0037] Use the IVE operation to transfer the second YUV image to cover the filling YUV image to obtain the first target YUV image.

[0038] According to a third aspect of the present application, a computer-readable storage medium is proposed, on which a computer program is stored. When the program is executed by a processor, a video image processing method for a drone gimbal camera as provided in any embodiment of the first aspect is implemented.

[0039] The present application proposes a video image processing method, device and storage medium for a drone gimbal camera, which has at least the following beneficial effects:

[0040] For the first YUV image with different aspect ratios captured by multiple lenses, it is first scaled to a second YUV image with a preset resolution, and then a black pixel-filled YUV image with a target resolution is created through an IVE operation based on hardware DMA. The second YUV image is then transferred and covered to the filled YUV image to obtain the final first target YUV image, and the first target YUV image adaptively generates a black border area. This solution does not require the calculation of the display position of the second YUV image, nor does it require a large amount of resource scheduling through the processor, and can adaptively generate a black border area for the second YUV image, keeping multiple video streams at the same aspect ratio, and the IVE operation is completed using hardware DMA. The entire transfer process can be completed within 1ms, and the lens output has the advantages of low computing power, low latency, and fast response, thereby improving the user's lens switching viewing effect in the future.

[0041] For video storage, the target YUV image is encoded using the more compatible H264 encoding format, and since the multiple target YUV images have a unified aspect ratio, they can be encoded under the same parameters through the first VENC module and saved in the same MP4 file, which solves the problem of having to call multiple target YUV images separately and encode and save them in different MP4 files, resulting in multiple file synchronization, management difficulties, and large memory usage.

[0042] For video display streaming, the target YUV image is encoded using the H265 encoding format with higher compression rate and video quality. And because the multiple target YUV images have a unified aspect ratio, they can be encoded under the same parameters through the second VENC module. When the user switches the lens, the second VENC module does not need to be restarted, which greatly improves the switching speed. At the same time, multiple target YUV images can be streamed through the same RTSP channel, and RTSP does not need to be restarted, which further improves the switching speed. Users on the client will not notice any video freezes, which improves the user's lens switching viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and are used together with the description to explain the principles of the present invention. It will be easy to recognize other embodiments and many expected advantages of the embodiments because they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale with each other. The same reference numerals refer to corresponding similar parts.

[0044] Figure 1 is a flow chart of a video image processing method of a drone gimbal camera according to a specific embodiment of the present application;

[0045] Figure 2 is a schematic diagram of generating a first target yuv image according to the first specific embodiment of the present application;

[0046] Figure 3 is a schematic diagram of generating a first target yuv image according to a second specific embodiment of the present application;

[0047] Figure 4 is a schematic diagram of generating a first target yuv image according to the third specific embodiment of the present application;

[0048] Figure 5 is a complete framework diagram of video image processing of a drone gimbal camera according to a specific embodiment of the present application;

[0049] Figure 6 It is a flow chart of a method for saving multiple target YUV images to the same MP4 file according to a specific embodiment of the present application;

[0050] Figure 7 It is a flow chart of a method for real-time switching and streaming of multiple target YUV images to a client according to a specific embodiment of the present application;

[0051] Figure 8 It is an architecture diagram of a video image processing device for a drone gimbal camera according to a specific embodiment of the present application.

[0052] Description of the accompanying drawings: 21, first YUV image; 22, second YUV image; 23, blank YUV image; 24, filled YUV image; 25, first target YUV image. DETAILED DESCRIPTION

[0053] The features and exemplary embodiments of various aspects of the present invention will be described in detail below. In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and Examples. It should be understood that the specific embodiments described herein are only configured to explain the present invention and are not configured to limit the present invention. For those skilled in the art, the present invention can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present invention by illustrating examples of the present invention.

[0054] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0055] The present application proposes a video image processing method for a drone gimbal camera. Figure 1 FIG. 1 shows a flow chart of a video image processing method for a drone gimbal camera according to a specific embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0056] Step S11: Use multiple lenses of the drone gimbal camera to collect multiple first YUV images in real time.

[0057] In this embodiment, the multiple lenses include a zoom lens, a fixed focus lens, and a thermal imaging lens. In other implementations, the multiple lenses may also include other types of lenses, and the multiple lenses may also be a combination of two or more of a zoom lens, a fixed focus lens, and a thermal imaging lens.

[0058] It should be noted that, among the zoom lens, fixed focus lens and thermal imaging lens of this embodiment, at least two of them have different aspect ratios and resolutions, so it is necessary to perform video image processing on the first YUV images of multiple lenses to unify the aspect ratio.

[0059] Step S12, establish RTSP protocol communication between the client and the PTZ camera, obtain the client's instruction, and in response to obtaining the lens switching instruction sent by the client, obtain the corresponding first YUV image according to the lens switching instruction, where the lens switching instruction includes a lens ID.

[0060] In this embodiment, the client is an app on the ground remote control. Since the transmission distance between the drone gimbal camera and the ground remote control is far, the stability of the network will fluctuate to a certain extent. In actual usage scenarios, it is necessary to control the transmission of video streams of multiple lenses by switching lenses. It is more dependent on real-time performance, controllability and anti-interference ability. Therefore, the RTSP protocol is adopted. RTSP is designed for real-time streaming, has unique advantages in streaming media transmission control, and can better cope with high-latency networks.

[0061] It should be understood that in other implementations, without considering the above factors, the PTZ camera and the remote controller may also use other protocols such as RTP, RTMP, WebRTC, etc. to communicate. For example, for the RTP protocol, it is only responsible for video data transmission and cannot participate in video stream control; for the RTMP protocol, when the network fluctuates, the quality of its video stream transmission is poor; for the WebRTC protocol, it is based on P2P transmission design and focuses more on real-time communication between browsers, which makes it difficult to guarantee transmission quality in a complex Internet environment.

[0062] Step S13, determine whether the original resolution of the acquired first YUV image is in the same ratio as the target resolution, if so, execute step S14; if not, execute steps S15-S17 in sequence.

[0063] Step S14: scale the original resolution of the first YUV image to the target resolution to obtain a second target YUV image, create a third memory area in the physical memory based on the target resolution, and use the IVE operation to move and save the second target YUV image to the third memory area.

[0064] In some embodiments, for example, the original resolution of the first YUV image is 3840*2160 and the target resolution is 1920*1080. At this time, the original resolution and the target resolution of the first YUV image are in the same ratio, so the resolution of the first YUV image is directly reduced to 1920*1080 to obtain the second target YUV image, and then the second target YUV image is saved to the created third memory area through the IVE (Intelligent Video Engine, video processing engine) operation.

[0065] It should be noted that when the original resolution of the first YUV image is smaller than the target resolution, for example, the original resolution of the first YUV image is 960*540 and the target resolution is 1920*1080, the resolution of the first YUV image is enlarged to 1920*1080.

[0066] Step S15: scale the first yuv image to a preset resolution smaller than the target resolution to obtain a second yuv image, create a first memory area in the physical memory based on the preset resolution, and use the ive operation to move and save the second yuv image to the first memory area.

[0067] In some embodiments, for example, the original resolution of the first YUV image is 2880*1800, and the target resolution is 1920*1080. At this time, the original resolution and the target resolution of the first YUV image are in different ratios, so the resolution of the first YUV image can be reduced to a preset resolution of 1440*900 to obtain a second target YUV image, and then the second target YUV image is saved to the created first memory area through the IVE operation.

[0068] In some embodiments, for example, the original resolution of the first YUV image is 3840*1800, and the target resolution is 1920*1080. At this time, the original resolution and the target resolution of the first YUV image are in different proportions, but the width values ​​are in the same proportion, so the resolution of the first YUV image can be reduced to a preset resolution of 1920*900 to obtain a second target YUV image, and then the second target YUV image is saved to the created first memory area through the IVE operation.

[0069] In some embodiments, for example, the original resolution of the first yuv image is 3072*2160, and the target resolution is 1920*1080. At this time, the original resolution and the target resolution of the first yuv image are in different proportions, but the height values ​​are in the same proportion, so the resolution of the first yuv image can be reduced to a preset resolution of 1536*1080 to obtain a second target yuv image, and then the second target yuv image is saved to the created first memory area through the ive operation.

[0070] It should be noted that when the original resolution of the first yuv image is already smaller than the target resolution, for example, 1280*1024, the first yuv image may not be modified, or the resolution of the first yuv image may be appropriately enlarged.

[0071] Step S16: create a second memory area in the physical memory based on the target resolution, create a blank yuv image of the target resolution and save it to the second memory area, use the ive operation to set the yuv value of the blank yuv image to the black pixel value, and obtain the filled yuv image.

[0072] Specifically, the blank yuv image is equivalent to a blank template of the multi-channel first yuv images with a uniform aspect ratio, and the filled yuv image obtained by filling black pixels through the ive operation is equivalent to the black background of the image.

[0073] Step S17: Use the IVE operation to transfer the second YUV image to cover the filling YUV image to obtain the first target YUV image.

[0074] Specifically, after the second YUV image is moved and covered to the filling YUV image, the original black pixel value of the covered area will be replaced with the real value of the second YUV image, while the uncovered area is equivalent to the black edge area of ​​the second YUV image adaptively generated, thereby obtaining the first target YUV image that can be displayed on the client.

[0075] Figure 2 A schematic diagram of generating a first target YUV image according to a first specific embodiment of the present application is shown. Figure 2As shown, the original resolution of the first YUV image 21 is in a different ratio to the target resolution. The first YUV image 21 with a resolution of 2880*1800 is reduced to a second YUV image 22 with a resolution of 1440*900, and then a blank YUV image 23 with a target resolution of 1920*1080 is created. The blank YUV image 23 is filled with a filled YUV image 24 of black pixels through an IVE operation. The second YUV image 22 is moved and covered to the filled YUV image 24 through an IVE operation, and the second YUV image 22 is displayed in the center of the filled YUV image 24, so as to obtain the final first target YUV image 25.

[0076] Figure 3 A schematic diagram of generating a first target YUV image according to a second specific embodiment of the present application is shown. Figure 3 As shown, the original resolution of the first YUV image 21 is in different proportions to the target resolution, but their width values ​​are in the same proportion. The first YUV image 21 with a resolution of 3840*1800 is reduced to a second YUV image 22 with a resolution of 1920*900, and then a blank YUV image 23 with a target resolution of 1920*1080 is created. The blank YUV image 23 is filled with a filled YUV image 24 of black pixels through an IVE operation. The second YUV image 22 is moved and covered to the filled YUV image 24 through an IVE operation, and the second YUV image 22 is displayed in the center of the filled YUV image 24, so as to obtain the final first target YUV image 25.

[0077] Figure 4 A schematic diagram of generating a first target YUV image according to a third specific embodiment of the present application is shown. Figure 4 As shown, the original resolution of the first YUV image 21 is in different proportions to the target resolution, but their height values ​​are in the same proportion. The first YUV image 21 with a resolution of 3072*2160 is reduced to a second YUV image 22 with a resolution of 1536*1080, and then a blank YUV image 23 with a target resolution of 1920*1080 is created. The blank YUV image 23 is filled with a filled YUV image 24 of black pixels through an IVE operation. The second YUV image 22 is moved and covered to the filled YUV image 24 through an IVE operation, and the second YUV image 22 is displayed in the center of the filled YUV image 24, so as to obtain the final first target YUV image 25.

[0078] In this embodiment, the second yuv image is moved and covered to the filling yuv image and then displayed in the center. In other implementations, the second yuv image can also be moved and covered to the preset display position of the filling yuv image.

[0079] It should be noted that in the prior art, to fill the black border area of ​​the image, it is usually necessary to first calculate the image size and the display position on the device side through the processor, and then perform resource scheduling based on the calculation results to add black borders to the upper, lower, left and right areas of the original image. The solution of the prior art has a large amount of calculation and high latency, and is only suitable for real-time display and streaming of single-lens video. For real-time display and streaming of multi-lens switching, there will be obvious stuttering; and drone gimbal cameras can usually only be equipped with low-computing power processors, which are difficult to meet the needs. Therefore, in this embodiment, the ive operation is implemented based on hardware DMA, and there is no need to calculate the display position of the image on the device side and perform resource scheduling. The entire process can be completed within 1ms, and the lens output has the advantages of low computing power, low latency, and fast response, and there will be no obvious stuttering when switching lenses later.

[0080] Figure 5 FIG. 4 shows a complete framework diagram of video image processing of a drone gimbal camera according to a specific embodiment of the present application. Figure 5 As shown, after unifying the aspect ratio of multiple first YUV images, the final multiple target YUV images are recorded and saved and pushed to the client in real time, wherein the lens switching instruction sent by the client to multiple lenses through the RTSP protocol also includes a memory address, a first encoding information and a second encoding information.

[0081] Figure 6 A flowchart of a method for saving multiple target YUV images to the same MP4 file according to a specific embodiment of the present application is shown. Figure 6 As shown, the method includes:

[0082] Step S31, find the corresponding memory area through the memory address, and send the target YUV image in the memory area to the first VENC encoding module.

[0083] Specifically, the memory address can be used to determine in which second memory area or which third memory area the first YUV image output by the current lens is stored after being adjusted to a uniform aspect ratio, thereby sending the first target YUV image of the corresponding second memory area or the second target YUV image of the corresponding third memory area to the first VENC encoding module.

[0084] Step S32: The first VENC encoding module encodes the target YUV image according to the first encoding information and stores it into the same local MP4 file, wherein the first VENC module uses the same parameters for different target YUV images.

[0085] Specifically, when the user switches lenses, since the multi-channel target YUV images have a unified aspect ratio, the multi-channel target YUV images can be encoded under the same parameters through the first VENC module and saved in the same MP4 file. This solves the problem that the multi-channel target YUV images need to be called separately and encoded and saved in different MP4 files, which causes multiple file synchronization, management difficulties, and large memory usage.

[0086] In this embodiment, the first encoding information adopts the H264 encoding format with higher compatibility, so that the saved video can be compatible with more hardware devices.

[0087] Figure 7 A flowchart of a method for real-time switching and streaming of multiple target YUV images to a client according to a specific embodiment of the present application is shown. Figure 7 As shown, the method includes:

[0088] Step 41, find the corresponding memory area through the memory address, copy the target YUV image in the memory area and send it to the second VENC encoding module.

[0089] Step 42, the second VENC encoding module encodes the target YUV image according to the second encoding information and pushes the image to the client through the RTSP protocol, wherein for different target YUV images, the second VENC module uses the same parameters and pushes the image to the client through the same RTSP channel.

[0090] Specifically, for video display streaming, since the multi-channel target YUV images have a unified aspect ratio, the multi-channel target YUV images can be encoded under the same parameters through the second VENC module. When the user switches the lens, the second VENC module does not need to be restarted, which greatly improves the switching speed. At the same time, the multi-channel target YUV images can be streamed through the same RTSP channel, and RTSP does not need to be restarted, which further improves the switching speed. Therefore, when the user switches the lens on the client, he will not notice any video screen freeze, which improves the user's lens switching viewing experience.

[0091] In this embodiment, the second encoding information adopts the H265 encoding format with higher compression rate and video quality, which ensures smoother video data transmission while ensuring video quality.

[0092] Based on the above-mentioned video image processing method of the drone gimbal camera, the present application also proposes a video image processing device for the drone gimbal camera, Figure 8 FIG. 1 shows an architecture diagram of a video image processing device for a drone gimbal camera according to a specific embodiment of the present application. Figure 8 As shown, the device comprises:

[0093] The acquisition module 100 is configured to use multiple lenses of the drone gimbal camera to acquire multiple first YUV images in real time.

[0094] The switching module 200 is configured to establish RTSP protocol communication between the client and the gimbal camera and obtain the client's instructions; in response to obtaining the lens switching instruction sent by the client, the corresponding first YUV image is obtained according to the lens switching instruction, and the lens switching instruction includes a lens ID, a memory address, a first encoding information and a second encoding information.

[0095] The processing module 300 is configured to perform the following operations in response to the original resolution of the acquired first YUV image being in different ratios to the target resolution:

[0096] The first yuv image is scaled to a preset resolution smaller than the target resolution to obtain a second yuv image, a first memory area is created in the physical memory based on the preset resolution, and the second yuv image is moved and saved to the first memory area using the ive operation; a second memory area is created in the physical memory based on the target resolution, a blank yuv image of the target resolution is created and saved to the second memory area, and the ive operation is used to set the yuv value of the blank yuv image to a black pixel value to obtain a filled yuv image; the ive operation is used to move the second yuv image to cover the filled yuv image to obtain the first target yuv image.

[0097] The video storage module 400 is configured to find the corresponding memory area through the memory address and send the target YUV image in the memory area to the first VENC encoding module; the first VENC encoding module encodes the target YUV image according to the first encoding information and stores it in the same local MP4 file, wherein the first VENC module uses the same parameters for different target YUV images.

[0098] The streaming module 500 is configured to find the corresponding memory area through the memory address, copy the target YUV image in the memory area and send it to the second VENC encoding module; the second VENC encoding module encodes the target YUV image according to the second encoding information and streams it to the client through the RTSP protocol, wherein for different target YUV images, the second VENC module uses the same parameters and streams them to the client through the same RTSP channel.

[0099] Based on the above-mentioned video image processing method of the drone gimbal camera, the present application also proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the video image processing method of the drone gimbal camera provided in any of the above-mentioned embodiments is implemented.

[0100] In the embodiments of the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device / system / method embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0101] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0102] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0103] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0104] Obviously, those skilled in the art can make various modifications and changes to the embodiments of the present invention without departing from the spirit and scope of the present invention. In this way, if these modifications and changes are within the scope of the claims of the present invention and their equivalents, the present invention is also intended to cover these modifications and changes. The word "comprising" does not exclude the presence of other elements or steps not listed in the claims. The simple fact that certain measures are recorded in mutually different dependent claims does not indicate that the combination of these measures cannot be used to profit. Any reference numerals in the claims should not be considered to limit the scope.

Claims

1. A video image processing method for a drone gimbal camera, characterized in that: include: Use multiple lenses of the drone gimbal camera to collect multiple first yuv images in real time; Establish RTSP protocol communication between the client and the PTZ camera to obtain instructions from the client; In response to obtaining a lens switching instruction sent by the client, obtaining the first YUV image corresponding to the lens switching instruction according to the lens switching instruction, wherein the lens switching instruction includes a lens ID; In response to the acquired original resolution of the first YUV image being in different ratios to the target resolution, the following operations are performed: Scaling the first YUV image to a preset resolution smaller than the target resolution to obtain a second YUV image, creating a first memory area in a physical memory based on the preset resolution, and using a video processing engine to move and save the second YUV image to the first memory area; Based on the target resolution, a second memory area is created in the physical memory, a blank YUV image of the target resolution is created and saved in the second memory area, and a YUV value of the blank YUV image is set to a black pixel value using a video processing engine operation to obtain a filled YUV image; The video processing engine is used to operate to transfer the second YUV image to cover the filling YUV image to obtain a first target YUV image.

2. The method according to claim 1, characterized in that The method further comprises: In response to the acquired original resolution of the first YUV image being in the same ratio as the target resolution, the following operations are performed: The original resolution of the first YUV image is scaled to the target resolution to obtain a second target YUV image, a third memory area is created in the physical memory based on the target resolution, and the second target YUV image is moved and saved to the third memory area using a video processing engine operation.

3. The method according to claim 2, characterized in that The lens switching instruction also includes a memory address and first coding information, and the method further includes: Find the corresponding memory area through the memory address, and send the target YUV image in the memory area to the first VENC encoding module; The first VENC encoding module encodes the target YUV image according to the first encoding information and stores it in the same local MP4 file; Among them, for different target YUV images, the first VENC encoding module uses the same parameters.

4. The method according to claim 3, characterized in that The lens switching instruction further includes second encoding information, and the method further includes: Find the corresponding memory area through the memory address, copy the target yuv image in the memory area and send it to the second venc encoding module; The second venc encoding module encodes the target yuv image according to the second encoding information and streams it to the client through the RTSP protocol; Among them, for different target YUV images, the second VENC encoding module uses the same parameters and pushes the stream to the client through the same RTSP channel.

5. The method according to claim 1, characterized in that The step of scaling the first YUV image to a preset resolution smaller than the target resolution includes: When the width or height of the first yuv image is in the same ratio as the width or height of the target resolution, the width or height of the first yuv image is scaled to be the same as the width or height of the target resolution.

6. The method according to claim 4, characterized in that The first encoding information is in H264 encoding format, and the second encoding information is in H265 encoding format.

7. The method according to claim 1, characterized in that Use hardware DMA to implement video processing engine operations.

8. The method according to claim 1, characterized in that The multiple lenses include at least a combination of two or more of a zoom lens, a fixed-focus lens and a thermal imaging lens.

9. A video image processing device for a drone gimbal camera, characterized in that: include: A collection module configured to collect multiple first YUV images in real time using multiple lenses of a drone gimbal camera; A switching module is configured to establish RTSP protocol communication between the client and the PTZ camera, and obtain instructions from the client; in response to obtaining a lens switching instruction sent by the client, obtain the corresponding first YUV image according to the lens switching instruction, wherein the lens switching instruction includes a lens ID; The processing module is configured to perform the following operations in response to the original resolution of the first YUV image being acquired and the target resolution being in different ratios: Scaling the first YUV image to a preset resolution smaller than the target resolution to obtain a second YUV image, creating a first memory area in a physical memory based on the preset resolution, and using a video processing engine to move and save the second YUV image to the first memory area; Based on the target resolution, a second memory area is created in the physical memory, a blank YUV image of the target resolution is created and saved in the second memory area, and a YUV value of the blank YUV image is set to a black pixel value using a video processing engine operation to obtain a filled YUV image; The video processing engine is used to operate to transfer the second YUV image to cover the filling YUV image to obtain a first target YUV image.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images