Image digital zooming method and device, unmanned aerial vehicle, electronic equipment and storage medium

Abstract

The invention relates to an image digital zooming method and device, an unmanned aerial vehicle, electronic equipment and a storage medium, the device comprises a de-jitter control module and at least two cutting-zooming modules, and the cutting-zooming modules are cascaded and connected with the de-jitter control module; the de-jitter control module receives the original information and determines cascade configuration information corresponding to each cutting-zooming module according to the original information; the de-jitter control module transmits each cascade configuration information to the corresponding cutting-zooming module; and each cutting-zooming module carries out zooming on an input cascade input image according to the received cascade configuration information and then outputs a cascade output image. According to the device provided by the invention, the visual jitter problem in a small-amplitude and high-magnification continuous digital zoom scene can be remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image digital zoom method, device, drone, electronic device and storage medium. Background Art

[0002] Digital image zoom is a basic function in many electronic imaging devices and multimedia software, and is widely used in mobile phones, security / surveillance, video conferencing, classroom observation / recording, robots, drones, commercial displays and other fields.

[0003] Many embedded platforms' media (image / video) processing modules provide hardware-accelerated image cropping and zooming operations, on which developers can implement digital image zooming, with the maximum magnification of up to 10 times. However, when zooming continuously at a small step and high magnification, there is often visually visible image jitter (position shaking), and the higher the magnification, the more obvious the jitter, resulting in poor visual perception.

[0004] Therefore, the prior art cannot effectively solve the image digital zoom jitter problem of the embedded platform. Summary of the invention

[0005] To solve the above problems, the present application provides an image digital magnification method, device, drone, electronic device and storage medium, which can be implemented on existing embedded platforms, significantly improving or even eliminating the problem of image jitter during small-step, high-magnification continuous digital magnification, thereby achieving a more precise and smooth continuous magnification effect and enhancing the visual experience.

[0006] This application adopts the following technical solutions:

[0007] In a first aspect, a method for digital zooming of an image is provided, the method being applied to a digital zooming device for an image, the device comprising at least two cropping-zooming modules and a de-shaking control module respectively connected to each cropping-zooming module, each cropping-zooming module being cascaded;

[0008] The method is performed by a de-jitter control module, and the method includes:

[0009] Receiving original information, and determining cascade configuration information corresponding to each cropping-scaling module according to the original information;

[0010] Each cascade configuration information is transmitted to the corresponding cropping-scaling module respectively, so that each cropping-scaling module zooms the input cascade input image according to the received cascade configuration information and then outputs the cascade output image.

[0011] Optionally, the original information includes: the size of the input image, the size of the target image, and the original cropping area of ​​the floating point value, the cascade configuration information includes: the size of the cascade input image, the size of the cascade output image, and the cascade cropping area of ​​the integer value, and the method includes:

[0012] Receiving the size of an input image and the size of a target image, and determining the size of a cascade input image and the size of a cascade output image corresponding to each cropping-scaling module according to the size of the input image and the size of the target image;

[0013] Receiving an original cropping area, and determining a cascade cropping area corresponding to each cropping-scaling module according to the original cropping area;

[0014] The size of each cascade input image, the size of each cascade output image and each cascade cropping area are summarized as cascade configuration information corresponding to each cropping-zooming module and transmitted to the corresponding cropping-zooming modules respectively, so that each cropping-zooming module can magnify the input cascade input image according to the received cascade configuration information and output the cascade output image.

[0015] Optionally, there are two cropping-scaling modules, and the size of the cascade input image and the size of the cascade output image corresponding to each cropping-scaling module are determined according to the size of the input image and the size of the target image, including:

[0016] Determine the size of the first cascade input image of the cropping-scaling module 1 according to the size of the input image, and determine the size of the second cascade output image of the cropping-scaling module 2 according to the size of the target image;

[0017] According to the size relationship between the input image size and the target image size, the size of the first cascade output image of the cropping-scaling module one and the size of the second cascade input image of the cropping-scaling module two are determined, wherein the size of the first cascade output image is equal to the size of the second cascade input image.

[0018] Optionally, determining the size of the first cascade output image of the cropping-scaling module 1 and the size of the second cascade input image of the cropping-scaling module 2 according to the size relationship between the size of the input image and the size of the target image includes:

[0019] When the size of the input image is not less than the size of the target image, determining that the size of the first cascade output image is equal to the size of the input image, so that the cropping-scaling module 1 performs image cropping and image enlargement operations on the first cascade input image, or does not perform image cropping and image scaling operations;

[0020] When the size of the input image is smaller than the size of the target image, it is determined that the size of the first cascade output image is larger than the size of the input image, so that the cropping-scaling module performs image cropping and image enlargement operations on the first cascade input image, or does not perform image cropping but performs image enlargement operations.

[0021] Optionally, determining the size of the first cascade input image of the cropping-scaling module 1 according to the size of the input image, and determining the size of the second cascade output image of the cropping-scaling module 2 according to the size of the target image, comprises:

[0022] The size of the input image is recorded as The size of the target image is recorded as Where W represents width and H represents height;

[0023] Then the size of the first cascade input image is The size of the second cascade output image

[0024] Optionally, determining the size of the first cascade output image of the cropping-scaling module 1 and the size of the second cascade input image of the cropping-scaling module 2 according to the size relationship between the size of the input image and the size of the target image includes:

[0025] Define the size of the first cascade output image as Define the size of the second cascade input image as Define the conversion coefficient as λ;

[0026] When the size of the input image is not less than the size of the target image, the conversion coefficient λ is determined to be 1, and according to the following formula Determine that the size of the first cascade output image is equal to the size of the input image;

[0027] When the size of the input image is smaller than the size of the target image, the conversion coefficient λ is determined to be greater than 1, and according to the following formula It is determined that the size of the first cascade output image is larger than the size of the input image.

[0028] Optionally, receiving an original cropping area, and determining, according to the original cropping area, cascade cropping areas corresponding to each cropping-scaling module, including:

[0029] According to the size of the input image, multiple anchor frames are preset, and the sizes of the multiple anchor frames decrease from 1 times the size of the input image in even multiples;

[0030] Compare the original cropped area with each anchor frame, and take the anchor frame that contains the original cropped area and whose size is closest to that of the original cropped area as the reference anchor frame;

[0031] The coordinates of the reference anchor frame are rounded and numerically aligned as the first cascade cropping area of ​​the cropping-scaling module 1;

[0032] Converting the original cropped area into a first coordinate in the reference anchor frame, and converting the first coordinate into a second coordinate in the first cascade output image according to a proportional relationship between the sizes of the reference anchor frame and the first cascade output image;

[0033] The second coordinate is rounded and numerically aligned to serve as the second cascade cropping area of ​​the cropping-scaling module 2.

[0034] Optionally, converting the original cropped area into a first coordinate in the reference anchor frame, and converting the first coordinate into a second coordinate in the first cascade output image according to a proportional relationship between the reference anchor frame and the size of the first cascade output image, comprises:

[0035] Define the original cropping area as [x * ,y * , w * ,h * ], where x * The horizontal coordinate of the reference point of the original clipping area, y * The ordinate of the reference point of the original cropping area, w * Indicates the width of the original cropping area, h * Indicates the height of the original clipping area;

[0036] Define the base anchor box as [x anchor ,y anchor , w anchor ,h anchor ], where x anchor Represents the horizontal coordinate of the reference point of the benchmark anchor box, y anchor The vertical coordinate of the reference point of the benchmark anchor box, w anchor Indicates the width of the reference anchor box, h anchor Indicates the height of the benchmark anchor box;

[0037] According to the following formula [x * -x anchor ,y * -y anchor , w * ,h * ]Determine the first coordinate;

[0038] According to the following formula [s x (x * -x anchor ), s y (y * -y anchor ), s x w * ,sy h * ]Determine the second coordinate, where represents the width of the input image, represents the height of the input image, and λ represents the conversion coefficient determined based on the relationship between the size of the input image and the size of the target image.

[0039] Optionally, the method further includes:

[0040] When the size of the second cascade cropping area is larger than the size of the target image, the cropping-scaling module 2 performs image cropping and image reduction operations on the second cascade input image;

[0041] When the size of the second cascade cropping area is smaller than the size of the target image, the cropping-scaling module 2 performs image cropping and image magnification operations on the second cascade input image;

[0042] When the size of the second cascade cropping region is equal to the size of the target image, the cropping-scaling module 2 performs an image cropping operation on the second cascade input image but does not perform an image scaling operation.

[0043] Optionally, the method includes:

[0044] Define the scaling factor in, Indicates the width of the target image, represents the height of the target image, w2 represents the width of the second cascade cropping area, and h2 represents the height of the second cascade cropping area;

[0045] When the size of the second cascade cropping area is larger than the size of the target image, determine the scaling ratio s * Less than 1;

[0046] When the size of the second cascade cropping area is smaller than the size of the target image, determine the scaling ratio s * Greater than 1;

[0047] When the size of the second cascade cropping area is equal to the size of the target image, the scaling factor s is determined. * Equal to 1.

[0048] In a second aspect, an image digital zoom device is provided, the device comprising: a de-shaking control module and at least two cropping-scaling modules;

[0049] Each cropping-scaling module is cascaded and connected to the de-jitter control module;

[0050] A de-jitter control module, used for receiving original information and determining cascade configuration information corresponding to each cropping-scaling module according to the original information;

[0051] A de-jitter control module, used for transmitting each cascade configuration information to the corresponding crop-scaling module respectively;

[0052] Each cropping-scaling module is used to scale the input cascade input image according to the received cascade configuration information and then output the cascade output image.

[0053] In a third aspect, the present application also provides a drone, which includes the above-mentioned image digital magnification device.

[0054] In a fourth aspect, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned image digital zoom method when executing the computer program.

[0055] In a fifth aspect, the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program implements the steps of the above-mentioned image digital magnification method when instructed by a processor.

[0056] At least one of the above technical solutions adopted in this application can achieve the following beneficial effects:

[0057] The device and method proposed in this application can significantly improve or even eliminate the visual jitter problem in small-step, high-magnification continuous digital zoom scenarios, and improve user experience, targeting situations where embedded platform GPUs are not convenient to use. The device and method provided in this application are easy to implement and do not increase hardware costs. They have a wide range of applications and can be expanded and enhanced based on existing software and hardware modules or system resources without changing hardware. Although compared to the prior art, the device provided in this application uses at least two cropping-zooming modules to expand the cropping-zooming operation from 1 stage to N stages, the total zoom ratio is equivalent and there is no excessive image magnification. Therefore, the zoom picture clarity obtained based on the device and method provided in this application can be basically consistent with the prior art, that is, there are no obvious side effects in terms of visual experience. In addition, compared to the existing 1-stage cropping-zooming operation, the expansion of this application to N stages can support higher zoom ratios while meeting hardware limitations. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0059] Figure 1 A schematic diagram showing the structure of a digital image zoom device in the prior art is shown;

[0060] Figure 2A schematic diagram showing the structure of an image digital zoom device according to an embodiment of the present application is shown;

[0061] Figure 3 A schematic diagram showing a flow chart of a method for digitally zooming an image according to an embodiment of the present application;

[0062] Figure 4 A schematic structural diagram of an image digital zoom device according to another embodiment of the present application is shown;

[0063] Figure 5 A schematic diagram showing the principle of a zoom ratio of 1.45x according to Example 1 of the present application;

[0064] Figure 6 A schematic diagram showing the principle of a zoom ratio of 2.76x according to Example 1 of the present application;

[0065] Figure 7 A schematic diagram showing the principle of a zoom ratio of 1.45x according to Example 2 of the present application;

[0066] Figure 8 A schematic diagram showing the principle of a zoom ratio of 9.76x according to Example 2 of the present application;

[0067] Fig. 9 A schematic structural diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0069] The digital image zoom device in the prior art is briefly introduced.

[0070] Many embedded platforms used to develop camera and media applications usually have hardware-supported cropping-zooming modules. The cropping-zooming module may include other image processing functions besides cropping and zooming. It is highly efficient and easy to integrate, so it is preferred when digital zooming is required.

[0071] Figure 1 The structure diagram of the digital image zoom device in the prior art is shown. Figure 1As shown, the image digital zoom device of the prior art has a cropping-zooming module as its core. The input of the cropping-zooming module includes: an input image and configuration information, and the output of the cropping-zooming module includes: a target image. The specific configuration information includes at least: the input image size, the target image size and the cropping area coordinates, and the cropping area coordinates are integer values. The cropping-zooming module is supported by hardware acceleration provided by the embedded platform and has high processing performance, but due to platform limitations, the cropping area coordinates usually need to be aligned to 2 (i.e., an even number). During the continuous digital zooming process, the cropping area coordinates will gradually change, and the input image size and the target image size generally remain unchanged.

[0072] use Figure 1 The digital image zoom device in the prior art shown in the figure performs digital image zooming. When zooming continuously with small steps and high magnification, there is often a visually visible picture jitter (position shake), and the higher the zoom ratio, the more obvious the jitter, resulting in poor visual perception.

[0073] The reason for the jitter of the zoom image is usually the loss of image sampling accuracy. The cropping area is calculated based on the zoom ratio, and its coordinates should be floating-point values, but when it is actually passed to the cropping-zooming module, it must be regularized into an integer value and aligned to 2, that is, it must be an even number (because the underlying processed images are mostly in YUV420 format, and odd coordinates are inconvenient to process), which directly leads to a maximum error of 1 or even 2 pixels in the cropping operation. The smaller the continuous zoom step, the smaller the change in the coordinates of the front and back cropping areas, the greater the proportion of the corresponding error, and the more severe the jitter. In addition, when the zoom ratio is large, the corresponding zoom operation is image magnification, which will further amplify the cropping error. The greater the image magnification ratio, the greater the final error amplification and the more severe the jitter.

[0074] That is to say, the existing technology is directly based on the cropping-zooming module provided by the platform, and cannot achieve fine and smooth continuous digital zooming.

[0075] In actual situations, for example, the cropping-scaling modules available on the SigmaStar platform include: VPE, DIVP, SCL, etc. The IVE module of the platform also provides a variety of common image processing operations, including image scaling operators. Therefore, based on custom cropping + IVE image scaling operators, a cropping-scaling execution unit can also be formed. If the configuration, resource usage, and performance overhead of the execution unit can meet business requirements, it can be used as an effective cropping-scaling module. The SigmaStar platform also has an LDC module, which is used for lens distortion correction itself and can also be used to complete the image cropping-scaling function when necessary.

[0076] For example, the cropping-scaling module available on the Hisi platform is VPSS, and its VGS module provides image scaling operations. Therefore, a cropping-scaling execution unit can also be formed based on custom cropping + VGS. If the configuration, resource usage, and performance overhead of the execution unit can meet business needs, it can also be used as an effective cropping-scaling module. In addition, similar to the SigmaStar platform, the Hisi platform also has an IVE module, and the image scaling operator based on custom cropping + IVE can also be used as a cropping-scaling module under appropriate circumstances. In addition, the LDC module of the Hisi platform can be used for image cropping and scaling in some cases.

[0077] Other similar embedded platforms provide some hardware-supported cropping-zooming modules, some modules can be connected in series (cascaded), or some modules can be time-division multiplexed, and the same modules can also be cascaded. Therefore, on many embedded platforms used to develop image digital zoom functions, there are usually more than one cropping-zooming module available.

[0078] In addition, the root cause of the jitter of the zoom image is the sampling error caused by rounding and aligning the input cropping area (platform limitation). When the zoom operation is image enlargement, the cropping error will be further amplified.

[0079] Based on this, the concept of the present application is that the coordinates of the input original cropping area are floating-point values ​​to ensure the numerical accuracy of the intermediate calculation process. At least two cascaded cropping-scaling modules are used to replace the existing single cropping-scaling module. When there are two cropping-scaling modules, the input image is cropped and enlarged by the cropping-scaling module one, or the image cropping and image scaling operations are not performed, and then the target area is cropped from the image output by the cropping-scaling module one by the cropping-scaling module two and scaled to the size of the target image to obtain the target image. In this process, the coordinates of the first cascade cropping area and the coordinates of the second cascade cropping area still comply with the platform restrictions, are rounded and numerically aligned.

[0080] The technical solutions provided by each embodiment of the present application are described in detail below.

[0081] Figure 2 The structure diagram of the digital image zoom device proposed in one embodiment of the present application is shown. Figure 2 The image digital zoom device proposed in the present application includes: a de-jitter control module and at least two cropping-zooming modules (respectively recorded as cropping-zooming module 1, cropping-zooming module 2...cropping-zooming module N), each cropping-zooming module is cascaded and connected to the de-jitter control module.

[0082] The de-jitter control module receives the original information, and after processing, outputs corresponding cascade configuration information (respectively recorded as configuration information 1, configuration information 2, ..., configuration information N) to each cascaded cropping-scaling module.

[0083] The original information at least includes the size of the input image, the size of the target image, and the original cropping area. In particular, the coordinates of the original cropping area in the original information are floating point values. The input image is used to indicate the original image before digital zooming, and the original image has a certain original size. The target image is used to indicate the desired output image after digital zooming, and the output image has an expected target size. The original cropping area is used to indicate the image content area that needs to be displayed when digital zooming is performed at a certain multiple, and the image content area has certain coordinates based on the zoom multiple.

[0084] In some actual scenarios, for example, when a user views an original image through an interactive display device, the user can zoom in / out on the original image by sliding his / her finger or selecting input. At this time, the displayed original image is the size of the input image, and the image displayed after zooming in / out is the size of the target image.

[0085] In other practical scenarios, for example, when a user views an original image through an interactive display device, he or she may need to focus on a certain image content area in the original image. The image content area that needs to be focused on is determined by selecting input or manually selecting a frame, and the image content area is the original cropped area. The size coordinates of the original cropped area are determined based on the zoom factor.

[0086] In more common scenarios, selecting a certain image content area and reducing / enlarging the image screen are performed in a superimposed manner. For example, when a user is viewing an original image, after selecting an image content area in the original image, the image content area is enlarged. At this time, the original image viewed by the user is the size of the input image, the enlarged image is the size of the target image, and the image content area is the original cropped area.

[0087] In each cascaded cropping-scaling module, the first cropping-scaling module (cropping-scaling module 1) receives an external input image, processes the input image according to the corresponding cascade configuration information (configuration information 1) received from the de-jitter control module, and sends the output processing result as input to the cropping-scaling module of the next level. Similarly, the last cropping-scaling module (cropping-scaling module N) processes the input according to the corresponding cascade configuration information (configuration information N) received from the de-jitter control module, and outputs the final target image.

[0088] The cascade configuration information at least includes the size of the cascade input image, the size of the cascade output image and the cascade cropping area. In particular, the coordinates of the cascade cropping area in the cascade configuration information are integer values ​​and are aligned (eg aligned to 2).

[0089] The cascade input image is used to indicate the image input to the corresponding crop-zoom module, which has a determined size of the cascade input image based on the output result of the previous cascade crop-zoom module. The cascade output image is used to indicate the image output from the corresponding crop-zoom module, which has the expected size of the cascade output image based on the requirements of the cascade configuration information transmitted to the corresponding crop-zoom module by the de-jitter control module. The cascade cropping area is used to indicate the cropping area corresponding to the corresponding crop-zoom module after the coordinate transformation of the original cropping area based on the size of the cascade input image corresponding to the crop-zoom module.

[0090] The functions implemented by a single crop-zoom module are basically the same as those of the crop-zoom module in the prior art. Each crop-zoom module can be partially or completely supported by hardware acceleration provided by the embedded platform, and has higher processing performance. As far as the intention of image cropping and scaling operations is concerned, each crop-zoom module can be regarded as a logical unit with equivalent functions, that is, there is no restriction on their actual structures being exactly the same. For example, each crop-zoom module can be two identical crop-zoom execution units provided / supported by the embedded platform, or the same crop-zoom execution unit for time-sharing multiplexing, or two different crop-zoom execution units.

[0091] In addition, each cropping-scaling module is not limited to being completely provided by the platform or hardware accelerated. That is, even if some parts of the cropping-scaling module are custom implemented (for example, in the actual execution unit of a cropping-scaling module, the cropping part is implemented by the developer or system software and runs on the CPU), as long as it does not affect the performance of its logical functions and the resource overhead and performance overhead meet the requirements, it can also be included.

[0092] The most important difference between the present application and the prior art is that a de-jitter control module is provided. The de-jitter control module processes the original information (including the original cropping area of ​​the floating point value) into the cascade configuration information corresponding to each cropping-zooming module (including the cascade cropping area of ​​the integer value), so that each cropping-zooming module can change the magnification according to the cascade configuration information.

[0093] The de-jitter control module can be used to implement the following image digital zoom method. Figure 3 The figure shows a flow chart of a method for digitally zooming an image according to an embodiment of the present application. Figure 3 The method includes steps S310 to S330.

[0094] Step S310 , receiving the size of the input image and the size of the target image, and determining the size of the cascade input image and the size of the cascade output image corresponding to each cropping-scaling module according to the size of the input image and the size of the target image.

[0095] Step S320: receiving an original cropping area, and determining, according to the original cropping area, cascade cropping areas corresponding to each cropping-scaling module.

[0096] In step S330, the size of each cascade input image, the size of each cascade output image, and the size of each cascade cropping area are summarized as cascade configuration information corresponding to each cropping-zooming module and transmitted to each corresponding cropping-zooming module respectively, so that each cropping-zooming module magnifies the input cascade input image according to the received cascade configuration information and outputs the cascade output image.

[0097] The de-jitter control module receives original information, which includes at least the size of the input image and the size of the target image. The input image is the original image before digital zooming, so its size is the determined original size. The target image is the expected output image after digital zooming, so its size is the expected target size.

[0098] According to the size of the input image and the size of the target image, the de-jitter control module determines the size of the cascade input image and the size of the cascade output image in the configuration information that needs to be transmitted to each crop-zoom module. In other words, the de-jitter control module first implements the function of configuring the size of the cascade input image and the size of the cascade output image of each crop-zoom module.

[0099] The de-jitter control module receives the original information, which at least includes the original cropping area, and the coordinates of the original cropping area are floating point values. The original cropping area is the image content area that needs to be displayed when digital zooming is performed at a certain multiple, so it has coordinates determined based on the zoom multiple. According to the original cropping area, the de-jitter control module determines the cascade cropping area in the configuration information that needs to be transmitted to each crop-zoom module. In other words, the de-jitter control module dynamically realizes the function of calculating the cascade cropping area of ​​each crop-zoom module.

[0100] The de-jitter control module transmits the calculated cascade configuration information corresponding to each cropping-scaling module to the corresponding cropping-scaling module. Each cropping-scaling module performs zoom processing on the image according to the corresponding cascade configuration information.

[0101] Depend on Figure 2 The device shown and Figure 3It can be seen from the method shown that the image digital zoom device provided by the present application adopts at least two cropping-zooming modules in cascade, and each cropping-zooming module is connected to the de-jitter control module. The image digital zoom method applied to the de-jitter control module provided by the present application receives the size of the input image and the size of the target image, and determines the size of the cascade input image and the size of the cascade output image corresponding to each cropping-zooming module according to the size of the input image and the size of the target image; receives the original cropping area, and determines the cascade cropping area corresponding to each cropping-zooming module according to the original cropping area; summarizes the size of each cascade input image, the size of each cascade output image and the size of each cascade cropping area into the cascade configuration information corresponding to each cropping-zooming module and transmits them to the corresponding cropping-zooming modules respectively, so that each cropping-zooming module zooms the input cascade input image according to the received cascade configuration information and then outputs the cascade output image.

[0102] In the prior art, in order to solve the problem of image jitter during digital zooming, there are conventionally the following solutions.

[0103] First, independently implement a cropping-scaling module whose input cropping area coordinates are floating-point values, and then control the sampling error through an appropriate difference method. However, for embedded platforms / hardware such as PCs and mobile phones used to develop digital zoom functions, many do not have GPU support, their CPU computing power is limited and it is difficult to perform hardware acceleration on custom modules, and the zoom image size is generally large, and the processing time is too long, which cannot meet the performance requirements. However, for such embedded platforms, the present application can significantly improve or even eliminate the visual jitter problem in small-step, high-magnification continuous digital zoom scenarios when GPU is not convenient to use, thereby improving the user experience. The device and method provided in the present application are easy to implement and do not increase hardware costs, have a wide range of applications, and can be based on existing software and hardware modules or system resources for functional expansion and enhancement without changing the hardware.

[0104] Second, increase the zoom speed. Although increasing the zoom speed can reduce the image jitter effect to a certain extent, it cannot completely eliminate it, and the viewing experience will be unnatural if the zoom speed is too fast. The device and method provided in this application can still provide a fine and smooth zoom effect even when the zoom speed is slow, thereby improving the visual experience.

[0105] Third, when the size of the target image is constant, try to increase the size of the input image as much as possible. However, the input image has an upper limit on its original size. When the input image has reached the upper limit of its original size, there is no room for optimization. Compared with the prior art, the device provided in the present application adopts at least two cropping-scaling modules to expand the cropping-scaling operation from 1 stage to N stages, but the total scaling ratio is equivalent and there is no excessive image magnification. Therefore, the clarity of the zoom image obtained based on the device and method provided in the present application can be basically consistent with the prior art, and there is no excessive requirement on the size of the input image, that is, there is no obvious side effect in terms of visual experience.

[0106] Fourth, the first-order crop-zoom operations of many embedded platforms have a limit on the magnification ratio. The lower the magnification ratio, the less obvious the jitter. For example, the Hisi platform VPSS and VGS modules support up to 16x magnification ratio. In the pursuit of higher-magnification digital zoom, limiting the upper limit of the magnification ratio is inconsistent with the demand for high-magnification products. Compared with the existing one-stage crop-zoom operation, the expansion of this application to N stages can support higher magnification ratios while meeting hardware limitations.

[0107] The device provided in the present application adopts at least two crop-zoom modules. More than two crop-zoom modules can support higher zoom ratios, and in some cases, the effect is better, but it will generate greater resource overhead, or there will be no more crop-zoom modules available. Therefore, in the absence of special necessity, two crop-zoom modules are commonly used and can usually meet the requirements. Therefore, two crop-zoom modules are described in the following, but it is not limited to two.

[0108] When there are two crop-scale modules, Figure 4 FIG. 2 is a schematic diagram showing the structure of an image digital zoom device proposed in another embodiment of the present application. Figure 4 The two cropping-scaling modules are respectively recorded as cropping-scaling module 1 and cropping-scaling module 2. The de-shaking control module receives the original information, and after processing, outputs configuration information 1 to the cropping-scaling module 1 and outputs configuration information 2 to the cropping-scaling module 2. The cropping-scaling module 1 and the cropping-scaling module 2 change the image magnification based on the corresponding cascade configuration information.

[0109] On this basis, the de-jitter control module can be used to achieve the following image digital zooming.

[0110] In some embodiments of the present application, in the above method, step S310, determines the size of the cascade input image and the size of the cascade output image corresponding to each cropping-scaling module according to the size of the input image and the size of the target image, including: determining the size of the first cascade input image of the cropping-scaling module one according to the size of the input image, and determining the size of the second cascade output image of the cropping-scaling module two according to the size of the target image; determining the size of the first cascade output image of the cropping-scaling module one and the size of the second cascade input image of the cropping-scaling module two according to the size relationship between the size of the input image and the size of the target image, wherein the size of the first cascade output image is equal to the size of the second cascade input image.

[0111] The de-jitter control module first configures the size of the cascade input image and the size of the cascade output image corresponding to the cropping-scaling module one and the cropping-scaling module two.

[0112] Specifically, the size of the first cascade output image of the cropping-scaling module one and the size of the second cascade input image of the cropping-scaling module two are determined according to the size relationship between the size of the input image and the size of the target image, including: when the size of the input image is not less than the size of the target image, the size of the first cascade output image is determined to be equal to the size of the input image, so that the cropping-scaling module one performs image cropping and image enlargement operations on the first cascade input image, or does not perform image cropping and image enlargement operations; when the size of the input image is smaller than the size of the target image, the size of the first cascade output image is determined to be larger than the size of the input image, so that the cropping-scaling module one performs image cropping and image enlargement operations on the first cascade input image, or does not perform image cropping but performs image enlargement operations.

[0113] The configuration information one corresponding to the cropping-scaling module one configured by the de-jitter control module needs to at least ensure that the cropping-scaling module does not perform an image reduction operation.

[0114] When the size of the input image is greater than or equal to the size of the target image, the size of the first cascade output image is equal to the size of the input image. At this time, if the size of the first cascade cropping area is smaller than the size of the input image, the cropping-scaling module 1 performs a cropping operation and performs an image zooming operation; if the size of the first cascade cropping area is equal to the size of the input image, the cropping-scaling module 1 does not perform a cropping operation and does not perform an image zooming operation.

[0115] When the size of the input image is smaller than the size of the target image, the size of the first cascade output image is larger than the size of the input image. At this time, if the size of the first cascade cropping area is smaller than the size of the input image, the cropping-scaling module 1 performs a cropping operation and performs an image magnification operation; if the size of the first cascade cropping area is equal to the size of the input image, the cropping-scaling module 1 does not perform a cropping operation but performs an image magnification operation.

[0116] Specifically, the size of the first cascade input image of the cropping-scaling module 1 is determined according to the size of the input image, and the size of the second cascade output image of the cropping-scaling module 2 is determined according to the size of the target image, including: defining the size of the input image as Define the size of the target image as Where W represents width and H represents height; then the size of the first cascade input image is The size of the second cascade output image

[0117] The de-shaking control module first converts the size of the first cascade input image of the cropping-scaling module into Configured to the size of the input image (known), the size of the second cascade output image of the crop-scaling module 2 Configured as the size of the target image (Known).

[0118] Specifically, according to the size relationship between the input image size and the target image size, determining the size of the first cascade output image of the cropping-scaling module 1 and the size of the second cascade input image of the cropping-scaling module 2 includes: defining the size of the first cascade output image as Define the size of the second cascade input image as Define the conversion coefficient as λ; when the size of the input image is not less than the size of the target image, determine the value of the conversion coefficient λ to be 1, and according to the following formula Determine that the size of the first cascade output image is equal to the size of the input image; when the size of the input image is smaller than the size of the target image, determine that the value of the conversion coefficient λ is greater than 1, and according to the following formula It is determined that the size of the first cascade output image is larger than the size of the input image.

[0119] Due to the cascade, the size of the first cascade output image The size of the second cascade input image The debounce control module configures both Among them, λ≥1.

[0120] The value of λ may be determined according to the relationship between the size of the input image and the size of the target image, as well as the corresponding resource overhead and performance consumption.

[0121] If the size of the input image is greater than or equal to the size of the target image, then λ=1 is taken. At this time, the size of the first cascade output image is equal to the size of the input image. When the size of the first cascade cropping area is equal to the size of the input image, the cropping-scaling module 1 neither performs image cropping nor image scaling. When the size of the first cascade cropping area is smaller than the size of the input image, the cropping-scaling module 1 performs image cropping and image magnification operations.

[0122] If the size of the input image is smaller than the size of the target image, λ takes a larger value greater than 1, but the corresponding resource overhead and performance consumption must meet the requirements. At this time, the size of the first cascade output image is larger than the size of the input image. Regardless of the size of the first cascade cropping area, the cropping-scaling module performs image magnification operation.

[0123] In some embodiments of the present application, in the above method, step S320, receiving the original cropping area, and determining the cascade cropping area corresponding to each cropping-scaling module according to the original cropping area, including: according to the size of the input image, presetting multiple anchor frames, and the sizes of the multiple anchor frames decrease from 1 times the size of the input image in an even multiple; comparing the original cropping area with each anchor frame, and taking the anchor frame that contains the original cropping area and the size of the anchor frame closest to the size of the original cropping area as the reference anchor frame; rounding the coordinates of the reference anchor frame and aligning the values ​​as the first cascade cropping area of ​​the cropping-scaling module one; converting the original cropping area into a first coordinate in the reference anchor frame, and according to the proportional relationship between the size of the reference anchor frame and the first cascade output image, converting the first coordinate into a second coordinate in the first cascade output image; rounding the second coordinate and aligning the values ​​as the second cascade cropping area of ​​the cropping-scaling module two.

[0124] The de-jitter control module then dynamically calculates the cascade cropping areas corresponding to the cropping-zooming module 1 and the cropping-zooming module 2. During the continuous digital zooming process, the de-jitter control module dynamically calculates the first cascade cropping area (integer value and numerical alignment) of the cropping-zooming module 1 and the second cascade cropping area (integer value and numerical alignment) of the cropping-zooming module 2 based on the input original cropping area (floating point value), thereby controlling the cropping-zooming module 1 and the cropping-zooming module 2.

[0125] Specifically, according to the size of the input image, multiple anchor frames are preset, and the sizes of the multiple anchor frames decrease in even multiples starting from 1 times the size of the input image.

[0126] Based on the size of the input image Several anchor boxes (or anchor regions) are pre-set. The number of anchor boxes is multiple, and the size of the anchor boxes gradually decreases from 1 times the size of the input image, that is, the size of the input image is 1 times, 1 / 2 times, 1 / 4 times, 1 / 8 times, 1 / 16 times...

[0127] Specifically, the original cropped area is compared with each anchor frame, and the anchor frame that includes the original cropped area and whose size is closest to the size of the original cropped area is used as the reference anchor frame.

[0128] The original cropping area is recorded as [x * ,y * , w * ,h * ](known), where x * The horizontal coordinate of the reference point of the original clipping area, y * The ordinate of the reference point of the original cropping area, w * Indicates the width of the original cropping area, h * Indicates the height of the original cropping area. In particular, the reference point may include the upper left corner, upper right corner, lower left corner, lower right corner, center point, etc. For example, x * Can represent the horizontal coordinate of the upper left corner of the original cropping area, y * It can represent the vertical coordinate of the upper left corner of the original cropping area. * It can also represent the horizontal coordinate of the center point of the original clipping area, y * The vertical coordinate of the center point of the original cropping area can also be expressed. The horizontal / vertical coordinates expressed as the upper left corner or the horizontal / vertical coordinates expressed as the center point can be converted to each other, that is, no matter expressed as the horizontal / vertical coordinates of the upper left corner or the horizontal / vertical coordinates of the center point, they all indicate the same original cropping area.

[0129] Compare the original cropped area with the preset anchor frames, and select the anchor frame that contains the original cropped area and is closest in size to it as the reference anchor frame. The coordinates of the reference anchor frame are marked as [x anchor ,y anchor , w anchor ,h anchor ], where x anchor Represents the horizontal coordinate of the reference point of the benchmark anchor box, y anchor The vertical coordinate of the reference point of the benchmark anchor box, w anchor Indicates the width of the reference anchor box, h anchor Indicates the height of the reference anchor box. In particular, the reference point may include the upper left corner, upper right corner, lower left corner, lower right corner, center point, etc. anchor It can represent the horizontal coordinate of the upper left corner of the reference anchor box, y anchorIt can represent the vertical coordinate of the upper left corner of the reference anchor box. anchor It can also represent the horizontal coordinate of the center point of the reference anchor frame, y anchor The vertical coordinate of the center point of the reference anchor frame can also be expressed. The horizontal / vertical coordinates expressed as the upper left corner or the horizontal / vertical coordinates expressed as the center point can be converted to each other, that is, whether expressed as the horizontal / vertical coordinates of the upper left corner or the horizontal / vertical coordinates of the center point, they all indicate the same reference anchor frame.

[0130] It is also specially noted that if x * Indicates the horizontal coordinate of the upper left corner of the original cropping area, y * Represents the ordinate of the upper left corner of the original cropping area, then the corresponding x anchor Indicates the horizontal coordinate of the upper left corner of the reference anchor box, y anchor Indicates the ordinate of the upper left corner of the reference anchor box; if x * Indicates the horizontal coordinate of the center point of the original clipping area, y * represents the ordinate of the center point of the original clipping area, then the corresponding x anchor Represents the horizontal coordinate of the center point of the reference anchor box, y anchor Indicates the ordinate of the center point of the reference anchor box. In other words, the reference point of the original cropped area and the reference point of the anchor box should be in the same orientation.

[0131] Specifically, the coordinates of the reference anchor frame are rounded and numerically aligned as the first cascade cropping area of ​​the cropping-scaling module.

[0132] The coordinates of the reference anchor frame are rounded and numerically aligned according to the requirements of the cropping-scaling module 1 as the first cascade cropping area. The first cascade cropping area is recorded as [x1, y1, w1, h1], where x1 represents the horizontal coordinate of the reference point of the first cascade cropping area, y1 represents the vertical coordinate of the reference point of the first cascade cropping area, w1 represents the width of the first cascade cropping area, and h1 represents the height of the first cascade cropping area.

[0133] When the size of the input image is not smaller than the size of the target image (λ = 1), and the reference anchor box is selected as the size of the input image When the size of the input image is not less than the size of the target image (λ = 1), and the reference anchor box is selected as the size of the input image When the size of the input image is smaller than the size of the target image (λ>1), no matter the reference anchor box is selected as the size of the input image, No matter how many times the image is enlarged, the cropping-scaling module 1 will perform image enlargement operation.

[0134] Specifically, converting the original cropped area into a first coordinate in the reference anchor frame, and converting the first coordinate into a second coordinate in the first cascade output image according to the proportional relationship between the size of the reference anchor frame and the size of the first cascade output image, including: defining the original cropped area as [x * ,y * , w * ,h * ], where x * The horizontal coordinate of the reference point of the original clipping area, y * The ordinate of the reference point of the original cropping area, w * Indicates the width of the original cropping area, h * Indicates the height of the original cropped area; the base anchor box is defined as [x anchor ,y anchor , w aachor ,h anchor ], where x anchor Represents the horizontal coordinate of the reference point of the benchmark anchor box, y anchor The vertical coordinate of the reference point of the benchmark anchor box, w anchor Indicates the width of the reference anchor box, h anchor represents the height of the reference anchor frame; then according to the following formula [x * -x anchor ,y * -y anchor , w * ,h * ] determine the first coordinate; then according to the following formula [s x (x * -x anchor ), s y (y * -y anchor ), s x w * ,s y h * ]Determine the second coordinate, where represents the width of the input image, represents the height of the input image, and λ represents the conversion coefficient determined based on the relationship between the size of the input image and the size of the target image.

[0135] When the original cropping area is recorded as [x * ,y * , w * ,h * ], the reference anchor frame is denoted as [x anchor ,y anchor , w aachor ,h anchor ], the first coordinate of the original cropping area in the reference anchor frame is marked as [x *-x anchor ,y * -y anchor , w * ,h * ].

[0136] Since the size of the first cascade output image (equal to the size of the second cascade input image) is recorded as Then the first coordinate is converted to the second coordinate in the first cascade output image as [s x (x * -x anchor ), s y (y * -y anchor ), s x w * ,s y h * ],in,

[0137] Specifically, the second coordinate is rounded and the values ​​are aligned to serve as the second cascade cropping area of ​​the cropping-scaling module 2.

[0138] After the second coordinate is rounded and numerically aligned according to the requirements of the cropping-scaling module 2, it is used as the second cascade cropping area. The second cascade cropping area is recorded as [x2, y2, w2, h2], where x2 represents the horizontal coordinate of the reference point of the second cascade cropping area, y2 represents the vertical coordinate of the reference point of the second cascade cropping area, w2 represents the width of the second cascade cropping area, and h2 represents the height of the second cascade cropping area.

[0139] In some embodiments of the present application, the above method also includes: when the size of the second cascade cropping area is larger than the size of the target image, the cropping-scaling module 2 performs image cropping and image reduction operations on the second cascade input image; when the size of the second cascade cropping area is smaller than the size of the target image, the cropping-scaling module 2 performs image cropping and image enlargement operations on the second cascade input image; when the size of the second cascade cropping area is equal to the size of the target image, the cropping-scaling module 2 performs image cropping operations on the second cascade input image but does not perform image scaling operations.

[0140] Based on the size relationship between the second cascade cropping area and the target image, the specific operation performed by the cropping-scaling module 2 is determined. It is particularly noted that when the value of the conversion coefficient λ is large enough, in theory, the size of the second cascade cropping area can always be made larger than the size of the target image, so that the cropping-scaling module 2 performs image cropping and image reduction operations. This situation is the most effective in overcoming jitters, and even eliminates jitters. However, due to resource overhead and processing time constraints, in some cases it is not possible to make λ large enough to make the size of the second cascade cropping area larger than the size of the target image. In this case, the size of the second cascade cropping area can be slightly smaller than or equal to the size of the target image, thereby reducing the magnification multiple required when the cropping-scaling module 2 performs an image magnification operation or not performing image magnification, thereby reducing the jitter phenomenon.

[0141] In some embodiments of the present application, the above method includes: defining a scaling ratio in, Indicates the width of the target image, represents the height of the target image, w2 represents the width of the second cascade cropping area, and h2 represents the height of the second cascade cropping area; when the size of the second cascade cropping area is larger than the size of the target image, the scaling ratio s is determined. * less than 1; when the size of the second cascade cropping area is smaller than the size of the target image, determining the scaling ratio s * greater than 1; when the size of the second cascade cropping area is equal to the size of the target image, determine the scaling ratio s * Equal to 1.

[0142] Calculate the scaling factor s from the second cascade cropping region to the size of the destination image * ,in If * >1, it means that the size of the second cascade cropping area is smaller than the size of the target image, and the cropping-scaling module 2 performs image cropping and image magnification operations. Although the cropping-scaling module 2 performs image magnification operations, the magnification ratio of the image by the cropping-scaling module 2 is greatly reduced, thereby improving the jitter effect. * <1, it means that the size of the second cascade cropping area is larger than the size of the target image, and the cropping-scaling module 2 performs image cropping and image reduction operations. At this time, there is basically no jitter effect. * =1, it means that the size of the second cascade cropping area is equal to the size of the target image, and the cropping-scaling module 2 performs an image cropping operation but does not perform an image scaling operation. At this time, there is basically no jitter effect.

[0143] Image reduction can be achieved by downsampling or other methods, and image enlargement can be achieved by upsampling or other methods, which are not specifically limited here.

[0144] Image jitter is reflected in the change of pixel size. The first cascade cropping area comes from the pre-set anchor frame, and the number of anchor frames is limited. In the corresponding continuous zoom process, the coordinates of the first cascade cropping area will not change frequently. Therefore, the error of the first cascade cropping area has no direct impact on the final jitter effect.

[0145] During the continuous zooming process, the coordinates of the second cascade cropping area will change continuously, and its error is directly related to the final jitter effect. The key role of the size of the first cascade cropping area and even the first cascade output image of the cropping-zooming module one (λ>1) is to enlarge the input image (including the original cropping area). After the enlargement to obtain a large image, the size of the second cascade cropping area is larger than the original cropping area, so that the subsequent operation of the cropping-zooming module two reduces the magnification ratio, and even converts image magnification into image reduction. This reduces the error of the cropping-zooming module two, and can therefore significantly improve or even eliminate the jitter effect.

[0146] The following is a specific embodiment of the image digital zoom method proposed in this application.

[0147] Example 1

[0148] Zoom requirement: The input image size is 3840×2160 (4K), the target image size is 1920×1080 (1080P), and a 1x to 4x digital zoom is performed on this basis.

[0149] The following method proposed in this application is used to change the magnification.

[0150] Step 1: The size of the input image is recorded as The size of the target image is recorded as Then the size of the first cascade input image is The size of the second cascade output image

[0151] The size of the first cascade output image is recorded as The size of the second cascade input image is recorded as The conversion coefficient is λ. Since the size of the input image is larger than the size of the target image, the size of the first cascade output image is determined to be equal to the size of the input image. Determine λ=1. So

[0152] Step 2: Preset two anchor boxes, which remain unchanged relative to the center point of the input image:

[0153] Anchor box 1, in, Represents the horizontal coordinate of the upper left corner of anchor box 1, Represents the ordinate of the upper left corner of anchor box 1, represents the width of anchor box 1, represents the height of anchor box 1,

[0154] Anchor box 2, in, Represents the horizontal coordinate of the upper left corner of anchor box 2, Represents the ordinate of the upper left corner of anchor box 2, represents the width of anchor box 2, represents the height of anchor box 2,

[0155] Step 3, when the magnification is 1x to 2x (inclusive), take the magnification of 1.45x as an example. Since the magnification is determined, the original cropping area can be determined based on the magnification. Taking the center point unchanged as an example, the original cropping area is Among them, x *1 (1) represents the horizontal coordinate of the upper left corner of the original cropping area, y *1 (1) represents the ordinate of the upper left corner of the original cropping area, Indicates the width of the original cropping area. Indicates the height of the original cropped area. The original cropped area is compared with anchor frame 1 and anchor frame 2, and the anchor frame that contains the original cropped area and is closest in size to it is selected as the reference anchor frame - anchor frame 1. The coordinates of anchor frame 1 are rounded and numerically aligned as the first cascade cropped area of ​​the cropping-scaling module 1.

[0156] According to the formula Determine the first coordinate, according to the formula sx111.45Win*1,sy111.45Hin*1 determine the second coordinate, where The second coordinate is rounded and the values ​​are aligned as the second cascade cropping area of ​​the cropping-scaling module 2. Figure 5 The figure shows the principle schematic diagram of the digital zoom of the image when the zoom ratio is 1.45x.

[0157] Step 4: Transfer to the cropping-scaling module 1, The image is transmitted to the cropping-scaling module 2 so that the cropping-scaling module 1 and the cropping-scaling module 2 perform zoom processing on the image.

[0158] At this time, it is equivalent to that the cropping-scaling module 1 does not perform image cropping and image scaling operations, and the cropping-scaling module 2 performs image cropping and image reduction operations. Therefore, there is no obvious jitter effect at this stage.

[0159] Step 5, when the magnification is 2x (not included) to 4x, take the magnification of 2.76x as an example. Since the magnification is determined, the original cropping area can be determined based on the magnification. Taking the center point unchanged as an example, the original cropping area is Among them, x *2 (1) represents the horizontal coordinate of the upper left corner of the original cropping area, y *2 (1) represents the ordinate of the upper left corner of the original cropping area, Indicates the width of the original cropping area. Indicates the height of the original cropped area. The original cropped area is compared with anchor frame 1 and anchor frame 2, and the anchor frame that contains the original cropped area and is closest in size to it is selected as the reference anchor frame - anchor frame 2. The coordinates of anchor frame 2 are rounded and numerically aligned as the first cascade cropped area of ​​the cropping-scaling module 1.

[0160] According to the formula Determine the first coordinate, according to the formula Determine the second coordinate, where The second coordinate is rounded and the values ​​are aligned as the second cascade cropping area of ​​the cropping-scaling module 2. Figure 6 The figure shows the principle schematic diagram of the digital zoom of the image when the zoom ratio is 2.76x.

[0161] Step 6: Transfer to the cropping-scaling module 1,

[0162] The image is transmitted to the cropping-scaling module 2 so that the cropping-scaling module 1 and the cropping-scaling module 2 perform zoom processing on the image.

[0163] At this time, the process of the cropping-scaling module 1 processing the input image into the first cascade output image is image cropping and image magnification. The cropping-scaling module 2 processes the second cascade input image (first cascade output image) into the second cascade output image (target image) is image cropping and image reduction. After this processing, the final zoom ratio and zoom effect are equivalent to the prior art using a 1st-order cropping-scaling module, but the error is reduced. The zoom effect when the zoom ratio is 2x (not included) to 4x is similar to the zoom effect when the zoom ratio is 1x to 2x (inclusive), and there is no obvious jitter effect.

[0164] Comparative Example 1

[0165] Zoom requirement: The size of the input image is 3840×2160 (4K), the size of the target image is 1920×1080 (1080P), and a 1x to 4x digital zoom is performed on this basis (the same as in Example 1).

[0166] Using the existing technology, when the magnification is 1x to 2x (inclusive), the cropping-zooming operation is image cropping and image reduction, and the jitter effect is not obvious at this stage; but when the magnification is 2x (not included) to 4x, the cropping-zooming operation is image cropping and image enlargement, and the jitter effect is obvious at this stage.

[0167] Example 2

[0168] Zoom requirement: The input image size is 1920×1080 (1080P), the target image size is 3840×2160 (4K), and a 1x to 16x digital zoom is performed on this basis.

[0169] The following method proposed in this application is used to change the magnification.

[0170] Step 1: The size of the input image is recorded as The size of the target image is recorded as Then the size of the first cascade input image is The size of the second cascade output image

[0171] The size of the first cascade output image is recorded as The size of the second cascade input image is recorded as The conversion coefficient is λ. Since the size of the input image is smaller than the size of the target image, it is determined that the size of the first cascade output image is larger than the size of the input image. Determine λ>1. Considering the resource cost constraint, take λ=2, so

[0172] Step 2: Pre-set 4 anchor boxes, which remain unchanged relative to the center point of the input image:

[0173] Anchor box 1, in, Represents the horizontal coordinate of the upper left corner of anchor box 1, Represents the ordinate of the upper left corner of anchor box 1, represents the width of anchor box 1, represents the height of anchor box 1,

[0174] Anchor box 2, in, Represents the horizontal coordinate of the upper left corner of anchor box 2, Represents the ordinate of the upper left corner of anchor box 2, represents the width of anchor box 2, represents the height of anchor box 2,

[0175] Anchor box 3, in, represents the horizontal coordinate of the upper left corner of anchor box 3, represents the ordinate of the upper left corner of anchor box 3, represents the width of anchor box 3, represents the height of anchor box 3,

[0176] Anchor box 4. in, represents the horizontal coordinate of the upper left corner of anchor box 4, represents the ordinate of the upper left corner of anchor box 4, represents the width of anchor box 4, represents the height of anchor box 4,

[0177] Step 3, when the magnification is 1x to 2x (inclusive), take the magnification of 1.45x as an example. Since the magnification is determined, the original cropping area can be determined based on the magnification. Taking the center point unchanged as an example, the original cropping area is Among them, x *1 (2) represents the horizontal coordinate of the upper left corner of the original cropping area, y *1 (2) represents the ordinate of the upper left corner of the original cropped area, Indicates the width of the original cropping area. Indicates the height of the original cropped area. The original cropped area is compared with anchor frames 1 to 4, and the anchor frame that contains the original cropped area and is closest in size to it is selected as the reference anchor frame - anchor frame 1. The coordinates of anchor frame 1 are rounded and numerically aligned as the first cascade cropped area of ​​the cropping-scaling module 1.

[0178] According to the formula Determine the first coordinate, according to the formula

[0179] Determine the second coordinate, where The second coordinate is rounded and the values ​​are aligned as the second cascade cropping area of ​​the cropping-scaling module 2. Figure 7 The figure shows the principle schematic diagram of the digital zoom of the image when the zoom ratio is 1.45x.

[0180] Step 4: Transfer to the cropping-scaling module 1,

[0181] The image is transmitted to the cropping-scaling module 2 so that the cropping-scaling module 1 and the cropping-scaling module 2 perform zoom processing on the image.

[0182] At this time, the process of the cropping-scaling module 1 processing the input image into the first cascade output image is not to crop the image but to enlarge the image, and the image enlargement factor is 2 times. Although there will still be numerical / sampling errors at this stage, they will not cause jitter. After this processing, the cropping-scaling module 2 processes the second cascade input image (first cascade output image) into the second cascade output image (target image) by cropping and enlarging the image, and the image enlargement factor is reduced to If the prior art directly uses the first-order cropping-scaling module for zooming, the image magnification is 2x to 4x when the zoom ratio is 1x to 2x (the image magnification is 2.9x when the zoom ratio is 1.45x). In contrast, the jitter effect is significantly improved by the method of the present application.

[0183] Step 5, when the magnification is 2x (not included) to 16x, take the magnification of 9.76x as an example. Since the magnification is determined, the original cropping area can be determined based on the magnification. Taking the center point unchanged as an example, the original cropping area is Among them, x *2 (2) represents the horizontal coordinate of the upper left corner of the original cropping area, y *2 (2) represents the ordinate of the upper left corner of the original cropped area, Indicates the width of the original cropping area. Indicates the height of the original cropped area. The original cropped area is compared with anchor frames 1 to 4, and the anchor frame that contains the original cropped area and is closest in size to it is selected as the reference anchor frame - anchor frame 4. The coordinates of anchor frame 4 are rounded and numerically aligned as the first cascade cropped area of ​​the cropping-scaling module 1.

[0184] According to the formula Determine the first coordinate, according to the formula

[0185] Determine the second coordinate, where The second coordinate is rounded and the values ​​are aligned as the second cascade cropping area of ​​the cropping-scaling module 2. Figure 8 The figure shows the principle schematic diagram of the digital zoom of the image when the zoom ratio is 9.76x.

[0186] Step 6: Transfer to the cropping-scaling module 1,

[0187] The image is transmitted to the cropping-scaling module 2 so that the cropping-scaling module 1 and the cropping-scaling module 2 perform zoom processing on the image.

[0188] At this time, the process of the cropping-scaling module 1 processing the input image into the first cascade output image is image cropping and image enlargement, and the image enlargement multiple is 16 times. After this processing, the cropping-scaling module 2 processes the second cascade input image (first cascade output image) into the second cascade output image (target image) by image cropping and image enlargement, and the image enlargement multiple is reduced to If the prior art is used to directly use the first-order cropping-scaling module for zooming, the image magnification is 4x to 32x when the zoom ratio is 2x to 16x (the image magnification is 19.52x when the zoom ratio is 9.76x). In contrast, the jitter effect is significantly improved by the method of the present application, and even almost disappears (refer to comparative example 2).

[0189] Comparative Example 2

[0190] Zoom requirement: The size of the input image is 1920×1080 (1080P), the size of the target image is 3840×2160 (4K), and a 1x to 16x digital zoom is performed on this basis (the same as in Example 2).

[0191] If the existing technology is used, when the zoom ratio is 1x, the size of the original cropped area is equal to the size of the input image, that is, 1920×1080, and the size of the target image is 3840×2160. The crop-zoom operation does not crop the image but enlarges the image, and the image magnification factor is 2 times. In other words, the image is enlarged starting from 1x, and the image magnification factor is twice the zoom ratio. When the zoom ratio is 16x, the image magnification factor reaches 32x. When zooming to a high magnification, the jitter effect is very severe and becomes more and more serious, and the visual experience is extremely poor.

[0192] It can be seen from Example 2 and Comparative Example 2 that when the prior art is used to perform 1x to 16x magnification, the image magnification of the first-order cropping-zooming module is 2x to 32x, and the jitter becomes very severe as the magnification increases, resulting in a very poor visual experience. However, using the method proposed in this application, the actual image magnification of the cropping-zooming module 2 during the magnification process can always be controlled at 1x to 2x, and the jitter effect is not obvious, which significantly improves or even eliminates the jitter problem.

[0193] When the de-jitter control module configures the size of the first cascade output image and the size of the second cascade input image, when the size of the input image is smaller than the size of the target image, if λ takes a larger value, the image magnification of the second cropping-scaling module will be further reduced, and even become image reduction, and the corresponding jitter effect will be improved more thoroughly. However, the larger the value of λ, the larger the image size of the intermediate link, the greater the resource overhead, and the longer the processing time. Therefore, the value of λ can be greater than 1 according to the resource overhead constraint.

[0194] Embodiment 1 and Embodiment 2 are both narrow-sense digital image magnification, i.e., digital zoom. However, it should be understood that the digital image magnification proposed in this application should be understood in a broad sense, and its application scenarios include but are not limited to: digital zoom, AutoFraming, close-up of the speaker, close-up animation or EPTZ in classroom directing and other applications.

[0195] In the zooming process shown in Example 1 and Example 2, the center of the cropped area remains stationary, and only the size changes. However, the present application does not limit whether the center point of the cropped area moves. In other words, in a broad sense, regardless of whether the center point of the cropped area is offset during the zooming process and the degree of offset, the method proposed in the present application is applicable. If necessary, this can be achieved by appropriately adjusting the position of the anchor frame or dynamically setting the anchor frame.

[0196] Taking the VPSS and VGS modules of the Hisi platform as an example, they support a maximum zoom ratio of 16x. In the scenario of Comparative Example 2, when the zoom ratio is 16x, the image magnification is 32x, which is far beyond the hardware limitation. It can actually only support a maximum zoom ratio of 8x. However, according to the method proposed in this application, in the process of 1x to 16x zoom, the maximum image magnification of the crop-zoom module 1 is 16x, and the image magnification of the crop-zoom module 2 is only 1x to 2x, both of which do not exceed the hardware limitation. Therefore, the method proposed in this application can support a zoom ratio of 16x.

[0197] Based on the same inventive concept, Figure 2 In the image digital zoom device shown:

[0198] A de-jitter control module, used for receiving original information and determining cascade configuration information corresponding to each cropping-scaling module according to the original information;

[0199] A de-jitter control module, used for transmitting each cascade configuration information to the corresponding crop-scaling module respectively;

[0200] Each cropping-scaling module is used to scale the input cascade input image according to the received cascade configuration information and then output the cascade output image.

[0201] Specifically, the original information includes: the size of the input image, the size of the target image, and the original cropping area of ​​the floating point value, and the cascade configuration information includes: the size of the cascade input image, the size of the cascade output image, and the cascade cropping area of ​​the integer value;

[0202] A de-jitter control module, used for receiving the size of an input image and the size of a target image, and determining the size of a cascaded input image and the size of a cascaded output image corresponding to each cropping-scaling module according to the size of the input image and the size of the target image;

[0203] A de-jitter control module, used for receiving an original cropping area, and determining a cascade cropping area corresponding to each cropping-scaling module according to the original cropping area;

[0204] A de-jitter control module, used for summarizing the size of each cascade input image, the size of each cascade output image and each cascade cropping area into cascade configuration information corresponding to each cropping-scaling module and transmitting the information to the corresponding cropping-scaling modules respectively;

[0205] Each cropping-scaling module is used to scale the input cascade input image according to the received cascade configuration information and then output the cascade output image.

[0206] Based on the same inventive concept, Figure 4 In the image digital zoom device shown:

[0207] The de-jitter control module is used to determine the size of the first cascade input image of the cropping-scaling module one according to the size of the input image, and determine the size of the second cascade output image of the cropping-scaling module two according to the size of the target image; according to the size relationship between the size of the input image and the size of the target image, determine the size of the first cascade output image of the cropping-scaling module one and the size of the second cascade input image of the cropping-scaling module two, wherein the size of the first cascade output image is equal to the size of the second cascade input image.

[0208] In some embodiments of the present application, the de-jitter control module is further configured to, when the size of the input image is not less than the size of the target image, determine that the size of the first cascade output image is equal to the size of the input image, so that the cropping-scaling module performs image cropping and image magnification operations on the first cascade input image, or does not perform image cropping and image scaling operations;

[0209] When the size of the input image is smaller than the size of the target image, it is determined that the size of the first cascade output image is larger than the size of the input image, so that the cropping-scaling module performs image cropping and image enlargement operations on the first cascade input image, or does not perform image cropping but performs image enlargement operations.

[0210] In some embodiments of the present application, the de-jitter control module is further used to preset multiple anchor frames according to the size of the input image, and the sizes of the multiple anchor frames decrease from 1 times the size of the input image in even multiples; compare the original cropped area with each anchor frame, and take the anchor frame that contains the original cropped area and whose size is closest to the size of the cropped area as the reference anchor frame; round the coordinates of the reference anchor frame and align the values ​​as the first cascade cropping area of ​​the cropping-scaling module one; convert the original cropped area into a first coordinate in the reference anchor frame, and convert the first coordinate into a second coordinate in the first cascade output image according to the proportional relationship between the size of the reference anchor frame and the first cascade output image; round the second coordinate and align the values ​​as the second cascade cropping area of ​​the cropping-scaling module two.

[0211] In some embodiments of the present application, the de-jitter control module is also used to, when the size of the second cascade cropping area is larger than the size of the target image, enable the cropping-scaling module 2 to perform image cropping and image reduction operations on the second cascade input image; when the size of the second cascade cropping area is smaller than the size of the target image, enable the cropping-scaling module 2 to perform image cropping and image enlargement operations on the second cascade input image; when the size of the second cascade cropping area is equal to the size of the target image, enable the cropping-scaling module 2 to perform image cropping operations on the second cascade input image but not perform image scaling operations.

[0212] It should be noted that the above-mentioned image digital zooming device can implement the above-mentioned image digital zooming method, which will not be described in detail.

[0213] The present application also provides a drone, which may include the image digital magnification device described in the above embodiment.

[0214] Fig. 9 A schematic diagram of the structure of an electronic device according to an embodiment of the present application is shown. Fig. 9 As shown, the internal structure of the electronic device may include a processor, a memory, a network interface and a database connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, the functions or steps of the image digital zoom method are implemented.

[0215] In one embodiment, the electronic device provided by the present application includes a memory and a processor, the memory stores a database and a computer program that can be run on the processor, and the aforementioned methods and steps of the present application are implemented when the processor executes the computer program.

[0216] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the aforementioned methods and steps of the present application are implemented.

[0217] It should be noted that the above functions or steps that can be implemented by the electronic device or computer-readable storage medium can refer to the relevant description in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0218] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0219] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0220] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for digitally zooming an image, characterized in that: The method is applied to an image digital zoom device, the device comprising at least two cropping-zooming modules and a de-shaking control module respectively connected to each of the cropping-zooming modules, and each of the cropping-zooming modules is cascaded; The method is performed by the de-jitter control module, and the method includes: Receiving original information, and determining cascade configuration information corresponding to each of the cropping-scaling modules according to the original information; Each of the cascade configuration information is transmitted to the corresponding cropping-scaling module, so that each of the cropping-scaling modules scales the input cascade input image according to the received cascade configuration information and then outputs a cascade output image.

2. The image digital zoom method according to claim 1, characterized in that: The original information includes: the size of the input image, the size of the target image, and the original cropping area of ​​the floating point value, the cascade configuration information includes: the size of the cascade input image, the size of the cascade output image, and the cascade cropping area of ​​the integer value, and the method includes: Receiving the size of the input image and the size of the target image, and determining the size of the cascade input image and the size of the cascade output image corresponding to each cropping-scaling module according to the size of the input image and the size of the target image; Receiving the original cropping area, and determining the cascaded cropping area corresponding to each cropping-scaling module according to the original cropping area; The size of each of the cascade input images, the size of each of the cascade output images and the size of each of the cascade cropping areas are summarized as the cascade configuration information corresponding to each of the cropping-zooming modules and transmitted to the corresponding each of the cropping-zooming modules respectively, so that each of the cropping-zooming modules can magnify the input cascade input image according to the received cascade configuration information and then output the cascade output image.

3. The image digital zoom method according to claim 2, characterized in that: There are two cropping-scaling modules, and determining the size of the cascade input image and the size of the cascade output image corresponding to each cropping-scaling module according to the size of the input image and the size of the target image includes: Determine the size of the first cascade input image of the cropping-scaling module 1 according to the size of the input image, and determine the size of the second cascade output image of the cropping-scaling module 2 according to the size of the target image; The size of the first cascade output image of the cropping-scaling module one and the size of the second cascade input image of the cropping-scaling module two are determined according to the size relationship between the size of the input image and the size of the target image, wherein the size of the first cascade output image is equal to the size of the second cascade input image.

4. The image digital zoom method according to claim 3, characterized in that: The determining, according to the size relationship between the size of the input image and the size of the target image, the size of the first cascade output image of the cropping-scaling module 1 and the size of the second cascade input image of the cropping-scaling module 2 comprises: When the size of the input image is not less than the size of the target image, determining that the size of the first cascade output image is equal to the size of the input image, so that the cropping-scaling module performs image cropping and image enlargement operations on the first cascade input image, or does not perform image cropping and image scaling operations; When the size of the input image is smaller than the size of the target image, it is determined that the size of the first cascade output image is larger than the size of the input image, so that the cropping-scaling module performs image cropping and image enlargement operations on the first cascade input image, or does not perform image cropping but performs image enlargement operations.

5. The image digital zoom method according to claim 3, characterized in that: The step of determining the size of the first cascade input image of the cropping-scaling module 1 according to the size of the input image, and determining the size of the second cascade output image of the cropping-scaling module 2 according to the size of the target image, comprises: Define the size of the input image as Define the size of the target image as Where W represents width and H represents height; Then the size of the first cascade input image is The size of the second cascade output image 6. The image digital zoom method according to claim 4, characterized in that: The determining, according to the size relationship between the size of the input image and the size of the target image, the size of the first cascade output image of the cropping-scaling module 1 and the size of the second cascade input image of the cropping-scaling module 2 comprises: Define the size of the first cascade output image as Define the size of the second cascade input image as Define the conversion coefficient as λ; When the size of the input image is not less than the size of the target image, the conversion coefficient λ is determined to be 1, and according to the following formula: Determine that the size of the first cascade output image is equal to the size of the input image; When the size of the input image is smaller than the size of the target image, the value of the conversion coefficient λ is determined to be greater than 1, and according to the following formula: It is determined that a size of the first cascade output image is larger than a size of the input image.

7. An image digital zoom device, characterized in that: The device comprises: a de-jitter control module and at least two cropping-scaling modules; Each of the cropping-scaling modules is cascaded and connected to the de-jitter control module; The de-jitter control module is used to receive original information and determine cascade configuration information corresponding to each of the cropping-scaling modules according to the original information; The de-jitter control module is used to transmit each of the cascade configuration information to the corresponding cropping-scaling module respectively; Each of the cropping-scaling modules is used to scale the input cascade input image according to the received cascade configuration information and then output a cascade output image.

8. A drone, characterized in that: The drone comprises the image digital magnification device according to claim 11.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the image digital zooming method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: The computer program implements the steps of the image digital zooming method according to any one of claims 1 to 6 when instructed by a processor.

Images