Image scaling processing method and device, terminal and storage medium

Abstract

The application discloses an image zoom processing method and device, a terminal and a storage medium. The method comprises the following steps: acquiring eye movement information when a user observes a target image, then performing coordinate conversion on the eye movement information to obtain a center coordinate corresponding to the eye movement information, determining a non-gaze area of the target image according to the center coordinate, compressing the non-gaze area of the target image to obtain a compressed target image, then performing calculation on the compressed target image to obtain a processed target image, and finally obtaining a decompressed target image according to the center coordinate and the processed target image. The application mainly saves storage bandwidth and calculation hardware resources by compression, and effectively reduces the demand for hardware resources on the premise of ensuring the image effect of the gaze point area.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to an image scaling processing method and device, a terminal and a storage medium. BACKGROUND

[0002] In computer image processing and computer graphics, image scaling refers to the process of adjusting the size of a digital image. Image scaling is a non-trivial process that requires a trade-off between processing efficiency and the smoothness and clarity of the result. When the size of an image is increased, the visibility of the pixels that make up the image will become higher, making the image appear "soft". Conversely, reducing an image will enhance its smoothness and clarity.

[0003] At present, the deep learning (Deep Learing) method is generally used for scaling the target image.

[0004] However, the above method can cause the problem of image quality degradation, resulting in poor scaling effect. SUMMARY

[0005] The main purpose of the present application is to provide an image scaling processing method, device, terminal and storage medium to solve the problem of image quality degradation in related technologies.

[0006] In order to achieve the above purpose, in a first aspect, the present application provides an image scaling processing method, comprising:

[0007] Obtaining eye movement information when a user observes a target image;

[0008] Converting the coordinates of the eye movement information to obtain the center coordinates corresponding to the eye movement information;

[0009] Determining the non-gaze area of the target image according to the center coordinates, compressing the non-gaze area of the target image, and obtaining the compressed target image;

[0010] Calculating the compressed target image to obtain the processed target image;

[0011] Obtaining the decompressed target image according to the center coordinates and the processed target image.

[0012] In a possible implementation, determining the non-gaze area of the target image according to the center coordinates, compressing the non-gaze area of the target image, and obtaining the compressed target image, comprises:

[0013] Determining the coordinates of each area in the non-gaze area of the target image according to the center coordinates;

[0014] The coordinates of each region in the target image are composed of the center coordinates and the coordinates of each region in the non-gaze region;

[0015] The target image is compressed based on the coordinates of each region in the target image to obtain a compressed target image.

[0016] In a possible implementation, the target image is compressed based on the coordinates of each region in the target image to obtain a compressed target image, comprising:

[0017] Based on the coordinates of each region in the non-gaze region of the target image, a coordinate mapping table of the non-gaze region is determined;

[0018] According to the gaze region, the center coordinates, the non-gaze region, and the coordinate mapping table, the mapping coordinates and the scaling factor of each pixel position in the target image are calculated by table lookup;

[0019] According to the scaling factor, the scaling weight of the gaze region, the scaling weight of the non-gaze region, and the fusion weight of the transition region in the target image are calculated, wherein the continuous factor scaling curve corresponding to the scaling weight is a derivative smoothing curve, the points on the derivative smoothing curve are used to represent the scaling factor of the pixel coordinate position, and the scaling factor is increasing,

[0020] The transition fusion weight curve corresponding to the fusion weight is a derivative smoothing curve, the points on the derivative smoothing curve are used to represent the excessive fusion weight of the pixel position and the edge distance of the gaze region, the excessive fusion weight is fused with the difference curve weight and the identity mapping weight, and the pixel position close to the edge of the gaze region has a large weight, and the pixel position close to the gaze region has a large identity mapping weight.

[0021] The scaling weight of the gaze region, the scaling weight of the non-gaze region, and the fusion weight of the transition region are weighted and synthesized to output the compressed target image.

[0022] In a possible implementation, the image algorithm operation adopted for the calculation processing of the compressed target image at least includes a denoising operation and a specific image algorithm operation.

[0023] In a possible implementation, the denoising algorithm is a neural network algorithm.

[0024] The compressed target image is calculated and processed to obtain a processed target image, comprising:

[0025] The compressed target image is denoised by using a neural network algorithm to obtain a processed target image.

[0026] In a possible implementation, the decompressed target image is obtained according to the center coordinates and the processed target image, comprising:

[0027] According to the center coordinates, coordinates of each region in the processed target image are calculated;

[0028] The processed target image is decompressed based on the coordinates of each region in the processed target image, to obtain a decompressed target image.

[0029] In a possible implementation, the center coordinates are positions corresponding to a gaze region of the target image;

[0030] According to the center coordinates, coordinates of each region in the processed target image are calculated, including:

[0031] According to the center coordinates, coordinates of each region in the non-gaze region of the processed target image are determined;

[0032] The coordinates of each region in the target image are composed of the center coordinates and the coordinates of each region in the non-gaze region.

[0033] In a possible implementation, the processed target image is decompressed based on the coordinates of each region in the processed target image, to obtain a decompressed target image, including:

[0034] Based on the coordinates of each region in the non-gaze region of the processed target image, a coordinate mapping table of the non-gaze region is determined;

[0035] According to the gaze region, the center coordinates, the non-gaze region and the coordinate mapping table, a reverse lookup table is performed to calculate mapping coordinates and a scaling factor of each pixel position in the processed target image;

[0036] According to the scaling factor, a scaling weight of the gaze region, a scaling weight of the non-gaze region and a fusion weight of the transition region in the processed target image are calculated;

[0037] The scaling weight of the gaze region, the scaling weight of the non-gaze region and the fusion weight of the transition region are weighted and synthesized to output the decompressed target image.

[0038] In a possible implementation, the center coordinates are positions corresponding to a gaze region of the target image or positions corresponding to a gaze point set on the target image.

[0039] In a possible implementation, the gaze point is dynamic information obtained according to eye tracking or a static gaze point that does not change according to eye movement information.

[0040] In a possible implementation, the target image at least includes a Bayer image, an RGB image, a RAW image and a YUV image.

[0041] In a possible implementation, the calculation manner for the compressed target image at least includes NPU calculation and neural network model calculation.

[0042] In a second aspect, an embodiment of the present application provides an image scaling processing apparatus, comprising:

[0043] An acquisition module is configured to acquire eye movement information when a user observes a target image.

[0044] A first calculation module is configured to perform coordinate conversion on the eye movement information to obtain a center coordinate corresponding to the eye movement information.

[0045] A compression module is configured to determine a non-gaze area of the target image according to the center coordinate, compress the non-gaze area of the target image, and obtain a compressed target image.

[0046] A second calculation module is configured to perform calculation on the compressed target image to obtain a processed target image.

[0047] A decompression module is configured to obtain a decompressed target image according to the center coordinate and the processed target image.

[0048] In a third aspect, an embodiment of the present application provides a terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of any of the above image scaling processing methods when executing the computer program.

[0049] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the steps of any of the above image scaling processing methods.

[0050] The embodiments of the present application provide an image scaling processing method, apparatus, terminal and storage medium, comprising: acquiring eye movement information when a user observes a target image, performing coordinate conversion on the eye movement information to obtain a center coordinate corresponding to the eye movement information, determining a non-gaze area of the target image according to the center coordinate, compressing the non-gaze area of the target image to obtain a compressed target image, performing calculation on the compressed target image to obtain a processed target image, and obtaining a decompressed target image according to the center coordinate and the processed target image. The present application scales the non-gaze area based on the center coordinate, so that the scaled image needs to be quickly restored to the original size after NPU processing, ensures that other spatial domain related modules of ISP are not affected, avoids the decline of the quality of the target image, and improves the scaling effect of the target image. In addition, the present application mainly saves storage bandwidth and computing hardware resources by compression, effectively reduces the demand for hardware resources under the premise of ensuring the image effect of the gaze point area. BRIEF DESCRIPTION OF DRAWINGS

[0051] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The illustrative embodiments of the application, and their description, do not limit the application. In the drawings:

[0052] Figure 1 is a schematic diagram of coordinate mapping of an image provided by an embodiment of the application;

[0053] Figure 2 is a schematic diagram of image partition setting magnification provided by an embodiment of the application;

[0054] Figure 3 is a flowchart of implementation of an image scaling processing method provided by an embodiment of the application;

[0055] Figure 4 is a flowchart of implementation of an image scaling processing method provided by another embodiment of the application;

[0056] Figure 5 is a flowchart of implementation of an image compression method provided by an embodiment of the application;

[0057] Figure 6 is a schematic diagram of setting magnification of each region of an image provided by an embodiment of the application;

[0058] Figure 7 is a schematic diagram of RGB image format and Bayer image format provided by an embodiment of the application;

[0059] Figure 8 is a schematic diagram of setting magnification of each region of an image provided by another embodiment of the application;

[0060] Figure 9 is a continuous magnification scaling curve diagram provided by an embodiment of the application;

[0061] Figure 10 is a transition zone curve diagram provided by an embodiment of the application;

[0062] Figure 11 is a flowchart of implementation of an image decompression method provided by an embodiment of the application;

[0063] Figure 12 is a structural schematic diagram of an image scaling processing device provided by an embodiment of the application;

[0064] Figure 13 is a schematic diagram of a terminal provided by an embodiment of the application. DETAILED DESCRIPTION

[0065] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0066] The terms "first", "second", "third", "fourth" and the like in the description, claims, and drawings of the present application, and the above-described drawings (if any) are used to distinguish similar objects, and do not necessarily indicate a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0067] It should be understood that, in various embodiments of the present application, the magnitude of the serial number of each process does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0068] It should be understood that, in the present application, "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0069] It should be understood that, in the present application, "a plurality of" means two or more. "And / or" is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents that the front and rear associated objects are in an "or" relationship. "Including A, B and C", "including A, B, C" means that A, B and C are all included, "including A, B or C" means that one of A, B and C is included, and "including A, B and / or C" means that any one or any two or three of A, B and C is included.

[0070] It should be understood that, in the present application, "B corresponding to A", "B corresponding to A", "A corresponding to B" or "B corresponding to A" means that B is associated with A, and B can be determined according to A. Determining B according to A does not mean that B is determined only according to A, but also can be determined according to A and / or other information. The matching of A and B means that the similarity of A and B is greater than or equal to a preset threshold.

[0071] Depending on the context, "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection."

[0072] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0073] Furthermore, to make the objectives, technical solutions, and advantages of the present invention clearer, specific embodiments will be described below in conjunction with the accompanying drawings.

[0074] In computer image processing and computer graphics, image scaling refers to the process of adjusting the size of a digital image.

[0075] In existing technologies, image scaling is mainly achieved through the Scaler method, such as... Figure 1 As shown, its essence is coordinate mapping. The target point D is located at the corresponding point Si in the source image, and then the target point D is obtained by interpolation through interpolation calculation using the interpolation formula.

[0076] The Scaler method includes Scaler down and Scaler up. For RGB images, it primarily involves partitioning the image and performing Scaler down and Scaler up operations at different magnifications. For example... Figure 2 As shown, the image is divided into multiple blocks, and the scaling factor used for each block is marked. For example, the first block in the upper left corner of the image is scaled by a factor of 4.

[0077] However, existing technologies suffer from image quality degradation, resulting in poor scaling performance.

[0078] Therefore, this application provides an image scaling processing method to solve the above problems and improve the scaling effect of images.

[0079] In one embodiment, such as Figure 3 As shown, an image scaling method is provided, including the following steps:

[0080] Step S301: Obtain eye movement information when the user observes the target image.

[0081] Eye-tracking information can be obtained through eye-tracking devices or other devices. Eye-tracking information mainly includes the user's gaze position on the screen or image and the corresponding timestamps.

[0082] Step S302: Perform coordinate conversion on the eye movement information to obtain the center coordinates corresponding to the eye movement information.

[0083] The MCU is mainly used for coordinate conversion of eye movement information. Through the conversion, the fixation position on the user image, that is, the center coordinate, can be obtained.

[0084] Step S303: determining a non-gaze area of the target image according to the center coordinate, compressing the non-gaze area of the target image, and obtaining a compressed target image.

[0085] The target image includes images in various formats, including but not limited to images in Bayer, RAW, RGB, and YUV formats.

[0086] Step S304: calculating the compressed target image to obtain a processed target image.

[0087] Step S305: obtaining a decompressed target image according to the center coordinate and the processed target image.

[0088] In combination with Figure 4 Taking the target image as a Bayer image as an example, the compressed Bayer image obtained according to the center coordinate and the Bayer image is realized by a Bayer Group module, and the decompressed Bayer image obtained according to the center coordinate and the processed Bayer image is realized by a Bayer Ungroup module.

[0089] The process of realizing the compressed Bayer image obtained according to the center coordinate and the Bayer image by the Bayer Group module will be described in detail below in combination with Figure 5 The main execution steps are as follows: first, the coordinates of each area in the Bayer image are calculated according to the center coordinate, and then the Bayer image is compressed based on the coordinates of each area in the Bayer image to obtain the compressed Bayer image. The center coordinate is the position corresponding to the gaze area of the Bayer image.

[0090] The coordinates of each area in the Bayer image are calculated according to the center coordinate, which requires that the coordinates of each area in the non-gaze area of the Bayer image are first determined according to the center coordinate, and then the coordinates of each area in the Bayer image are composed of the center coordinate and the coordinates of each area in the non-gaze area.

[0091] The coordinates of each region in the non-gaze region of the Bayer image are used to determine a coordinate mapping table of the non-gaze region, and then the gaze region, the center coordinates, the non-gaze region, and the coordinate mapping table are used to look up a table, calculate the mapping coordinates and the scaling factor of each pixel position in the Bayer image, and then calculate the scaling weight of the gaze region, the scaling weight of the non-gaze region, and the fusion weight of the transition region in the Bayer image according to the scaling factor. Finally, the scaling weight of the gaze region, the scaling weight of the non-gaze region, and the fusion weight of the transition region are weighted and synthesized to output the compressed Bayer image.

[0092] In combination Figure 6 , the center coordinates are the positions corresponding to the gaze region of the target image, and the gaze region is scaled by a fixed factor. In addition, the edge of the gaze region is provided with a transition band, which can ensure natural transition of the image. The size of the gaze region can be adjusted according to requirements.

[0093] The non-gaze region is outside the transition band (transition region), and the non-gaze region is divided into multiple blocks by lines, and each block is scaled by a variable factor, and the variable factor is calculated by functions f(i) and f(j) and is related to the distance from the gaze region. The functions f(i) and f(j) are existing calculation functions and are not specifically limited here and can be set according to requirements. For example, the scaling of the first block in the upper left corner uses f(i) and f(j) rate sampling respectively.

[0094] In addition, the center coordinates can also be the positions corresponding to the static gaze point set on the target image. The static gaze point is a point set by the user on the target image, which can be set according to the user's requirements and is not specifically limited here.

[0095] Taking the target image as a Bayer image, the Bayer image format naturally has resolution loss compared to the RGB image format. When resampling, it is assumed that the target image is mapped to the Bayer image center frame near Figure 7 in the source image position, at this time there are 9 values available for R channel value calculation in the 5x5 Bayer image neighborhood, 12 values available for G channel, and 4 values available for B channel. While assuming that the target image is mapped to the RGB image center frame near Figure 7 in the source image position, and there are 25 values available for each of the three channels in the RGB domain.

[0096] In combination Figure 8 and Figure 9When the gaze region moves, since the zooming magnification of other regions is related to the distance from the gaze region, in order to ensure that the overall zooming magnification is constant, the application designs a zooming magnification lookup table according to the image size and the overall zooming magnification, and the zooming magnification lookup table (i.e. the coordinate mapping table) records the pixel coordinates before Grouping and the pixel coordinates after Grouping, and the pixel coordinates before Ungrouping and the pixel coordinates after Ungrouping. The continuous magnification zooming curve as shown in Figure 9 is formed by the pixel coordinates before Grouping and the pixel coordinates after Grouping, and the pixel coordinates before Ungrouping and the pixel coordinates after Ungrouping. In order to dynamically calculate the zooming magnification of each region when the gaze region moves.

[0097] The continuous magnification zooming curve adopts a derivative-smoothed curve, and taking a cubic polynomial curve as an example, the corresponding derivative quadratic polynomial curve is the corresponding coordinate zooming magnification, and the increment of the zooming magnification is relatively stable, the interpolation weight is selected according to the zooming magnification and the blur requirement in the interpolation curve (nearest, bilinear, lanczos, etc.), and the maximum zooming magnification, the minimum zooming magnification and the overall zooming magnification are set according to the requirement, and the coefficients of the polynomial are solved.

[0098] The fusion weight of the transition region and the distance of the pixel position from the edge of the gaze region form the transition band curve as shown in Figure 10 , i.e. the transition fusion weight curve, which makes the image change more smoothly when the gaze region is an identity transformation, and the transition band curve selects a point on the sine curve according to the distance from the center region, and the lanczos weight is larger when it is closer to the edge of the attention region, and the identity weight is larger when it is closer to the center of the attention region.

[0099] The vertical axis "transition fusion weight" of the transition fusion weight curve fuses the interpolation curve weight and the identity mapping weight, thereby generating the transition band weighted synthesis weight c, and the calculation formula is as follows:

[0100] c = b * w1 + a * (1-w1)

[0101] Wherein, b is the identity weight, a is the interpolation curve weight, and w1 is the transition fusion weight coefficient.

[0102] After the compressed target image is obtained based on the above embodiment, the compressed target image needs to be calculated to obtain a processed target image.

[0103] The calculation is performed on the compressed target image, including but not limited to NPU calculation on the target image or processing of the target image through units such as DSP, CPU, GPU, ASIC, and storage (SRAM / DDR / NAND), and the like, and the processing mode is not specifically limited here.

[0104] For example, the calculation on the compressed target image is denoising of the target image, which mainly utilizes a neural network algorithm to perform denoising processing on the compressed target image to obtain a processed target image.

[0105] Then, the decompressed target image is obtained according to the center coordinates and the processed target image.

[0106] In combination with Figure 11 The main execution steps for obtaining the decompressed target image according to the center coordinates and the processed target image are as follows: first, the coordinates of each region in the processed target image are calculated according to the center coordinates, and then the processed target image is decompressed based on the coordinates of each region in the processed target image to obtain the decompressed target image. The center coordinates are the positions corresponding to the gaze region of the target image.

[0107] According to the center coordinates, the coordinates of each region in the processed target image are calculated, which requires first determining the coordinates of each region in the non-gaze region of the processed target image according to the center coordinates, and then the coordinates of each region in the target image are composed of the center coordinates and the coordinates of each region in the non-gaze region.

[0108] According to the coordinates of each region in the processed target image, the decompressed target image is obtained by decompressing the processed target image, which requires first determining the coordinate mapping table of the non-gaze region based on the coordinates of each region in the non-gaze region of the processed target image, then performing reverse table lookup according to the gaze region, the center coordinates, the non-gaze region, and the coordinate mapping table to calculate the mapping coordinates and the scaling factor of each pixel position in the processed target image, then calculating the scaling weight of the gaze region, the scaling weight of the non-gaze region, and the fusion weight of the transition region according to the scaling factor, and finally weighting and synthesizing the scaling weight of the gaze region, the scaling weight of the non-gaze region, and the fusion weight of the transition region to output the decompressed target image.

[0109] In addition, the compressed target image can be input into an algorithm model such as an ISP, an NPU, etc. for continuous processing, where the ISP is an image signal processing architecture that can run various algorithm programs and process image signals in real time. The ISP mainly includes a CPU, a SUB IP, an IF, etc. The CPU is a central processing unit that can run various image processing algorithms such as AF, LSC, etc. and control peripheral devices. The CPU inside a modern ISP is generally an ARM Cortex-A series, such as Cortex-A5, Cortex-A7. The SUB IP is a general term for various functional modules that perform professional processing on images. Common SUB IPs include DIS, CSC, VRA, etc. The NPU is a neural network processing unit, which is a special processor for neural network calculation. The main function of the NPU is to accelerate neural network calculation, which can quickly realize data processing, transmission, and back propagation of neural networks. The working principle of the NPU is based on the calculation model of the neural network, and the core is the implementation of the deep learning algorithm. Generally, the NPU processes various computing operations in the neural network in parallel to achieve efficient calculation. In the neural network, each neuron is connected through weights, and during the training process, the NPU adjusts these weights according to sample data to make the neural network eventually achieve the best recognition effect.

[0110] The embodiment of the present application provides an image scaling processing method, device, terminal and storage medium, which comprises: obtaining eye movement information when a user observes a target image, then performing coordinate conversion on the eye movement information to obtain a center coordinate corresponding to the eye movement information, then determining a non-gaze area of the target image according to the center coordinate, compressing the non-gaze area of the target image to obtain a compressed target image, then performing calculation on the compressed target image to obtain a processed target image, and finally obtaining a decompressed target image according to the center coordinate and the processed target image. The present application scales the non-gaze area based on the center coordinate, so that the scaled image needs to be quickly restored to the original size after NPU processing, which ensures that other spatial domain related modules of the ISP are not affected, avoids the decline of the target image quality, and improves the scaling effect of the target image. In addition, the present application mainly saves storage bandwidth and computing hardware resources by compression, effectively reduces the demand for hardware resources under the premise of ensuring the image effect of the gaze point area. It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0111] The following is a device embodiment of the present application. For details not described in detail, reference can be made to the corresponding method embodiments described above.

[0112] Figure 12 A structural diagram of an image scaling processing apparatus provided by an embodiment of the present application is shown. For ease of illustration, only parts related to the embodiment of the present application are shown. The image scaling processing apparatus includes an acquisition module 1201, a first calculation module 1202, a compression module 1203, a second calculation module 1204, and a decompression module 1205, and specifically as follows:

[0113] The acquisition module 1201 is configured to acquire eye movement information when a user observes a target image.

[0114] The first calculation module 1202 is configured to perform coordinate conversion on the eye movement information to obtain a center coordinate corresponding to the eye movement information.

[0115] The compression module 1203 is configured to determine a non-gaze region of the target image according to the center coordinate, compress the non-gaze region of the target image, and obtain a compressed target image.

[0116] The second calculation module 1204 is configured to perform calculation on the compressed target image to obtain a processed target image.

[0117] The decompression module 1205 is configured to obtain a decompressed target image according to the center coordinate and the processed target image.

[0118] In a possible implementation, the compression module 1203 is further configured to determine coordinates of each region in the non-gaze region of the target image according to the center coordinate.

[0119] The coordinates of each region in the target image are composed of the center coordinate and the coordinates of each region in the non-gaze region.

[0120] The target image is compressed based on the coordinates of each region in the target image to obtain the compressed target image.

[0121] In a possible implementation, the compression module 1203 is further configured to determine a coordinate mapping table of the non-gaze region based on the coordinates of each region in the non-gaze region of the target image.

[0122] The mapping coordinates and the scaling factor of each pixel position in the target image are calculated by table lookup according to the gaze region, the center coordinate, the non-gaze region, and the coordinate mapping table.

[0123] The scaling weight of the gaze region, the scaling weight of the non-gaze region, and the fusion weight of the transition region in the target image are calculated according to the scaling factor, wherein a continuous factor scaling curve corresponding to the scaling weight is a derivative smooth curve, a point on the derivative smooth curve is used to represent the scaling factor of the pixel coordinate position, and the scaling factor is increasing,

[0124] The transition fusion weight curve corresponding to the fusion weight is a derivative smoothing curve, and points on the derivative smoothing curve are used to represent the excessive fusion weight of the pixel position and the distance from the edge of the gaze region. The excessive fusion weight fuses the difference curve weight and the identity mapping weight, and the weight of the pixel position close to the edge of the gaze region is large, and the identity mapping weight of the pixel position close to the gaze region is large.

[0125] The scaling weight of the gaze region, the scaling weight of the non-gaze region and the fusion weight of the transition region are weighted and synthesized to output the compressed target image.

[0126] In a possible implementation, the image algorithm operation used for the calculation processing of the compressed target image at least includes a denoising operation and a specific image algorithm operation.

[0127] In a possible implementation, the denoising algorithm is a neural network algorithm.

[0128] The decompression module is also used to perform denoising processing on the compressed target image by using the neural network algorithm to obtain the processed target image.

[0129] In a possible implementation, the decompression module 1205 is also used to calculate the coordinates of each region in the processed target image according to the center coordinates.

[0130] The decompression module is also used to perform denoising processing on the compressed target image by using the neural network algorithm to obtain the processed target image.

[0131] In a possible implementation, the center coordinates are the positions corresponding to the gaze region of the target image.

[0132] The decompression module 1205 is also used to determine the coordinates of each region in the non-gaze region of the processed target image according to the center coordinates.

[0133] The coordinates of each region in the target image are composed of the center coordinates and the coordinates of each region in the non-gaze region.

[0134] In a possible implementation, the decompression module 1205 is also used to determine the coordinate mapping table of the non-gaze region based on the coordinates of each region in the non-gaze region of the processed target image.

[0135] According to the gaze region, the center coordinates, the non-gaze region and the coordinate mapping table, the mapping coordinates and the scaling factor of each pixel position in the processed target image are calculated by reverse lookup table.

[0136] According to the scaling factor, the scaling weight of the gaze region, the scaling weight of the non-gaze region and the fusion weight of the transition region in the processed target image are calculated.

[0137] The scaling weights of the gaze region, the scaling weights of the non-gaze region, and the fusion weights of the transition region are weighted and synthesized to output the decompressed target image.

[0138] In one possible implementation, the center coordinates are the position corresponding to the gaze area of ​​the target image or the position corresponding to the gaze point set on the target image.

[0139] In one possible implementation, the gaze point is either a dynamic gaze point obtained from eye tracking or a static gaze point that does not change based on eye tracking information.

[0140] In one possible implementation, the target image includes at least a Bayer image, an RGB image, a RAW image, and a YUV image.

[0141] In one possible implementation, the computational methods for calculating the compressed target image include at least NPU computation and neural network model computation.

[0142] Figure 13 This is a schematic diagram of a terminal provided in an embodiment of the present invention. Figure 13 As shown, the terminal 13 in this embodiment includes: a processor 1301, a memory 1302, and a computer program 1303 stored in the memory 1302 and executable on the processor 1301. When the processor 1301 executes the computer program 1303, it implements the steps in the various image scaling processing method embodiments described above, for example... Figure 3 Steps 301 to 305 are shown. Alternatively, when processor 1301 executes computer program 1303, it implements the functions of each module / unit in the above-described image scaling processing device embodiments, for example... Figure 12 The functions of modules / units 1201 to 1205 shown.

[0143] The present invention also provides a readable storage medium storing a computer program, which, when executed by a processor, is used to implement the image scaling processing method provided in the various embodiments described above.

[0144] The readable storage medium can be a computer storage medium or a communication medium. A communication medium includes any medium that facilitates the transfer of computer programs from one location to another. A computer storage medium can be any available medium accessible to a general-purpose or special-purpose computer. For example, a readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application-Specific Integrated Circuit (ASIC). Alternatively, the ASIC can be located in a user equipment. Of course, the processor and the readable storage medium can also exist as discrete components in a communication device. The readable storage medium can be a read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0145] The present invention also provides a program product including executable instructions stored in a readable storage medium. At least one processor of the device can read the executable instructions from the readable storage medium, and the execution of the executable instructions by the at least one processor causes the device to implement the image scaling processing methods provided in the various embodiments described above.

[0146] In the embodiments of the above-described device, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0147] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An image scaling processing method, characterized in that, include: Acquire eye movement information when the user observes the target image; The eye movement information is converted into coordinates to obtain the center coordinates corresponding to the eye movement information; The non-gazing region of the target image is determined based on the center coordinates, and the non-gazing region of the target image is compressed to obtain a compressed target image; The compressed target image is processed to obtain the processed target image; Based on the center coordinates and the processed target image, the decompressed target image is obtained; The step of determining the non-gazing region of the target image based on the center coordinates, and compressing the non-gazing region of the target image to obtain a compressed target image includes: Based on the center coordinates, determine the coordinates of each region in the non-gazing region of the target image; The coordinates of each region in the target image are constituted by the center coordinates and the coordinates of each region in the non-gazing region; The target image is compressed based on the coordinates of each region in the target image to obtain the compressed target image; The step of compressing the target image based on the coordinates of each region in the target image to obtain the compressed target image includes: Based on the coordinates of each region in the non-gazing region of the target image, determine the coordinate mapping table of the non-gazing region; The mapping coordinates and scaling factor of each pixel in the target image are calculated by looking up the coordinate mapping table based on the gaze area, center coordinates, non-gaze area and the coordinate mapping table. The scaling weights of the gaze region, non-gaze region, and transition region in the target image are calculated based on the scaling factor. The continuous scaling curve corresponding to these scaling weights is a derivative-smoothed curve. Points on this continuous scaling curve represent the scaling factor at pixel coordinate positions, and the scaling factor increases incrementally. The transition fusion weight curve corresponding to the fusion weight is a derivative smoothing curve. The points on the transition fusion weight curve are used to characterize the transition fusion weight of the distance between the pixel position and the edge of the gaze region. The transition fusion weight combines the difference curve weight and the identity mapping weight. The pixel position closer to the edge of the gaze region has a larger weight, and the identity mapping weight of the pixel position closer to the gaze region has a larger weight. The scaling weights of the gaze region, the scaling weights of the non-gaze region, and the fusion weights of the transition region are weighted and synthesized to output the compressed target image.

2. The image scaling processing method as described in claim 1, characterized in that, The image algorithm used to process the compressed target image includes at least a denoising algorithm.

3. The image scaling processing method as described in claim 2, characterized in that, The denoising algorithm is a neural network algorithm; The step of performing calculations on the compressed target image to obtain the processed target image includes: The compressed target image is denoised using a neural network algorithm to obtain the processed target image.

4. The image scaling processing method as described in claim 1, characterized in that, The process of obtaining the decompressed target image based on the center coordinates and the processed target image includes: Based on the center coordinates, calculate the coordinates of each region in the processed target image; The processed target image is decompressed based on the coordinates of each region in the processed target image to obtain the decompressed target image.

5. The image scaling processing method as described in claim 4, characterized in that, The center coordinates are the positions corresponding to the gaze area of ​​the target image; The step of calculating the coordinates of each region in the processed target image based on the center coordinates includes: Based on the center coordinates, determine the coordinates of each region in the non-gazing region of the processed target image; The coordinates of each region in the target image are constituted by the center coordinates and the coordinates of each region in the non-gazing region.

6. The image scaling processing method as described in claim 5, characterized in that, The step of decompressing the processed target image based on the coordinates of each region in the processed target image to obtain the decompressed target image includes: Based on the coordinates of each region in the non-gazing region of the processed target image, a coordinate mapping table for the non-gazing region is determined. Based on the gaze area, center coordinates, non-gaze area, and coordinate mapping table, a reverse lookup table is performed to calculate the mapped coordinates and scaling factor of each pixel in the processed target image. The scaling weights of the gaze region, the non-gaze region, and the fusion weights of the transition region in the processed target image are calculated based on the scaling factor. The scaling weights of the gaze region, the scaling weights of the non-gaze region, and the fusion weights of the transition region are weighted and synthesized to output the decompressed target image.

7. The image scaling processing method as described in claim 1, characterized in that, The center coordinates are the position corresponding to the gaze area of ​​the target image or the position corresponding to the gaze point set on the target image.

8. The image scaling processing method as described in claim 7, characterized in that, The gaze point is either a dynamic information obtained from eye tracking or a static gaze point that does not change based on eye tracking information.

9. The image scaling processing method according to any one of claims 1-8, characterized in that, The target image includes at least Bayer images, RGB images, RAW images, and YUV images.

10. The image scaling processing method according to any one of claims 1-8, characterized in that, The calculation method for the compressed target image includes at least NPU calculation and neural network model calculation.

11. An image scaling processing apparatus, characterized in that, include: The acquisition module is used to acquire eye movement information when the user observes the target image; The first calculation module is used to perform coordinate conversion on the eye movement information to obtain the center coordinates corresponding to the eye movement information; A compression module is used to determine the non-gazing region of the target image based on the center coordinates, and to compress the non-gazing region of the target image to obtain a compressed target image; The second calculation module is used to calculate the compressed target image to obtain the processed target image; The decompression module is used to obtain the decompressed target image based on the center coordinates and the processed target image; The step of determining the non-gazing region of the target image based on the center coordinates, and compressing the non-gazing region of the target image to obtain a compressed target image includes: Based on the center coordinates, determine the coordinates of each region in the non-gazing region of the target image; The coordinates of each region in the target image are constituted by the center coordinates and the coordinates of each region in the non-gazing region; The target image is compressed based on the coordinates of each region in the target image to obtain the compressed target image; The step of compressing the target image based on the coordinates of each region in the target image to obtain the compressed target image includes: Based on the coordinates of each region in the non-gazing region of the target image, determine the coordinate mapping table of the non-gazing region; The mapping coordinates and scaling factor of each pixel in the target image are calculated by looking up the coordinate mapping table based on the gaze area, center coordinates, non-gaze area and the coordinate mapping table. The scaling weights of the gaze region, non-gaze region, and transition region in the target image are calculated based on the scaling factor. The continuous scaling curve corresponding to these scaling weights is a derivative-smoothed curve. Points on this continuous scaling curve represent the scaling factor at pixel coordinate positions, and the scaling factor increases incrementally. The transition fusion weight curve corresponding to the fusion weight is a derivative smoothing curve. The points on the transition fusion weight curve are used to characterize the transition fusion weight of the distance between the pixel position and the edge of the gaze region. The transition fusion weight combines the difference curve weight and the identity mapping weight. The pixel position closer to the edge of the gaze region has a larger weight, and the identity mapping weight of the pixel position closer to the gaze region has a larger weight. The scaling weights of the gaze region, the scaling weights of the non-gaze region, and the fusion weights of the transition region are weighted and synthesized to output the compressed target image.

12. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the image scaling processing method as described in any one of claims 1 to 10.

13. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the image scaling processing method as described in any one of claims 1 to 10.

Images