Image scaling processing method and device, terminal and storage medium

Abstract

The invention discloses an image zooming processing method and device, a terminal and a storage medium, and the method comprises the steps: obtaining eye movement information when a user observes a target image, carrying out the coordinate conversion of the eye movement information, obtaining a center coordinate corresponding to the eye movement information, determining a non-gazing region of the target image according to the center coordinate, and carrying out the zooming processing of the target image. The method comprises the steps of compressing a non-gazing area of a target image to obtain a compressed target image, then calculating the compressed target image to obtain a processed target image, and finally obtaining a decompressed target image according to a center coordinate and the processed target image. According to the method, storage bandwidth and calculation hardware resources are mainly saved through compression, and the hardware resource demand is effectively reduced on the premise that the image effect of the gazing point area is guaranteed.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image scaling processing method, device, terminal and storage medium. Background Art

[0002] In computer image processing and computer graphics, image scaling is the process of adjusting the size of a digital image. Image scaling is a non-trivial process that requires a trade-off between processing efficiency and the smoothness and clarity of the result. When the size of an image is increased, the pixels that make up the image become more visible, making the image appear "soft". Conversely, reducing an image will enhance its smoothness and clarity.

[0003] At present, deep learning is generally used to scale the target image.

[0004] However, the above method may cause the problem of image quality degradation, resulting in poor scaling effect. Summary of the invention

[0005] The main purpose of the present application is to provide an image scaling processing method, device, terminal and storage medium to solve the problem of image quality degradation in the related art.

[0006] In order to achieve the above objectives, in a first aspect, the present application provides an image scaling processing method, comprising:

[0007] Obtaining eye movement information when the user observes the target image;

[0008] Perform coordinate conversion on the eye movement information to obtain the center coordinates corresponding to the eye movement information;

[0009] Determine a non-gaze area of ​​the target image according to the central coordinates, compress the non-gaze area of ​​the target image, and obtain a compressed target image;

[0010] Calculate the compressed target image to obtain a processed target image;

[0011] The decompressed target image is obtained according to the center coordinates and the processed target image.

[0012] In a possible implementation, determining a non-gaze area of ​​a target image according to a central coordinate, compressing the non-gaze area of ​​the target image, and obtaining a compressed target image includes:

[0013] Determine the coordinates of each area in the non-gaze area of ​​the target image according to the center coordinates;

[0014] The coordinates of each area in the target image are formed by the central coordinates and the coordinates of each area in the non-gaze area;

[0015] The target image is compressed based on the coordinates of each region in the target image to obtain a compressed target image.

[0016] In a possible implementation, compressing the target image based on the coordinates of each region in the target image to obtain a compressed target image includes:

[0017] Determine a coordinate mapping table of the non-gaze area based on the coordinates of each area in the non-gaze area of ​​the target image;

[0018] According to the gaze area, center coordinates, non-gaze area and coordinate mapping table, a table lookup is performed to calculate the mapping coordinates and zoom ratio of each pixel position in the target image;

[0019] The scaling weight of the gaze area, the scaling weight of the non-gaze area and the fusion weight of the transition area in the target image are calculated according to the scaling ratio, wherein the continuous scaling curve corresponding to the scaling weight is a derivative smoothing curve, and the points on the derivative smoothing curve are used to represent the scaling ratio of the pixel coordinate position, and the scaling ratio is increasing.

[0020] The transition fusion weight curve corresponding to the fusion weight is a derivative smoothing curve. The points on the derivative smoothing curve are used to characterize the over-fusion weight of the distance between the pixel position and the edge of the gaze area. The over-fusion weight combines the difference curve weight and the identity mapping weight. The pixel position close to the edge of the gaze area has a large weight, and the pixel position close to the gaze area has a large identity mapping weight.

[0021] The scaling weight of the gaze area, the scaling weight of the non-gaze area and the fusion weight of the transition area are weighted and synthesized to output a compressed target image.

[0022] In a possible implementation, the image algorithm operation used to perform computational processing on the compressed target image includes at least a denoising operation and a specific image algorithm operation.

[0023] In one possible implementation, the denoising algorithm is a neural network algorithm;

[0024] The compressed target image is subjected to computational processing to obtain a processed target image, including:

[0025] The compressed target image is denoised using a neural network algorithm to obtain a processed target image.

[0026] In a possible implementation, obtaining a decompressed target image according to the center coordinates and the processed target image includes:

[0027] According to the center coordinates, the coordinates of each area in the processed target image are calculated;

[0028] The processed target image is decompressed based on the coordinates of each region in the processed target image to obtain a decompressed target image.

[0029] In a possible implementation, the center coordinates are positions corresponding to the gaze area of ​​the target image;

[0030] Based on the center coordinates, the coordinates of each area in the processed target image are calculated, including:

[0031] Determine the coordinates of each area in the non-gaze area of ​​the processed target image according to the center coordinates;

[0032] The coordinates of each area in the target image are composed of the central coordinates and the coordinates of each area in the non-gaze area.

[0033] In a possible implementation, decompressing the processed target image based on the coordinates of each region in the processed target image to obtain the decompressed target image includes:

[0034] Determine a coordinate mapping table of the non-gaze area based on the coordinates of each area in the non-gaze area of ​​the processed target image;

[0035] According to the gaze area, the center coordinates, the non-gaze area and the coordinate mapping table, a reverse lookup is performed to calculate the mapping coordinates and the zoom factor of each pixel position in the processed target image;

[0036] Calculate the scaling weight of the gaze area, the scaling weight of the non-gaze area, and the fusion weight of the transition area in the processed target image according to the scaling ratio;

[0037] The scaling weight of the gaze area, the scaling weight of the non-gaze area, and the fusion weight of the transition area are weighted and synthesized to output the decompressed target image.

[0038] In a possible implementation manner, the center coordinates are a position corresponding to a gaze area of ​​the target image or a position corresponding to a gaze point set on the target image.

[0039] In a possible implementation, the gaze point is dynamic information obtained according to eye tracking or a static gaze point that does not change according to eye movement information.

[0040] In a possible implementation manner, the target image includes at least a Bayer image, an RGB image, a RAW image, and a YUV image.

[0041] In a possible implementation, a calculation method for calculating the compressed target image includes at least NPU calculation and neural network model calculation.

[0042] In a second aspect, an embodiment of the present invention provides an image scaling processing device, including:

[0043] An acquisition module, used to acquire eye movement information when a user observes a target image;

[0044] A first calculation module is used to convert the eye movement information into coordinates to obtain the center coordinates corresponding to the eye movement information;

[0045] A compression module, used for determining a non-gaze area of ​​a target image according to a central coordinate, and compressing the non-gaze area of ​​the target image to obtain a compressed target image;

[0046] A second calculation module is used to calculate the compressed target image to obtain a processed target image;

[0047] The decompression module is used to obtain a decompressed target image according to the center coordinates and the processed target image.

[0048] In a third aspect, an embodiment of the present invention provides a terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above image scaling processing methods when executing the computer program.

[0049] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above image scaling processing methods are implemented.

[0050] The embodiments of the present invention provide an image scaling processing method, device, terminal and storage medium, including: obtaining eye movement information when a user observes a target image, then performing coordinate conversion on the eye movement information to obtain the center coordinates corresponding to the eye movement information, then determining the non-gaze area of ​​the target image based on the center coordinates, compressing the non-gaze area of ​​the target image to obtain a compressed target image, then calculating the compressed target image to obtain a processed target image, and finally obtaining a decompressed target image based on the center coordinates and the processed target image. The present invention scales the non-gaze area based on the center coordinates, so that the scaled image needs to be quickly restored to its original size after the NPU processing is completed, ensuring that other spatial domain-related modules of the ISP will not be affected, avoiding the degradation of the target image quality, and improving the scaling effect of the target image. In addition, the present invention mainly saves storage bandwidth and computing hardware resources through compression, and effectively reduces the hardware resource requirements while ensuring the image effect of the gaze point area. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The drawings constituting a part of this application are used to provide a further understanding of this application, so that other features, purposes and advantages of this application become more obvious. The schematic embodiment drawings and their descriptions of this application are used to explain this application and do not constitute an improper limitation on this application. In the drawings:

[0052] Figure 1 is a schematic diagram of coordinate mapping of an image provided by an embodiment of the present invention;

[0053] Figure 2 is a schematic diagram of setting magnification of an image partition provided by an embodiment of the present invention;

[0054] Figure 3 is a flowchart of an implementation method of an image scaling processing method provided by an embodiment of the present invention;

[0055] Figure 4 is a flow chart of an implementation of an image scaling method provided by another embodiment of the present invention;

[0056] Figure 5 is a flow chart of an implementation of an image compression method provided by an embodiment of the present invention;

[0057] Figure 6 is a schematic diagram of setting the magnification of each area of ​​an image provided by an embodiment of the present invention;

[0058] Figure 7 is a schematic diagram of an RGB image format and a Bayer image format provided by an embodiment of the present invention;

[0059] Figure 8 is a schematic diagram of another image magnification setting for each region provided by an embodiment of the present invention;

[0060] Fig. 9 is a continuous magnification scaling curve diagram provided by an embodiment of the present invention;

[0061] Fig.10 is a transition zone curve diagram provided by an embodiment of the present invention;

[0062] Fig.11 is a flow chart of an implementation of an image decompression method provided by an embodiment of the present invention;

[0063] Fig.12 is a structural schematic diagram of an image scaling processing device provided by an embodiment of the present invention;

[0064] Fig.13 is a schematic diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein.

[0067] It should be understood that in various embodiments of the present invention, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0068] It should be understood that in the present invention, "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0069] It should be understood that in the present invention, "plurality" refers to two or more than two. "And / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "Contains A, B and C", "Contains A, B, C" means that A, B, and C are all included, "Contains A, B or C" means that one of A, B, and C is included, and "Contains A, B and / or C" means that any one, any two, or any three of A, B, and C are included.

[0070] It should be understood that in the present invention, "B corresponding to A", "B corresponding to A", "A corresponds to B" or "B corresponds to A" means that B is associated with A and B can be determined based on A. Determining B based on A does not mean determining B based only on A, but B can also be determined based on A and / or other information. A and B match when the similarity between A and B is greater than or equal to a preset threshold.

[0071] Depending on the context, "if" as used herein may be interpreted as "when" or "when" or "in response to determining" or "in response to detecting."

[0072] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0073] Furthermore, in order to make the objectives, technical solutions and advantages of the present invention more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0074] In computer image processing and computer graphics, image scaling refers to the process of adjusting the size of a digital image.

[0075] In the prior art, images are mainly scaled by using a scaler, such as Figure 1 As shown in the figure, its essence is coordinate mapping. The point D of the target image finds the corresponding point Si in the source image, and then interpolates the target point D through the interpolation formula.

[0076] Among them, the scaler mode includes scaler down and scaler up. For RGB images, the scaler down and scaler up operations with different magnifications are mainly performed on the image through partitioning. Figure 2 As shown, the image is divided into multiple blocks, and then the magnifications used for scaling are marked on different blocks. For example, the first block in the upper left corner of the image is scaled at a magnification of 4.

[0077] However, the existing technology may suffer from the problem of image quality degradation, resulting in poor scaling effect.

[0078] Therefore, the present application provides an image scaling processing method to solve the above-mentioned problem and improve the image scaling effect.

[0079] In one embodiment, Figure 3 As shown, a method for image scaling processing is provided, comprising the following steps:

[0080] Step S301: Acquire eye movement information of a user when observing a target image.

[0081] Among them, eye movement information can be obtained through eye movement equipment or other equipment. The eye movement information mainly includes information such as the user's gaze position on the screen or image and the corresponding timestamp.

[0082] Step S302: performing coordinate conversion on the eye movement information to obtain the center coordinates corresponding to the eye movement information.

[0083] This application mainly uses MCU to convert the coordinates of eye movement information. Through the conversion, the gaze position of the user on the image, that is, the center coordinates, can be obtained.

[0084] Step S303: determining a non-gaze area of ​​the target image according to the central coordinates, and compressing the non-gaze area of ​​the target image to obtain a compressed target image.

[0085] The target image includes images in various formats, including but not limited to images in Bayer, RAW, RGB, and YUV formats.

[0086] Step S304: Calculate the compressed target image to obtain a processed target image.

[0087] Step S305: Obtain a decompressed target image based on the center coordinates and the processed target image.

[0088] Combination Figure 4 Taking the target image as a Bayer image as an example, the compressed Bayer image is obtained based on the center coordinates and the Bayer image, which is implemented by the Bayer Group module, while the decompressed Bayer image is obtained based on the center coordinates and the processed Bayer image, which is implemented by the Bayer Ungroup module.

[0089] The following is a detailed description of the process of obtaining the compressed Bayer image based on the center coordinates and the Bayer image by the Bayer Group module. Figure 5 The main execution steps are: first, according to the center coordinates, calculate the coordinates of each area in the Bayer image, and then compress the Bayer image based on the coordinates of each area in the Bayer image to obtain a compressed Bayer image. Among them, the center coordinates are the positions corresponding to the gaze area of ​​the Bayer image.

[0090] Among them, according to the central coordinates, the coordinates of each area in the Bayer image are calculated. It is necessary to first determine the coordinates of each area in the non-gaze area of ​​the Bayer image according to the central coordinates, and then the coordinates of each area in the Bayer image are constructed by the central coordinates and the coordinates of each area in the non-gaze area.

[0091] Among them, the Bayer image is compressed based on the coordinates of each area in the Bayer image to obtain a compressed Bayer image. It is necessary to first determine the coordinate mapping table of the non-gaze area based on the coordinates of each area in the non-gaze area of ​​the Bayer image, and then look up the table according to the gaze area, center coordinates, non-gaze area and coordinate mapping table to calculate the mapping coordinates and scaling factor of each pixel position in the Bayer image, and then calculate the scaling weight of the gaze area, the scaling weight of the non-gaze area and the fusion weight of the transition area in the Bayer image according to the scaling factor, and finally perform weighted synthesis on the scaling weight of the gaze area, the scaling weight of the non-gaze area and the fusion weight of the transition area to output the compressed Bayer image.

[0092] Combination Figure 6 , the center coordinate is the position corresponding to the gaze area of ​​the target image, and the gaze area is scaled at a fixed rate. In addition, a transition zone is set at the edge of the gaze area, which can ensure a natural image transition. The size of the gaze area can be adjusted according to demand.

[0093] The non-gaze area is outside the transition zone (transition zone). The non-gaze area is divided into multiple blocks by lines, and each block is scaled by a change factor. The change factor is calculated by functions f(i) and f(j), which is related to the distance from the gaze area. Among them, functions f(i) and f(j) are existing calculation functions, which are not specifically limited here and can be set according to needs. For example, the scaling of the first block in the upper left corner adopts f(i) sampling and f(j) sampling respectively.

[0094] In addition, the center coordinates may also be a position corresponding to a static gaze point set on the target image. The static gaze point is a point set by the user on the target image, and the point may be set according to user needs, and is not specifically limited here.

[0095] The target image is a Bayer image. The Bayer image format naturally has a loss of resolution compared to the RGB image format. When resampling, it is assumed that the target image is mapped to the source image position. Figure 7 Near the center box of the Bayer image in , there are 9 values ​​available for calculating the R channel value in the Bayer image neighborhood 5x5, 12 values ​​available for the G channel, and 4 values ​​available for the B channel. Assuming that the target image is mapped to the source image position Figure 7 The RGB image is near the center box, and in the RGB domain, 25 values ​​are available for each of the three RGB channels.

[0096] Combination Figure 8 and Fig. 9When the gaze area moves, since the zoom ratio of other areas is related to the distance from the gaze area, in order to ensure that the overall zoom ratio remains unchanged, the present application designs a zoom ratio lookup table according to the image size and the overall zoom ratio. The zoom ratio lookup table (i.e., the coordinate mapping table) records the pixel coordinates before Group and the pixel coordinates after Group, as well as the pixel coordinates before Ungroup and the pixel coordinates after Ungroup. Among them, the pixel coordinates before Group and the pixel coordinates after Group, as well as the pixel coordinates before Ungroup and the pixel coordinates after Ungroup are formed as follows: Fig. 9 The continuous magnification scaling curve shown in Figure 1 is used to dynamically calculate the scaling factor of each area when the gaze area moves.

[0097] The continuous magnification scaling curve adopts a derivative smoothing curve. Taking the cubic polynomial curve as an example, its derivative quadratic polynomial curve is the corresponding coordinate scaling factor. The increasing scaling factor is relatively stable. The interpolation weight selects points on the interpolation curve (nearest, bilinear, lanczos, etc.) according to the scaling factor and fuzzy requirements, and sets the maximum scaling factor, minimum scaling factor, and overall scaling factor according to the requirements to solve the coefficients of the polynomial.

[0098] The fusion weight of the transition area and the distance between the pixel position and the edge of the gaze area form Fig.10 The transition band curve shown, that is, the transition fusion weight curve, makes the image change smoother when the gaze area is an identical transformation. The transition band curve selects points on the sine curve according to the distance from the central area, and the closer to the edge of the focus area, the larger the lanczos weight, and the closer to the center of the focus area, the larger the identity weight.

[0099] Among them, the vertical axis "transition fusion weight" of the transition fusion weight curve combines the interpolation curve weight and the identity mapping weight to generate the transition zone weighted synthesis weight c. The calculation formula is as follows:

[0100] c=b*w1+a*(1-w1)

[0101] Among them, b is the constant weight, a is the interpolation curve weight, and w1 is the transition fusion weight coefficient.

[0102] After the compressed target image is obtained based on the above embodiment, it is necessary to perform calculations on the compressed target image to obtain a processed target image.

[0103] Among them, calculations are performed on the compressed target image, which includes but is not limited to NPU calculations on the target image or processing of the target image through units such as DSP, CPU, GPU, ASIC and storage (SRAM / DDR / NAND), etc. The processing method here is not specifically limited.

[0104] For example, calculating the compressed target image is to denoise the target image, which mainly uses a neural network algorithm to denoise the compressed target image to obtain a processed target image.

[0105] Afterwards, it is necessary to obtain the decompressed target image based on the center coordinates and the processed target image.

[0106] Combination Fig.11 , for obtaining a decompressed target image based on the center coordinates and the processed target image, the main execution steps are: firstly, according to the center coordinates, the coordinates of each area in the processed target image are calculated, and then the processed target image is decompressed based on the coordinates of each area in the processed target image to obtain the decompressed target image. Among them, the center coordinates are the position corresponding to the gaze area of ​​the target image.

[0107] Among them, the coordinates of each area in the processed target image are calculated based on the central coordinates. It is necessary to first determine the coordinates of each area in the non-gaze area of ​​the processed target image based on the central coordinates, and then the coordinates of each area in the target image are constructed by the central coordinates and the coordinates of each area in the non-gaze area.

[0108] Among them, the processed target image is decompressed based on the coordinates of each area in the processed target image to obtain the decompressed target image. It is necessary to first determine the coordinate mapping table of the non-gaze area based on the coordinates of each area in the non-gaze area of ​​the processed target image, and then perform a reverse lookup table according to the gaze area, the center coordinates, the non-gaze area and the coordinate mapping table to calculate the mapping coordinates and the scaling factor of each pixel position in the processed target image, and then calculate the scaling weight of the gaze area, the scaling weight of the non-gaze area and the fusion weight of the transition area in the processed target image according to the scaling factor, and finally perform weighted synthesis on the scaling weight of the gaze area, the scaling weight of the non-gaze area and the fusion weight of the transition area to output the decompressed target image.

[0109] In addition, the compressed target image can be input into algorithm models such as ISP and NPU for further processing. Among them, ISP is an image signal processing architecture that can run various algorithm programs and process image signals in real time. ISP mainly includes CPU, SUP IP, IF and other devices. CPU is the central processing unit, which can run various image processing algorithms such as AF and LSC and control peripheral devices. The CPU inside the modern ISP is generally ARM Cortex-A series, such as Cortex-A5 and Cortex-A7. SUB IP is a general name for various functional modules, which perform their own professional processing on the image. Common SUB IPs include DIS, CSC, VRA, etc. Among them, NPU is a neural network processing unit, which is a dedicated processor for neural network calculation. The main function of NPU is to accelerate neural network calculation, and can quickly realize the calculation operations such as data processing, transmission and back propagation of neural network. The working principle of NPU is based on the calculation model of neural network, and its core is the implementation of deep learning algorithm. Generally speaking, NPU realizes efficient calculation by parallelizing various calculation operations in neural network. In a neural network, neurons are connected through weights. During the training process, the NPU will continuously adjust these weights based on sample data so that the neural network can ultimately achieve the optimal recognition effect.

[0110] The embodiment of the present invention provides an image scaling processing method, device, terminal and storage medium, including: obtaining eye movement information when a user observes a target image, then performing coordinate conversion on the eye movement information to obtain the center coordinates corresponding to the eye movement information, and then determining the non-gaze area of ​​the target image according to the center coordinates, compressing the non-gaze area of ​​the target image to obtain a compressed target image, and then calculating the compressed target image to obtain a processed target image, and finally obtaining a decompressed target image according to the center coordinates and the processed target image. The present invention scales the non-gaze area based on the center coordinates, so that the scaled image needs to be quickly restored to its original size after the NPU processing is completed, ensuring that other spatial domain-related modules of the ISP will not be affected, avoiding the degradation of the target image quality, and improving the scaling effect of the target image. In addition, the present invention mainly saves storage bandwidth and computing hardware resources through compression, and effectively reduces the hardware resource requirements while ensuring the image effect of the gaze point area. It should be understood that the size of the sequence number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0111] The following is an embodiment of the device of the present invention. For details not described in detail therein, reference may be made to the corresponding method embodiment described above.

[0112] Fig.12 The structure diagram of an image scaling processing device provided by an embodiment of the present invention is shown. For the convenience of description, only the part related to the embodiment of the present invention is shown. The image scaling processing device includes an acquisition module 1201, a first calculation module 1202, a decompression module 1203, a second calculation module 1204 and a decompression module 1205, which are as follows:

[0113] An acquisition module 1201 is used to acquire eye movement information when a user observes a target image;

[0114] A first calculation module 1202 is used to convert the eye movement information into coordinates to obtain the center coordinates corresponding to the eye movement information;

[0115] The compression module 1203 is used to determine the non-gaze area of ​​the target image according to the central coordinates, and compress the non-gaze area of ​​the target image to obtain a compressed target image.

[0116] The second calculation module 1204 is used to calculate the compressed target image to obtain a processed target image;

[0117] The decompression module 1205 is used to obtain a decompressed target image according to the center coordinates and the processed target image.

[0118] In a possible implementation, the compression module 1203 is further configured to determine the coordinates of each area in the non-gaze area of ​​the target image according to the center coordinates;

[0119] The coordinates of each area in the target image are formed by the central coordinates and the coordinates of each area in the non-gaze area;

[0120] The target image is compressed based on the coordinates of each region in the target image to obtain a compressed target image.

[0121] In a possible implementation, the compression module 1203 is further configured to determine a coordinate mapping table of the non-gaze area based on the coordinates of each area in the non-gaze area of ​​the target image;

[0122] According to the gaze area, center coordinates, non-gaze area and coordinate mapping table, a table lookup is performed to calculate the mapping coordinates and zoom ratio of each pixel position in the target image;

[0123] The scaling weight of the gaze area, the scaling weight of the non-gaze area and the fusion weight of the transition area in the target image are calculated according to the scaling ratio, wherein the continuous scaling curve corresponding to the scaling weight is a derivative smoothing curve, and the points on the derivative smoothing curve are used to represent the scaling ratio of the pixel coordinate position, and the scaling ratio is increasing.

[0124] The transition fusion weight curve corresponding to the fusion weight is a derivative smoothing curve. The points on the derivative smoothing curve are used to characterize the over-fusion weight of the distance between the pixel position and the edge of the gaze area. The over-fusion weight combines the difference curve weight and the identity mapping weight. The pixel position close to the edge of the gaze area has a large weight, and the pixel position close to the gaze area has a large identity mapping weight.

[0125] The scaling weight of the gaze area, the scaling weight of the non-gaze area and the fusion weight of the transition area are weighted and synthesized to output a compressed target image.

[0126] In a possible implementation, the image algorithm operation used to perform computational processing on the compressed target image includes at least a denoising operation and a specific image algorithm operation.

[0127] In one possible implementation, the denoising algorithm is a neural network algorithm;

[0128] The decompression module is also used to perform denoising on the compressed target image using a neural network algorithm to obtain a processed target image.

[0129] In a possible implementation, the decompression module 1205 is further used to calculate the coordinates of each area in the processed target image according to the center coordinates;

[0130] The processed target image is decompressed based on the coordinates of each region in the processed target image to obtain a decompressed target image.

[0131] In a possible implementation, the center coordinates are positions corresponding to the gaze area of ​​the target image;

[0132] The decompression module 1205 is further used to determine the coordinates of each area in the non-gaze area of ​​the processed target image according to the center coordinates;

[0133] The coordinates of each area in the target image are composed of the central coordinates and the coordinates of each area in the non-gaze area.

[0134] In a possible implementation, the decompression module 1205 is further configured to determine a coordinate mapping table of the non-gaze area based on the coordinates of each area in the non-gaze area of ​​the processed target image;

[0135] According to the gaze area, the center coordinates, the non-gaze area and the coordinate mapping table, a reverse lookup is performed to calculate the mapping coordinates and the zoom factor of each pixel position in the processed target image;

[0136] Calculate the scaling weight of the gaze area, the scaling weight of the non-gaze area, and the fusion weight of the transition area in the processed target image according to the scaling ratio;

[0137] The scaling weight of the gaze area, the scaling weight of the non-gaze area, and the fusion weight of the transition area are weighted and synthesized to output the decompressed target image.

[0138] In a possible implementation manner, the center coordinates are a position corresponding to a gaze area of ​​the target image or a position corresponding to a gaze point set on the target image.

[0139] In a possible implementation, the gaze point is dynamic information obtained according to eye tracking or a static gaze point that does not change according to eye movement information.

[0140] In a possible implementation manner, the target image includes at least a Bayer image, an RGB image, a RAW image, and a YUV image.

[0141] In a possible implementation, a calculation method for calculating the compressed target image includes at least NPU calculation and neural network model calculation.

[0142] Fig.13 is a schematic diagram of a terminal provided by an embodiment of the present invention. Fig.13 As shown, the terminal 13 of this embodiment includes: a processor 1301, a memory 1302, and a computer program 1303 stored in the memory 1302 and executable on the processor 1301. When the processor 1301 executes the computer program 1303, the steps in the above-mentioned various image scaling processing method embodiments are implemented, for example Figure 3 Alternatively, when the processor 1301 executes the computer program 1303, the functions of each module / unit in the above-mentioned various image scaling processing device embodiments are implemented, for example Fig.12 Functions of modules / units 1201 to 1205 are shown.

[0143] The present invention further provides a readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the image scaling processing methods provided in the above-mentioned various implementation modes.

[0144] Among them, the readable storage medium can be a computer storage medium or a communication medium. The communication medium includes any medium that facilitates the transmission of a computer program from one place to another. The computer storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer. For example, a readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application-specific integrated circuit (Application Specific Integrated Circuits, abbreviated as: ASIC). In addition, the ASIC can be located in a user device. Of course, the processor and the readable storage medium can also exist in a communication device as discrete components. The readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0145] The present invention also provides a program product, which includes an execution instruction, which is stored in a readable storage medium. At least one processor of a device can read the execution instruction from the readable storage medium, and at least one processor executes the execution instruction so that the device implements the image scaling processing method provided by the various embodiments described above.

[0146] In the embodiments of the above-mentioned devices, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0147] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. An image scaling method, characterized in that: include: Obtaining eye movement information when the user observes the target image; Performing coordinate conversion on the eye movement information to obtain the center coordinates corresponding to the eye movement information; determining a non-gaze area of ​​the target image according to the central coordinates, and compressing the non-gaze area of ​​the target image to obtain a compressed target image; Performing computational processing on the compressed target image to obtain a processed target image; A decompressed target image is obtained according to the center coordinates and the processed target image.

2. The image scaling method according to claim 1, wherein: The step of determining the non-gaze area of ​​the target image according to the central coordinates and compressing the non-gaze area of ​​the target image to obtain a compressed target image includes: Determining coordinates of each area in the non-gaze area of ​​the target image according to the center coordinates; The coordinates of each area in the target image are formed by the center coordinates and the coordinates of each area in the non-gaze area; The target image is compressed based on the coordinates of each area in the target image to obtain the compressed target image.

3. The image scaling method according to claim 2, wherein: The compressing the target image based on the coordinates of each area in the target image to obtain the compressed target image includes: Determining a coordinate mapping table of the non-gaze area based on coordinates of each area in the non-gaze area of ​​the target image; Performing a table lookup according to the gaze area, the center coordinates, the non-gaze area, and the coordinate mapping table to calculate the mapping coordinates and the zoom factor of each pixel position in the target image; The scaling weight of the gaze area, the scaling weight of the non-gaze area and the fusion weight of the transition area in the target image are calculated according to the scaling ratio, wherein the continuous scaling curve corresponding to the scaling weight is a derivative smooth curve, and the points on the derivative smooth curve are used to represent the scaling ratio of the pixel coordinate position, and the scaling ratio is increasing. The transition fusion weight curve corresponding to the fusion weight is a derivative smooth curve, and the points on the derivative smooth curve are used to characterize the transition fusion weight of the distance between the pixel position and the edge of the gaze area, and the transition fusion weight is a fusion of the difference curve weight and the identity mapping weight, and the pixel position close to the edge of the gaze area has a large weight, and the pixel position close to the gaze area has a large identity mapping weight; The scaling weight of the gaze area, the scaling weight of the non-gaze area, and the fusion weight of the transition area are weighted and synthesized to output a compressed target image.

4. The image scaling method according to claim 1, wherein: The image algorithm operation used in the computational processing of the compressed target image at least includes a denoising operation and a specific image algorithm operation.

5. The image scaling method according to claim 4, wherein: The denoising algorithm is a neural network algorithm; The step of performing computational processing on the compressed target image to obtain a processed target image comprises: The compressed target image is subjected to denoising processing using a neural network algorithm to obtain the processed target image.

6. The image scaling method according to claim 1, wherein: The step of obtaining the decompressed target image according to the center coordinates and the processed target image includes: Calculating the coordinates of each region in the processed target image according to the center coordinates; The processed target image is decompressed based on the coordinates of each region in the processed target image to obtain the decompressed target image.

7. The image scaling method according to claim 6, wherein: The center coordinates are positions corresponding to the gaze area of ​​the target image; The step of calculating the coordinates of each region in the processed target image based on the center coordinates includes: Determining coordinates of each area in the non-gaze area of ​​the processed target image according to the center coordinates; The coordinates of each area in the target image are formed by the center coordinates and the coordinates of each area in the non-gaze area.

8. The image scaling method according to claim 7, wherein: The decompressing the processed target image based on the coordinates of each region in the processed target image to obtain the decompressed target image includes: Determining a coordinate mapping table of the non-gaze area based on the coordinates of each area in the non-gaze area of ​​the processed target image; Performing a reverse lookup according to the gaze area, the center coordinates, the non-gaze area, and the coordinate mapping table to calculate the mapping coordinates and the zoom factor of each pixel position in the processed target image; Calculating the scaling weight of the gaze area, the scaling weight of the non-gaze area, and the fusion weight of the transition area in the processed target image according to the scaling ratio; The scaling weight of the gaze area, the scaling weight of the non-gaze area, and the fusion weight of the transition area are weighted and synthesized, and a decompressed target image is output.

9. The image scaling method according to claim 1, wherein: The center coordinates are a position corresponding to a gaze area of ​​the target image or a position corresponding to a gaze point set on the target image.

10. The image scaling method according to claim 9, characterized in that: The gaze point is dynamic information obtained according to eye tracking or a static gaze point that does not change according to eye movement information.

11. The image scaling method according to any one of claims 1 to 10, characterized in that: The target image at least includes a Bayer image, an RGB image, a RAW image, and a YUV image.

12. The image scaling method according to any one of claims 1 to 10, characterized in that: The calculation method for calculating the compressed target image at least includes NPU calculation and neural network model calculation.

13. An image scaling processing device, characterized in that: include: An acquisition module, used to acquire eye movement information when a user observes a target image; A first calculation module, used for performing coordinate conversion on the eye movement information to obtain the center coordinates corresponding to the eye movement information; A compression module, configured to determine a non-gaze area of ​​the target image according to the central coordinates, and compress the non-gaze area of ​​the target image to obtain a compressed target image; A second calculation module is used to calculate the compressed target image to obtain a processed target image; The decompression module is used to obtain a decompressed target image according to the center coordinates and the processed target image.

14. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the image scaling method according to any one of claims 1 to 12 are implemented.

15. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the image scaling method according to any one of claims 1 to 12 are implemented.

Images