Image scaling processing method and apparatus, and terminal and storage medium

Abstract

Disclosed in the present disclosure are an image scaling processing method and apparatus, and a terminal, a storage medium, a computer program product and a computer program. The method comprises: acquiring eye movement information when a user observes a target image; then, performing coordinate conversion on the eye movement information, so as to obtain center coordinates corresponding to the eye movement information; determining a non-gaze area of the target image on the basis of the center coordinates, and compressing the non-gaze area of the target image, so as to obtain a compressed target image; next, performing calculation on the compressed target image, so as to obtain a processed target image; and finally, obtaining a decompressed target image on the basis of the center coordinates and the processed target image. In the present disclosure, compression is mainly performed for saving on storage bandwidths and computing hardware resources, and thus requirements for hardware resources are effectively reduced while ensuring an image effect of a gaze-point area.

Description

Image scaling processing method, device, terminal and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to the Chinese patent applications filed with the China Patent Office on September 25, 2023, with application number 202311249554.0 and application name “Image scaling processing method, device, terminal and storage medium” and the Chinese patent application filed with the China Patent Office on September 25, 2023, with application number 202311246202.X and application name “Image compression processing method, device, equipment and storage medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of computer technology, and in particular to an image scaling processing method, device, terminal, and storage medium. Background Art

[0004] In computer image processing and computer graphics, image scaling refers to the process of adjusting the size of a digital image. Image scaling is a nontrivial process that requires a trade-off between processing efficiency and the smoothness and clarity of the resulting image. When an image is increased in size, the pixels that make up the image become more visible, resulting in a "soft" appearance. Conversely, reducing an image's size enhances its smoothness and clarity.

[0005] Currently, deep learning is generally used to scale the target image.

[0006] However, the above method may cause image quality degradation, resulting in poor scaling effect.

[0007] Summary of the Invention

[0008] The main purpose of the present disclosure is to provide an image scaling processing method, apparatus, terminal, storage medium, computer program product and computer program to solve the problem of image quality degradation in related technologies.

[0009] To achieve the above objectives, in a first aspect, the present disclosure provides an image scaling method, comprising:

[0010] Obtaining eye movement information when the user observes the target image;

[0011] Perform coordinate conversion on the eye movement information to obtain the center coordinates corresponding to the eye movement information;

[0012] Determine a non-gaze area of ​​the target image according to the center coordinates, compress the non-gaze area of ​​the target image, and obtain a compressed target image;

[0013] Calculating the compressed target image to obtain a processed target image;

[0014] The decompressed target image is obtained according to the center coordinates and the processed target image.

[0015] In a possible implementation, determining a non-gaze area of ​​a target image based on a center coordinate, compressing the non-gaze area of ​​the target image to obtain a compressed target image includes:

[0016] Determining the coordinates of each region in the non-gaze area of ​​the target image based on the center coordinates, wherein the coordinates of each region in the target image include the center coordinates and the coordinates of each region in the non-gaze area;

[0017] The target image is compressed based on the coordinates of each region in the target image to obtain a compressed target image.

[0018] In a possible implementation, the target image further includes a gaze area, and the method further includes:

[0019] Determining the coordinates of each region of the gaze area of ​​the target image based on the center coordinates, wherein the coordinates of each region in the target image also include the coordinates of each region in the gaze area, and the gaze area includes a transition area located at the edge; compressing the target image based on the coordinates of each region in the target image to obtain the compressed target image. In one possible implementation, compressing the target image based on the coordinates of each region in the target image to obtain the compressed target image includes:

[0020] Determining a coordinate mapping table of the non-gaze area based on coordinates of each area in the non-gaze area of ​​the target image;

[0021] Based on the gaze area, center coordinates, non-gaze area, and coordinate mapping table, the mapping coordinates and scaling factor of each pixel position in the target image are calculated.

[0022] Calculating the scaling weight of the gaze area, the scaling weight of the non-gaze area, and the fusion weight of the transition area in the target image according to the scaling ratio, wherein the continuous scaling curve corresponding to the scaling weight is a first derivative smooth curve, and the points on the first derivative smooth curve are used to represent the scaling ratio of the pixel coordinate position, and the scaling ratio is increasing;

[0023] The transition fusion weight curve corresponding to the fusion weight is the second derivative smooth curve. The points on the second derivative smooth curve are used to represent the transition fusion weight of the distance between the pixel position and the edge of the gaze area. The transition fusion weight combines the interpolation curve weight and the identity mapping weight. The interpolation curve weight of the pixel position close to the edge of the gaze area is large, and the identity mapping weight of the pixel position close to the center of the gaze area is large.

[0024] The scaling weight of the gaze area, the scaling weight of the non-gaze area, and the fusion weight of the transition area are weighted and synthesized to output a compressed target image.

[0025] In a possible implementation, the image algorithm operation used to perform computational processing on the compressed target image includes at least a denoising algorithm operation and a specific image algorithm operation.

[0026] In one possible implementation, the denoising algorithm is a neural network algorithm;

[0027] Performing computational processing on the compressed target image to obtain a processed target image includes:

[0028] The compressed target image is denoised using a neural network algorithm to obtain a processed target image.

[0029] In a possible implementation, obtaining a decompressed target image according to the center coordinates and the processed target image includes:

[0030] Based on the center coordinates, calculate the coordinates of each area in the processed target image;

[0031] The processed target image is decompressed based on the coordinates of each region in the processed target image to obtain a decompressed target image.

[0032] In one possible implementation, the center coordinates are positions corresponding to the gaze area of ​​the target image;

[0033] Based on the center coordinates, calculate the coordinates of each area in the processed target image, including:

[0034] The coordinates of each region in the non-gaze area of ​​the processed target image are determined according to the central coordinates. The coordinates of each region in the target image include the central coordinates and the coordinates of each region in the non-gaze area.

[0035] In a possible implementation, decompressing the processed target image based on the coordinates of each region in the processed target image to obtain the decompressed target image includes:

[0036] determining a coordinate mapping table of the non-gaze area based on the coordinates of each area in the non-gaze area of ​​the processed target image;

[0037] Perform a reverse lookup based on the gaze area, center coordinates, non-gaze area, and coordinate mapping table to calculate the mapping coordinates and scaling factor of each pixel position in the processed target image;

[0038] Calculate the scaling weight of the gaze area, the scaling weight of the non-gaze area, and the fusion weight of the transition area in the processed target image according to the scaling ratio;

[0039] The scaling weight of the gaze area, the scaling weight of the non-gaze area, and the fusion weight of the transition area are weighted and synthesized to output the decompressed target image.

[0040] In a possible implementation, the center coordinates are a position corresponding to a gaze area of ​​the target image or a position corresponding to a gaze point set on the target image.

[0041] In a possible implementation, the gaze point is dynamic information obtained according to eye tracking or a static gaze point that does not change according to eye movement information.

[0042] In a possible implementation, the target image includes at least a Bayer image, an RGB image, a RAW image, and a YUV image.

[0043] In a possible implementation, a calculation method for calculating the compressed target image includes at least neural network processor (Neural Processing Unit, NPU) calculation and neural network model calculation.

[0044] In a possible implementation, the target image further includes a gaze area, and the method further includes:

[0045] Determine the gaze area and non-gaze area of ​​the target image based on the center coordinates;

[0046] Determining a first sampling rate for the gaze area and a second sampling rate for the non-gaze area, wherein the first sampling rate is greater than the second sampling rate;

[0047] Sampling the gaze area and the non-gaze area based on the first sampling rate and the second sampling rate respectively to obtain a sampled image;

[0048] A first compression rate and a second compression rate of the sampled image are determined, and compression processing is performed on the gaze area and the non-gaze area of ​​the sampled image based on the first compression rate and the second compression rate, respectively, to obtain a target compressed image.

[0049] In one possible implementation, determining the gaze area and the non-gaze area of ​​the target image based on the center coordinates includes:

[0050] Divide the target image in the horizontal and vertical directions based on the center coordinates to obtain N*M image units;

[0051] Determine a gaze area based on the center coordinates, where the gaze area includes X*Y image units, 1≤X≤N, 1≤Y≤M;

[0052] The other regions of the target image are determined as non-gaze regions.

[0053] In a possible implementation, sampling the gaze area and the non-gaze area based on the first sampling rate and the second sampling rate to obtain a sampled image includes:

[0054] Obtaining the abscissa and ordinate of the target image, and determining the sampling initial position and step distance based on the abscissa, ordinate, and center coordinates of the target image;

[0055] Set the data window and sampling points, and determine the movement parameters of the data window based on the sampling points;

[0056] The gaze area and the non-gaze area are sampled based on the movement parameters, the sampling initial position and the step distance to obtain a sampled image.

[0057] In a possible implementation, setting a data window and a sampling point, and determining a movement parameter of the data window based on the sampling point, includes:

[0058] Determining relative positions of the data window and the first sampling point and / or the second sampling point;

[0059] Movement parameters are determined based on the relative positions.

[0060] In a possible implementation, within the non-gaze area, the sampling rates of image units located in the same horizontal direction are the same, and the sampling rates of image units located in the same vertical direction are the same.

[0061] In one possible implementation, determining a first sampling rate for a gaze area and a second sampling rate for a non-gaze area includes:

[0062] Setting a first sampling rate for the gaze area, wherein the first sampling rate is within a gaze compression range;

[0063] Determine the distance between the image unit in the non-gaze area and the image in the gaze area;

[0064] determining a plurality of second sub-sampling rates based on the second sampling rate, wherein different second sub-sampling rates have different sampling parameters;

[0065] The second subsampling rate corresponding to the image units in the non-gaze area is configured based on the image distance, wherein a larger image distance corresponds to a smaller second subsampling rate.

[0066] In one possible implementation, the method further includes:

[0067] The target compressed image is stored and / or decompressed based on the first compression ratio and the second compression ratio to obtain an output image.

[0068] In a second aspect, an embodiment of the present disclosure provides an image scaling processing device, comprising:

[0069] An acquisition module, used to acquire eye movement information when the user observes the target image;

[0070] The first calculation module is used to convert the eye movement information into coordinates to obtain the center coordinates corresponding to the eye movement information;

[0071] a compression module, configured to determine a non-gaze area of ​​the target image based on the center coordinates, and compress the non-gaze area of ​​the target image to obtain a compressed target image;

[0072] A second calculation module is used to calculate the compressed target image to obtain a processed target image;

[0073] The decompression module is used to obtain a decompressed target image according to the center coordinates and the processed target image.

[0074] In a third aspect, an embodiment of the present disclosure provides a terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above image scaling methods when executing the computer program.

[0075] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of any of the above image scaling processing methods are implemented.

[0076] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program, so that the electronic device implements the steps of any of the above image scaling processing methods when executing the computer program.

[0077] In a sixth aspect, an embodiment of the present disclosure further provides a computer program, which is stored in a readable storage medium. At least one processor of an electronic device can read the above-mentioned computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device implements the steps of any of the above image scaling processing methods when executing the computer program.

[0078] The present disclosure provides an image scaling method, apparatus, terminal, and storage medium, including: obtaining eye movement information when a user observes a target image; performing coordinate conversion on the eye movement information to obtain center coordinates corresponding to the eye movement information; determining a non-gaze area of ​​the target image based on the center coordinates; compressing the non-gaze area of ​​the target image to obtain a compressed target image; performing calculations on the compressed target image to obtain a processed target image; and finally, obtaining a decompressed target image based on the center coordinates and the processed target image. The present disclosure scales the non-gaze area based on the center coordinates, ensuring that the scaled image is quickly restored to its original size after NPU processing is completed. This ensures that other spatial domain-related modules of the ISP are not affected, avoids degradation of the target image quality, and improves the scaling effect of the target image. Furthermore, according to an exemplary embodiment of the invention, the gaze area of ​​the target image can also be determined based on the center coordinates. By setting the sampling rate of the gaze area to be higher than the sampling rate of the non-gaze area and the compression rate of the gaze area to be lower than the compression rate of the non-gaze area, the size of the compressed image can be reduced while ensuring image quality, effectively saving storage resources. The present disclosure mainly saves storage bandwidth and computing hardware resources through compression, and effectively reduces hardware resource requirements while ensuring the image effect of the gaze point area. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] The accompanying drawings, which constitute part of this disclosure, are intended to provide a further understanding of the disclosure and to make other features, objectives, and advantages of the disclosure more apparent. The accompanying drawings of the exemplary embodiments of the disclosure and their descriptions are intended to explain the disclosure and do not constitute an improper limitation of the disclosure. In the accompanying drawings:

[0080] FIG1 is a schematic diagram of coordinate mapping of an image provided by an embodiment of the present disclosure;

[0081] FIG2 is a schematic diagram of setting a magnification of an image partition provided by an embodiment of the present disclosure;

[0082] FIG3 is a flowchart of an implementation method of an image scaling process provided by an embodiment of the present disclosure;

[0083] FIG4 is a flowchart of an implementation method of an image scaling process provided by another embodiment of the present disclosure;

[0084] FIG5 is a flowchart of an implementation of an image compression method provided by an embodiment of the present disclosure;

[0085] FIG6 is a schematic diagram of a magnification setting for each area of ​​an image provided by an embodiment of the present disclosure;

[0086] FIG7 is a schematic diagram of an RGB image format and a Bayer image format provided by an embodiment of the present disclosure;

[0087] FIG8 is a schematic diagram of another image magnification setting for each region provided by an embodiment of the present disclosure;

[0088] FIG9 is a continuous magnification scaling curve diagram provided by an embodiment of the present disclosure;

[0089] FIG10 is a transition zone curve diagram provided by an embodiment of the present disclosure;

[0090] FIG11 is a flowchart of an implementation method of an image decompression method provided by an embodiment of the present disclosure;

[0091] FIG12 is a flowchart of an implementation method of an image scaling process provided by another embodiment of the present disclosure;

[0092] FIG13 is a flowchart of an implementation of determining a gaze area and a non-gaze area of ​​a target image provided by an embodiment of the present disclosure;

[0093] FIG14 is a schematic diagram of a gaze area and a non-gaze area provided by an embodiment of the present disclosure;

[0094] FIG15 is a flowchart of an implementation of determining a first sampling rate and a second sampling rate provided by an embodiment of the present disclosure;

[0095] FIG16 is a schematic diagram of configuring a second subsampling rate for a non-gaze area according to an embodiment of the present disclosure;

[0096] FIG17 is a flowchart of an implementation of sampling processing of gaze areas and non-gaze areas provided by an embodiment of the present disclosure;

[0097] FIG18 is a schematic structural diagram of an image scaling processing device provided by an embodiment of the present disclosure;

[0098] FIG19 is a block diagram of an image scaling processing device provided by another embodiment of the present disclosure;

[0099] FIG20 is a block diagram of an image scaling processing device provided by another embodiment of the present disclosure;

[0100] FIG21 is a schematic diagram of a terminal provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0101] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0102] The terms "first," "second," "third," "fourth," and so forth (if any) in the description and claims of the present disclosure and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, such that the embodiments of the present disclosure described herein can be practiced in sequences other than those illustrated or described herein.

[0103] It should be understood that in the various embodiments of the present disclosure, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.

[0104] It should be understood that in the present disclosure, "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product or apparatus.

[0105] It should be understood that in the present disclosure, "plurality" refers to two or more. "And / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "Contains A, B and C", "Contains A, B, C" means that A, B, and C are all included, "Contains A, B or C" means that one of A, B, and C is included, and "Contains A, B and / or C" means that any one, any two, or any three of A, B, and C are included.

[0106] It should be understood that, in this disclosure, "B corresponding to A," "B corresponding to A," "A corresponds to B," or "B corresponds to A" means that B is associated with A and B can be determined based on A. Determining B based on A does not mean determining B based solely on A; B can also be determined based on A and / or other information. A and B match when the similarity between A and B is greater than or equal to a preset threshold.

[0107] Depending on the context, "if" as used herein may be interpreted as "when" or "when" or "in response to determining" or "in response to detecting."

[0108] The following specific embodiments are used to describe the technical solution of the present disclosure in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0109] Furthermore, in order to make the objectives, technical solutions and advantages of the present disclosure more clear, specific embodiments will be described below with reference to the accompanying drawings.

[0110] In computer image processing and computer graphics, image scaling refers to the process of adjusting the size of a digital image.

[0111] In the prior art, images are mainly scaled using a scaler, as shown in FIG1 . Its essence is coordinate mapping, where a point D of a target image finds a corresponding point Si in a source image, and then interpolation is performed using an interpolation formula to obtain the target point D.

[0112] Scaling methods include downscaling and upscaling. For RGB images, downscaling and upscaling are performed at different magnifications by partitioning the image. As shown in Figure 2, the image is divided into multiple blocks, and the scaling factor used is marked on each block. For example, the first block in the upper left corner of the image is scaled at a factor of 4.

[0113] However, the existing technology may suffer from the problem of image quality degradation, resulting in poor scaling effect.

[0114] Therefore, the present disclosure provides an image scaling method to solve the above problems and improve the image scaling effect.

[0115] In one embodiment, as shown in FIG3 , a method for image scaling is provided, comprising the following steps:

[0116] Step S301: Acquire eye movement information of a user when observing a target image.

[0117] Among them, eye movement information can be obtained through eye movement equipment or other devices. The eye movement information mainly includes information such as the user's gaze position on the screen or image and the corresponding timestamp.

[0118] Step S302: performing coordinate conversion on the eye movement information to obtain the center coordinates corresponding to the eye movement information.

[0119] The present disclosure mainly uses MCU to convert the eye movement information into coordinates, and through the conversion, the gaze position on the user's image, that is, the center coordinates, can be obtained.

[0120] Step S303: determining a non-gaze area of ​​the target image according to the central coordinates, and compressing the non-gaze area of ​​the target image to obtain a compressed target image.

[0121] The target image includes images in various formats, including but not limited to Bayer, RAW, RGB, and YUV formats.

[0122] Step S304: Calculate the compressed target image to obtain a processed target image.

[0123] Step S305: Obtain a decompressed target image based on the center coordinates and the processed target image.

[0124] 4 , taking the target image as a Bayer image as an example, the compressed Bayer image obtained based on the center coordinates and the Bayer image is implemented by the Bayer Group module, while the decompressed Bayer image obtained based on the center coordinates and the processed Bayer image is implemented by the Bayer Ungroup module.

[0125] The following details how the Bayer Group module generates a compressed Bayer image based on the center coordinates and the Bayer image. Referring to Figure 5, the main steps are: first, the coordinates of each region in the Bayer image are calculated based on the center coordinates; then, based on the coordinates of each region in the Bayer image, the Bayer image is compressed to obtain the compressed Bayer image. The center coordinates are the location corresponding to the gaze region in the Bayer image.

[0126] Among them, according to the center coordinates, the coordinates of each area in the Bayer image are calculated. It is necessary to first determine the coordinates of each area in the non-gaze area of ​​the Bayer image based on the center coordinates, and then the coordinates of each area in the Bayer image are constructed by the center coordinates and the coordinates of each area in the non-gaze area.

[0127] Among them, the Bayer image is compressed based on the coordinates of each area in the Bayer image to obtain a compressed Bayer image. It is necessary to first determine the coordinate mapping table of the non-gaze area based on the coordinates of each area in the non-gaze area of ​​the Bayer image, and then look up the table according to the gaze area, center coordinates, non-gaze area and coordinate mapping table to calculate the mapping coordinates and scaling factor of each pixel position in the Bayer image. Then, the scaling weight of the gaze area, the scaling weight of the non-gaze area and the fusion weight of the transition area in the Bayer image are calculated according to the scaling factor. Finally, the scaling weight of the gaze area, the scaling weight of the non-gaze area and the fusion weight of the transition area are weighted and synthesized to output the compressed Bayer image.

[0128] In Figure 6, the center coordinates correspond to the gaze area of ​​the target image, and the gaze area is scaled at a fixed factor. Furthermore, a transition zone is provided at the edge of the gaze area to ensure a natural transition between images. The size of the gaze area can be adjusted as needed.

[0129] The non-gaze area is located outside the transition zone (transition region). This non-gaze area is divided into multiple blocks by lines, and each block is scaled at a variable rate. The rate of change is calculated by functions f(i) and f(j), which are related to the distance from the gaze area. Functions f(i) and f(j) are existing calculation functions and are not specifically limited here. They can be set as needed. For example, the first block in the upper left corner is scaled using the f(i) and f(j) magnifications, respectively.

[0130] In addition, the center coordinates may also be the position corresponding to a static gaze point set on the target image. This static gaze point is a point set by the user on the target image, which can be set according to user needs and is not specifically limited here.

[0131] Assuming the target image is a Bayer image, the Bayer image format inherently loses resolution compared to the RGB image format. When resampling, assume that the target image is mapped to the source image near the center box of the Bayer image in Figure 7. In this case, 9 values ​​are available for the R channel, 12 values ​​for the G channel, and 4 values ​​for the B channel in the 5x5 Bayer image neighborhood. Assuming the target image is mapped to the source image near the center box of the RGB image in Figure 7, 25 values ​​are available for each of the three RGB channels in the RGB domain.

[0132] In conjunction with Figures 8 and 9, when the gaze area moves, since the zoom ratio of other areas is related to the distance from the gaze area, in order to ensure that the overall zoom ratio remains unchanged, the present disclosure designs a zoom ratio lookup table based on the image size and the overall zoom ratio. The zoom ratio lookup table (i.e., the coordinate mapping table) records the pixel coordinates before and after Grouping, as well as the pixel coordinates before and after Ungrouping. Among them, the pixel coordinates before and after Grouping, as well as the pixel coordinates before and after Ungrouping, form a continuous zoom ratio curve as shown in Figure 9. This allows the zoom ratio of each area to be dynamically calculated when the gaze area moves.

[0133] The continuous magnification scaling curve uses a derivative smoothing curve. Taking the cubic polynomial curve as an example, its derivative quadratic polynomial curve is the corresponding coordinate scaling factor. The increasing scaling factor is relatively stable. The interpolation weight selects points on the interpolation curve (nearest, bilinear, lanczos, etc.) according to the scaling factor and fuzzy requirements, and the maximum scaling factor, minimum scaling factor, and overall scaling factor are set according to the requirements to solve the coefficients of the polynomial.

[0134] The fusion weight of the transition area and the distance between the pixel position and the edge of the gaze area form a transition band curve as shown in Figure 10, that is, the transition fusion weight curve. When the gaze area is an identity transformation, the transition band curve makes the image change smoother. The transition band curve selects points on the sine curve according to the distance from the center area. The closer to the edge of the focus area (gaze area), the larger the Lanczos weight, and the closer to the center of the focus area, the larger the identity weight.

[0135] The vertical axis of the transition fusion weight curve, "transition fusion weight," combines the interpolation curve weight and the identity mapping weight to generate a transition band weighted composite weight c. The calculation formula is as follows: c = b*w1 + a*(1-w1)

[0136] Among them, b is the constant weight, a is the interpolation curve weight, and w1 is the transition fusion weight coefficient.

[0137] After the compressed target image is obtained based on the above embodiment, it is necessary to perform calculations on the compressed target image to obtain a processed target image.

[0138] Among them, calculations are performed on the compressed target image, including but not limited to NPU calculations on the target image or processing of the target image through units such as DSP, CPU, GPU, ASIC and storage (SRAM / DDR / NAND), etc. The processing method here is not specifically limited.

[0139] For example, performing calculation on the compressed target image is to denoise the target image, which mainly utilizes a neural network algorithm to perform denoising processing on the compressed target image to obtain a processed target image.

[0140] Afterwards, it is necessary to obtain the decompressed target image based on the center coordinates and the processed target image.

[0141] In conjunction with Figure 11, the main steps for obtaining a decompressed target image based on the center coordinates and the processed target image are: first, based on the center coordinates, calculate the coordinates of each region in the processed target image; then, based on the coordinates of each region in the processed target image, decompress the processed target image to obtain the decompressed target image. The center coordinates are the position corresponding to the gaze area in the target image.

[0142] Among them, based on the center coordinates, the coordinates of each area in the processed target image are calculated. It is necessary to first determine the coordinates of each area in the non-gaze area of ​​the processed target image based on the center coordinates, wherein the coordinates of each area in the target image include the center coordinates and the coordinates of each area in the non-gaze area.

[0143] Among them, the processed target image is decompressed based on the coordinates of each area in the processed target image to obtain the decompressed target image. It is necessary to first determine the coordinate mapping table of the non-gaze area based on the coordinates of each area in the non-gaze area of ​​the processed target image, and then perform a reverse lookup table based on the gaze area, center coordinates, non-gaze area and coordinate mapping table to calculate the mapping coordinates and scaling factor of each pixel position in the processed target image, and then calculate the scaling weight of the gaze area, the scaling weight of the non-gaze area and the fusion weight of the transition area in the processed target image according to the scaling factor, and finally perform weighted synthesis on the scaling weight of the gaze area, the scaling weight of the non-gaze area and the fusion weight of the transition area to output the decompressed target image.

[0144] Furthermore, the compressed target image can be fed into algorithmic models such as the ISP and NPU for further processing. The ISP is an image signal processing architecture capable of running various algorithms and processing image signals in real time. The ISP primarily comprises a CPU, SUP IP, and IF. The CPU, or central processing unit, runs image processing algorithms such as AF and LSC and controls peripherals. Modern ISPs typically utilize CPUs from the ARM Cortex-A series, such as the Cortex-A5 and Cortex-A7. SUB IP is a general term for various functional modules that perform specialized image processing. Common SUB IPs include DIS, CSC, and VRA. The NPU, or neural network processing unit, is a specialized processor designed for neural network computing. Its primary function is to accelerate neural network computations, enabling rapid implementation of computational operations such as data processing, transfer, and backpropagation. The NPU operates based on a neural network computing model, with its core being the implementation of deep learning algorithms. Generally speaking, the NPU achieves efficient computation by parallelizing various computational operations within a neural network. In a neural network, neurons are connected through weights. During the training process, the NPU will continuously adjust these weights based on sample data so that the neural network can ultimately achieve the optimal recognition effect.

[0145] The present disclosure provides an image scaling method, apparatus, terminal, and storage medium, including: obtaining eye movement information when a user observes a target image; performing coordinate conversion on the eye movement information to obtain the center coordinates corresponding to the eye movement information; determining a non-gaze area of ​​the target image based on the center coordinates; compressing the non-gaze area of ​​the target image to obtain a compressed target image; performing calculations on the compressed target image to obtain a processed target image; and finally, obtaining a decompressed target image based on the center coordinates and the processed target image. The present disclosure scales the non-gaze area based on the center coordinates, ensuring that the scaled image is quickly restored to its original size after NPU processing is completed. This ensures that other spatial domain-related modules of the ISP are not affected, avoids degradation of the target image quality, and improves the scaling effect of the target image. Furthermore, the present disclosure primarily saves storage bandwidth and computing hardware resources through compression, effectively reducing hardware resource requirements while ensuring image quality in the gaze point area. It should be understood that the sequence numbers of the steps in the above embodiments do not necessarily indicate the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the present disclosure.

[0146] FIG12 is a flowchart of an implementation of an image scaling method 1000 provided in another embodiment of the present disclosure. As shown in FIG12 , the image scaling method 1000 may include:

[0147] Step S100: acquiring a target image and center coordinates, and determining a gaze area and a non-gaze area of ​​the target image based on the center coordinates;

[0148] Step S200: determining a first sampling rate of a gaze area and a second sampling rate of a non-gaze area, wherein the first sampling rate is greater than the second sampling rate;

[0149] Step S300: Sampling the gaze area and the non-gaze area based on the first sampling rate and the second sampling rate respectively to obtain a sampled image;

[0150] Step S400: determining a first compression ratio and a second compression ratio of a sampled image, and performing compression processing on the sampled image based on the first compression ratio and the second compression ratio to obtain a target compressed image.

[0151] The specific contents of each step of the above-mentioned image scaling method 1000 will be described in detail below.

[0152] Step S100

[0153] In an exemplary embodiment of the present disclosure, the target image and the center coordinates are first acquired, and the gaze area and the non-gaze area of ​​the target image are determined based on the center coordinates. For example, an eye-movement device can be used to acquire the target image and the center coordinates, wherein the center coordinates can be the position where the eye is looking in the target image. For example, the center coordinates can include the gaze point coordinates, and the information of the gaze point can change accordingly as the direction of the user's eye gaze changes. The target image is then divided into the gaze area and the non-gaze area based on the gaze point information. Figure 13 is a flowchart of an implementation of determining the gaze area and the non-gaze area of ​​the target image provided by an embodiment of the present disclosure. As shown in Figure 13, determining the gaze area and the non-gaze area of ​​the target image may include the following steps:

[0154] Step S110: Divide the target image in the horizontal direction and the vertical direction based on the gaze point coordinates to obtain N*M image units;

[0155] Step S120: Acquire eye movement information, and determine a gaze area based on the eye movement information, wherein the gaze area includes X*Y image units, 1≤X≤N, 1≤Y≤M;

[0156] Step S130: determining other areas of the target image as non-gaze areas.

[0157] Figure 14 is a schematic diagram of the gaze area and non-gaze area provided by an embodiment of the present disclosure. As shown in Figure 14, the target image may include a gaze point 10, a gaze area 11, and a non-gaze area 12. First, based on the coordinates of the gaze point, the image is divided in the horizontal and vertical directions to obtain N*M image units (not shown in the figure), wherein the size of each image unit can be the same, and the image units of the present disclosure can also be pixels. Then, eye movement information is obtained, and the gaze area is determined based on the eye movement information. The gaze area may include X*Y image units, wherein M, N, X, and Y are all positive integers, and 1≤X≤N, 1≤Y≤M. The other areas of the target image are determined as non-gaze areas 12.

[0158] Step S200

[0159] In an exemplary embodiment of the present disclosure, after determining the gaze area and the non-gaze area, a first sampling rate for the gaze area and a second sampling rate for the non-gaze area may also be determined, wherein the first sampling rate is greater than the second sampling rate. FIG15 is a flowchart of an implementation of determining the first sampling rate and the second sampling rate provided by an embodiment of the present disclosure. As shown in FIG15 , determining the first sampling rate and the second sampling rate may include the following steps:

[0160] Step S210: setting a first sampling rate of the gaze area, wherein the first sampling rate is within the gaze compression range;

[0161] Step S220: determining the image distance between the image unit in the non-gaze area and the image in the gaze area;

[0162] Step S230: determining a plurality of second sub-sampling rates based on the second sampling rate, wherein different second sub-sampling rates have different sampling parameters; and

[0163] Step S240: configuring a second subsampling rate corresponding to the image units in the non-gaze area based on the image distance, wherein a larger image distance corresponds to a smaller second subsampling rate.

[0164] Exemplarily, the first sampling rate of the gaze area is first set, wherein the first sampling rate is within the gaze compression range. Since the eye is more sensitive to the image quality of the gaze area, the size of the first sampling rate can be controlled by setting the gaze compression range to further ensure the imaging quality of the gaze area. Then, multiple second sub-sampling rates can be set inside the non-gaze area, wherein the sampling parameters of the multiple second sub-sampling rates are different. In the non-gaze area, the sampling rates of the image units located in the same horizontal direction are the same, and the sampling rates of the image units located in the same vertical direction are the same. Figure 16 is a schematic diagram of configuring the second sub-sampling rate for the non-gaze area provided by an embodiment of the present disclosure. As shown in Figure 16, taking the gaze area being located in the center of the target image area, configuring the second sub-sampling rate based on the horizontal direction in the non-gaze area as an example. The non-gaze area is divided according to the image distance between the image unit in the non-gaze area and the image of the gaze area, for example, the second sub-image area 121 and the second sub-image area 122 are divided, wherein the second sub-image area 121 and the second sub-image area 122 can each contain multiple image units, and then a second sub-sampling rate is configured for the second sub-image area 121 and the second sub-image area 122, wherein the second sub-sampling rate corresponding to the second sub-image area 121 is greater than the second sub-sampling rate corresponding to the second sub-image area 122.

[0165] Step S300

[0166] In an exemplary embodiment of the present disclosure, after obtaining the first sampling rate and the second sampling rate, the gaze area and the non-gaze area can be sampled based on the first sampling rate and the second sampling rate to obtain a sampled image. Figure 17 is a flowchart of an implementation of sampling the gaze area and the non-gaze area provided by an embodiment of the present disclosure. As shown in Figure 17, the sampling process of the gaze area and the non-gaze area may include the following steps:

[0167] Step S310: obtaining the abscissa and ordinate of the target image, and determining the sampling initial position and step distance based on the abscissa, ordinate and center coordinates of the target image;

[0168] Step S320: setting a data window and sampling points, and determining a movement parameter of the data window based on the sampling points;

[0169] Step S330: sampling the gaze area and the non-gaze area based on the movement parameters, the sampling initial position, and the stepping distance to obtain a sampled image.

[0170] Exemplarily, the initial sampling position and step distance are determined based on the horizontal and vertical coordinates of the target image and the location of the gaze point. Different sampling rates correspond to different step distances, where a higher sampling rate corresponds to a smaller step distance. For example, the initial sampling position and step distance can be set based on the first and second sampling rates set in the gaze and non-gaze areas. A data window and sampling points are set in the gaze and non-gaze areas, and movement parameters of the data window are determined based on the set sampling points. The movement parameters of the data window include the movement speed and movement distance of the data window. By setting the sampling points and then setting the movement parameters of the data window based on the sampling points, different sampling frequencies can be accommodated within the same data window, improving sampling flexibility and efficiency. The gaze and non-gaze areas are then sampled based on the movement parameters, the initial sampling position, and the step distance to obtain a sampled image. During the sampling process of the gaze and non-gaze areas, the relative positions of the data window and the first and / or second sampling points can also be determined, and the movement parameters determined based on the relative positions. The first sampling point can be the current sampling point, and the second sampling point can be the next sampling point. For example, the relative positions of the current sampling point, the next sampling point, and the data window may include: (1) the current sampling point is not within the data window; (2) the current sampling point is within the data window, and the next sampling point is not within the data window; (3) the current sampling point and the next sampling point are both within the data window, but the distance between the current sampling point and the data window is greater than the distance threshold; (4) the next sampling point is located at the starting sampling position of the next row. If the current sampling point, the next sampling point, and the data window meet any of the above relative position conditions, the data window needs to be moved in units of step distance, and then the data window at the next moment is obtained, and the relative positions of the current sampling point, the next sampling point, and the data window at the next moment are determined again. If the current sampling point, the next sampling point, and the data window do not meet the above relative position conditions, the data window does not move. The present disclosure optimizes the sampling process by adjusting the relative positions of the sampling points and the data window, which can avoid back pressure on the front-stage data flow to a certain extent and improve the performance of the sampling operation.

[0171] Step S400

[0172] In an exemplary embodiment of the present disclosure, after obtaining a sampled image, a first compression rate and a second compression rate of the sampled image may be determined, and the sampled image may be compressed based on the first and second compression rates to obtain a target compressed image. For example, after sampling the target image to obtain the sampled image, different compression rates or compression methods may be set based on the gaze area and non-gaze area of ​​the sampled image. For example, the gaze area may be compressed using a lossless compression method, while the non-gaze area may be compressed using a lossy compression method. For example, a first compression rate may be set for the gaze area, and a second compression rate may be set for the non-gaze area, where the first compression rate is less than the second compression rate. The present disclosure may also set multiple second sub-compression rates within the non-gaze area. The second compression rate may include multiple second sub-compression rates, and the second sub-compression rates may be set based on the second sub-sampling rate. For example, the smaller the second sub-sampling rate, the larger the second sub-compression rate. After setting the first and second compression rates, the sampled image may be compressed based on the first and second compression rates to obtain a target compressed image.

[0173] According to an exemplary embodiment of the present disclosure, by dividing the target image into a gaze area and a non-gaze area, and setting the sampling rate of the gaze area to be greater than the sampling rate of the non-gaze area, and the compression rate of the gaze area to be less than the compression rate of the non-gaze area, the size of the compressed image can be reduced while ensuring the image quality, thereby effectively saving storage resources.

[0174] In an exemplary embodiment of the present disclosure, after obtaining a target compressed image, the target compressed image can be transmitted and stored, and the target compressed image can be decompressed. For example, the target compressed image can be stored and / or decompressed based on a first compression ratio and a second compression ratio to obtain an output image, where the output image can be a sampled image. The present disclosure can also further restore the sampled image based on the first sampling ratio and the second sampling ratio to obtain the target image. The following is an apparatus embodiment of the present disclosure. For details not fully described therein, reference can be made to the corresponding method embodiment described above.

[0175] FIG18 shows a schematic diagram of the structure of an image scaling processing device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. The image scaling processing device includes an acquisition module 1201, a first calculation module 1202, a compression module 1203, a second calculation module 1204, and a decompression module 1205. The details are as follows:

[0176] An acquisition module 1201 is used to acquire eye movement information of a user when observing a target image;

[0177] A first calculation module 1202 is used to perform coordinate conversion on the eye movement information to obtain the center coordinates corresponding to the eye movement information;

[0178] The compression module 1203 is configured to determine a non-gaze area of ​​the target image according to the center coordinates, and compress the non-gaze area of ​​the target image to obtain a compressed target image.

[0179] A second calculation module 1204 is used to calculate the compressed target image to obtain a processed target image;

[0180] The decompression module 1205 is configured to obtain a decompressed target image based on the center coordinates and the processed target image.

[0181] In one possible implementation, the compression module 1203 is further configured to determine, based on the center coordinates, the coordinates of each region in the non-gaze area of ​​the target image, wherein the coordinates of each region in the target image include the center coordinates and the coordinates of each region in the non-gaze area;

[0182] The target image is compressed based on the coordinates of each region in the target image to obtain a compressed target image.

[0183] In a possible implementation, the compression module 1203 is further configured to determine a coordinate mapping table of the non-gaze area based on the coordinates of each area in the non-gaze area of ​​the target image;

[0184] Based on the gaze area, center coordinates, non-gaze area, and coordinate mapping table, the mapping coordinates and scaling factor of each pixel position in the target image are calculated.

[0185] Calculating the scaling weight of the gaze area, the scaling weight of the non-gaze area, and the fusion weight of the transition area in the target image according to the scaling ratio, wherein the continuous scaling curve corresponding to the scaling weight is a first derivative smooth curve, and the points on the first derivative smooth curve are used to represent the scaling ratio of the pixel coordinate position, and the scaling ratio is increasing;

[0186] The transition fusion weight curve corresponding to the fusion weight is the second derivative smooth curve. The points on the second derivative smooth curve are used to represent the transition fusion weight of the distance between the pixel position and the edge of the gaze area. The transition fusion weight combines the interpolation curve weight and the identity mapping weight. The interpolation curve weight of the pixel position close to the edge of the gaze area is large, and the identity mapping weight of the pixel position close to the center of the gaze area is large.

[0187] The scaling weight of the gaze area, the scaling weight of the non-gaze area, and the fusion weight of the transition area are weighted and synthesized to output a compressed target image.

[0188] In a possible implementation, the image algorithm operation used to perform computational processing on the compressed target image includes at least a denoising algorithm operation and a specific image algorithm operation.

[0189] In one possible implementation, the denoising algorithm is a neural network algorithm;

[0190] The decompression module is also used to perform denoising on the compressed target image using a neural network algorithm to obtain a processed target image.

[0191] In a possible implementation, the decompression module 1205 is further configured to calculate the coordinates of each region in the processed target image based on the center coordinates;

[0192] The processed target image is decompressed based on the coordinates of each region in the processed target image to obtain a decompressed target image.

[0193] In one possible implementation, the center coordinates are positions corresponding to the gaze area of ​​the target image;

[0194] The decompression module 1205 is further configured to determine the coordinates of each region in the non-gaze area of ​​the processed target image based on the center coordinates, wherein the coordinates of each region in the target image include the center coordinates and the coordinates of each region in the non-gaze area.

[0195] In a possible implementation, the decompression module 1205 is further configured to determine a coordinate mapping table of the non-gaze area based on the coordinates of each area in the non-gaze area of ​​the processed target image;

[0196] Perform a reverse lookup based on the gaze area, center coordinates, non-gaze area, and coordinate mapping table to calculate the mapping coordinates and scaling factor of each pixel position in the processed target image;

[0197] Calculate the scaling weight of the gaze area, the scaling weight of the non-gaze area, and the fusion weight of the transition area in the processed target image according to the scaling ratio;

[0198] The scaling weight of the gaze area, the scaling weight of the non-gaze area, and the fusion weight of the transition area are weighted and synthesized to output the decompressed target image.

[0199] In a possible implementation, the center coordinates are a position corresponding to a gaze area of ​​the target image or a position corresponding to a gaze point set on the target image.

[0200] In a possible implementation, the gaze point is dynamic information obtained according to eye tracking or a static gaze point that does not change according to eye movement information.

[0201] In a possible implementation, the target image includes at least a Bayer image, an RGB image, a RAW image, and a YUV image.

[0202] In a possible implementation, a calculation method for calculating the compressed target image includes at least NPU calculation and neural network model calculation.

[0203] The present disclosure also provides an image scaling device. FIG19 is a block diagram of an image scaling device 2000 according to an exemplary embodiment of the present disclosure. As shown in FIG19 , the image scaling device 2000 may include an image preprocessing module 2100, a sampling rate acquisition module 2200, an image sampling module 2300, and an image compression module 2400.

[0204] In an exemplary embodiment of the present disclosure, the image pre-processing module 2100 may be configured to acquire a target image and center coordinates, and determine a gaze area and a non-gaze area of ​​the target image based on the center coordinates.

[0205] Exemplarily, the image preprocessing module 2100 can be used to divide the target image in the horizontal and vertical directions based on the gaze point coordinates to obtain N*M image units; obtain eye movement information, and determine the gaze area based on the eye movement information, wherein the gaze area includes X*Y image units, 1≤X≤N, 1≤Y≤M; and determine other areas of the target image as non-gaze areas.

[0206] In an exemplary embodiment of the present disclosure, the sampling rate acquisition module 2200 may be used to determine a first sampling rate for the gaze area and a second sampling rate for the non-gaze area, wherein the first sampling rate is greater than the second sampling rate.

[0207] For example, the sampling rate acquisition module 2200 can be used to set a first sampling rate for the gaze region, where the first sampling rate is within the gaze compression range; determine the image distance between image units in the non-gaze region and the gaze region; determine multiple second sub-sampling rates based on the second sampling rate, where different second sub-sampling rates have different sampling parameters; and configure the second sub-sampling rates corresponding to image units in the non-gaze region based on the image distance, where a larger image distance corresponds to a smaller second sub-sampling rate. Within the non-gaze region, the sampling rates of image units in the same horizontal direction are the same, and the sampling rates of image units in the same vertical direction are the same.

[0208] In an exemplary embodiment of the present disclosure, the image sampling module 2300 may be configured to perform sampling processing on the gaze area and the non-gaze area based on a first sampling rate and a second sampling rate, respectively, to obtain a sampled image.

[0209] Exemplarily, the image sampling module 2300 can be used to obtain the horizontal and vertical coordinates of the target image, and determine the sampling initial position and step distance based on the horizontal and vertical coordinates and center coordinates of the target image; set the data window and sampling points, and determine the movement parameters of the data window based on the sampling points; sample the gaze area and non-gaze area based on the movement parameters, sampling initial position and step distance to obtain a sampled image.

[0210] In an exemplary embodiment of the present disclosure, the image compression module 2400 may be configured to determine a first compression ratio and a second compression ratio of a sampled image, and compress the sampled image based on the first compression ratio and the second compression ratio to obtain a target compressed image.

[0211] In an exemplary embodiment of the present disclosure, the image scaling processing device 2000 also includes: a storage unit that can be used to store the target compressed image; a decompression unit that can be used to obtain the target compressed image from the storage unit, and store and / or decompress the target compressed image to obtain an output image.

[0212] The functions and effects of the various modules of the image scaling processing apparatus 2000 of the present disclosure have been specifically described in the various steps of the image scaling processing method 1000 and will not be elaborated on here.

[0213] According to an exemplary embodiment of the present disclosure, by dividing the target image into a gaze area and a non-gaze area, and setting the sampling rate of the gaze area to be greater than the sampling rate of the non-gaze area, and the compression rate of the gaze area to be less than the compression rate of the non-gaze area, the size of the compressed image can be reduced while ensuring the image quality, thereby effectively saving storage resources.

[0214] The present disclosure also provides an image scaling processing device. Figure 20 is a block diagram of an image scaling processing device according to an exemplary embodiment of the present disclosure. As shown in Figure 20, the image scaling processing device may include an eye movement acquisition unit 20, a micro control unit 30, a storage unit 40, and a decompression unit 50. Among them, the eye movement acquisition unit 20 can be used to obtain a target image and center coordinates based on an eye movement device, and determine the gaze area and non-gaze area of ​​the target image based on the center coordinates; the micro control unit 30 can be used to obtain corresponding coordinate points based on the gaze area and non-gaze area, and compress the gaze area and non-gaze area based on the coordinate points to obtain a target compressed image; the storage unit 40 can be used to store the target compressed image; the decompression unit 50 can be used to obtain the target compressed image of the storage unit 40, and store and / or decompress the target compressed image to obtain an output image.

[0215] The functions and roles of the various units of the image scaling processing device of the present disclosure have been specifically described in the various steps of the image scaling processing method 1000, and will not be repeated here. Figure 21 is a schematic diagram of the terminal provided by an embodiment of the present disclosure. As shown in Figure 21, the terminal 13 of this embodiment includes: a processor 1301, a memory 1302, and a computer program 1303 stored in the memory 1302 and executable on the processor 1301. When the processor 1301 executes the computer program 1303, the steps in the above-mentioned embodiments of the image scaling processing method are implemented. Alternatively, when the processor 1301 executes the computer program 1303, the functions of the various modules / units in the above-mentioned embodiments of the image scaling processing device are implemented.

[0216] The present disclosure further provides a readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the image scaling processing methods provided in the various embodiments described above.

[0217] The readable storage medium may be a computer storage medium or a communication medium. A communication medium includes any medium that facilitates the transfer of a computer program from one location to another. A computer storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer. For example, a readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium may also be an integral part of the processor. The processor and the readable storage medium may be located in an application-specific integrated circuit (ASIC). Alternatively, the ASIC may be located in a user device. Of course, the processor and the readable storage medium may also exist as discrete components in a communication device. The readable storage medium may be a read-only memory (ROM), a random-access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, or an optical data storage device.

[0218] The present disclosure also provides a program product, which includes execution instructions stored in a readable storage medium. At least one processor of a device can read the execution instructions from the readable storage medium, and at least one processor executes the execution instructions so that the device implements the image scaling processing methods provided in the various embodiments described above.

[0219] The present disclosure also provides a computer program, which is stored in a readable storage medium. At least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program, so that the electronic device implements the image scaling processing method provided by the various embodiments described above.

[0220] In the embodiments of the above-mentioned devices, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present disclosure may be directly implemented by a hardware processor or implemented by a combination of hardware and software modules in the processor.

[0221] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit the same. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present disclosure, and should all be included in the scope of protection of the present disclosure.

Claims

1. An image scaling processing method, comprising: Obtaining eye movement information when the user observes the target image; Performing coordinate conversion on the eye movement information to obtain the center coordinates corresponding to the eye movement information; determining a non-gaze area of ​​the target image according to the central coordinates, and compressing the non-gaze area of ​​the target image to obtain a compressed target image; Performing computational processing on the compressed target image to obtain a processed target image; A decompressed target image is obtained according to the center coordinates and the processed target image.

2. The image scaling method according to claim 1, wherein: The step of determining the non-gaze area of ​​the target image according to the central coordinates and compressing the non-gaze area of ​​the target image to obtain a compressed target image includes: Determining the coordinates of each area in the non-gaze area of ​​the target image according to the central coordinates, wherein the coordinates of each area in the target image include the central coordinates and the coordinates of each area in the non-gaze area; The target image is compressed based on the coordinates of each area in the target image to obtain the compressed target image.

3. The image scaling method according to claim 2, wherein: The target image also includes a gaze area, and the method further includes: Determining coordinates of each area of ​​the gaze area of ​​the target image according to the central coordinates, wherein the coordinates of each area in the target image also include coordinates of each area in the gaze area, and the gaze area includes a transition area located at an edge; The target image is compressed based on the coordinates of each area in the target image to obtain the compressed target image.

4. The image scaling method according to claim 3, wherein: The compressing the target image based on the coordinates of each area in the target image to obtain the compressed target image includes: Determining a coordinate mapping table of the non-gaze area based on coordinates of each area in the non-gaze area of ​​the target image; Performing a table lookup according to the gaze area, the center coordinates, the non-gaze area, and the coordinate mapping table to calculate the mapping coordinates and the zoom factor of each pixel position in the target image; Calculating the scaling weight of the gaze area, the scaling weight of the non-gaze area and the fusion weight of the transition area in the target image according to the scaling ratio, wherein the continuous scaling curve corresponding to the scaling weight is a first derivative smooth curve, the points on the first derivative smooth curve are used to represent the scaling ratio of the pixel coordinate position, and the scaling ratio is increasing; The transition fusion weight curve corresponding to the fusion weight is a second derivative smooth curve, and the points on the second derivative smooth curve are used to characterize the transition fusion weight of the distance between the pixel position and the edge of the gaze area. The transition fusion weight integrates the interpolation curve weight and the identity mapping weight, and the interpolation curve weight of the pixel position close to the edge of the gaze area is large, and the identity mapping weight of the pixel position close to the center of the gaze area is large. The scaling weight of the gaze area, the scaling weight of the non-gaze area, and the fusion of the transition area The weights are weighted and synthesized to output the compressed target image.

5. The image scaling method according to claim 1, wherein: The image algorithm operation used in the computational processing of the compressed target image at least includes a denoising algorithm operation and a specific image algorithm operation.

6. The image scaling method according to claim 5, wherein: The denoising algorithm is a neural network algorithm; The step of performing computational processing on the compressed target image to obtain a processed target image comprises: The compressed target image is denoised using the neural network algorithm to obtain the processed target image.

7. The image scaling method according to claim 1, wherein: The step of obtaining the decompressed target image according to the center coordinates and the processed target image includes: Calculating the coordinates of each region in the processed target image according to the center coordinates; The processed target image is decompressed based on the coordinates of each region in the processed target image to obtain the decompressed target image.

8. The image scaling method according to claim 7, wherein: The center coordinates are positions corresponding to the gaze area of ​​the target image; The step of calculating the coordinates of each region in the processed target image based on the center coordinates includes: The coordinates of each area in the non-gaze area of ​​the processed target image are determined according to the central coordinates, wherein the coordinates of each area in the target image include the central coordinates and the coordinates of each area in the non-gaze area.

9. The image scaling method according to claim 8, wherein: The decompressing the processed target image based on the coordinates of each region in the processed target image to obtain the decompressed target image includes: Determining a coordinate mapping table of the non-gaze area based on the coordinates of each area in the non-gaze area of ​​the processed target image; Performing a reverse lookup according to the gaze area, the center coordinates, the non-gaze area, and the coordinate mapping table to calculate the mapping coordinates and the zoom factor of each pixel position in the processed target image; Calculating the scaling weight of the gaze area, the scaling weight of the non-gaze area, and the fusion weight of the transition area in the processed target image according to the scaling ratio; The scaling weight of the gaze area, the scaling weight of the non-gaze area, and the fusion weight of the transition area are weighted and synthesized, and a decompressed target image is output.

10. The image scaling method according to claim 1, wherein: The center coordinates are a position corresponding to a gaze area of ​​the target image or a position corresponding to a gaze point set on the target image.

11. The image scaling method according to claim 10, wherein: The gaze point is dynamic information obtained according to eye tracking or a static gaze point that does not change according to eye movement information.

12. The image scaling method according to any one of claims 1 to 11, wherein: The target image at least includes a Bayer image, an RGB image, a RAW image, and a YUV image.

13. The image scaling method according to any one of claims 1 to 11, wherein: The calculation method for calculating the compressed target image at least includes neural network processor NPU calculation and neural network model calculation.

14. The image scaling method according to claim 1 or 2, wherein: The target image also includes a gaze area, and the method further includes: Determining the gaze area and the non-gaze area of ​​the target image according to the central coordinates; Determining a first sampling rate of the gaze area and a second sampling rate of the non-gaze area, wherein the first sampling rate is greater than the second sampling rate; Based on the first sampling rate and the second sampling rate, sampling the gaze area and the non-gaze area respectively to obtain a sampled image; A first compression rate and a second compression rate of the sampled image are determined, and based on the first compression rate and the second compression rate, the gaze area and the non-gaze area of ​​the sampled image are compressed respectively to obtain a target compressed image.

15. The image scaling method according to claim 14, wherein: Determining a gaze area and a non-gaze area of ​​the target image according to the center coordinates includes: Dividing the target image in the horizontal direction and the vertical direction based on the center coordinate to obtain N*M image units; Determining the gaze area based on the central coordinates, wherein the gaze area includes X*Y image units, 1≤X≤N, 1≤Y≤M; Other areas of the target image are determined as the non-gaze areas.

16. The image scaling method according to claim 15, wherein: The method of sampling the gaze area and the non-gaze area based on the first sampling rate and the second sampling rate to obtain a sampled image includes: Acquire the abscissa and ordinate of the target image, and determine the sampling initial position and step distance based on the abscissa and ordinate of the target image and the center coordinates; Setting a data window and a sampling point, and determining a movement parameter of the data window based on the sampling point; The gaze area and the non-gaze area are sampled based on the movement parameter, the sampling initial position, and the stepping distance to obtain the sampled image.

17. The image scaling method according to claim 16, wherein: Setting a data window and a sampling point, and determining a movement parameter of the data window based on the sampling point, includes: Determining the relative position of the data window and the first sampling point and / or the second sampling point; The movement parameter is determined based on the relative position.

18. The image scaling method according to claim 15, wherein: In the non-gaze area, the sampling rates of the image units located in the same horizontal direction are the same, and the sampling rates of the image units located in the same vertical direction are the same.

19. The image scaling method according to claim 16, wherein: Determining a first sampling rate of the gaze area and a second sampling rate of the non-gaze area includes: Setting the first sampling rate of the gaze area, wherein the first sampling rate is within a gaze compression range; Determining an image distance between the image unit in the non-gaze area and the gaze area; Determining a plurality of second sub-sampling rates based on the second sampling rate, wherein different second sub-sampling rates have different sampling parameters; The second sub-sampling rate corresponding to the image unit in the non-gaze area is configured based on the image distance, wherein a larger image distance corresponds to a smaller second sub-sampling rate.

20. The image scaling method according to any one of claims 14 to 19, wherein: The method further comprises: The target compressed image is stored and / or decompressed based on the first compression ratio and the second compression ratio to obtain an output image.

21. An image scaling processing device, comprising: An acquisition module, used to acquire eye movement information when a user observes a target image; A first calculation module, used for performing coordinate conversion on the eye movement information to obtain the center coordinates corresponding to the eye movement information; A compression module, configured to determine a non-gaze area of ​​the target image according to the central coordinates, and compress the non-gaze area of ​​the target image to obtain a compressed target image; A second calculation module is used to calculate the compressed target image to obtain a processed target image; The decompression module is used to obtain a decompressed target image according to the center coordinates and the processed target image.

22. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the image scaling processing method according to any one of claims 1 to 20 is implemented.

23. A computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a processor, the image scaling processing method according to any one of claims 1 to 20 is implemented.

24. A computer program product, wherein: The computer program product comprises computer program instructions, and the computer program instructions enable a computer to execute the image scaling processing method according to any one of claims 1 to 20.

25. A computer program, wherein The computer program enables a computer to execute the image scaling method according to any one of claims 1 to 20.

Images