Image resolution conversion method and device based on deep neural network, and terminal equipment

Abstract

A method and device for image resolution conversion based on a deep neural network, and a terminal device, the method for image resolution conversion includes: obtaining an image to be converted and a conversion ratio; mapping the image to be converted to a latent space to obtain a plurality of implicit feature maps; determining coordinates to be interpolated in the converted image, the pixel values ​​of the coordinates to be interpolated being empty; mapping each coordinate to be interpolated to the plurality of implicit feature maps according to the conversion ratio to obtain an implicit feature vector corresponding to each coordinate to be interpolated; converting the implicit feature vector corresponding to each coordinate to be interpolated into a pixel value, and filling each pixel value into the coordinate position to be interpolated in the converted image to obtain the converted image. The technical solution of the present invention can realize the conversion of image resolution at any ratio.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image resolution conversion method and device based on a deep neural network, and a terminal device. Background Art

[0002] The main task of single image super resolution (SISR) is to reconstruct a visually natural high resolution (HR) image from a degraded low resolution (LR) image. Currently, there are many single image super resolution reconstruction (SISR) algorithms based on deep learning, such as SRCNN, ESPCNN, EDSR, RDN, RCAN, etc.

[0003] The reconstruction quality of the SISR algorithm based on deep learning at a specific magnification far exceeds that of the traditional interpolation algorithm. A single model built by the SISR algorithm based on deep learning can handle super-resolution tasks at a certain magnification, and the magnifications are all integers.

[0004] However, real scenes have the need to handle non-integer, arbitrary-magnification super-resolution tasks. Existing deep learning-based SISR algorithms cannot support high-definition multiplication of low-resolution images. Summary of the invention

[0005] The technical problem solved by the present invention is how to realize the conversion of image resolution to any magnification.

[0006] To solve the above technical problems, an embodiment of the present invention provides an image resolution conversion method based on a deep neural network. The image resolution conversion method based on a deep neural network includes: obtaining an image to be converted and a conversion ratio; mapping the image to be converted to a latent space to obtain multiple implicit feature maps; determining coordinates to be interpolated in the converted image, wherein the pixel values ​​of the coordinates to be interpolated are empty; mapping each coordinate to be interpolated to the multiple implicit feature maps according to the conversion ratio to obtain an implicit feature vector corresponding to each coordinate to be interpolated; converting the implicit feature vector corresponding to each coordinate to be interpolated into a pixel value, and filling each pixel value into the position of the coordinate to be interpolated in the converted image to obtain the converted image.

[0007] Optionally, mapping each coordinate to be interpolated to the multiple implicit feature maps according to the conversion ratio includes: in each implicit feature map, determining a feature coordinate having a mapping relationship with the coordinate to be interpolated according to the conversion ratio; in each implicit feature map, determining an implicit feature value corresponding to the coordinate to be interpolated by interpolating other coordinates within the neighborhood of the feature coordinate, and the implicit feature vector includes multiple implicit feature values.

[0008] Optionally, determining the feature coordinates having a mapping relationship with the coordinates to be interpolated according to the conversion ratio includes: calculating a quotient of the coordinates to be interpolated and the conversion ratio as the feature coordinates.

[0009] Optionally, a neural network model is used to map the image to be converted to a latent space and to convert the implicit feature vectors corresponding to each coordinate to be interpolated into pixel values. The neural network model includes an encoder and a multi-layer perceptron. The encoder is used to map the image to be converted to a latent space, and the multi-layer perceptron is used to convert the implicit feature vectors corresponding to each coordinate to be interpolated into pixel values.

[0010] Optionally, the encoder is trained in the following manner: constructing the encoder and the decoder; obtaining a first training sample, wherein the first training sample includes an image with a first resolution; and using the first training sample to train the encoder and the decoder.

[0011] Optionally, the multilayer perceptron is trained in the following manner: a second training sample is obtained, where the second training sample includes a plurality of implicit feature maps and a second resolution image, and a second resolution of the second resolution image is greater than a first resolution of the first resolution image; and the multilayer perceptron is trained using the second training sample.

[0012] Optionally, the first resolution image is obtained by downsampling the second resolution image.

[0013] The embodiment of the present invention also discloses an image resolution conversion device based on a deep neural network, and the image resolution conversion device based on a deep neural network includes: an acquisition module, used to acquire an image to be converted and a conversion ratio; a feature mapping module, used to map the image to be converted to a latent space to obtain a plurality of implicit feature maps; a module for determining coordinates to be interpolated, used to determine the coordinates to be interpolated in the converted image, the pixel value of the coordinates to be interpolated is empty; a coordinate mapping module, used to map each coordinate to be interpolated to the plurality of implicit feature maps according to the conversion ratio to obtain an implicit feature vector corresponding to each coordinate to be interpolated; and a conversion module, used to convert the implicit feature vector corresponding to each coordinate to be interpolated into a pixel value, and fill each pixel value into the position of the coordinate to be interpolated in the converted image to obtain the converted image.

[0014] An embodiment of the present invention further discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the image resolution conversion method based on a deep neural network are executed.

[0015] An embodiment of the present invention also discloses a terminal device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the steps of the image resolution conversion method based on a deep neural network when running the computer program.

[0016] Compared with the prior art, the technical solution of the embodiment of the present invention has the following beneficial effects:

[0017] In the technical solution of the present invention, when converting the image to be converted, the mapped implicit feature map is mapped to each coordinate in the converted image, so that each coordinate in the converted image has a corresponding implicit feature vector, and then the final converted pixel image is obtained by converting the implicit feature vector and the pixel value. The technical solution of the present invention has no limit on the conversion ratio by combining the neural network and the interpolation process, and can realize the conversion of arbitrary ratios and non-integer ratios of low-resolution images; in addition, each coordinate position in the converted image has a pixel value, so that the converted image has higher clarity.

[0018] Furthermore, when mapping the coordinates to be interpolated to multiple implicit feature maps, the feature coordinates that have a ratio conversion relationship with the coordinates to be interpolated can be determined first, and then the implicit feature vectors can be obtained by interpolating other coordinates in the neighborhood of the feature coordinates. The technical solution of the present invention uses the real eigenvalues ​​in the implicit feature map to determine the implicit feature vectors of the coordinates to be interpolated through interpolation, avoiding the use of predicted values ​​in the prior art to reconstruct high-resolution images, and can improve the clarity of the image on the basis of achieving arbitrary ratio super-resolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of an image resolution conversion method based on a deep neural network according to an embodiment of the present invention;

[0020] Figure 2 yes Figure 1 A flowchart of a specific implementation of step 104 is shown;

[0021] Figure 3 is a schematic diagram of a specific application scenario of an embodiment of the present invention;

[0022] Figure 4 It is a structural schematic diagram of an image resolution conversion device based on a deep neural network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] As described in the background, real scenes have the need to process non-integer, arbitrary-magnification super-resolution tasks. Existing deep learning-based SISR algorithms cannot support high-definition processing of low-resolution images at multiple magnifications.

[0024] The inventor of this application has found through research that the neural network model in the prior art reconstructs high-resolution images based on image features, which is essentially a process of upsampling feature maps. However, there is no floating-point multiplier operator in the current image upsampling process, and only integer multiplier super-resolution can be achieved.

[0025] In the technical solution of the present invention, when converting the image to be converted, the mapped implicit feature map is mapped to each coordinate in the converted image, so that each coordinate in the converted image has a corresponding implicit feature vector, and then the final converted pixel image is obtained by converting the implicit feature vector and the pixel value. The technical solution of the present invention has no limit on the conversion ratio by combining the neural network and the interpolation process, and can realize the conversion of arbitrary ratios and non-integer ratios of low-resolution images; in addition, each coordinate position in the converted image has a pixel value, so that the converted image has higher clarity.

[0026] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0027] Figure 1 It is a flowchart of an image resolution conversion method based on a deep neural network according to an embodiment of the present invention.

[0028] The technical solution of the present invention can be used in any terminal device with computing capability, that is, each step of the method can be executed by the terminal device, which may include but is not limited to a mobile phone, a computer, a tablet computer, etc.

[0029] For details, please refer to Figure 1 , the image resolution conversion method may include the following steps:

[0030] Step 101: Obtain an image to be converted and a conversion ratio;

[0031] Step 102: Mapping the image to be converted to a latent space to obtain a plurality of latent feature maps;

[0032] Step 103: determining the coordinates to be interpolated in the converted image, wherein the pixel values ​​of the coordinates to be interpolated are empty;

[0033] Step 104: mapping each coordinate to be interpolated to the plurality of implicit feature maps according to the conversion ratio to obtain an implicit feature vector corresponding to each coordinate to be interpolated;

[0034] Step 105: converting the implicit feature vectors corresponding to the coordinates to be interpolated into pixel values, and filling the pixel values ​​into the coordinate positions to be interpolated in the converted image, so as to obtain the converted image.

[0035] It should be pointed out that the serial numbers of the steps in this embodiment do not limit the execution order of the steps.

[0036] It can be understood that, in a specific implementation, the image resolution conversion method can be implemented in the form of a software program, and the software program runs in a processor integrated inside a chip or a chip module.

[0037] The image to be converted in the embodiment of the present invention can be called a low-resolution image, and the converted image can be called a high-resolution image. The conversion ratio is greater than 1. That is, the image to be converted needs to be converted into an image with a higher resolution.

[0038] In the specific implementation of step 102, the image to be converted can be mapped to the latent space, and multiple implicit feature maps can be obtained. The number of implicit feature maps is related to the number of convolution kernels used by the neural network algorithm. For example, when the number of convolution kernels is 64, 64 implicit feature maps can be obtained. Since the process of mapping to the latent space can extract feature information of the image to be converted, such as edge information, local information, etc., the implicit feature map contains rich feature information and is suitable for reconstructing high-resolution images.

[0039] In the specific implementation of step 103 and step 104, interpolation of each coordinate in the converted image can be achieved. Specifically, since the resolution of the converted image is higher than the resolution of the image to be converted, there are coordinates with empty pixel values ​​in the converted image, and these coordinates can be called coordinates to be interpolated. The coordinate values ​​of the coordinates to be interpolated can be calculated by the interpolation process of the coordinate values ​​of the feature coordinates in the implicit feature map.

[0040] The specific interpolation process can be: first determine the coordinates to be interpolated with empty pixel values, and then map each coordinate to be interpolated to multiple implicit feature maps. Each coordinate to be interpolated can correspond to an implicit feature vector through mapping, and the implicit feature vector includes the values ​​of the feature coordinates corresponding to the coordinates to be interpolated in multiple implicit feature maps. For example, when the number of convolution kernels is 64, the implicit feature vector is a 64-dimensional vector, and each dimension represents the value of the feature coordinate corresponding to the coordinate to be interpolated in an implicit feature map.

[0041] Among them, each coordinate in the converted image is a coordinate to be interpolated.

[0042] Then, in the specific implementation of step 105, the implicit feature vectors corresponding to the coordinates to be interpolated are converted into pixel values, so as to obtain the final converted image.

[0043] Specifically, the conversion from implicit feature vectors to pixel values ​​refers to converting the feature values ​​in each implicit feature map into pixel values ​​of three channels (such as RGB or YUV).

[0044] In a specific embodiment, a multi-layer perceptron (MLP) may be used to convert the implicit feature vectors corresponding to the coordinates to be interpolated into pixel values.

[0045] It should be noted that in actual application scenarios, any other feasible neural network architecture can also be used to realize the conversion from implicit feature vectors to pixel values, and the embodiments of the present invention are not limited to this.

[0046] The embodiments of the present invention can realize super-resolution at any magnification, and can also realize super-resolution at non-integer magnification.

[0047] In a non-limiting embodiment of the present invention, please refer to Figure 2 , Figure 1 Step 104 may include the following steps:

[0048] Step 201: In each implicit feature map, determining feature coordinates having a mapping relationship with the coordinates to be interpolated according to the conversion ratio;

[0049] Step 202: In each implicit feature map, the implicit feature value corresponding to the coordinate to be interpolated is determined by interpolating other coordinates in the neighborhood of the feature coordinate, and the implicit feature vector includes multiple implicit feature values.

[0050] In a specific implementation, each coordinate in the implicit feature map can be referred to as a feature coordinate. The feature coordinate has a mapping relationship with the coordinate to be interpolated, and the mapping relationship can be represented by a conversion ratio. After obtaining the mapping relationship between the coordinate to be interpolated and the feature coordinate, the coordinate value of the coordinate to be interpolated (i.e., the implicit feature value) can be obtained by interpolating other coordinates in the neighborhood of the feature coordinate.

[0051] Furthermore, the coordinate values ​​of the coordinates used for interpolation (ie, other coordinates) are integer coordinates. The feature coordinates can be integer coordinates or non-integer coordinates.

[0052] In a non-limiting embodiment, bilinear interpolation can be used to interpolate other coordinates in the neighborhood to determine the implicit eigenvalues ​​corresponding to the coordinates to be interpolated. Bicubic interpolation and nearest neighbor interpolation can also be used to interpolate other coordinates in the neighborhood to determine the implicit eigenvalues ​​corresponding to the coordinates to be interpolated.

[0053] It is understandable that any existing implementable interpolation algorithm may be used to interpolate other coordinates, and the embodiment of the present invention does not limit this.

[0054] In a specific embodiment, the quotient of the coordinate to be interpolated and the conversion magnification is calculated as the feature coordinate. That is, the mapping relationship between the feature coordinate and the coordinate to be interpolated means that the quotient of the coordinate to be interpolated and the conversion magnification is the feature coordinate corresponding to the coordinate to be interpolated. In other words, the product of the feature coordinate and the conversion magnification is the coordinate to be interpolated corresponding to the feature coordinate.

[0055] For example, when the coordinate to be interpolated is (10,10) and the conversion factor is 2.5, the corresponding feature coordinate can be calculated to be (4,4). Then, in each feature map, other coordinates in the neighborhood of the feature coordinate (4,4), such as the coordinate values ​​of the coordinates (3,3), (3,5), (5,3) and (5,5), can be bilinearly interpolated to obtain the coordinate value of the interpolated coordinate (10,10).

[0056] In another variation, for the coordinate to be interpolated (10,10), if the conversion ratio is 3.5, the corresponding feature coordinate is (2.85,2.85). Then, in each feature map, four integer feature coordinates near the feature coordinate (2.85,2.85) can be selected, such as (2,2), (2,3), (3,2) and (3,3) for bilinear interpolation to obtain the coordinate value of the coordinate to be interpolated (10,10).

[0057] In a non-limiting embodiment of the present invention, a neural network model is used to map the image to be converted to a latent space and convert the implicit feature vectors corresponding to each coordinate to be interpolated into pixel values. The neural network model includes an encoder and a multi-layer perceptron. The encoder is used to map the image to be converted to a latent space, and the multi-layer perceptron is used to convert the implicit feature vectors corresponding to each coordinate to be interpolated into pixel values.

[0058] In a specific implementation, the encoder and the multi-layer perceptron can be constructed using neural networks respectively. The encoder and the multi-layer perceptron can also be obtained through training respectively.

[0059] In a non-limiting embodiment, the encoder is trained in the following manner: constructing the encoder and decoder (Decode); obtaining a first training sample, wherein the first training sample includes a first resolution image; and using the first training sample to train the encoder and decoder.

[0060] In a non-limiting embodiment, a multilayer perceptron is trained in the following manner: a second training sample is obtained, wherein the second training sample includes a plurality of implicit feature maps and a second resolution image, wherein a second resolution of the second resolution image is greater than a first resolution of the first resolution image; and the multilayer perceptron is trained using the second training sample.

[0061] Furthermore, the first resolution image is obtained by downsampling the second resolution image. That is to say, in practical applications, the encoder and the multilayer perceptron can be trained by collecting one type of image (i.e., the second resolution image), thereby achieving the convenience of training.

[0062] In a specific application scenario of the present invention, please refer to Figure 3 The neural network model used in the embodiment of the present invention includes an encoder 301 and a multi-layer perceptron 302. First, the encoder 301 and decoder 304 models are trained. After the model training is completed, the parameters of the encoder 301 model are frozen for later acquisition of the implicit feature map of the input image. The multi-layer perceptron 302 is then trained for subsequent conversion of the feature map into a pixel map.

[0063] The input image is mapped to the latent space by the encoder 301, so as to obtain an implicit feature map that can represent the input image (that is, the image to be converted). When the number of channels is 64, the number of implicit feature maps is also 64. The implicit feature map output by the encoder 301 is interpolated in step S31. Specifically, bilinear interpolation can be used to map the coordinates to be interpolated of the output image to a low-resolution implicit feature map, and then bilinear interpolation is performed on the low-resolution implicit feature map to obtain the implicit feature vector (64 dimensions) of the coordinates to be interpolated, that is, a high-resolution implicit feature map. The implicit feature vector of the coordinates to be interpolated is mapped to the pixel value of the coordinates to be interpolated by the multi-layer perceptron 302, and the output image (that is, the converted image) is output.

[0064] The embodiment of the present invention realizes the process of obtaining corresponding pixel values ​​through coordinate values ​​on a high-resolution image (output image). In addition, the embodiment of the present invention realizes resolution conversion by combining a neural network model and an interpolation process, so that the conversion magnification is no longer limited to an integer magnification, thereby realizing the conversion of arbitrary magnifications and non-integer magnifications of low-resolution images.

[0065] Please refer to Figure 4 The embodiment of the present invention also discloses an image resolution conversion device based on a deep neural network. The image resolution conversion device 40 based on a deep neural network may include:

[0066] An acquisition module 401 is used to acquire an image to be converted and a conversion ratio;

[0067] A feature mapping module 402 is used to map the image to be converted into a latent space to obtain a plurality of latent feature maps;

[0068] The to-be-interpolated coordinates determining module 403 is used to determine the to-be-interpolated coordinates in the converted image, wherein the pixel values ​​of the to-be-interpolated coordinates are empty;

[0069] A coordinate mapping module 404, used to map each to-be-interpolated coordinate to the plurality of implicit feature maps according to the conversion ratio, so as to obtain an implicit feature vector corresponding to each to-be-interpolated coordinate;

[0070] The conversion module 405 is used to convert the implicit feature vector corresponding to each coordinate to be interpolated into a pixel value, and fill each pixel value into the coordinate position to be interpolated in the converted image to obtain the converted image.

[0071] For more information about the working principle and working method of the image resolution conversion device 40, please refer to Figures 1 to 3 The relevant description in will not be repeated here.

[0072] In a specific implementation, the above-mentioned image resolution conversion device can correspond to a chip with a PDCCH detection function in a terminal device, such as a SOC (System-On-a-Chip), a baseband chip, etc.; or correspond to a chip module with a PDCCH detection function in the terminal device; or correspond to a chip module with a data processing function chip, or correspond to a terminal device.

[0073] Regarding the various modules / units included in the various devices and products described in the above embodiments, they can be software modules / units, or hardware modules / units, or they can be partially software modules / units and partially hardware modules / units. For example, for various devices and products applied to or integrated in a chip, the various modules / units included therein can all be implemented in the form of hardware such as circuits, or at least some of the modules / units can be implemented in the form of software programs, which run on a processor integrated inside the chip, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits; for various devices and products applied to or integrated in a chip module, the various modules / units included therein can all be implemented in the form of hardware such as circuits, and different modules / units can be located in the same component (such as a chip, circuit module, etc.) or different components of the chip module, or at least some of the modules / units can be implemented in the form of hardware such as circuits. The element can be implemented in the form of a software program, which runs on a processor integrated inside the chip module, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits; for various devices and products applied to or integrated in the terminal, the various modules / units contained therein can be implemented in the form of hardware such as circuits, and different modules / units can be located in the same component (for example, chip, circuit module, etc.) or in different components in the terminal, or, at least some modules / units can be implemented in the form of a software program, which runs on a processor integrated inside the terminal, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits.

[0074] The embodiment of the present invention further discloses a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is run, it can execute Figure 1 or Figure 2 The steps of the method shown in . The storage medium may include ROM, RAM, magnetic disk or optical disk, etc. The storage medium may also include non-volatile memory (non-volatile) or non-transitory memory, etc.

[0075] The embodiment of the present invention further discloses a terminal device, which may include a memory and a processor, wherein the memory stores a computer program that can be run on the processor. When the processor runs the computer program, it can execute Figure 1 or Figure 2 The steps of the method shown in .

[0076] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article indicates that the associated objects before and after are in an "or" relationship.

[0077] The "plurality" appearing in the embodiments of the present application refers to two or more.

[0078] The first, second, etc. descriptions appearing in the embodiments of the present application are only used for illustration and distinction of the description objects. There is no order, nor do they indicate any special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0079] It should be understood that in the embodiments of the present application, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0080] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM) and direct rambus RAM (DR RAM).

[0081] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0082] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0083] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and systems can be implemented in other ways. For example, the device embodiments described above are merely schematic; for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0084] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0085] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0086] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0087] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. A method for image resolution conversion based on deep neural network, characterized in that: include: Obtain the image to be converted and the conversion ratio; Mapping the image to be converted to a latent space to obtain a plurality of implicit feature maps; Determine a coordinate to be interpolated in the converted image, wherein the pixel value of the coordinate to be interpolated is empty; Mapping each coordinate to be interpolated to the multiple implicit feature maps according to the conversion ratio to obtain an implicit feature vector corresponding to each coordinate to be interpolated, wherein the implicit feature vector includes the value of the feature coordinate corresponding to the coordinate to be interpolated in the multiple implicit feature maps, and the coordinate value of the coordinate to be interpolated is obtained by interpolating other coordinates in the neighborhood of the feature coordinate having a mapping relationship with the coordinate to be interpolated, and the coordinate values ​​of the other coordinates used for interpolation are integer coordinates; The implicit feature vectors corresponding to the coordinates to be interpolated are converted into pixel values, and the pixel values ​​are filled into the coordinate positions to be interpolated in the converted image to obtain the converted image.

2. The image resolution conversion method based on deep neural network according to claim 1 is characterized in that: The step of mapping each to-be-interpolated coordinate to the plurality of implicit feature maps according to the conversion ratio comprises: In each implicit feature map, determining feature coordinates having a mapping relationship with the coordinates to be interpolated according to the conversion ratio; In each implicit feature map, the implicit feature value corresponding to the coordinate to be interpolated is determined by interpolating other coordinates in the neighborhood of the feature coordinate, and the implicit feature vector includes multiple implicit feature values.

3. The image resolution conversion method based on deep neural network according to claim 2 is characterized in that: The step of determining the characteristic coordinates having a mapping relationship with the coordinates to be interpolated according to the conversion ratio comprises: The quotient of the coordinate to be interpolated and the conversion factor is calculated as the characteristic coordinate.

4. The image resolution conversion method based on deep neural network according to claim 1 is characterized in that: A neural network model is used to map the image to be converted into a latent space and to convert the implicit feature vectors corresponding to each coordinate to be interpolated into pixel values. The neural network model includes an encoder and a multi-layer perceptron. The encoder is used to map the image to be converted into a latent space, and the multi-layer perceptron is used to convert the implicit feature vectors corresponding to each coordinate to be interpolated into pixel values.

5. The image resolution conversion method based on deep neural network according to claim 4 is characterized in that: The encoder is trained in the following manner: constructing the encoder and decoder; Acquire a first training sample, where the first training sample includes a first resolution image; The encoder and decoder are trained using the first training sample.

6. The image resolution conversion method based on deep neural network according to claim 5 is characterized in that: The multilayer perceptron is trained in the following manner: Acquire a second training sample, where the second training sample includes a plurality of implicit feature maps and a second resolution image, wherein a second resolution of the second resolution image is greater than a first resolution of the first resolution image; The multilayer perceptron is trained using the second training sample.

7. The image resolution conversion method based on deep neural network according to claim 6 is characterized in that: The first-resolution image is obtained by downsampling the second-resolution image.

8. An image resolution conversion device based on a deep neural network, characterized in that: include: An acquisition module is used to acquire the image to be converted and the conversion ratio; A feature mapping module, used for mapping the image to be converted into a latent space to obtain a plurality of latent feature maps; A module for determining coordinates to be interpolated, used for determining coordinates to be interpolated in the converted image, wherein the pixel value of the coordinates to be interpolated is empty; A coordinate mapping module, used for mapping each coordinate to be interpolated to the multiple implicit feature maps according to the conversion ratio to obtain an implicit feature vector corresponding to each coordinate to be interpolated, wherein the implicit feature vector includes the value of the feature coordinate corresponding to the coordinate to be interpolated in the multiple implicit feature maps, and the coordinate value of the coordinate to be interpolated is obtained by interpolating other coordinates in the neighborhood of the feature coordinate having a mapping relationship with the coordinate to be interpolated, and the coordinate values ​​of the other coordinates used for interpolation are integer coordinates; The conversion module is used to convert the implicit feature vectors corresponding to the coordinates to be interpolated into pixel values, and fill the pixel values ​​into the coordinate positions to be interpolated in the converted image to obtain the converted image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image resolution conversion method based on a deep neural network described in any one of claims 1 to 7 are performed.

10. A terminal device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor runs the computer program, the steps of the image resolution conversion method based on deep neural network described in any one of claims 1 to 7 are performed.

Images