Training data acquisition method, image quality enhancement model and method, and electronic device

Abstract

The present application provides a training data acquisition method, an image quality enhancement model, an image quality enhancement method, an electronic device and a machine-readable storage medium. The training data is used for the image quality enhancement model. The training data acquisition method includes: acquiring a high-resolution image; converting the high-resolution image into an original image collected by a sensor; superimposing noise that satisfies a specified distribution on the original image to obtain an intermediate image; converting the intermediate image into a low-resolution image; wherein the high-resolution image is an image with a resolution greater than a first threshold, and the low-resolution image is an image with a resolution less than a second threshold.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a training data acquisition method, an image quality enhancement model and method, an electronic device, and a machine-readable storage medium. Background Art

[0002] With the continuous development and progress of live broadcast technology, more and more users use terminal devices to live broadcast and record and share food, beautiful scenery, talents, etc. However, due to the limitations of terminal device shooting and data upload compression rate, when users upload live video streams through terminal devices, the resolution of the live screen received by the audience will be relatively low and the image quality will be poor, affecting the user experience.

[0003] Although there have been many studies and applications of image restoration, enhancement and super-resolution technologies based on deep learning networks in the field of non-video live broadcast. However, due to the high real-time nature of video live broadcast, the application of deep learning networks in the field of video live broadcast is still relatively rare. In addition, most of the low-resolution training data used in image restoration, enhancement and super-resolution technologies based on deep learning networks are full-color images (such as RGB images) collected using high-performance image acquisition devices. Low-resolution images are obtained by artificially introducing noise or blurring into full-color images. The low-resolution training images obtained by this method are often quite different from the low-resolution images generated in real live broadcast scenes. Summary of the invention

[0004] In order to overcome the problem in the related art that the low-resolution images trained by the image quality enhancement model are significantly different from the low-resolution images generated in the real live broadcast scene, the present application provides a training data acquisition method, an image quality enhancement model, an electronic device and a machine-readable storage medium.

[0005] According to a first aspect of an embodiment of the present application, a method for acquiring training data is provided, wherein the training data is used for an image quality enhancement model, and the method comprises: acquiring a high-resolution image; converting the high-resolution image into an original image acquired by a sensor; superimposing noise satisfying a specified distribution on the original image to obtain an intermediate image; and converting the intermediate image into a low-resolution image; wherein the high-resolution image is an image having a resolution greater than a first threshold, and the low-resolution image is an image having a resolution less than a second threshold.

[0006] According to the second aspect of an embodiment of the present application, an image quality enhancement model is provided, which is determined based on the training data described in the first aspect of the present application, and is used to achieve image quality enhancement of an input image. The image quality enhancement model includes: an encoding end and a decoding end; the encoding end includes an input convolution layer and a specified number of first layers, the first layers are connected in series in sequence, and each first layer includes a downsampling layer and a first number of residual modules; the decoding end includes a specified number of second layers and an output convolution layer, the second layers are connected in series in sequence, and each second layer includes an upsampling layer and a second number of residual modules; wherein the residual module of the last first layer of the encoding end is connected to the residual module of the first second layer of the decoding end, and the second number is greater than the first number.

[0007] According to a third aspect of an embodiment of the present application, there is provided an image quality enhancement method based on the image quality enhancement model described in the second aspect of the present application, the method comprising: obtaining an image to be processed from a host terminal; inputting the image to be processed into the image quality enhancement model to obtain a high-resolution image with a resolution greater than a first threshold; wherein the image to be processed undergoes encoding processing and decoding processing in the image quality enhancement model in sequence, the encoding processing in sequence comprising an input convolution processing and a specified number of first processings, each first processing comprising a downsampling processing and a first number of feature extractions, the decoding processing comprising a specified number of second processings and an output convolution processing, each second processing comprising an upsampling processing and a second number of feature recoverys, and the second number is greater than the first number

[0008] The image to be processed is input into the image quality enhancement model to obtain a high-resolution image; wherein the resolution of the high-resolution image is greater than a first threshold.

[0009] According to a fourth aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of the first aspect of the present application and the third aspect of the present application is implemented.

[0010] According to a fifth aspect of an embodiment of the present application, a machine-readable storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the first aspect and the third aspect of the present application is implemented.

[0011] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0012] In the embodiment of the present application, the training data is determined by converting the high-resolution image into the original image collected by the sensor, superimposing noise that satisfies the specified distribution on the original image, and converting it into a low-resolution image. Since it is based on the principle of image imaging, it simulates the process of users using terminal devices to obtain images in the real world. Therefore, the low-resolution image generated based on the method described in the present application is closer to the low-resolution image generated in the real scene, and the low-resolution effect of the image is more accurate.

[0013] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the application.

[0015] Figure 1 It is a flowchart of a method for obtaining training data according to an exemplary embodiment of the present application.

[0016] Figure 2 This is a flow chart of generating a full-color image according to an exemplary embodiment of the present application.

[0017] Figure 3 A color filter array in RGGB format is shown in the present application according to an exemplary embodiment.

[0018] Figure 4 It is a schematic diagram showing the principle of obtaining an RGB image by demosaicing a Bayer array image in RGGB format according to an exemplary embodiment of the present application.

[0019] Figure 5 It is a schematic diagram of an image effect before and after tone mapping according to an exemplary embodiment of the present application.

[0020] Figure 6 It is a schematic diagram of generating a low-resolution image by performing blur kernel convolution on an image according to an exemplary embodiment of the present application.

[0021] Figure 7 It is a schematic diagram of the principle of obtaining a blur kernel by interpolation based on a Markov random process according to an exemplary embodiment of the present application.

[0022] Figure 8 It is a structural schematic diagram of an image quality enhancement model shown in the present application according to an exemplary embodiment.

[0023] Fig. 9It is a structural schematic diagram of another image quality enhancement model shown in the present application according to an exemplary embodiment.

[0024] Fig.10 The present application shows a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0025] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0026] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0027] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0028] With the continuous development and progress of live broadcast technology, more and more users use terminal devices to live broadcast and record and share food, beautiful scenery, talents, etc. However, due to the limitations of terminal device shooting, for example, the terminal device is unstable or the subject is moving, resulting in blurred imaging effects; the image acquisition device of the terminal device is of poor quality, resulting in low image quality, etc., when users upload live video streams through terminal devices, the resolution of the live screen received by the audience will be relatively low and the image quality will be poor, affecting the user experience.

[0029] Although image restoration, enhancement and super-resolution based on deep learning networks have been widely studied and applied in non-video live broadcast fields, due to the high real-time nature of video live broadcast, the application of deep learning networks in the field of video live broadcast is relatively rare.

[0030] In addition, deep learning-based visual algorithms require a lot of data to drive them. According to the learning method of deep learning networks, they are divided into supervised and unsupervised (or weakly) supervised learning. Currently, especially in the field of image processing based on deep learning, the effects of supervised methods are almost always stronger than those of unsupervised methods. This makes the effects of deep learning-based image quality enhancement methods strongly dependent on the quantity and quality of paired image data (such as noisy images and clean images, blurred images and clear images). However, paired image data is difficult to obtain.

[0031] For noisy images, there are currently some studies that use the short and long exposure of the camera to obtain images with more noise and less noise in a specific scene as a pair of training data for the deep learning network - input and label data. However, this method can only be used for specific imaging devices, and the image acquisition cost is very high. For blurred images, one acquisition method is to use a high-speed camera to shoot a high-frame rate video, and a motion blurred image can be obtained by adding and averaging multiple consecutive frames. However, due to the limitation of the camera shooting frame rate (for example, 120Hz or 240Hz), this method will cause the synthetic image to have large ghosting. In addition, there are many other methods to obtain low-resolution images for deep learning network training, but there is still a large difference between the low-resolution images obtained and the low-resolution images generated in the real live broadcast scene. In order to solve the above-mentioned defects existing in the related art, the present application provides a method for obtaining training data, an image quality enhancement model and method, an electronic device, and a machine-readable storage medium.

[0032] Next, the embodiments of the present application are described in detail.

[0033] like Figure 1 As shown, Figure 1 This is a flowchart of a method for acquiring training data according to an exemplary embodiment of the present application, wherein the method for acquiring training data is used to acquire a corresponding low-resolution image based on a high-resolution image, so that the high-resolution image and the acquired low-resolution image constitute a set of training data - label data and input data, which are used to train image processing models such as image quality restoration models and image quality enhancement models based on deep learning to obtain models with excellent performance. The method for acquiring training data comprises the following steps:

[0034] Step S101, obtaining a high-resolution image;

[0035] Step S102, converting the high-resolution image into an original image captured by a sensor;

[0036] Step S103, superimposing noise satisfying a specified distribution on the original image to obtain an intermediate image;

[0037] Step S104, converting the intermediate image into a low-resolution image;

[0038] The high-resolution image is an image with a resolution greater than a first threshold, and the low-resolution image is an image with a resolution less than a second threshold.

[0039] The above-mentioned method for obtaining training data can be executed by a server, by an intelligent device with higher processing capabilities, or by a device with corresponding computing capabilities, and this application does not impose any restrictions on this.

[0040] In some embodiments, in step S101, the high-resolution image can be acquired through a high-performance image acquisition device, or can be acquired from an existing high-resolution image database, or can be acquired by other means, for example, by extracting frames from a high-resolution video, and the present application does not impose any restrictions on this.

[0041] In some embodiments, the high-resolution image is an RGB image with a resolution greater than a specified threshold. Of course, the high-resolution image can also be a full-color image in other formats, such as a full-color image in YUV format, and this application does not impose any restrictions on this.

[0042] Before introducing the subsequent embodiments, for the sake of easy understanding, the process of generating a full-color image is first introduced. Figure 2 , a schematic diagram of the process of generating a full-color image for a scene 201 is given. The scene 201 is first imaged by an optical lens 202 to generate an optical image; the optical image is photoelectrically converted by a sensor 203 and converted into an analog electrical signal; the analog electrical signal is converted by an analog-to-digital converter 204 (A / D conversion) to obtain a digital electrical signal. The digital electrical signal is processed by a series of image signal processing 205 to obtain a full-color image 206.

[0043] In the above process, the sensor 203 does not have the ability to perceive colors and can only obtain grayscale images. In order to generate a full-color image of the scene 201, a color filter structure is usually added in front of the photosensitive array of the sensor 203. Accordingly, the image signal processing 205 includes a processing module for color information, so that a full-color image with realistic colors and high resolution can be obtained. The color filter structure covered by the sensor photosensitive array is usually called a color filter array (CFA).

[0044] The most commonly used color filter array is the Bayer array, and there are many types. Figure 3As shown in FIG. 1 , a specific Bayer array color filter array is provided, namely, a color filter array 301 in RGGB format, where R, G, and B represent red, green, and blue filter array units, respectively. Figure 3 It can be seen that in the color filter array of RGGB format, every 4 filter array units form a repeating unit 302. The reason is that human vision is most sensitive to green, so in the repeating unit of the color filter array, the G component is usually twice that of R and B. When the color filter array of RGGB format is added before the photosensitive array of the sensor 203, the sensor 203 generates a Bayer array image of the corresponding format after photoelectric conversion, that is, the raw image (Raw Image) of RGGB format. The raw image of RGGB format is the image corresponding to the color filter array 301 of RGGB format, and every 4 pixels of the image have the color information of red, green, green and blue respectively. In addition to the Bayer array color filter array of RGGB format, there are many other types of color filter arrays, such as: RGBG, GRBG, BGGR, etc.

[0045] The image obtained by the sensor with the added color filter array has the same color distribution as the color filter array, and looks similar to the image effect with mosaic added. In addition, due to the different responses of the sensor and the human eye to color, after the analog-to-digital conversion 204 of the signal obtained by the sensor is performed to obtain the Bayer array image of the specified format, it is necessary to perform at least one of the following image signal processing 205: de-mosaicing, color correction, tone mapping, etc., to obtain a full-color image. Among them, each specific image signal processing step is described in detail in the following text in conjunction with the embodiments of the present application, and the present application will not repeat it here.

[0046] Based on the above-mentioned full-color image generation process, in some embodiments, the original image involved in step S102 and step S103 of the training data acquisition method described in the present application is a sensor array image of a specified format. For example, when the original image is collected by a sensor based on an RGGB color filter array, the original image is a sensor array image in the RGGB format. Of course, those skilled in the art should understand that when the original image is collected by a sensor using a color filter array in other formats, the original image is a sensor array image in other formats, including but not limited to Bayer array images in RGBG, GRBG, BGGR formats, etc.

[0047] As mentioned above, the full-color image needs to undergo image signal processing 205 during the generation process. However, the noise of the original image obtained based on the sensor usually conforms to a specific distribution, while the noise of the full-color image based on image signal processing usually has very complex characteristics and no longer conforms to the specific distribution conformed to by the original image. In the relevant technology for generating low-resolution images, it is usually to directly add noise that satisfies the specified distribution to the high-resolution image to simulate the noise distribution in the low-resolution image. However, this does not conform to the superposition order of noise in the low-resolution image in the real scene (in the real scene, the noise that satisfies the specified distribution is superimposed on the original image). Therefore, in the training data acquisition method described in the present application, the high-resolution image is first converted into the original image collected by the sensor, and then the noise that satisfies the specified distribution is superimposed on the original image to simulate the superposition of noise that satisfies the specified distribution in the real scene during the image generation process.

[0048] Therefore, in some embodiments, in the training data acquisition method described in the present application, step S102, converting the high-resolution image into the original image captured by the sensor, includes: performing an inverse image signal processing transform on the high-resolution image; wherein the inverse image signal processing transform is the inverse process of the image signal processing transform performed during the generation of the high-resolution image, and the image signal processing transform includes at least one of the following: de-mosaicing, color correction, and tone mapping.

[0049] As mentioned above, the image obtained by the sensor is similar to the color distribution of the color filter array, and looks like a mosaic effect. Therefore, the image signal processing for generating a high-resolution image may include demosaicing. Demosaicing refers to converting the original image obtained by the sensor (usually a sensor array image in a specified format, such as a Bayer array image) into a full-color image, such as an RGB image, by a color interpolation method.

[0050] Combination Figure 4 , taking the original image as a Bayer array image in RGGB format and the full-color image as an RGB image as an example, an exemplary description of de-mosaicing is given. Based on the sensor with the added color filter array, an original image in RGGB format as shown in 401 can be obtained. The original image 401 is interpolated to obtain a red image 402, a green image 403 and a blue image 404 respectively. Based on the three color images, color synthesis is performed to obtain a full-color image 405. The interpolation process can be implemented with reference to the relevant technology, and this application will not elaborate on it.

[0051] Therefore, in some embodiments, obtaining the original image based on the high-resolution image may be achieved by using the inverse process of the above-mentioned demosaicing process. Figure 4 To explain, based on the high-resolution image 405 , the red image 402 , the green image 403 , and the blue image 404 are first acquired, and then the pixel values ​​of the designated colors at the designated pixel positions are extracted to acquire the original image of the RGGB Bayer array shown in 401 .

[0052] Of course, those skilled in the art should understand that the inverse process of the above-mentioned demosaicing process is only an exemplary description. The inverse process can also be designed accordingly based on the demosaicing process in the image signal processing process experienced in the specific high-resolution image generation process, and this application does not limit this.

[0053] Due to the difference between the spectrum responsivity of visible light of the human eye and the spectrum responsivity of the sensor, as well as the influence of the chromatic aberration of the optical lens itself, the image signal processing for generating high-resolution images may also include color correction. The usual method for implementing color correction is to obtain a color-corrected image by performing a dot multiplication operation on a preset color change matrix of a specified size and the image to be processed. For example, a 3*3 color change matrix M is selected and dot-multiplied with the image to be processed A to obtain a color-corrected image B, that is, A*M=B.

[0054] Accordingly, in some embodiments, the original image is obtained based on the high-resolution image, and a preset color change matrix of a specified size can be used to perform a dot multiplication with the high-resolution image to implement the inverse process of color correction and obtain the original image. Still referring to the previous example, for the high-resolution image B after color correction, the inverse matrix M of the color change matrix M is selected. -1 , the high-resolution image B and the inverse matrix M -1 , we can get the image A to be processed without color correction, that is, B*M -1 =A.

[0055] The specific size of the color change matrix can be determined according to the size of the high-resolution image and the computing power of the processor. The specific value of the color change matrix can be determined based on experience, or an existing color change matrix can be used. Of course, it can also be a color change matrix designed independently by a person skilled in the art, and this application does not limit this.

[0056] Since the high-resolution images collected by the sensor and processed by the image signal usually need to be presented to people on the display device. However, the contrast of the display device is usually low, which is different from the response of the human eye to color. In order to make the displayed image effect as close as possible to the effect people see in the real world. In the image signal processing process of generating a high-resolution image from the original image collected by the sensor, tone mapping can also be included, mapping the pixel values ​​in the original image according to a preset mapping relationship to obtain new pixel values.

[0057] Combination Figure 5 Provide explanation. Figure 5 The left image in the figure is an image without tone mapping. It can be seen that some important information in the original image, such as global and local contrast and important details, is lost. The image effect is quite different from what people see in the real world. Figure 5 Each pixel value in the left image is mapped to another new pixel value to adjust the contrast of the original image and make the details more clearly visible. Figure 5 The image shown in the right figure, this process is tone mapping. In some embodiments, for the image to be processed with a pixel value range of 0 to 255, a mapping table can be preset, and the high-resolution image after tone mapping processing can be obtained by looking up the new pixel value corresponding to each pixel value in the mapping table.

[0058] Then, accordingly, in some embodiments, the original image is obtained based on the high-resolution image, and the inverse process of color mapping can be performed on the high-resolution image based on a preset mapping relationship to obtain the original image. Taking the mapping relationship as an example, the inverse process of color mapping the high-resolution image is to find the original pixel value corresponding to each pixel value in the high-resolution image in the mapping table. Of course, those skilled in the art should understand that the above-mentioned embodiment of the preset mapping relationship in the form of a mapping table is only an exemplary description. The mapping relationship can also be implemented by other algorithms, and this application does not limit this.

[0059] In the method for acquiring training data described in the present application, step S102, converting the high-resolution image into the original image captured by the sensor, can be achieved only through de-mosaicing processing, color correction, or the inverse process of tone mapping in the image signal processing process, or can be achieved through multiple of the inverse processes of de-mosaicing, color correction, and tone mapping, and the present application does not impose any restrictions on this.

[0060] In addition, those skilled in the art should understand that the image signal processing process in the generation process of the high-resolution image may include not only de-mosaicing, color correction and tone mapping, but also lens correction, gamma correction, white balance processing, etc. Then, converting the high-resolution image into the original image captured by the sensor can also be achieved through the inverse process of the above-mentioned lens correction, gamma correction, white balance processing, etc., and the present application does not impose any restrictions on this.

[0061] Through the above embodiments, the original image collected by the corresponding sensor can be obtained based on the high-resolution image. In the training data acquisition method described in the present application, step S103 superimposes noise that satisfies a specified distribution on the original image to obtain an intermediate image. In some embodiments, the noise superimposed on the original image includes the shot noise of the sensor and / or the read noise of the sensor.

[0062] When using a sensor to capture and image objects in the real world, there is often noise in the original image obtained. The noise mainly comes from the noise of the sensor. The noise of the sensor mainly comes from the shot noise (also known as shot noise) caused by the incoming light shooting and the read noise (also known as read noise) caused by the inaccurate reading of the circuit. Among them, the shot noise comes from the accumulation of light and can be expressed as a Poisson distribution. The read noise can be expressed as a Gaussian distribution with a fixed variance.

[0063] In some embodiments, the shot noise of the sensor obeys a uniform distribution, and the mathematical expression of the uniform distribution is:

[0064] log(λ shot )~U(a=log(0.0001),b=log(0.012)) (1)

[0065] Among them, λ shot is the shot noise of the sensor, U represents uniform distribution, a represents the minimum value of uniform distribution, and b represents the maximum value of uniform distribution.

[0066] In some embodiments, the read noise of the sensor obeys a Gaussian distribution, and the expression of the Gaussian distribution is:

[0067] log(λ read )~N(μ=2.18log(λ shot )+1.2,σ=0.26) (2)

[0068] Among them, λ shot is the shot noise of the sensor, λ read is the sensor’s read noise, N represents Gaussian distribution, μ represents mean, and σ represents variance.

[0069] It can be seen from the above formula that the read noise of the sensor is related to the shot noise of the sensor. The shot noise of the sensor can be determined by formula (1) or by other distribution formulas, which is not limited in this application.

[0070] Those skilled in the art should understand that the specific expressions of the shot noise and read noise of the above sensor are only exemplary. The shot noise and read noise of the sensor can also be noise expressed by other expressions, which is not limited in this application.

[0071] According to the above embodiment, it can be seen that when the noise of the sensor mainly comes from shot noise and read noise, then, in step S103, noise satisfying a specified distribution is superimposed on the original image to obtain an intermediate image, and the pixel distribution intensity of the intermediate image can be expressed as a Gaussian distribution whose variance is related to the shot noise and the read noise:

[0072] y~N(μ=x,σ 2 =λ read +λ shot x) (3)

[0073] Wherein, x is the true pixel intensity of the original image without the introduction of noise, and y is the pixel value of the original image with the introduction of noise.

[0074] Based on the above embodiments, it can be seen that first converting the high resolution into the original image collected by the sensor and then superimposing the noise that satisfies the specified distribution on the original image can more realistically simulate the impact of noise on full-color image generation in the real world.

[0075] Those skilled in the art should understand that when noise satisfying a specified distribution is superimposed on the original image to obtain an intermediate image, the noise satisfying the specified distribution may be other noises, such as thermal noise of the sensor, in addition to shot noise and read noise of the sensor, and the present application does not impose any limitation on this.

[0076] In order to more accurately simulate the generation process of low-resolution images in the real world, in the training data acquisition method described in the present application, step S103, converting the intermediate image into a low-resolution image, can be achieved in the following way: performing a specified image signal processing transformation on the intermediate image to obtain a first low-resolution image; wherein the specified image signal processing transformation includes at least one of the following processing methods: de-mosaicing, color correction and tone mapping.

[0077] Based on the above-mentioned steps S101 and S102, the original image with noise of specified distribution superimposed can be obtained, which is equivalent to obtaining Figure 2Then, the original image superimposed with the noise of the specified distribution is subjected to the A / D conversion 204. Figure 2 The image signal processing transformation 205 shown obtains a first low-resolution image, which can simulate the generation of a low-resolution image in a real scene.

[0078] The specified image signal processing transformation includes at least one of the following processing methods: de-mosaicing, color correction and tone mapping. De-mosaicing, color correction and tone mapping in the image processing transformation process have been described in detail above and will not be repeated here.

[0079] Through the above embodiments, it can be seen that by converting a high-resolution image into an original image captured by a sensor, superimposing noise that satisfies a specified distribution on the original image, and processing it through a specified image signal, a low-resolution image is obtained. Because it is based on the principle of image imaging, it simulates the process of a user using a terminal device to obtain an image in the real world. Therefore, the low-resolution image generated based on the method described in this application is closer to the low-resolution image generated in a real scene.

[0080] For application scenarios in which images are acquired using an image acquisition device, the low-resolution image obtained may be blurred due to relative motion between the shooting device and the object, such as the object moving, the shooting device shaking, and the lack of focus. In order to more accurately simulate the generation of low-resolution images, in some embodiments, step S103, converting the intermediate image into a low-resolution image, may also include: convolving the first low-resolution image with a specified blur kernel to obtain a second low-resolution image. The first low-resolution image is the first low-resolution image obtained by performing specified image signal processing after superimposing noise that satisfies a specified distribution on the original image as described above.

[0081] Combination Figure 6 For explanation. Figure 6 , 601 is the first low-resolution image (expressed in the form of pixel values) mentioned above, 602 is the blur kernel expressed in the form of a matrix, and 603 is the second low-resolution image obtained by convolving the first low-resolution image 601 with the blur kernel 602. Based on the physical meaning of convolution, it can be seen that convolving the blur kernel with the first low-resolution image is essentially to superimpose the information of the neighborhood of each pixel on the pixel, and the ideal imaging process is that each pixel point uniquely corresponds to the information of one object point. Therefore, convolving the blur kernel simulates the effect of superimposing the information of a certain object point on multiple pixels due to relative motion or lack of focus in a real shooting scene.

[0082] The blur kernel represented in matrix form has no restriction on the size of the matrix in this application. When the size of the blur kernel is larger, the number of neighboring pixels superimposed on each pixel after the convolution operation is more, and the blur effect of the obtained second resolution is more obvious. However, the larger the size of the blur kernel, the greater the amount of computation required for the convolution operation, and the higher the requirement for the computing power of the processor. Therefore, the size of the blur kernel can be selected by weighing the blur effect and the amount of computation.

[0083] The blur kernel can be determined based on random motion or other methods, such as a blur kernel set based on experience, a blur kernel extracted from a low-resolution image based on deep learning, etc. This application does not impose any restrictions on this.

[0084] There are multiple options for the blur kernel used to obtain the second low-resolution image. In some embodiments, the blur kernel can be determined based on a trajectory function of a Markov random process.

[0085] The so-called Markov random process means that a particle has an initial velocity, and its velocity at the next moment is only related to the current moment, and the velocity change at the next moment can be determined by a specific random distribution. Figure 7 As shown in , it is the random motion trajectory of a particle moving in a Markov random process within a certain period of time (represented by gray continuous line segments). Figure 7 As shown, by performing sub-pixel interpolation on the motion trajectory of the particle (ie, the pixel blocks around the motion trajectory), a matrix of a certain size can be randomly extracted therefrom to obtain a blur kernel of a corresponding size.

[0086] Of course, those skilled in the art should understand that in addition to determining the blur kernel based on the trajectory function of the Markov random process, the blur kernel can also be determined based on functions corresponding to other random processes, such as Brownian motion, Poisson process, etc. The present application does not limit the specific random process used to determine the blur kernel.

[0087] Through the above embodiments, it can be seen that by converting a high-resolution image into an original image collected by a sensor, superimposing noise that satisfies a specified distribution on the original image, and converting it into a low-resolution image through specified image signal processing and blur kernel convolution processing, the training data is determined. Because it is based on the principle of image imaging and simulates the process of users using terminal devices to obtain images in the real world, it not only considers the influence of sensor noise and image signal processing, but also considers the influence of blur factors such as relative motion and lack of focus. Therefore, the low-resolution image generated based on the method described in this application is closer to the low-resolution image generated in the real scene.

[0088] In the live broadcast scenario, the anchor end obtains the images constituting the video through a terminal device with an image acquisition device, and transmits it to the user end through a communication method such as a wireless network or a wired network. In this process, the formation of the low-resolution image is also affected by image encoding. Therefore, in order to simulate the formation of low-resolution images in this scenario, in some embodiments, in step S104 of the method for acquiring training data described in the present application, the intermediate image is converted into a low-resolution image, and further includes: performing a specified encoding process on the second resolution image to obtain a third resolution image, wherein the specified encoding process has a preset compression rate.

[0089] In the transmission process of images or video frames, in order to save communication bandwidth and increase transmission speed, the images or video frames are usually encoded to obtain images or video frames with a certain compression rate. Most encoding processes are lossy compression processes, such as JPEG compression encoding and the like. In order to simulate the decrease in image resolution caused by the encoding process in real scenes, for the second resolution image obtained in the foregoing, the specified encoding process used in the real scene can also be performed. The encoding process can refer to the relevant technical implementation to achieve the resolution loss caused by encoding in the real live scene, and this application will not go into details. The specified encoding process performed can have a preset compression rate. In addition, a preset bit stream, quantization parameter (Quantization Parameter, QP value), etc. can also be set, and this application does not limit this.

[0090] Through the above embodiments, it can be seen that by converting a high-resolution image into an original image collected by a sensor, superimposing noise that satisfies a specified distribution on the original image, and converting it into a low-resolution image through specified image signal processing, blur kernel convolution, and encoding processing, the training data is determined. Because it is based on the principle of image imaging and simulates the process of users using terminal devices to obtain images in the real world, in addition to the above-mentioned other factors that cause image quality degradation, the encoding loss of the image during transmission is also considered. Therefore, the low-resolution image generated based on the method described in this application is closer to the low-resolution image generated in a real scene.

[0091] Those skilled in the art should understand that it is also possible to first introduce coding loss to the high-resolution image, then convert the high-resolution image with the coding loss into the original image, and then superimpose noise that satisfies the specified distribution, perform image signal processing, and then perform convolution operation with the blur kernel to obtain a low-resolution image. The obtained low-resolution image is also closer to the low-resolution image obtained in the real scene, and can be used for training image quality enhancement models, image quality restoration models, etc. based on deep learning networks.

[0092] Based on the method for acquiring training data described above in this application, a low-resolution image that is closer to the low-resolution image generated in the real scene can be acquired. The low-resolution image obtained based on the above-mentioned method for acquiring training data and the high-resolution image corresponding to the generated low-resolution image can constitute a pair of training data for determining the parameters of the image quality enhancement model based on the deep learning network.

[0093] Based on the training data acquisition method described in the present application, the present application also provides an image quality enhancement model, which is determined based on the training data obtained by the method described above and is used to enhance the image quality of the input image, such as Figure 8 As shown, the image quality enhancement model includes: an encoding end 801 and a decoding end 802.

[0094] Among them, the encoding end 801 includes an input convolution layer and a specified number of first layers 803, the first layers 803 are connected in series in sequence, and each first layer 803 includes a downsampling layer of a preset size and a first number of residual modules; the decoding end 802 includes a specified number of second layers 804 and an output convolution layer, the second layers 804 are connected in series in sequence, and each second layer includes an upsampling layer of a preset size and a second number of residual modules; the residual module of the last first layer 803 of the encoding end 801 is connected to the residual module of the first second layer 804 of the decoding end 802, and the second number is greater than the first number, that is, the number of residual modules in the second layer 804 of the decoding end is greater than the residual module in the first layer 803 of the encoding end.

[0095] In the related art, the deep learning network used for noise removal adopts U-Net network or improved U-Net network, which usually adopts a symmetrical structure, that is, at the encoding end of the network, a specified number of convolutional layers and downsampling layers are adopted, and correspondingly, at the decoding end of the network, the same number of upsampling layers and convolutional layers are adopted. In this application, in order to obtain a deep learning network for image quality enhancement with better performance, the following is constructed: Figure 8 The image quality enhancement model shown in the figure adopts an asymmetric structure, and a residual module is added after the downsampling layer of the encoding end and before the upsampling layer of the decoding end for further extraction and recovery of high-level features. In addition, since the decoding end is used to restore the image, the number of residual modules at the decoding end is greater than the number of residual modules at the encoding end, so as to restore the features of the image at a deeper level. The specific structure of the residual module can refer to the relevant technology, and this application will not elaborate on it.

[0096] Based on the above embodiment, Figure 8The image quality enhancement model shown uses the low-resolution image obtained based on the method for acquiring training data described above and its corresponding high-resolution image as a pair of training data, that is, the low-resolution image is the input data of the image quality enhancement model, and the high-resolution image is the label corresponding to the input data. Based on multiple pairs of training data, the image quality enhancement model is trained until the pre-constructed loss function meets the preset training conditions, and then a trained image quality enhancement model is obtained. Among them, the pre-constructed loss function can be a loss function for measuring the similarity, absolute distance, square distance, etc. between the input of the deep learning network and the label (i.e., high-resolution image), and this application does not limit this. Using the trained image quality enhancement model, the low-resolution image to be enhanced and without a label is input into the trained image quality enhancement model, and a high-resolution image with enhanced image quality can be obtained.

[0097] It can be seen from the above embodiments that the present application obtains a low-resolution image based on the training data acquisition method described above, and uses the corresponding high-resolution image as a pair of training data to train the image quality enhancement model constructed by the present application. Since the constructed image quality enhancement model adopts a deep learning network with an asymmetric structure, and the number of residual modules at the decoding end is greater than the number of residual modules at the encoding end, it is possible to restore the features of the low-resolution image at a deeper level, thereby obtaining an image quality enhancement model with better performance.

[0098] In some embodiments, the decoding end of the image quality enhancement model also includes a feature series layer, which is connected after the last second-level upsampling layer of the encoding end, and is used to perform feature fusion on the input of the input convolution layer and the input result of the input convolution layer with the output result of the last second-level upsampling layer; and\or, the feature series layer is connected after other non-last second-level upsampling layers of the encoding end, and is used to perform feature fusion on the output result of the residual module of the encoding end with the output result of the corresponding second-level upsampling layer.

[0099] like Fig. 9 As shown in FIG. 1 , an exemplary image quality enhancement model described in the present application is provided, wherein the encoding end of the image quality enhancement model includes two first levels 803, and the decoding end includes two second levels 804. Fig. 9It can be seen that a feature series layer is added to the two second levels 804 of the encoding end, and the feature series layer is located after the downsampling layer of each second level 804. The last feature series layer of the decoding end performs feature fusion on the input data, the output result of the input convolution layer, and the upsampling output result of the previous layer. Due to the fusion of the three, the image quality enhancement model can learn more information about the input data and the encoding end, and thus can obtain better training results. In addition, the non-last feature series layer of the decoding end, such as Fig. 9 The first feature concatenation layer at the decoding end shown in the figure fuses the features of the residual module of the corresponding size with the upsampling result of the previous layer of the feature concatenation layer. Similarly, more features of the input data and the encoding end can be learned, which is conducive to obtaining better training results.

[0100] It can be seen from the above embodiments that a feature series layer is further introduced into the image quality enhancement model described in the present application, and the features extracted from the input data and / or the encoding end are fused with the output results of the upsampling at the decoding end, so that the image quality enhancement model can learn more information about the input data and the encoding end, and obtain an image quality enhancement model with better performance.

[0101] In some embodiments, the image quality enhancement model, at its decoding end, may further include an upsampling layer of a specified size after the output convolution layer, for upsampling the output result of the output convolution layer to further obtain a super-resolution image.

[0102] Still combined Fig. 9 To explain, we can see that in Fig. 9 In the above example, the number of upsampling layers at the decoding end of the image quality enhancement model is the same as the number of downsampling layers at the encoding end. Therefore, Fig. 9 The trained image quality enhancement model can restore the image quality of the input low-resolution image. Fig. 9 After the output convolution layer of the image quality enhancement model shown, an upsampling layer is added, and based on the training data obtained by the method for obtaining training data described in the present application, the image quality enhancement model with an upsampling layer added after the output convolution layer is trained until the preset loss function meets the training conditions. The trained image quality enhancement model can further improve the resolution of the low-resolution image to be processed, that is, obtain a super-resolution image corresponding to the low-resolution image to be processed.

[0103] It can be seen from the above embodiments that when the image quality enhancement model adopts an asymmetric structure, that is, the number of residual modules at the decoding end is greater than the number of residual modules at the encoding end, and a feature series layer is connected, the input data and / or the features of the encoding end are fused to the decoding end, and an upsampling layer is added to the output convolution layer of the decoding end, it is possible to obtain a super-resolution image for the low-resolution image to be processed that is input into the trained image quality enhancement model.

[0104] When the image quality enhancement models described in the foregoing text of this application are applied in application scenarios where the generation of image frames such as live broadcasts is highly real-time, the number of first layers at the encoding end and the number of second layers at the decoding end of the image quality enhancement model can be respectively lower than the first threshold and the second threshold, that is, the number of layers of the deep learning network of the image quality enhancement model is limited, so that the image quality enhancement model is a lightweight deep learning network that can complete training faster and obtain output data faster when applied.

[0105] Based on the training data obtained in the above-mentioned embodiment of the present application, the above-mentioned image quality enhancement model is trained to obtain a trained image quality enhancement model. The trained image quality enhancement model is pre-installed on the server side of the live broadcast manufacturer, and the image quality of the image pushed from the anchor end is enhanced based on the installed image quality enhancement model, so that a high-resolution image with a resolution greater than a preset threshold can be obtained, and forwarded to the user end, so that the user end receives a high-resolution image with a resolution greater than the preset threshold. Alternatively, the trained image quality enhancement model is pre-installed on the user end, and when the user receives the image pushed from the anchor end, the image quality is enhanced based on the installed image quality enhancement model, and a high-resolution image with a resolution greater than the preset threshold can also be obtained. Of course, the image quality enhancement model can also be installed on a third server to enhance the image quality of the image pushed from the anchor end, so that the user end receives a high-resolution image with a resolution greater than the preset threshold. Of course, those skilled in the art should understand that the high-resolution image can be used as an image frame to constitute a high-definition video, that is, based on the image quality enhancement model, the user end can eventually obtain a high-definition video with a resolution higher than the preset threshold.

[0106] Therefore, corresponding to the image quality enhancement model described in the present application, the present application also provides an image quality enhancement method based on the image quality enhancement model, the method comprising: obtaining an image to be processed from a host terminal; inputting the image to be processed into the image quality enhancement model to obtain a high-resolution image with a resolution greater than a first threshold;

[0107] Among them, the image to be processed undergoes encoding processing and decoding processing in the image quality enhancement model in sequence, the encoding processing includes input convolution processing and a specified number of first processings in sequence, each first processing includes a downsampling process and a first number of feature extractions, the decoding processing includes a specified number of second processings and output convolution processing, each second processing includes an upsampling process and a second number of feature recovery, and the second number is greater than the first number.

[0108] The training data obtained by the method for acquiring training data described above is used to train the image quality enhancement model described above based on a preset loss function to obtain a trained image quality enhancement model. Since a large amount of data has been trained, the image to be processed obtained from the anchor end is input into the image quality enhancement model for processing including encoding and decoding. The image quality enhancement model can automatically learn the characteristics of the input data and then generate a corresponding high-resolution image. The trained image quality enhancement model can be set on a terminal device, on a server of a live broadcast application manufacturer, or on a third-party computing device. Accordingly, the image quality enhancement method can be executed directly by a terminal device, on a server of a live broadcast application manufacturer, or on a third-party computing device. This application does not impose any restrictions on this.

[0109] Corresponding to the image quality enhancement model, in some embodiments, after the last upsampling process of the decoding process, the method may further include: performing feature fusion on the input of the input convolution process and the output result of the output convolution process with the output result of the last upsampling process;

[0110] and\or,

[0111] After the other non-last upsampling process of the decoding process, the method further comprises:

[0112] The output result of the feature extraction of the encoding process is feature fused with the output result of the upsampling process of the corresponding size.

[0113] Through the above embodiments, it can be seen that the image quality enhancement model trained using the training data of the present application has the ability to generate a corresponding high-resolution image based on the input image to be processed. When the image quality enhancement model and the corresponding image quality enhancement method are applied in a real live broadcast scene, it can deal with the problem of image quality degradation caused by reasons such as poor imaging effect of the image acquisition device and relative motion of the photographed object relative to the image acquisition device, and can provide users with a better viewing experience.

[0114] Corresponding to the image quality enhancement model, in some embodiments, after the output convolution processing of the decoding process, the method may further include an upsampling process for realizing super-resolution processing of the image to be processed.

[0115] Through the above embodiments, it can be seen that the image quality enhancement model trained using the training data of the present application has the ability to generate a super-resolution image corresponding to the input image generator to be processed. When the image quality enhancement model and the corresponding image quality enhancement method are applied in a real live broadcast scene, it can deal with the problem of image quality degradation caused by reasons such as poor imaging effect of the image acquisition device and relative motion of the photographed object relative to the image acquisition device, and can provide users with a better viewing experience.

[0116] The method for acquiring training data described in the present application can simulate the formation process of low-resolution images in real scenes based on high-resolution images to obtain low-resolution images with image quality closer to that in the real world. The obtained low-resolution image and its corresponding high-resolution image can be used as a pair of training data. The training data can be used for the training of the image quality enhancement model described in the present application, and can also be used for the training of other image quality enhancement models, image quality restoration models, super-resolution models, etc. based on deep learning networks, and the present application does not impose any restrictions on this.

[0117] Corresponding to the above-mentioned method embodiments, the present application also provides an embodiment of an electronic device. The electronic device may be a server or a terminal device, etc.

[0118] See also Fig.10 , is a hardware structure diagram of an electronic device embodiment of the present application, and the electronic device may include a memory 1001 and a processor 1002. The memory 1001 may be an internal storage unit of the electronic device, such as a hard disk or memory of the device. The memory 1001 may also be an external storage device of the device, such as a plug-in hard disk equipped on the device, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. Further, the memory 1001 may also include both an internal storage unit of the device and an external storage device. The memory 1001 is used to store the computer program and other programs and data required by the device. The memory 1001 can also be used to temporarily store data that has been output or is to be output. When the program stored in the memory 1001 is executed, the processor 1002 calls the program stored in the memory 1001 to execute the methods of the aforementioned embodiments, which have been described in detail in the previous text and will not be repeated here.

[0119] Of course, those skilled in the art should understand that, generally, other hardware may be included according to the actual functions of the electronic device, such as a network interface, etc., which will not be described in detail in this application.

[0120] In an embodiment of the present application, a machine-readable storage medium is further provided, wherein the machine-readable storage medium stores a computer program. When the computer program is executed by a processor, all embodiments of the above method of the present application are implemented, which will not be described in detail here.

[0121] The machine-readable storage medium may be an internal storage unit of the device described in any of the foregoing embodiments, such as a hard disk or memory of the device. The machine-readable storage medium may also be an external storage device of the device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the machine-readable storage medium may also include both an internal storage unit of the device and an external storage device. The machine-readable storage medium is used to store the computer program and other programs and data required by the device. The machine-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0122] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0123] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0124] Those skilled in the art will readily appreciate other embodiments of the present application after considering the application and practicing the inventions claimed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary techniques in the art that are not claimed in the present application. The application and embodiments are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0125] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0126] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for obtaining training data, characterized in that: The training data is used for an image quality enhancement model, and the method comprises: Acquire high-resolution images; Converting the high-resolution image into an original image collected by a sensor, wherein the original image is a sensor array image in a specified format; Superimposing noise satisfying a specified distribution on the original image to obtain an intermediate image, wherein the noise superimposed on the original image includes shot noise of the sensor and / or read noise of the sensor; converting the intermediate image into a low-resolution image; The high-resolution image is an image with a resolution greater than a first threshold, and the low-resolution image is an image with a resolution less than a second threshold.

2. The method according to claim 1, characterized in that The high-resolution image is a high-resolution RGB image.

3. The method according to claim 1, characterized in that Converting the high-resolution image into an original image collected by a sensor, including: Performing image signal processing inverse transformation on the high-resolution image; The image signal processing inverse transform is the inverse process of the image signal processing transform performed in the process of generating the high-resolution image, and the image signal processing transform includes at least one of the following: de-mosaicing, color correction and tone mapping.

4. The method according to claim 1, characterized in that: The shot noise of the sensor obeys uniform distribution, and the mathematical expression of the uniform distribution is: log(λ shot )~U(a)log(0.0001),b)log(0.012)) and\or, The read noise of the sensor obeys Gaussian distribution, and the expression of Gaussian distribution is: log(λ read )~N(μ=2.18log(λ shot )+1.2,σ=0.26); Among them, λ shot is the shot noise of the sensor, λ read is the reading noise of the sensor, U represents uniform distribution, N represents Gaussian distribution, a represents the minimum value of uniform distribution, b represents the maximum value of uniform distribution, μ represents the mean, and σ represents the variance.

5. The method according to claim 1, characterized in that Converting the intermediate image into a low-resolution image comprises: Performing a specified image signal processing transformation on the intermediate image to obtain a first low-resolution image; The specified image signal processing transformation includes at least one of the following: de-mosaicing, color correction and tone mapping.

6. The method according to claim 5, characterized in that Converting the intermediate image into a low-resolution image further comprises: The first low-resolution image is convolved with a specified blur kernel to obtain a second low-resolution image.

7. The method according to claim 6, characterized in that The specified blur kernel is determined based on a trajectory function of a Markov random process.

8. The method according to claim 7, characterized in that Converting the intermediate image into a low-resolution image further comprises: Performing a specified encoding process on the second low-resolution image to obtain a third low-resolution image; The specified encoding process has a preset compression rate.

9. An image quality enhancement model, characterized in that: The image quality enhancement model is determined based on the training data described in any one of claims 1 to 8, and is used to achieve image quality enhancement of input data. The image quality enhancement model includes: an encoding end and a decoding end; The encoding end includes an input convolution layer and a specified number of first layers, the first layers are sequentially connected in series, and each first layer includes a downsampling layer and a first number of residual modules; The decoding end includes a specified number of second layers and an output convolution layer, the second layers are sequentially connected in series, and each second layer includes an upsampling layer and a second number of residual modules; The last first-level residual module of the encoding end is connected to the first second-level residual module of the decoding end, and the second number is greater than the first number.

10. The image quality enhancement model according to claim 9, characterized in that: The decoding end also includes a feature concatenation layer; The feature series layer is connected after the last upsampling layer at the decoding end, and is used to perform feature fusion on the input of the input convolution layer and the output result of the input convolution layer with the output result of the last upsampling layer of the second level; and\or, The feature series layer is connected after the other non-last second-level upsampling layers at the decoding end, and is used to perform feature fusion on the output result of the residual module at the encoding end and the output result of the second-level upsampling layer of the corresponding size.

11. The image quality enhancement model according to claim 9, characterized in that: The decoding end further includes an upsampling layer after the output convolution layer, which is used to upsample the output result of the output convolution layer.

12. An image quality enhancement method based on the image quality enhancement model according to claim 9, characterized in that: The method comprises: Obtain the image to be processed from the anchor end; Inputting the image to be processed into the image quality enhancement model to obtain a high-resolution image with a resolution greater than a first threshold; Among them, the image to be processed undergoes encoding processing and decoding processing in the image quality enhancement model in sequence, the encoding processing includes input convolution processing and a specified number of first processings in sequence, each first processing includes a downsampling process and a first number of feature extractions, the decoding processing includes a specified number of second processings and output convolution processing, each second processing includes an upsampling process and a second number of feature recovery, and the second number is greater than the first number.

13. The method according to claim 12, characterized in that After the last upsampling process of the decoding process, the method further comprises: Performing feature fusion on the input of the input convolution processing, the output result of the output convolution processing, and the output result of the last upsampling processing; and\or, After the other non-last upsampling process of the decoding process, the method further comprises: The output result of the feature extraction of the encoding process is feature fused with the output result of the upsampling process of the corresponding size.

14. The method according to claim 13, characterized in that After the output convolution process of the decoding process, the method also includes an upsampling process.

15. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 8 or 12 to 14 is implemented.

16. A machine-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 or 12 to 14 is implemented.

Images