A method of image processing and related apparatus

Abstract

The application discloses a kind of method and related device of image processing, applied to image processing field, for improving the quality of output image.The method comprises: first, the first image to be processed is processed to obtain the second image, and whether there is target texture in the first image is judged using a discrimination model.If the first image has target texture, then the residual image of the first image is generated using a residual image generation model, and the second image is corrected using the residual image to output the third image, thereby improving the image quality.A kind of image processing device is also provided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular, to an image processing method and related apparatus. BACKGROUND

[0002] With the development of intelligent terminal technology, intelligent terminals (such as mobile phones) integrate photography functions, and users can use the photography functions of intelligent terminals to take pictures or live broadcast at any time, which brings great convenience to people's life.

[0003] Currently, the process of digital photography processing can be as follows: first, image information is collected by an image sensor (such as a complementary metal-oxide-semiconductor (CMOS) or a charge-coupled device (CCD)) to generate a raw image. Then, the terminal pre-processes the raw image, and after image processing such as demosaicing or denoising on the pre-processed raw image, outputs the processed image. Taking demosaicing processing as an example, CMOS or CCD sensors can only record the received photon energy, that is, brightness, and cannot record the wavelength of the received light wave, that is, color information. In this case, using CMOS or CCD alone can only take black and white pictures. To address this situation, a color filter array composed of different color filters can be used to collect different color information. However, using the color filter array can only collect information of one color at one pixel, and complete color information at least requires red, blue, and green primary color information, so the collected color information is actually incomplete. Therefore, it is necessary to reconstruct the color information to obtain complete color information (i.e., demosaic).

[0004] Using a traditional image processing method (such as image interpolation) to perform image processing (such as demosaicing), since the data collected by the color filter array is incomplete, in some cases, it is easy to make errors in reconstructing all color information, and thus some image distortion phenomena can occur. SUMMARY

[0005] Embodiments of the present application provide an image processing method and related apparatus. The method is applied to an image processing apparatus. For example, the apparatus can be a mobile phone, an iPad, a camera, or other electronic devices with a photography function. The method is used to improve the quality of output images.

[0006] In a first aspect, an embodiment of the present application provides a method for image processing, which can include: first, an electronic device acquires a first image, the first image being an original image collected by an image sensor or a preprocessed original image; then, the electronic device performs image processing on the first image to obtain a second image; the electronic device uses a discrimination model to determine whether the first image contains a target texture; when the discrimination model determines that the first image contains the target texture, the electronic device uses a residual image generation model to generate a residual image of the first image; and finally, the electronic device corrects the second image using the residual image to obtain a third image. In the embodiment of the present application, the first image to be processed is processed (such as demosaicing) to obtain a processed second image. If the first image contains a target texture, the second image can have pseudo-color (such as moire, purple edge, etc.). In the present application, the discrimination model is used to determine whether the first image contains a target texture. If the first image contains a target texture, the residual image generation model is used to generate a residual image corresponding to the first image. The second image is corrected by the residual image, and the third image is output, such as an image after the pseudo-color is removed, thereby improving the image quality.

[0007] In an optional implementation, the use of the discrimination model to determine whether the first image contains a target texture can further include: the electronic device divides the first image into a plurality of sub-block images; then uses the discrimination model to determine whether the plurality of sub-block images contain a target texture; when the discrimination model determines that the first image contains a target texture, the use of the residual image generation model to generate a residual image of the first image can further include: when the discrimination model determines that a first sub-block image in the plurality of sub-block images contains a target texture, the electronic device uses the residual image generation model to generate a residual sub-block image of the first sub-block image; and the use of the residual image to correct the second image to obtain the third image can include: the electronic device uses the residual sub-block image to correct a target sub-block image in the second image corresponding to the first sub-block image to obtain a target sub-block correction image in the third image. In the embodiment, the electronic device divides the first image into a plurality of sub-block images, so that the discrimination model can directly determine which sub-block image contains a target texture, thereby positioning the target sub-block containing the target texture. In this way, the residual sub-block image generated by the residual image generation network directly corresponds to the target sub-block, and then the residual sub-block image is used to correct the target sub-block image in the second image corresponding to the sub-block image. The residual image generation model does not need to generate residual sub-block images of all sub-block images, thereby reducing the calculation amount and saving the calculation resources.

[0008] In an optional implementation, the determining whether the plurality of sub-block images contain the target texture by using the discrimination model can specifically include: the electronic device extracts image features in each of the plurality of sub-block images by using the discrimination model; and then the electronic device determines whether each of the plurality of sub-block images contains the target texture according to the image features. The generating the residual sub-block image of the first sub-block image by using the residual image generation model can specifically include: the electronic device generates the residual sub-block image of the first sub-block image by using the residual image generation model according to the image features of the first sub-block image. In this embodiment, the discrimination model first extracts image features of all sub-block images, and determines whether each of the sub-block images contains the target texture based on the extracted image features. When the discrimination model determines that the first sub-block image contains the target texture, the discrimination model outputs the image features of the first sub-block image to the residual image generation model. The discrimination model and the residual image generation model share the feature extraction part, which reduces the complexity of the model and further reduces the calculation amount.

[0009] In an optional implementation, the method can further include: when the discrimination model determines that the first image does not contain the target texture, the electronic device does not start the residual image generation model, and directly outputs the second image. Not all cases use the residual image generation model. Only when the discrimination model determines that the first image contains the target texture, the residual image generation model is started to correct the second image by using the residual image. When the first image does not contain the target texture, the second image obtained by processing the first image by using the first processing device does not have distortion, and the terminal can directly output the second image, thereby reducing the calculation overhead of the terminal.

[0010] In an optional implementation, the image processing on the first image further includes: performing image demosaicing processing on the first image.

[0011] In an optional implementation, at least one of the discrimination model and the residual image generation model is a convolutional neural network model. The discrimination model and the residual image generation model are pre-trained convolutional neural network models. The discrimination task of the discrimination model is relatively simple, and the residual image generation network is started only when the first image contains the target texture, without occupying a large amount of processing resources of the terminal. The method can guarantee the image quality and the operation amount within the range that can be borne by the electronic device (such as a mobile phone) with limited processing performance, without affecting the processing performance of the terminal. The method can be truly applied to the electronic device (such as a mobile phone) with limited processing performance.

[0012] In an optional implementation, the image demosaicing processing includes: image interpolation processing.

[0013] In an optional implementation, the discrimination model comprises an input layer, a first convolutional layer, a classifier, and a gate; the input layer is connected to the first convolutional layer, the first convolutional layer is connected to the classifier and the gate respectively, and the classifier and the gate are connected. The input layer is configured to receive an input image, the first convolutional layer is configured to extract image features of the input image, the classifier is configured to determine whether the target texture is contained in the first image according to the image features, and the gate is configured to pass or block the image features. In this example, when the classifier determines that the target texture is contained in the first image, the gate outputs the image features to the residual image generation network; when the classifier determines that the target texture is not contained in the first image, the gate does not output the image features to the residual image generation network, that is, the terminal does not start the residual image generation model, thereby saving the computing resources of the electronic device.

[0014] In a second aspect, the embodiments of the present application provide an image processing apparatus, comprising: a first processing apparatus configured to acquire a first image, perform image processing on the first image, and obtain a second image, the first image being an original image or a preprocessed original image; a discrimination model configured to determine whether the target texture is contained in the first image; a residual image generation model configured to generate a residual image of the first image when the discrimination model determines that the target texture is contained in the first image; and a correction module configured to correct the second image by using the residual image to obtain a third image.

[0015] In an optional implementation, the apparatus further comprises an image segmentation module; the image segmentation module is configured to segment the first image into a plurality of sub-block images; the discrimination model is further configured to determine whether the target texture is contained in the plurality of sub-block images; the residual image generation model is further configured to generate a residual sub-block image of a first sub-block image when the discrimination model determines that the target texture is contained in the first sub-block image; and the correction module is further configured to correct a target sub-block image in the second image corresponding to the first sub-block image by using the residual sub-block image to obtain a target sub-block correction image in the third image.

[0016] In an optional implementation, the discrimination model is further configured to extract image features in each of the plurality of sub-block images, and determine whether the target texture is contained in each of the plurality of sub-block images according to the image features; and the residual image generation model is further configured to generate the residual sub-block image of the first sub-block image according to the image features of the first sub-block image.

[0017] In an optional implementation, the first processing apparatus is further configured to perform image demosaicing processing on the first image.

[0018] In an optional implementation, at least one of the discrimination model and the residual image generation model is a convolutional neural network model.

[0019] In an optional implementation, the first processing apparatus is further configured to perform image interpolation processing.

[0020] In a third aspect, an embodiment of the present application provides an image processing apparatus, which comprises a processor and a memory coupled to the processor; the memory is configured to store instructions; and the processor is configured to execute the instructions in the memory, so that the apparatus performs the method in any of the first aspect.

[0021] In a fourth aspect, an embodiment of the present application provides a computer readable medium, which is configured to store a computer program, and when the computer program is executed on a computer, the computer program causes the computer to perform the method in any of the first aspect.

[0022] In a fifth aspect, an embodiment of the present application provides a chip, which comprises a processor and a communication interface, for example, an input / output interface, a pin or a circuit, etc. The processor is configured to read instructions to perform the method performed by the electronic device in any of the first aspect.

[0023] In a sixth aspect, an embodiment of the present application provides a computer program product, which, when executed by a computer, implements the method performed by the electronic device in any of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is a schematic diagram of a Bayer color filter array;

[0025] Figure 2 is a schematic diagram of a principle of inferring the color of a pixel from the color distribution around the pixel;

[0026] Figure 3 is a schematic diagram of a Bayer color filter array used in an example of demosaicing by image interpolation;

[0027] Figure 4 is a schematic diagram of moiré;

[0028] Figure 5 is a schematic diagram of an ideal image, a demosaiced image and a residual image in an embodiment of the present application;

[0029] Figure 6 is a schematic diagram of an apparatus for image processing in an embodiment of the present application;

[0030] Figure 7 is a schematic diagram of an embodiment of a system comprising a neural network processor in an embodiment of the present application;

[0031] Figure 8 is a schematic diagram of another apparatus for image processing in an embodiment of the present application;

[0032] Figure 9 A flowchart of an example of a method of image processing in an embodiment of the present application;

[0033] Figure 10 A flowchart of an example of a method of image processing in an embodiment of the present application;

[0034] Figure 11 A schematic diagram of an example of the structure of a discriminant model and a residual image generation model in an embodiment of the present application;

[0035] Figure 12A A schematic diagram of an example of dividing a first image into a plurality of sub-block images in an embodiment of the present application;

[0036] Figure 12B An embodiment schematic diagram of correcting a target sub-block image in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. The terms "first", "second", etc. in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the terms used in this way can be interchanged as appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or modules does not necessarily have to be limited to only those steps or modules clearly listed, but can include other steps or modules that are not clearly listed or inherent to the process, method, product or device.

[0038] In order to better understand the present application, first, the words involved in the present application are explained:

[0039] Bayer pattern filter: a mosaic color filter array formed by arranging RGB color filters on the square of light sensing components. Digital image sensors mostly use this specific arrangement of color filter array to make color images. Please refer to Figure 1 As shown in FIG. 1, taking the Bayer filter array as an example, 50% of the squares in the Bayer filter array are green, 25% of the squares are red, and the other 25% of the squares are blue, so the Bayer filter array is usually represented by the RGGB symbol, wherein R (red) represents red information, G (green) represents green information, and B (blue) represents blue information.

[0040] Demosaicing: Each pixel of the color filter array only filters and records one of the three colors of RGB, and the information obtained from a single pixel cannot fully represent the composition value of red, green and blue. In order to obtain a full-color image, the composition value of a specific pixel is estimated by using the pixels of the same color around it, so as to reconstruct all colors. The process of reconstructing all colors is called demosaicing. For example, the filter points of other colors are regularly distributed around each filter point, so the original color of the light can be determined by combining the values of the filter points. Please refer to Figure 2 For example, yellow light is a mixture of red light and green light, so after the light passes through the filter array, the pixels under the red and green points will have values, but the pixels under the blue point will have no value. Therefore, according to the color distribution around a pixel, which has red and green but no blue, it can be inferred that the original color of the pixel should be yellow.

[0041] Image interpolation: A method of image processing, which predicts the value of a pixel point according to the information of the pixel point. In general, image interpolation can be used for demosaicing. Image interpolation includes but is not limited to nearest neighbor interpolation, bilinear interpolation and high-order interpolation, etc.

[0042] In order to better understand the image interpolation algorithm, first, an image interpolation algorithm (Example A) is taken as an example for illustrative description. Please refer to Figure 3 Figure 3 The Bayer color filter array in Figure 3 has 9 positions (or squares) marked with corresponding numbers (from No. 1-9). It can be understood that in the Bayer color filter array, each pixel has only one color information, so it is necessary to restore the full color of each pixel. For example, taking No. 4 pixel as an example, the green information of No. 4 pixel can be obtained through the Bayer color filter array, which can be expressed as "G4", and the task of image interpolation is to reconstruct the red information (denoted as "R4") and the blue information (denoted as "B4") of No. 4 pixel. For example, taking No. 5 pixel as an example, the red information of No. 5 pixel can be obtained through the Bayer color filter array, which can be expressed as "R5", and the task of image interpolation is to reconstruct the green information (denoted as "G5") and the blue information (denoted as "B5") of No. 5 pixel.

[0043] For example, Figure 3 The color sensing values of the 5x5 Bayer color filter array shown in

[0044] Table 1

[0045] ​ 20 78 52 43 18 31 65 64 58 29 45 102 154 142 62 15 58 104 112 157 36 11 85 96 74

[0046] Combining Figure 3 As shown in Table 1, the pixel value "52" in the first row and the third column is a red sensing value, which can be represented as R1=52, wherein "R" represents "red", and "1" represents Figure 3 The "1st pixel" is marked in Table 1, and the sensing value is "52". Similarly, the pixel value "64" in the second row and the third column can be represented as G2=64, the pixel value "154" in the third row and the third column can be represented as R5=154, and so on. It should be noted that the values in Table 1 are only examples for the convenience of description, and do not constitute a limiting description of the present application.

[0047] The color information of the 5th pixel is taken as an example for description as follows:

[0048] First, the gradients of the red channel in the horizontal and vertical directions are calculated to interpolate the green channel, i.e., to reconstruct the green information of the 5th pixel.

[0049] a, ΔH=|G4-G6|+|R5-R3+R5-R7|

[0050] b, ΔV=|G2-G8|+|R5-R1+R5-R9|

[0051] c, if ΔH>ΔV, then

[0052] d, if ΔH<ΔV, then

[0053] e, if ΔH=ΔV, then

[0054] wherein ΔH is an auxiliary calculation parameter in the horizontal direction, and ΔV is an auxiliary calculation parameter in the vertical direction.

[0055] In the example shown in Table 3 above:

[0056] ΔH=|G4-G6|+|R5-R3+R5-R7|=|102-142|+|154-45+154-62|=40+201=241

[0057] ΔV=|G2-G8|+|R5-R1+R5-R9|=|64-104|+|154-52+154-85|=40+171=211

[0058] ΔH>ΔH, then

[0059] After the interpolation of the green channel is completed, the blue channel is interpolated in the same way, i.e. the blue information of the No. 5 pixel is reconstructed. The principle of reconstructing the blue information of the No. 5 pixel is the same as that of reconstructing the green information, which is not described herein. The matrix representation of the three channels of the image can be obtained through the above image interpolation algorithm. It should be noted that the above image interpolation is only illustrative, and there are many improved methods for image interpolation for demosaicing, which are not described herein.

[0060] Since the principle of image interpolation demosaicing is to estimate and calculate the color of the target pixel based on the sensing values of the adjacent pixels, and the data collected by the color filter array is incomplete, errors are prone to occur in the reconstruction of all color information, and some image distortion phenomena occur. Some distorted scenes are described as follows: Figure 4 As shown in FIG. 6, when the original image contains periodic textures, the demosaiced image will have special lines after image interpolation demosaicing. Such lines are called moire. For example, when the original image has textures with large color contrast between light and dark, purple edges will occur, and the main reason for the purple edges is that light is diffracted at the junction of light and dark. The shorter the wavelength, the more likely it is to diffract. The light corresponding to the short wavelength (mainly the near-ultraviolet spectrum) will be diffracted to the dark edge of the light-dark junction, and the corresponding color filter array will have the phenomenon that the "B pixel" has light response, the "G pixel" is next, and the "R pixel" has basically no response. The output of the final color is in the form of image interpolation, i.e. the color of the target pixel is determined by referring to the colors of the surrounding pixels of the target pixel. Therefore, the area covered by the diffracted light will appear in the color corresponding to the "B pixel", which will have a slight impact on the "G pixel". The greater the light contrast, the more obvious it is. At this time, the purple edge will appear at the position where the light and dark are alternated, i.e. the purple edge. In addition to the moire and purple edge described above, the image distortion scene can also have the ghosting effect (as shown in FIG. 7), the zipper effect (as shown in FIG. 8), and the near color confusion, which are not described herein. Figure 5 Figure 5

[0061] ​​The embodiment of the present application provides a kind of image processing method, the method is used to improve the quality of output image, i.e. effectively handle the pseudo color such as moire, purple edge, which is difficult to handle by traditional demosaicing algorithm.The method is applied to the device of image processing, the device includes but is not limited to smart phone, ipad, camera and other electronic equipment with photographic function.In the embodiment of the present application, still use traditional image processing method (such as image interpolation) to the first image to be processed (such as demosaicing), obtain the second image after demosaicing, but the discrimination model and residual image generation model are added in the present application, the discrimination model is used to discriminate whether there is image interpolation difficult to handle texture in the first image, if there is image interpolation difficult to handle texture in the first image, then the residual image corresponding to the first image is generated by using residual image generation model, the second image is corrected by residual image, and the third image after correction is output, i.e. the image after removing pseudo color, so as to improve the image quality.The discrimination model and residual image generation model are neural network models.

[0062] In order to better illustrate the present application, further explain other words involved in the present application:

[0063] Target texture: the texture described in the present example includes not only the texture of the surface of the object, but also the color pattern on the surface of the object.The target texture is the texture that is difficult to handle by the first image processing device (such as image interpolation processing device), or in other words, the target texture is the texture that will cause image distortion after image processing by the first image processing device.For example, the target texture is periodic texture (such as vertical stripe or horizontal stripe image on clothes), and so on.

[0064] Residual image: please refer to Figure 5 As shown, the residual image is the difference image (denoted as "Res") between the ideal image (or called "original image") (denoted as "Img1") without correction and the image (denoted as "Img2") after image processing by the first image processing device.It can be understood that the relationship between Img1, Img2 and Res is shown in formula 1: Res = Img2-Img1 (formula 1).Exemplarily, Img1, Img2 and Res can all be RGB format images, or Img1, Img2 and Res can all be CMYK format images, etc., which is not limited in particular.In the embodiment of the present application, Img1, Img2 and Res are all taken as RGB format images for example.

[0065] Discrimination model: a pre-trained convolutional neural network model, used to discriminate whether there is target texture in the input image.

[0066] Residual image generation model: a pre-trained convolutional neural network model used to generate a residual image.

[0067] Relationship between the discrimination model and the residual image generation model: the discrimination model connects the residual image generation model, and when the discrimination model determines that the first image contains the target texture, the discrimination model outputs the extracted image features to the residual image generation model, that is, when the discrimination model determines that the first image to be processed contains the target texture, the residual image generation model is started. When the discrimination model determines that the first image does not contain the target texture, the discrimination model does not output the image features, that is, when the discrimination model determines that the first image does not contain the target texture, the residual image generation model is not started.

[0068] Process of training the discrimination model: the electronic device trains the convolutional neural network model using a first training data set to obtain the discrimination model. The first training data set includes a plurality of samples, wherein each sample includes input data and a label having a mapping relationship. The input data is an image with texture, and the label is "0" or "1", "0" indicating that the input data does not contain the target texture, and "1" indicating that the input data contains the target texture. The electronic device can be a terminal or a server, and the specific implementation is not limited.

[0069] Process of training the residual image generation model: the electronic device trains the neural convolutional network using a second training data set to obtain the residual image generation model. The second training data set includes a plurality of samples, wherein each sample includes input data and a label having a mapping relationship. The input data is the image features output by the discrimination model, and the label is the residual image.

[0070] The present application provides an image processing method, which is applied to an image processing device. Please refer to Figure 6 As shown in the figure, the image processing device 1000 includes a first processing device 1001, a discrimination model 1002, a residual image generation model 1003, and a correction module 1004. Optionally, the image processing device further includes an output module 1005 and an image segmentation module 1006.

[0071] The first processing apparatus 1001 is configured to acquire a first image, perform image processing on the first image to obtain a second image, the first image being an original image or a pre-processed original image; the discrimination model 1002 is configured to determine whether the first image contains a target texture; the residual image generation model 1003 is configured to generate a residual image of the first image when the discrimination model 1002 determines that the first image contains the target texture; and the correction module 1004 is configured to correct the second image by using the residual image to obtain a third image. For example, the first processing apparatus 1001 includes, but is not limited to, an image signal processor having an image interpolation processing apparatus, a filter, a binary image processing apparatus or a histogram processing apparatus, etc. The discrimination model 1002 and the residual image generation model 1003 are neural network models, which can be embedded, integrated or run in a neural network processing unit (NPU). The correction module 1004 can be a processor or a software module executed thereon, or a combination of both, which is not limited in the embodiment. The processor includes, but is not limited to, one or more of a central processing unit (CPU), a digital signal processor (DSP) or an image signal processor.

[0072] Further, the first processing apparatus 1001 is configured to perform the following Figure 9 The method performed by the first image processing apparatus in steps 601 and 602 in the corresponding embodiment will be specifically described below Figure 9 The description in steps 601 and 602 in the corresponding embodiment will not be repeated here. The discrimination model 1002 is configured to perform the following Figure 9 The method performed by the discrimination model in step 603 in the corresponding embodiment will be specifically described below Figure 9 The description in step 603 in the corresponding embodiment will not be repeated here. The residual image generation model 1003 is configured to perform the following Figure 9 The method performed by the residual image generation model in step 604 in the corresponding embodiment will be specifically described below Figure 9 The description in step 604 in the corresponding embodiment will not be repeated here. The correction module 1004 is configured to perform the following Figure 9 Step 605 in the corresponding embodiment will be specifically described below Figure 9 The description in step 605 in the corresponding embodiment will not be repeated here.

[0073] Optionally, the output module 1005 is configured to output the third image, or output the second image. The output module 1005 includes, but is not limited to, a display module or an interface connected to a display module.

[0074] Optionally, the image segmentation module 1006 is configured to segment the first image into a plurality of sub-block images, and is further configured to segment the second image into a plurality of sub-block images, the number of sub-block images in the first image being the same as that in the second image. The image segmentation module 1006 can be a part of the image signal processor.

[0075] In one possible design, the functions of the first processing device 1001, the discrimination model 1002, the residual image generation model 1003, the correction module 1004 and the image segmentation module 1006 are implemented by one processing device, and the functions of the processing device are implemented by software, hardware or a combination thereof. Therefore, it can be understood that the above modules can be implemented by software, hardware or a combination thereof. At this time, the processing device includes a memory and a processor, wherein the memory is configured to store a computer program, and the processor is configured to read and execute the computer program stored in the memory to perform the corresponding processing and / or steps in the above method embodiments. The processor includes but is not limited to one or more of a CPU, a DSP, an image signal processor, a neural network processor and a microcontroller.

[0076] Optionally, the processing device only includes the processor. The memory for storing the computer program is located outside the processing device, and the processor is connected with the memory through a circuit / wire to read and execute the computer program stored in the memory. Optionally, the functions of the processing device are implemented by hardware. At this time, the processing device includes an input interface circuit, a logic circuit and an output interface circuit. Optionally, the processing device can be one or more chips or one or more integrated circuits.

[0077] In one possible design, the functions of the first processing device 1001, the correction module 1004 and the image segmentation module 1006 are implemented by one processor. The functions of the discrimination model 1002 and the residual image generation model 1003 are implemented by another system as shown in Figure 8 , which can be implemented by a neural network processor.

[0078] Please refer to Figure 7As shown, a brief introduction to the neural network processor 110 is provided for ease of understanding. The neural network processor 110 is mounted as a coprocessor to a host processor, which can include a CPU for example, and the host processor is used to distribute tasks. The core part of the neural network processor is the operation circuit 1103, which extracts matrix data in the memory and performs multiplication operation through the controller 1104. In some implementations, the operation circuit 1103 internally includes a plurality of processing units (PEs). In some implementations, the operation circuit 1103 is a two-dimensional systolic array. The operation circuit 1103 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1103 is a general-purpose matrix processor.

[0079] For example, it is assumed that there are an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of the matrix B from the weight memory 1102 and buffers it on each PE of the operation circuit. The operation circuit takes the matrix A data from the input memory 1101 and performs matrix operation with the matrix B to obtain a partial result or a final result of the matrix, which is saved in the accumulator 1108.

[0080] The unified memory 1106 is used to store input data and output data. The weight data is transferred to the weight memory 1102 through the direct memory access controller (DMAC) 1105. The input data is also transferred to the unified memory 1106 through the DMAC.

[0081] The BIU (Bus Interface Unit) 1110 is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer 1109.

[0082] The BIU (Bus Interface Unit) 1110 is used for the instruction fetch buffer 1109 to obtain instructions from the external memory, and is also used for the direct memory access controller 1105 to obtain original data of the input matrix A or the weight matrix B from the external memory.

[0083] The DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1106, or to transfer weight data to the weight memory 1102, or to transfer input data to the input memory 1101.

[0084] The vector computation unit 1107 is a plurality of arithmetic processing units that further process the output of the arithmetic circuit as necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, and the like. The vector computation unit 1107 is mainly used for network computation in non-convolution / FC layers in a neural network, such as Pooling (pooling), Batch Normalization (batch normalization), Local Response Normalization (local response normalization), and the like.

[0085] In some implementations, the vector computation unit 1107 can store the processed output vector to the unified buffer 1106. For example, the vector computation unit 1107 can apply a non-linear function to the output of the arithmetic circuit 1103, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 1107 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit 1103, such as for use in a subsequent layer in a neural network. The controller 1104 is connected to an instruction fetch buffer 1109 for storing instructions used by the controller 1104; the unified memory 1106, the input memory 1101, the weight memory 1102, and the instruction fetch buffer 1109 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0086] Referring to Figure 8 The image processing apparatus 1200 can be a mobile phone, a tablet computer, a monitoring lens, or the like, which has an image processing function. The image processing apparatus 1200 includes a processor 1201, a memory 1202, an input unit 1203, a display unit 1204, a camera 1205, and the like. The memory 1202 can be used to store software programs and modules, and the processor 1201 can execute various function applications and data processing of the apparatus by running the software programs and modules stored in the memory 1202. The memory 1202 can include a high-speed random access memory, and can further include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. The processor 1201 can be a processing apparatus as mentioned above, and the processor 1201 includes but is not limited to one or more of various types of processors, such as a CPU, a DSP, an image signal processor, a neural network processor, and a microcontroller as mentioned above. Figure 6 The processor 1201 can be a processing apparatus as mentioned above, and the processor 1201 includes but is not limited to one or more of various types of processors, such as a CPU, a DSP, an image signal processor, a neural network processor, and a microcontroller as mentioned above.

[0087] The input unit 1203 can be used to receive inputted digital or character information, and to generate key signal input related to user settings of the device and function control. Specifically, the input unit 1203 can include a touch panel 1231. The touch panel 1231, also known as a touch screen, can collect touch operations of a user thereon or adjacent thereto, such as operations of the user using a finger, a stylus, or any suitable object or accessory on or adjacent to the touch panel 1231.

[0088] The display unit 1204 can be used to display various image information. The display unit 1204 can include a display panel 1241, which can optionally be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. In some embodiments, the touch panel 1231 can be integrated with the display panel 1241 to realize input and output functions of the device. For example, the display unit 1204 is used to display a second image and a third image in this application. The camera 1205 is used to capture a raw image.

[0089] The method provided by the present application will be described in detail below through specific embodiments. Please refer to Figure 9 and Figure 10 It is shown that the method provided by the present application is an image processing method, and the execution subject of the method can be an electronic device, which is exemplified by a terminal (such as a mobile phone). Alternatively, the execution subject of the method can also be a processor, a chip, or a chip system in the electronic device. In this embodiment, the execution subject of the method is exemplified by a terminal (such as a mobile phone).

[0090] In step 601, the terminal acquires a first image. The first image can be a raw image, or the first image can be a raw image after preprocessing. For example, the terminal captures a raw image through an image sensor. The image sensor includes a photosensitive component and a color filter array layer, and the photosensitive component is a CMOS or CCD photosensitive component. The color filter array layer is arranged in front of the CMOS or CCD photosensitive component. In this embodiment, the arrangement of colors in the color filter array is not limited, and the color filter array can be exemplified by a Bayer color filter array. The terminal captures image information of a photographed object through the image sensor to generate a raw image. Then, the raw image can be preprocessed, which includes but is not limited to white balance gain, lens vignetting correction, etc., to obtain a preprocessed raw image. The first image processing device in the terminal receives the raw image captured by the image sensor, or the first image processing device receives the preprocessed raw image.

[0091] Step 602, the terminal performs image processing on the first image to obtain a second image. Illustratively, the terminal performs image processing on the first image by using a first image processing device (which can also be referred to as a “first processing device” for short). The first image processing device is a device for performing image processing by using a non-neural network. The first image processing device includes, but is not limited to, an image interpolation processing device, a filter, a binary image processing device, a histogram processing device, etc. That is, the first image processing device is not a device for performing image processing by using a machine learning method, for example, can be an image signal processor including the above processing functions. In this example, the first image processing device is taken as an example of an image interpolation processing device for illustration. The “image processing” in this step includes, but is not limited to, demosaicing, denoising, lens shading correction (LSP), super-resolution (reconstructing a low-resolution image into a high-resolution image), etc.

[0092] For the convenience of illustration, in this example, the first image processing device is taken as an example of an image interpolation processing device, and the “image processing” is taken as an example of demosaicing. That is, the terminal performs demosaicing on the first image by using the image interpolation processing device to obtain a second image after demosaicing. In this example, the specific image interpolation algorithm is not limited, for example, the image interpolation algorithm can be a nearest neighbor interpolation method, a bilinear interpolation method, a high-order interpolation method, etc. The image interpolation algorithm can be the interpolation algorithm exemplified in Example A above, etc.

[0093] Step 603, the terminal determines whether the first image contains a target texture by using a discrimination model. Illustratively, the terminal inputs the first image into a second image processing device (which can also be referred to as a “second processing device” for short), and the second image processing device includes a discrimination model and a residual image generation model. The discrimination model is used to determine whether the first image contains a target texture. When the discrimination model determines that the first image contains a target texture, step 604 is performed. When the discrimination model determines that the first image does not contain a target texture, step 606 is performed. It should be noted that steps 602 and 603 are not limited in time sequence, and steps 602 and 603 can be performed synchronously.

[0094] The discrimination model and the residual image generation model are both pre-trained convolutional neural networks. Please refer to Figure 11As shown, the structure of the discriminative network is illustrated exemplarily: the discriminative model includes an input layer, a first convolutional layer, a classifier, and a gate. The input layer is connected to the first convolutional layer, which in turn is connected to both the classifier and the gate. The input layer receives the input image, the first convolutional layer extracts image features from the input image, and the classifier determines whether the first image contains a target texture based on these features. For example, the classifier might be a binary classification function (such as the sigmoid function) used to perform binary classification on the input data. The gate acts to allow or block the image features. Specifically, when the classifier determines that the first image contains a target texture, the gate outputs the image features to the residual image generation network; when the classifier determines that the first image does not contain a target texture, the gate does not output the image features to the residual image generation network, meaning the terminal does not activate the residual image generation model. It should be understood that the image features involved in this embodiment are image features summarized or extracted by the neural network model through training or learning; specific details can be found in existing neural network technologies.

[0095] For example, the input layer receives input data, which is then fed into the first convolutional layer ( Figure 11 In the example, the first convolutional layer (comprising two convolutional layers) is used to extract image features from the input image. These image features then simultaneously enter two branches: the data from the first branch goes to the classifier, and the data from the second branch goes to the gate. The classifier is connected to a gate, meaning its output goes to the gate. The gate determines whether to allow or block the image features based on the received output. For example, when the extracted image features enter the classifier, a "0" output indicates that the input image does not contain the target texture. When the gate receives a "0", it blocks the image features from the first convolutional layer; that is, the image features are not output to the residual image generation model. In other words, when the gate receives a "0", it does not output image features, and therefore the residual image generation model is not activated. When the classifier outputs a "1", it indicates that the input data contains the target texture. When the gate receives a "1", it allows the image features from the first convolutional layer. In this example, the discriminant model is used to extract image features from the first image. The discriminant model and the residual image generation model share the feature extraction part. That is, the discriminant model extracts the image features of the first image. When the first image contains target texture, the discriminant model directly outputs the extracted image features to the residual image generation model. The residual image generation model does not need to repeatedly extract the image features of the first image. Although the deeper the neural network, the better the performance can be, and the residual image generated by the residual image generation model is more accurate, the residual image generation model alone consumes a lot of computational resources and is not economical. The two models share the feature extraction part (i.e., the first convolutional layer), which can greatly reduce the amount of computation on the terminal.

[0096] Optionally, the discriminative model further comprises a first down-convolution layer and a second down-convolution layer. It can be understood that the first down-convolution layer and the second down-convolution layer each comprise a down-sampling layer and a convolution layer, and the first down-convolution layer and the second down-convolution layer are used to reduce the image size, thereby reducing the calculation amount of the terminal device. The image features output from the first convolution layer enter the first down-convolution layer and the second down-convolution layer respectively, the image features output from the first down-convolution layer are input to the classifier, and the image features output from the second down-convolution layer are input to the gate.

[0097] Optionally, the terminal divides the first image into a plurality of sub-block images, and then inputs the plurality of sub-block images to the discriminative model to determine whether each sub-block image contains the target texture. When the discriminative model determines that the target block image contains the target texture, the discriminative model outputs the image features of the target block image to the residual image generation model. When the discriminative model determines that none of the plurality of sub-block images contains the target texture, the terminal will not start the residual image generation model, thereby further reducing the calculation amount. In addition, the discriminative model first extracts image features of all sub-block images, and determines whether each sub-block image contains the target texture based on the extracted image features. When the discriminative model determines that the first sub-block image contains the target texture, the image features of the first sub-block image are output to the residual image generation model. The feature extraction part of the discriminative model and the residual image generation model are shared, thereby reducing the complexity of the model and further reducing the calculation amount. In addition, in the present example, the terminal divides the first image into a plurality of sub-block images, so that the discriminative model can directly determine which sub-block image contains the target texture, thereby positioning the target sub-block containing the target texture in the second image. In the above embodiment, the result of image processing of the first image is the second image. If the first image is divided into a plurality of sub-blocks, and the first sub-block contains the target texture, then the result of image processing corresponding to the first sub-block is a target sub-block in the second image, i.e., the target sub-block contains the target texture. In this way, the residual image generated by the residual image generation network directly corresponds to the target sub-block, and then the residual sub-block image of the sub-block image is used to correct the target sub-block image in the second image corresponding to the sub-block image, so that the residual image generation model does not need to generate residual sub-block images of all sub-block images, thereby reducing the calculation amount and saving the calculation resources.

[0098] For example, please refer to Figure 12AAs shown, in one application scenario, the terminal divides the first image into 9 sub-block images, in order to distinguish the 9 sub-block images, the 9 sub-block images can be respectively denoted as "sub-block image a" - "sub-block image i", the terminal inputs the 9 sub-block images into the discrimination model, when the discrimination model judges that there is a target texture in "sub-block image e", the discrimination model outputs the image feature of "sub-block image e" (denoted as "image feature e").

[0099] Step 604, when the discrimination model judges that the first image contains a target texture, the terminal generates a residual image of the first image by using the residual image generation model. The first convolutional layer in the discrimination model extracts the image feature of the first image, and then the classifier in the discrimination model judges whether the first image contains a target texture according to the image feature. When the discrimination model judges that the first image contains a target texture, the image feature is output to the residual image generation model. Please refer to Figure 11 As shown, the residual image generation model includes a deconvolutional layer (such as a convolutional kernel size of 3x3) and a convolutional layer (such as a convolutional kernel size of 3x3). Among them, the deconvolutional layer is used to increase the size of the image feature, so that the size of the residual image output by the residual image generation model is the same as the size of the input image input into the discrimination model. It should be noted that the structure of the residual image generation model is only an example for convenience of explanation, and does not constitute a limiting description of the present application. The residual image generation model is a fully convolutional neural network structure, which is not limited to the structure, such as the number of convolutional layers and the size of the convolutional kernel in the present example.

[0100] For example, the discrimination model outputs the image feature of "sub-block image e" (denoted as "image feature e") to the residual image generation model, and the image residual generation model receives the input "image feature e" and outputs "residual sub-block image e". Please refer to Figure 11 As shown, the process of generating a residual image is exemplarily illustrated for steps 603 and 604. For example, the size of an image is represented by h x w x c, where h represents the height, w represents the width, and c represents the number of channels. The size of the input image (such as a sub-block image) is taken as an example of h x w x 1, the size of the input image is h x w, and the number of channels is 1. Then the terminal rearranges the input image, and the size of the image becomes (the number of channels changes from 1 to 4, and the 4 channels are GRGB), and the size of the image is After the image of A tensor representing the size. For example, the image size is calculated according to convolution formula 2: Equation (2) is used, where i is the size of the input image, k is the size of the convolution kernel, s is the stride, and p is the padding. For example, if the input image size is 4×4, the convolution kernel size is 3×3, the stride is strides = 2, and the padding is padding = 0, that is, i = 4, k = 3, s = 1, p = 0, the discriminant model outputs an image of size 2×2. The "16" (i.e., number of channels) in the equation is determined by the number of convolution kernels, with one channel corresponding to one convolution kernel. It should be noted that in this example, the specific values ​​of each parameter in the above equation (2) are not limited, nor is the number of convolution kernels limited. The specific values ​​of each parameter are merely examples for the sake of illustration and do not constitute a limiting statement of this application.

[0101] The above The tensor of size h×w×16 is passed through the deconvolution layer of the residual image generation model, outputting a tensor of size h×w×16. This means the size of the image output by the discriminator model is smaller than the input image size, while the deconvolution layer in the residual image generation model increases the image size, thus making the output image size the same as the input image size. Then, the h×w×16 tensor is passed through a convolutional layer to obtain a residual image of size h×w×3. It should be noted that the size of the residual image generated by the residual image generation model (i.e., h×w) is the same as the size of the input image to the discriminator model, and the number of channels in the residual image (e.g., RGB, 3 channels) is the same as the number of channels in the second image (e.g., RGB, 3 channels), thus allowing the residual image to be used to correct the second image.

[0102] Step 605: The terminal uses the residual image to correct the second image, obtaining the third image. In this example, the residual image is the difference between the ideal image and the de-mosaiced image. According to Equation 1 above (Res = Img2 - Img1), the method for correcting the second image using the residual image is: subtract the second image (Img2) from the residual image to obtain the corrected third image. That is, Img1 = Img2 - Res. Here, Img1 is the third image (which can also be understood as the output ideal image), Res is the residual image, and Img1 is the de-mosaiced image (i.e., the second image).

[0103] Optionally, the residual sub-block image of the first sub-block image and the target sub-block image at the corresponding position of the first sub-block image in the second image are corrected to obtain the corrected target sub-block image in the third image, and then the third image is output. The correction of the second image (or the target sub-block image in the second image) is illustrated by example. Please refer again to... Figure 5and the following matrix (formula 3), generally, the image is represented in RGB format, for example, a picture with a size of 5x5 can be represented as a tensor with a size of 5x5x3. Taking one of the RGB channels (for example, the R channel) as an example, the matrix calculation for correcting the second image by the residual image is as follows:

[0104]

[0105] In the above formula (3), the first matrix corresponds to the second image, the second matrix corresponds to the residual image, and the third matrix corresponds to the third image. Alternatively, the first matrix corresponds to the target sub-block image in the second image, the second matrix corresponds to the residual sub-block image, and the third matrix corresponds to the target sub-block correction image. For example, the second matrix in the above formula (3) is used to represent the R channel of the residual image, and the residual image of the RGB three channels can be represented as the following matrix (4).

[0106]

[0107] It should be noted that in the example shown in the above formula (3), only the R channel is taken as an example for correction calculation. In fact, for matrix calculation, matrix calculation as shown in formula (3) also needs to be performed on the other two channels (i.e., the GB channel). The calculation of the other two channels is not exemplified here.

[0108] In one application scenario, please refer to Figure 12BAs shown, the terminal divides the first image into 9 sub-block images, which are respectively denoted as "sub-block image a" - "sub-block image i", and inputs the 9 sub-block images into the discrimination model. When the discrimination model determines that there is a target texture in the "sub-block image e", the discrimination model outputs the image feature of the "sub-block image e" (denoted as "image feature e") as the output result of the neural network discrimination model. The residual image generation model receives the image feature e, and generates the residual sub-block image of the "sub-block image e" (denoted as "residual sub-block image e") according to the "image feature e". The terminal corresponds the "sub-block image e" to the corresponding position in the second image. The second image also includes a plurality of corresponding sub-block images, and the sub-block image at the position e is the target sub-block image. It can be understood that the processing result of the first image is the second image, and therefore the number of sub-block images in the first image is the same as the number of sub-block images in the second image. The second labels of the plurality of sub-block images in the second image and the first labels of the plurality of sub-images in the first image have a mapping relationship, so that the sub-block images in the first image can be corresponded to the sub-block images in the second image through the first labels and the second labels. Further, the terminal corrects the target sub-block image (denoted as "target sub-block image e") corresponding to the "sub-block image e" in the second image by using the "residual sub-block image e", to obtain a target sub-block correction image (denoted as "target sub-block correction image e"), and finally the terminal outputs a third image.

[0109] Step 606, when the discrimination model determines that the first image does not contain the target texture, the terminal does not start the residual image generation model, and outputs the second image. When the discrimination model determines that the first image does not contain the target texture, that is, the classifier in the discrimination model outputs "0" to the gate, when the gate receives "0", the image feature extracted by the discrimination model will not be output to the residual image generation model, the residual image generation model is not started, and the terminal directly outputs the second image obtained after the first processing device processes the first image. It can be understood that the residual image generation model does not work in all cases. Only when the discrimination model determines that the first image contains the target texture, the residual image generation model is started. When the first image does not contain the target texture, the second image obtained by the terminal through the first processing device processing the first image will not have distortion phenomenon, so the terminal can directly output the second image, thereby reducing the computing overhead of the terminal.

[0110] In the embodiment of the present application, first, the first image is processed by the first image processing device to obtain the second image. For example, the first image is demosaiced based on the image interpolation processing device. It can be understood that in most scenarios, the first image will not be distorted when it is demosaiced by the image interpolation method. When the first image includes target texture, the second image will be distorted. Therefore, in the embodiment of the present application, the discrimination model is used to determine whether the first image includes target texture. When the first image includes target texture, the residual image generation model is started, the residual image is generated by using the residual image generation model, and then the second image is corrected by using the residual image to obtain the third image, thereby improving the image quality of the second image. Moreover, the first image processing device is a non-neural network processing device. In the image processing method, although the image quality of the image processing method based on the convolutional neural network is high, the calculation amount of the image processing method using the neural network is much larger than that of the image processing method using the non-neural network (such as image interpolation). For example, the current convolutional neural network is used for image processing in the computer vision field. When the input picture size is 420x 420, the total calculation amount is about 44.8 GFlops, that is, about 448 billion floating point operations. Such a huge amount of calculation greatly exceeds the current processor performance of handheld devices (such as mobile phones). In the embodiment of the present application, the non-neural network image processing method is mainly used for image processing, and then the neural network is used for correction. Although the discrimination model and the residual image generation model are used in the embodiment, the discrimination task is relatively simple, and the residual image generation network is started only when the first image includes target texture. This method can not only ensure the image quality, but also ensure that the calculation amount is within the range that can be tolerated by the terminal (such as a mobile phone) with limited processing performance, so as not to affect the processing performance of the intelligent terminal, thereby being truly applied to the terminal (such as a mobile phone) with limited processing performance.

[0111] The embodiment of the present application provides a computer readable medium, and the computer readable storage medium is used to store a computer program. When the computer program is run on a computer including components in the above device embodiments, the computer is caused to execute the method performed by the terminal in the above method embodiments.

[0112] The embodiment of the present application provides a chip, and the chip includes a processor and a communication interface, for example, an input / output interface, a pin, or a circuit. The processor is used to read instructions to execute the method performed by the terminal in the above method embodiments.

[0113] The embodiment of the present application provides a computer program product, which is executed by a computer including components in the above device embodiments to implement the method performed by the terminal in the above method embodiments.

[0114] The above-described embodiments are only used to illustrate the technical solutions of the present application, but not to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those ordinarily skilled in the art should understand: the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of image processing, characterized by, The method comprises the following steps: obtaining a first image, the first image being an original image or a preprocessed original image; performing image demosaicing on the first image to obtain a second image; segmenting the first image into a plurality of sub-block images; using a discrimination model to determine whether the plurality of sub-block images contain a target texture, the target texture being a texture that will cause image distortion after image processing by a first image processing device, the first image processing device being an image interpolation processing device, a filter, a binary image processing device, or a histogram processing device; when the discrimination model determines that a first sub-block image in the plurality of sub-block images contains the target texture, using a residual image generation model to generate a residual sub-block image of the first sub-block image; using the residual sub-block image to correct a target sub-block image in the second image corresponding to the first sub-block image to obtain a target sub-block corrected image in a third image.

2. The method of claim 1, wherein, The step of using the discrimination model to determine whether the plurality of sub-block images contain the target texture comprises the following steps: using the discrimination model to extract image features in each sub-block image in the plurality of sub-block images; determining whether each sub-block image contains the target texture according to the image features. The step of using the residual image generation model to generate the residual sub-block image of the first sub-block image comprises the following step: using the residual image generation model to generate the residual sub-block image of the first sub-block image according to image features of the first sub-block image.

3. The method according to any one of claims 1-2, characterized in that, At least one of the discrimination model and the residual image generation model is a convolutional neural network model.

4. The method of claim 1, wherein, The image demosaicing processing comprises image interpolation processing.

5. An apparatus for image processing, characterized by The method comprises the following steps: a first processing device is configured to obtain a first image, perform image demosaicing on the first image to obtain a second image, the first image being an original image or a preprocessed original image; an image segmentation module is configured to segment the first image into a plurality of sub-block images; a discrimination model is configured to determine whether the plurality of sub-block images contain a target texture, the target texture being a texture that will cause image distortion after image processing by a first image processing device, the first image processing device being an image interpolation processing device, a filter, a binary image processing device, or a histogram processing device; a residual image generation model is configured to generate a residual sub-block image of a first sub-block image in the plurality of sub-block images when the discrimination model determines that the first sub-block image contains the target texture; a correction module is configured to use the residual sub-block image to correct a target sub-block image in the second image corresponding to the first sub-block image to obtain a target sub-block corrected image in a third image.

6. The apparatus according to claim 5, wherein the discrimination model is specifically configured to extract image features in each sub-block image in the plurality of sub-block images, and determine whether each sub-block image contains the target texture according to the image features. ​ The residual image generation model is specifically configured to generate a residual sub-block image of the first sub-block image according to image features of the first sub-block image.

7. The apparatus of any one of claims 5-6, wherein, The at least one of the discrimination model and the residual image generation model is a convolutional neural network model.

8. The apparatus of claim 5, wherein, The first processing apparatus is specifically configured to perform image interpolation processing.

9. An apparatus for image processing, the apparatus comprising: Comprise: a processor and a memory coupled to the processor; The memory is configured to store instructions; The processor is configured to execute the instructions in the memory, so that the apparatus executes the method in any one of claims 1 to 4.

10. A computer readable medium characterized by The computer readable medium is configured to store a computer program, when the computer program runs on a computer, so that the computer executes the method in any one of claims 1 to 4.

Images