A system and method for blind denoising of images.

Abstract

This disclosure discloses a system and method for blind image denoising. The system and method may be included in a combined processing apparatus, which may further include a general interconnect interface and other processing devices. The computing device interacts with the other processing devices to jointly perform user-specified computational operations. The combined processing apparatus may also include a storage device connected to the device and other processing devices respectively, for storing data from the device and other processing devices. The solution disclosed herein can improve the operational efficiency of computations in various data processing fields, including, for example, artificial intelligence, thereby reducing the overall computational overhead and cost.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of computer vision, and more particularly, to image denoising in computer vision. BACKGROUND

[0002] Image and video denoising plays a very important role in the field of computer vision, and is also a core algorithm in ISP (image signal processing). Traditional ISP denoising algorithms in the Bayer domain or YUV domain usually use a combination of various traditional filters and adjusted parameters to achieve denoising.

[0003] Due to the lack of data sets and differences in sensors, in recent years, image and video methods based on deep learning mainly do the work of removing additive Gaussian white noise on RGB images, that is, artificial Gaussian white noise is added to clean RGB pictures, and then a neural network is used to output the denoised pictures. In deep learning video denoising, the inter-frame alignment method based on optical flow estimation is the mainstream method.

[0004] The prior art has the following defects:

[0005] The inter-frame alignment method using optical flow motion estimation has a large amount of calculation, and in the presence of noise, the error of the optical flow calculation result is too large. The existing denoising method is more based on Gaussian white noise, which cannot accurately describe the noise in actual deployment, and needs to input the intensity information of the noise, etc., and cannot realize blind denoising. In addition, the model in the prior art is too large or needs to rely on the result of optical flow, so real-time denoising cannot be achieved.

[0006] For denoising algorithms in the RGB or YUV domain, since the image has been processed by ISP upstream demosaicing algorithms, white balance algorithms, etc., the noise distribution is very complex, and it is difficult to describe it with a simple distribution compared to the Raw picture in the Bayer domain. The ground truth of the training data set is also difficult to obtain. SUMMARY

[0007] One purpose of the present disclosure is to overcome the defect that the prior art cannot perform blind denoising on images.

[0008] According to a first aspect of the present disclosure, a system for blind denoising of images is provided, comprising: a noise estimation module configured to receive a first image group comprising a plurality of first image frames, and to obtain a noise estimate of the first image group according to the first image group; a motion alignment module configured to obtain a second image group according to the first image group and the noise estimate, the second image group containing motion information between the plurality of first image frames to align the first image frames; and a denoising module configured to obtain a third image after denoising according to the second image group.

[0009] According to a second aspect of the present disclosure, there is provided a method for blind denoising of images, comprising: obtaining a noise estimate of a first image group from the first image group, wherein the first image group comprises a plurality of first image frames; obtaining a second image group from the first image group and the noise estimate, the second image group containing motion information between the plurality of first image frames to facilitate alignment of the first image frames; and obtaining a third image with reduced noise from the second image group.

[0010] According to a third aspect of the present disclosure, there is provided an electronic device comprising: one or more processors; and memory storing computer-executable instructions that, when executed by the one or more processors, cause the electronic device to perform the method as described above.

[0011] According to a fourth aspect of the present disclosure, there is provided a computer-readable storage medium comprising computer-executable instructions that, when executed by one or more processors, perform the method as described above.

[0012] The scheme of the present disclosure implicitly learns the motion information between frames without using the optical flow method in the prior art, and thereby aligns the adjacent frames. Finally, the noise in the images can be efficiently removed according to the aligned adjacent frames and the obtained noise estimate.

[0013] The present disclosure estimates various parameters of the noise through the noise estimation module without prior knowledge of the various parameters of the noise, and thereby facilitates the subsequent alignment of the adjacent frames, so that the blind denoising can be performed in real time. BRIEF DESCRIPTION OF DRAWINGS

[0014] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0015] Figure 1 A schematic diagram of a system for blind denoising of images according to one embodiment of the present disclosure is shown;

[0016] Figure 2 A schematic diagram of a noise estimation module 110 according to one embodiment of the present disclosure is shown;

[0017] Figure 3 A schematic diagram of a noise estimation module 110 according to another embodiment of the present disclosure is shown;

[0018] Figure 4aAn internal structure diagram of one processor group to which the method of the present disclosure can be applied is shown.

[0019] Figure 4b An internal structure diagram of one processor group to which the method of the present disclosure can be applied is shown.

[0020] Figure 5 A diagram showing the division of a first image group according to one embodiment of the present disclosure is shown.

[0021] Figure 6 A diagram showing the division of a first image group according to one embodiment of the present disclosure is shown.

[0022] Figures 7a to 7c A diagram showing the division of a first image group according to one embodiment of the present disclosure is shown.

[0023] Figure 8 A method of blind denoising of images according to one embodiment of the present disclosure is shown.

[0024] Figure 9 A combination processing device is shown; and

[0025] Figure 10 An exemplary board card is shown. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present disclosure.

[0027] It should be understood that the terms "first", "second", "third", and "fourth" and the like in the claims, specification, and drawings of the present disclosure are used to distinguish different objects, and are not used to describe a particular order. The terms "include" and "contain" used in the specification and claims of the present disclosure indicate the presence of the described features, whole, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, whole, steps, operations, elements, components, and / or sets thereof.

[0028] It should also be understood that the terms used herein are for the purpose of describing particular embodiments and are not intended to limit the present disclosure. As used in this specification and claims, the singular forms "a," "an" and "the" include plural referents unless the context clearly dictates otherwise. It should also be further understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items, and that the term "at least one of’ encompasses each possible combination of one or more of the associated listed items.

[0029] As used in this specification and claims, the term "if' can be construed to mean "when" or "once," or "in response to a determination" or "in response to the occurrence of" that follows, depending on the context in which it is used. Similarly, the phrase "if it is determined" or "if [a described condition or event] is detected" can be construed to mean "once it is determined" or "in response to a determination" or "once [the described condition or event] is detected" or "in response to the occurrence of [the described condition or event]," depending on the context in which it is used.

[0030] First, an example of noise involved in the present application is introduced.

[0031] Common noise is Gaussian additive white noise (AWGN). Assuming that the observed image is In, the image without noise is I, and the noise is n, their relationship can be expressed as:

[0032] In = I + n

[0033] Under this assumption, the probability distribution of AWGN noise can be described as

[0034]

[0035] wherein the above formula n i is the noise value at pixel i, which follows a Gaussian distribution with a mean of 0 and a variance of σ 2 w.

[0036] AWGN noise is independent of the signal, so it cannot describe the signal-dependent photon noise. The Poisson distribution can be used to describe this part of the noise. The expression of the Poisson distribution is:

[0037] n i ~αP(I i ) - I i

[0038] wherein I i is the signal without noise, and the mean and variance of the noise are both I i , and a is the amplification coefficient of the sensor.

[0039] The noise signal contains both signal-dependent Poisson noise and signal-independent Gaussian noise. A model that jointly describes the Poisson-Gaussian noise is

[0040]

[0041] More generally, a heteroscedastic Gaussian model can be used to approximate

[0042]

[0043] The heteroscedastic Gaussian model is also referred to as Noise Lever Function (NLF) to describe the noise level and pixel intensity.

[0044] However, due to the lack of data sets and the differences in sensors, the image and video methods based on deep learning in recent years mainly do the work of removing additive Gaussian white noise on RGB images, and less work is done on signal-dependent Poisson distribution noise.

[0045] Figure 1 A schematic diagram of a system for blind denoising of images according to an embodiment of the present disclosure is shown, including a noise estimation module 110 configured to receive a first image group including a plurality of first image frames, and obtain a noise estimation of the first image group according to the first image group; a motion alignment module 120 configured to obtain a second image group according to the first image group and the noise estimation, the second image group containing motion information between the plurality of first image frames to facilitate alignment of the first image frames; and a denoising module 130 configured to obtain a third image after denoising according to the second image group.

[0046] As Figure 1 shown, the noise estimation module 1 can be a conventional convolutional neural network that can be trained to identify images containing noise.

[0047] When training the noise estimation module 1, the noise estimation module can be trained by known noise estimation and image frames, i.e., the noise-containing image can be fed into the noise estimation module 1, wherein the noise used during training can be known noise (the description of the noise includes but is not limited to the mean, variance, distribution type, etc. of the noise). By training the noise estimation module 1, a trained noise estimation module can be obtained. During inference, the trained noise estimation module can obtain the noise estimation of the noise when receiving a real noise-containing image.

[0048] The first image group may include multiple image frames. For example, the first image group may be a video stream, which may include multiple video frames, each of which is an image. Preferably, the first image group may include multiple consecutive image frames. If an image frame at a certain time t is taken as the image frame of interest, then the consecutive image frames may include image frames at times t-2, t-1, t, t+1, t+2, etc. Multiple image frames represented by time are merely an example and do not necessarily mean that these image frames are entered in chronological order. For example, a certain number of video frames can be pre-stored, and the noise estimation module 110 simultaneously receives these pre-stored video frames and estimates the noise of these video frames. The following explanation will use five image frames at times t-2, t-1, t, t+1, and t+2 as an example.

[0049] According to one embodiment of this disclosure, the noise estimation module 110 may be configured to sequentially receive the plurality of first image frames to obtain a noise estimate for each first image frame.

[0050] Figure 2 A schematic diagram of a noise estimation module 110 according to one embodiment of the present disclosure is shown.

[0051] like Figure 2 As shown, each image frame sequentially enters the noise estimation module 110. Upon receiving each image frame, the noise estimation module 110 estimates the noise of that image frame, i.e., provides a noise estimate for that image frame. The noise estimate can be, for example, the mean, standard deviation, etc. of the noise. For example, the noise estimation module 110 receives image frame t-2 and obtains the noise estimate t-2 for image frame t-2; receives image frame t-1 and obtains the noise estimate t-1 for image frame t-1; receives image frame t and obtains the noise estimate t for image frame t; receives image frame t+1 and obtains the noise estimate t+1 for image frame t+1; and receives image frame t+2 and obtains the noise estimate t+2 for image frame t+2.

[0052] The noise estimates output by the noise estimation module 110 can be stored sequentially for use by the subsequent motion alignment module 120.

[0053] According to one embodiment of this disclosure, the noise estimation module 110 may be multiple, configured to receive the multiple first image frames in parallel and obtain a noise estimate for each first image frame.

[0054] Figure 3 A schematic diagram of a noise estimation module 110 according to another embodiment of the present disclosure is shown.

[0055] like Figure 3As shown, the noise estimation modules 110 can be multiple, for example, noise estimation module 110-1, noise estimation module 110-2, noise estimation module 110-3, noise estimation module 110-4 and noise estimation module 110-5. These noise estimation modules 110 can receive corresponding image frames in parallel, for example, noise estimation module 110-1 can receive the image frame at time t-2 to obtain noise estimation t-2, noise estimation module 110-2 can receive the image frame at time t-1 to obtain noise estimation t-1, noise estimation module 110-3 can receive the image frame at time t to obtain noise estimation t, noise estimation module 110-4 can receive the image frame at time t+1 to obtain noise estimation t+1, and noise estimation module 110-5 can receive the image frame at time t+2 to obtain noise estimation t+2.

[0056] For the noise estimation module, it can be executed in parallel by multiple threads of a single processor, or by different processing cores in a multi-core processor.

[0057] Figure 4a And Figure 4b A schematic diagram of a multi-core processor according to an embodiment of the present disclosure is shown, wherein, Figure 4a A schematic diagram of the internal structure of a processor group to which the method of the present disclosure can be applied is shown.

[0058] An artificial intelligence (AI) chip accelerates data computing capability and reduces memory access latency. The AI chip adopts a multi-core processor architecture, supports up to 16-core parallel computing, and adds a memory unit core (also referred to as on-chip or in-chip memory unit) to accelerate data reading, solving the memory access bottleneck problem of the processor core and DDR (also referred to as off-chip memory unit) of the AI chip. It provides users with stronger computing power in processing deep learning, network computing and other scenarios.

[0059] The AI chip has 16 processor cores for executing computing tasks. Every 4 processor cores form a processor group, i.e., there are 4 processor groups in total. Each processor group has a memory unit core. The memory unit core is mainly used for data exchange between the shared memory unit in the processor group and the processor core and data exchange between processor groups. When the memory core and the processor core access the DDR at the same time, after arbitration by the multiplexer, it is ensured that only one group of buses accesses the DDR.

[0060] Figure 4b A schematic diagram of the structure of an artificial intelligence processor to which the method of the present disclosure can be applied is shown. As Figure 4bAs shown, the DDR of the AI chip adopts a Non-Uniform Memory Access (NUMA) architecture, and each processor group can access different DDR channels through NOC0, but the delay of accessing different DDR channels is different. Each processor group corresponds to a DDR channel with the lowest access delay, and the delay of accessing other channels is relatively long. For example, Figure 1 As shown in the structure diagram of the processor group and the DDR in b, the processor group 0, the processor group 1, the processor group 2 and the processor group 3 respectively access the DDR0, the DDR1, the DDR2 and the DDR3 with the lowest delay. That is, each processor core accesses the DDR channel with the lowest access delay in the respective processor group.

[0061] Since the access bandwidth inside the processor group is higher than the access bandwidth between the processor core and the DDR, the AI chip can access the shared storage unit internally through the processor group to reduce the direct access of the processor core to the DDR, thereby improving the data throughput.

[0062] When 4-core parallel computing is required, the storage unit core can broadcast data from the shared storage unit to the 4 processor cores in the processor group through data broadcasting (through NOC1) to perform data computation. Compared with the way of reading data through the DDR by all processor cores, the access delay can be reduced in this case, and the computing performance is optimized.

[0063] When the computing demand increases, 16 processor cores can need to process multiple computing tasks at the same time. Direct access of the processor core to the DDR will inevitably cause data access delay, resulting in slow computing speed and other problems. The AI chip exchanges data between the processor groups to avoid direct communication between the 16 processor cores and the DDR, thereby reducing the data access delay.

[0064] Although the present disclosure exemplarily shows 5 parallel noise estimation modules, the present disclosure can adopt a corresponding number of noise estimation modules according to the number of image frames, so as to improve the operation efficiency in a parallel operation manner.

[0065] After obtaining the noise estimation of each image frame, the noise estimation and the first image group can be input into the motion alignment module 120, as shown in Figure 1 .

[0066] The first image group can be conducted to the motion alignment module 120 through the noise estimation module 110, as shown by the solid line in Figure 1 The motion alignment module 120 can also directly receive the first image group without the conduction of the noise estimation module 110, as shown by the dotted line in Figure 1 .

[0067] In the prior art, in order to fuse inter-frame motion information in video noise reduction, an optical flow method is usually used to align adjacent frames, but it is difficult to calculate accurately in the presence of noise. For example, if the noise in the video frame is black in the color domain, when a black object (e.g. a person's hair) in the video frame moves, it is difficult to determine whether the noise or the target (e.g. the hair) in the image has moved, so it is difficult to calculate the optical flow in the presence of noise.

[0068] In the present disclosure, according to one embodiment of the present disclosure, the first image group can be divided into a plurality of first image frame sets, each first image frame set including a portion of the plurality of first image frames, and the motion alignment module 120 can be configured to sequentially receive each first image frame set and a corresponding noise estimate to facilitate alignment of the first image frames.

[0069] Figure 5 A schematic diagram of dividing the first image group is shown according to one embodiment of the present disclosure.

[0070] As Figure 5 shown, the first image group includes n image frames, i.e. image frame 0 to image frame n-1, which can be divided into a plurality of sets, for example, image frame 0 and image frame 1 form the first image frame set 0, image frame 1 and image frame 2 form the first image frame set 1,..., image frame n-2 and image frame n-1 form the first image frame set m. By analyzing the changes between the image frames, the motion relationship between the image frames can be obtained.

[0071] According to one embodiment of the present disclosure, at least one same image frame can be included in adjacent first image frame sets.

[0072] For example, Figure 5 the first image frame set 0 includes image frame 0 and image frame 1, the first image frame set 1 includes image frame 1 and image frame 2, and the first image frame set 2 (not shown) includes image frame 2 and image frame 3, so the adjacent first image frame sets 0 and 1 include the same image frame 1, and the adjacent first image frame sets 1 and 2 include the same image frame 2. By such arrangement, the motion alignment module 120 can obtain the inter-frame motion information from these image frames and align the image frames.

[0073] It should be understood that Figure 5 It is only for the purpose of example that two image frames are included in each image frame set, and in the embodiments of the present disclosure, the number of image frames included in each image frame set is not limited, and the number of image frames included in different image frame sets can be the same or different.

[0074] According to one preferred embodiment of the present disclosure, the at least one first image frame at the back of the preceding first image frame set is at the front of the following first image frame set.

[0075] Figure 6 A diagram showing the division of the first image group according to one embodiment of the present disclosure is shown.

[0076] Figure 5 A diagram showing the division of the first image group as a whole, Figure 6 An example of a more specific embodiment is shown. As Figure 6 shown, the first image group exemplarily includes image frame t-2, image frame t-1, image frame t, image frame t+1 and image frame t+2, where t is time. In this case, the five image frames can be divided into three image sets, the first image frame set 1 includes image frame t-2, image frame t-1 and image frame t; the first image frame set 2 includes image frame t-1, image frame t and image frame t+1; and the first image frame set 3 includes image frame t, image frame t+1 and image frame t+2. As can be seen, each first image frame set includes three image frames, so when these image frames are input to the motion alignment module 120, the motion alignment module 120 can obtain the motion information between the frames from these image frames, and further align the image frames.

[0077] Further as Figure 6 shown, the first image frame set 1 is first input to the motion alignment module 120, and the motion alignment module 120 generates the second image 1 according to the three image frames t-2, t-1 and t and the noise estimates corresponding to each image frame; next, the first image frame set 2 is input to the motion alignment module 120, and the motion alignment module 120 generates the second image 2 according to the three image frames t-1, t and t+1 and the noise estimates corresponding to each image frame; then, the first image frame set 3 is input to the motion alignment module 120, and the motion alignment module 120 generates the second image 3 according to the three image frames t, t+1 and t+2 and the noise estimates corresponding to each image frame.

[0078] It should be understood that although Figure 6 three motion alignment modules 120 are shown, this is merely for ease of understanding, and in actual cases, the motion alignment module 120 can be only a single one, which receives the first image frame set 1 to the first image frame set 3 one by one, so that the motion alignment module 120 can obtain the changes between the image frames through the image frames and the noise estimates, and further facilitate the alignment of the information in the image frames.

[0079] It is also to be understood that the grouping of image frames described above can be varied and is not limited to the examples described above. For example, the image frames can be of other quantities and the grouping of each set of image frames is not limited to the manner described above.

[0080] Figures 7a to 7c A diagram illustrating division of a first image group according to other embodiments of the disclosure is shown.

[0081] As shown in FIG. 3, a first image group exemplarily includes image frames t-3, t-2, t-1, t, t+1, t+2, and t+3, where t is time. In this case, the seven image frames can be divided into four image sets, a first image frame set 1 including image frames t-3 and t-2, a first image frame set 2 including image frames t-2, t-1, and t, a first image frame set 3 including image frames t, t+1, and t+2, and a first image frame set 4 including image frames t+1, t+2, and t+3. Figure 7a As shown in FIG. 3, a first image group exemplarily includes image frames t-3, t-2, t-1, t, t+1, t+2, and t+3, where t is time. In this case, the seven image frames can be divided into four image sets, a first image frame set 1 including image frames t-3 and t-2, a first image frame set 2 including image frames t-2, t-1, and t, a first image frame set 3 including image frames t, t+1, and t+2, and a first image frame set 4 including image frames t+1, t+2, and t+3.

[0082] As shown in FIG. 3, a first image group exemplarily includes image frames t-3, t-2, t-1, t, t+1, t+2, and t+3, where t is time. In this case, the seven image frames can be divided into four image sets, a first image frame set 1 including image frames t-3 and t-2, a first image frame set 2 including image frames t-2, t-1, and t, a first image frame set 3 including image frames t, t+1, and t+2, and a first image frame set 4 including image frames t+1, t+2, and t+3. Figure 7b As shown in FIG. 3, a first image group exemplarily includes image frames t-3, t-2, t-1, t, t+1, t+2, and t+3, where t is time. In this case, the seven image frames can be divided into four image sets, a first image frame set 1 including image frames t-3 and t-2, a first image frame set 2 including image frames t-2, t-1, and t, a first image frame set 3 including image frames t, t+1, and t+2, and a first image frame set 4 including image frames t+1, t+2, and t+3.

[0083] As shown in FIG. 3, a first image group exemplarily includes image frames t-3, t-2, t-1, t, t+1, t+2, and t+3, where t is time. In this case, the seven image frames can be divided into four image sets, a first image frame set 1 including image frames t-3 and t-2, a first image frame set 2 including image frames t-2, t-1, and t, a first image frame set 3 including image frames t, t+1, and t+2, and a first image frame set 4 including image frames t+1, t+2, and t+3. Figure 7c As shown in FIG. 3, a first image group exemplarily includes image frames t-3, t-2, t-1, t, t+1, t+2, and t+3, where t is time. In this case, the seven image frames can be divided into four image sets, a first image frame set 1 including image frames t-3 and t-2, a first image frame set 2 including image frames t-2, t-1, and t, a first image frame set 3 including image frames t, t+1, and t+2, and a first image frame set 4 including image frames t+1, t+2, and t+3. Figure 7cThe illustrated implementation differs from other implementations in that the number of image frames in each first image frame set is not uniform. Those skilled in the art can choose the desired partitioning method according to actual needs; this disclosure does not impose any limitations on the number of image frames in each image group or its partitioning method.

[0084] According to one embodiment of this disclosure, the motion alignment module can be a neural network with a Unet structure.

[0085] The Unet network architecture is widely used in image processing, such as medical image processing. It typically includes a contracting path and an expanding path. The contracting path is mainly used to capture context information in the image, while the expanding path is used to accurately localize the parts of the image that need to be segmented.

[0086] Next, the denoising module 130 can receive multiple images output by the motion alignment module 120 and fuse these images to obtain the final denoised image. Since the second image group output by the motion alignment module 120 already contains motion information between image frames, after alignment by the motion alignment module 120, the denoising module 130 can clearly identify the noise part, thereby removing the noise from the image.

[0087] The denoising module 130 can adopt the same network structure as the motion alignment module 120, or the denoising module 130 can be implemented by a temporal attention-based fusion method. The temporal attention-based fusion method can be obtained from the paper "EDVR: Video Restoration with Enhanced Deformable Convolutional Networks" by Xintao Wang et al., which is incorporated herein by reference in its entirety.

[0088] Figure 8 A method for blind denoising of an image according to one embodiment of the present disclosure is shown, comprising: in operation S810, obtaining a noise estimate of a first image group based on a first image group, wherein the first image group includes a plurality of first image frames; in operation S820, obtaining a second image group based on the first image group and the noise estimate, the second image group containing motion information between the plurality of first image frames to facilitate alignment of the first image frames; and in operation S830, obtaining a denoised third image based on the second image group.

[0089] This disclosure also provides an electronic device, including: one or more processors; and a memory storing computer-executable instructions that, when executed by the one or more processors, cause the electronic device to perform the method described above.

[0090] This disclosure also provides a computer-readable storage medium including computer-executable instructions that, when executed by one or more processors, perform the method described above.

[0091] In this application, the technical solution disclosed herein can be applied to images of various formats, including but not limited to Raw format, RGB format, and YUV format, etc.

[0092] The scheme disclosed herein can implicitly learn inter-frame motion information without employing existing optical flow methods, thereby aligning adjacent frames. Finally, based on the aligned adjacent frames and the obtained noise estimate, noise in the image can be efficiently removed.

[0093] This disclosure allows for the estimation of various noise parameters without prior knowledge of them, through a noise estimation module. This facilitates the alignment of subsequent adjacent frames, enabling real-time blind denoising.

[0094] In this disclosure, the noise estimation module 110, motion alignment module 120, and denoising module 130 can be trained individually to obtain the desired parameter values; however, these modules are preferably trained in combination. Training them in combination yields better training results, resulting in better output during inference.

[0095] The technical solution disclosed herein can be applied to the field of artificial intelligence, and can be implemented as or in an artificial intelligence chip. This chip can exist independently or be included in a computing device.

[0096] Figure 9 A combined processing apparatus 900 is shown, comprising the aforementioned computing device 902, a general interconnect interface 904, and other processing devices 906. The computing device according to this disclosure interacts with the other processing devices to jointly perform user-specified operations. Figure 9 This is a schematic diagram of the combined processing device.

[0097] Other processing devices include one or more processor types such as central processing unit (CPU), graphics processing unit (GPU), and neural network processor. There is no limit to the number of processors included in other processing devices. These other processing devices serve as interfaces between the machine learning computing device and external data and control, including data transfer and basic control such as starting and stopping the machine learning computing device. Other processing devices can also collaborate with the machine learning computing device to complete computational tasks.

[0098] A general interconnect interface is used to transfer data and control commands between a computing device (including, for example, a machine learning computing device) and other processing devices. The computing device can obtain required input data from other processing devices and write it to on-chip storage; it can obtain control commands from other processing devices and write them to on-chip control caches; and it can read data from the computing device's storage modules and transmit it to other processing devices.

[0099] Optionally, the structure may further include a storage device 908, which is connected to both the computing device and the other processing device. The storage device is used to store data in the computing device and the other processing device, and is particularly suitable for data that cannot be fully stored in the internal storage of the computing device or other processing device, requiring computation.

[0100] This combined processing device can serve as a System-on-a-Chip (SoC) for devices such as mobile phones, robots, drones, and video surveillance equipment, effectively reducing the core area of ​​the control unit, increasing processing speed, and lowering overall power consumption. In this case, the universal interconnect interface of the combined processing device connects to certain components of the device, such as cameras, monitors, mice, keyboards, network cards, and Wi-Fi interfaces.

[0101] In some embodiments, this disclosure also discloses a chip package structure that includes the aforementioned chip.

[0102] In some embodiments, this disclosure also discloses a circuit board that includes the above-described chip packaging structure. See also... Figure 10 The present invention provides an exemplary board, which, in addition to the chip 1002, may also include other supporting components, including but not limited to: a storage device 1004, an interface device 1006, and a controller 1008.

[0103] The storage device is connected to the chip within the chip package structure via a bus and is used to store data. The storage device may include multiple sets of storage cells 1010. Each set of storage cells is connected to the chip via a bus. It is understood that each set of storage cells may be DDR SDRAM (Double Data Rate SDRAM).

[0104] DDR can double the speed of SDRAM without increasing the clock frequency. DDR allows data to be read on both the rising and falling edges of the clock pulse. DDR is twice as fast as standard SDRAM. In one embodiment, the storage device may include four groups of storage cells. Each group of storage cells may include multiple DDR4 chips. In one embodiment, the chip may internally include four 72-bit DDR4 controllers, of which 64 bits are used for data transmission and 8 bits are used for ECC verification. In one embodiment, each group of storage cells includes multiple Double Rate Synchronous Dynamic Random Access Memory (DRAM) units connected in parallel. DDR can transmit data twice per clock cycle. A controller for controlling DDR is provided in the chip for controlling data transmission and data storage in each storage cell.

[0105] The interface device is electrically connected to the chip within the chip package structure. The interface device is used to realize data transmission between the chip and an external device 1012 (e.g., a server or computer). For example, in one embodiment, the interface device can be a standard PCIe interface. For instance, data to be processed is transferred from the server to the chip via a standard PCIe interface, realizing data transfer. In another embodiment, the interface device can also be other interfaces; this disclosure does not limit the specific form of the other interfaces mentioned above, as long as the interface unit can realize the switching function. Furthermore, the calculation results of the chip are still transmitted back to the external device (e.g., the server) by the interface device.

[0106] The controller is electrically connected to the chip. The controller monitors the state of the chip. Specifically, the chip and the controller can be electrically connected via an SPI interface. The controller may include a microcontroller (MCU). The chip may include multiple processing chips, multiple processing cores, or multiple processing circuits, capable of driving multiple loads. Therefore, the chip can operate in different states, such as high load and low load. The controller can regulate the operating states of multiple processing chips, multiple processing cores, and / or multiple processing circuits within the chip.

[0107] In some embodiments, this disclosure also discloses an electronic device or apparatus that includes the aforementioned board.

[0108] Electronic devices or apparatuses include data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, mobile phones, dashcams, navigators, sensors, cameras, servers, cloud servers, cameras, camcorders, projectors, watches, headphones, mobile storage, wearable devices, vehicles, home appliances, and / or medical devices.

[0109] The means of transportation include airplanes, ships and / or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical equipment includes MRI scanners, ultrasound scanners and / or electrocardiographs.

[0110] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this disclosure. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are optional embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0111] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0112] In the embodiments provided in this disclosure, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, optical, acoustic, magnetic, or other forms.

[0113] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] Furthermore, the functional units in the various embodiments disclosed herein can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software program module.

[0115] If the integrated unit is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, when the technical solution disclosed herein can be embodied in the form of a software product, the computer software product is stored in a storage device and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage device includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0116] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. A system for blind denoising of images, comprising: a noise estimation module configured to receive a first image group comprising a plurality of consecutive first image frames and derive a noise estimate of the first image group from the first image group; a motion alignment module configured to learn changes between the first image frames from the first image group and the noise estimate and align information in the first image frames to derive a second image group containing motion information between the plurality of first image frames to facilitate removal of noise from the aligned first image frames, wherein adjacent frames corresponding to the plurality of first image frames in the second image group are aligned; and a denoising module configured to derive a third image with reduced noise from the second image group.

2. The system of claim 1, wherein, The noise estimation module is configured to receive the plurality of first image frames sequentially to derive a noise estimate for each first image frame.

3. The system of claim 1, wherein, The noise estimation module is a plurality of noise estimation modules configured to receive the plurality of first image frames in parallel and derive a noise estimate for each first image frame.

4. The system of claim 3, wherein, The plurality of noise estimation modules are executed by different processing cores in a multi-core processor.

5. The system of any of claims 1-4, wherein, The noise estimation module is trained by known noise estimates and image frames.

6. The system of any of claims 1-5, wherein, The first image group is divided into a plurality of first image frame sets, each first image frame set comprising a portion of the plurality of first image frames, The motion alignment module is configured to receive each first image frame set and corresponding noise estimate sequentially to facilitate alignment of the first image frames.

7. The system of claim 6, wherein, Adjacent first image frame sets comprise at least one same image frame.

8. The system of claim 7, wherein, At least one first image frame at the back of a preceding first image frame set is at the front of a following first image frame set.

9. The system of any of claims 1-8, wherein, The motion alignment module is a neural network of Unet structure.

10. The system of any of claims 1-9, wherein, The denoising module is of the same structure as the motion alignment module; or the denoising module is implemented by a fusion method based on temporal attention.

11. The system of any of claims 1-10, wherein, The noise is Gaussian white noise and / or Poisson distribution noise.

12. The system of any of claims 1-11, wherein, The noise estimate comprises a mean and a variance of the noise. 13.A method for blind denoising of images, comprising: deriving a noise estimate of a first image group from the first image group, wherein the first image group comprises a plurality of consecutive first image frames; learning changes between the first image frames from the first image group and the noise estimate and aligning information in the first image frames to derive a second image group containing motion information between the plurality of first image frames to facilitate removal of noise from the aligned first image frames, wherein adjacent frames corresponding to the plurality of first image frames in the second image group are aligned; and deriving a third image with reduced noise from the second image group. 14.An electronic device, comprising: one or more processors; and a memory having computer-executable instructions stored therein that, when executed by the one or more processors, cause the electronic device to perform the method of claim 13.

15. A computer-readable storage medium comprising computer-executable instructions that, when executed by one or more processors, perform the method of claim 13.

Images